Background: Kin selection is a driving force in the evolution of mammalian social complexity. Recognition of paternal kin using vocalizations occurs in taxa with cohesive, complex social groups. This is the first investigation of paternal kin recognition via vocalizations in a small-brained, solitary foraging mammal, the grey mouse lemur (Microcebus murinus), a frequent model for ancestral primates. We analyzed the high frequency/ultrasonic male advertisement (courtship) call and alarm call.

Results: Multi-parametric analyses of the calls’ acoustic parameters and discriminant function analyses showed that advertisement calls, but not alarm calls, contain patrilineal signatures. Playback experiments controlling for familiarity showed that females paid more attention to advertisement calls from unrelated males than from their fathers. Reactions to alarm calls from unrelated males and fathers did not differ.

Conclusions: 1) Findings provide the first evidence of paternal kin recognition via vocalizations in a small-brained, solitarily foraging mammal. 2) High predation, small body size, and dispersed social systems may select for acoustic paternal kin recognition in the high frequency/ultrasonic ranges, thus limiting risks of inbreeding and eavesdropping by predators or conspecific competitors. 3) Paternal kin recognition via vocalizations in mammals is not dependent upon a large brain and high social complexity, but may already have been an integral part of the dispersed social networks from which more complex, kin-based sociality emerged.

Introduction: Maternal kin selection is a driving force in the evolution of mammalian social complexity and it requires that kin are distinctive from nonkin. The transition from the ancestral state of asociality to the derived state of complex social groups is thought to have occurred via solitary foraging, in which individuals forage alone, but, unlike the asocial ancestors, maintain dispersed social networks via scent-marks and vocalizations. We hypothesize that matrilineal signatures in vocalizations were an important part of these networks. We used the solitary foraging gray mouse lemur (Microcebus murinus) as a model for ancestral solitary foragers and tested for matrilineal signatures in their calls, thus investigating whether such signatures are already present in solitary foragers and could have facilitated the kin selection thought to have driven the evolution of increased social complexity in mammals. Because agonism can be very costly, selection for matrilineal signatures in agonistic calls should help reduce agonism between unfamiliar matrilineal kin. We conducted this study on a well-studied population of wild mouse lemurs at Ankarafantsika National Park, Madagascar. We determined pairwise relatedness using seven microsatellite loci, matrilineal relatedness by sequencing the mitrochondrial D-loop, and sleeping group associations using radio-telemetry. We recorded agonistic calls during controlled social encounters and conducted a multi-parametric acoustic analysis to determine the spectral and temporal structure of the agonistic calls. We measured 10 calls for each of 16 females from six different matrilineal kin groups.

Results: Calls were assigned to their matriline at a rate significantly higher than chance (pDFA: correct = 47.1%, chance = 26.7%, p = 0.03). There was a statistical trend for a negative correlation between acoustic distance and relatedness (Mantel Test: g = -1.61, Z = 4.61, r = -0.13, p = 0.058).

Conclusions: Mouse lemur agonistic calls are moderately distinctive by matriline. Because sleeping groups consisted of close maternal kin, both genetics and social learning may have generated these acoustic signatures. As mouse lemurs are models for solitary foragers, we recommend further studies testing whether the lemurs use these calls to recognize kin. This would enable further modeling of how kin recognition in ancestral species could have shaped the evolution of complex sociality.

Arguments of human uniqueness emphasize our complex sociality, unusual cognitive capacities, and language skills, but the timing of the origin of these abilities and their evolutionary causes remain unsolved. Though not unique to primates, kin-biased sociality was key to the success of the primate order. In contrast to ancestral solitary mammals, the earliest primates are thought to have maintained dispersed (non-group living) social networks, communicating over distances via vocalizations and scent marks. If such ancestral primates recognized kin, those networks may have facilitated the evolution of kin-biased sociality in the primate order and created selection for increased cognitive and communicative abilities. I used the gray mouse lemur (Microcebus murinus) to model whether vocalizations could have facilitated matrilineal and patrilineal kin recognition in ancestral primates. Much like mouse lemurs today, ancestral primates are thought to have been small-bodied, nocturnal creatures that captured insects and foraged for fruit in the thin, terminal ends of tree branches. Thus, the mouse lemur is an excellent model species because its ecological niche is likely to be similar to that of ancestral primates 55-90 million years ago. I conducted playback experiments in Ankarafantsika National Park, Madagascar testing whether mouse lemur agonistic calls contain matrilineal kin signatures and whether the lemurs recognize matrilineal kin. In contrast to large-brained, socially complex monkeys with frequent coalitionary behavior, mouse lemurs did not react differently to the agonistic calls of matrilineal kin and nonkin, though moderate signatures were present in the calls. I tested for patrilineal signatures and patrilineal kin recognition via mating and alarm calls in a colony with known pedigree relationships. The results are the first to demonstrate that a nocturnal, solitary foraging mammal gives mating calls with patrilineal signatures and recognizes patrilineal kin. Interestingly, alarm calls did not have signatures and did not facilitate kin recognition, suggesting that selection for kin recognition is stronger in some call types than others. As this dissertation is the first investigation of vocal kin recognition in a dispersed-living, nocturnal strepsirrhine primate, it greatly advances our knowledge of the role of vocal communication in the evolution of primate social complexity.

As an evolutionary force, hybridization outcomes include introgression, admixture, speciation, and reproductive isolation. While hybridization has been studied in several primates, the marmoset genus Callithrix is an important, but little studied example of Neotropical hybridization. Varying degrees of reproductive isolation exist between Callithrix species, and hybridization occurs at species borders or regions containing introduced and native species. Interbreeding between Callithrix species carries important implications for biodiversity and genetic integrity within the genus. However, species origins and levels of genetic admixture in marmoset hybrid zones are generally unknown, and few population genetic studies of individual Callithrix species exist. Using the mitochondrial control region and 44 microsatellite markers, this work explored the genetic diversity and species origins of two C. penicillata and C. jacchus hybrid zones, as well as genetic diversity and divergence in the parental species. Both marker types showed that C. penicillata is more genetically diverse than C. jacchus. Based on mtDNA, C. jacchus seems to have experienced a past population expansion and C. penicillata evolved under constant population size. The data revealed the existence of a previously undocumented natural hybrid zone along the São Francisco River in NE Brazil and confirmed species origins of an anthropogenic zone in Rio de Janeiro state. The data also showed much lower levels of admixture and genetic diversity within the natural hybrid zone than in the anthropogenic zone. Further, the data suggested that the São Francisco River is an important geographic barrier to gene flow in the natural hybrid zone. On the other hand, admixture patterns within the anthropogenic hybrid zone suggested collapse of reproductive barriers, and the formation of a hybrid marmoset swarm. Thus, this work suggested different evolutionary dynamics in anthropogenic vs. natural animal hybrid zones. Restriction Associated DNA sequencing (RADseq) identified a large number of single nucleotide polymorphisms within C. jacchus and C. penicillata genomes. These preliminary data were used to measure intraspecific genomic diversity and interspecific divergence. In the future, RADseq will be used to study genus-wide diversity of Callithrix species, examine past and present marmoset demographic history, and applied to the evolutionary study of marmoset hybridization.