Mating behavior has profound consequences for two phenomena--individual reproductive success and the maintenance of species boundaries--that contribute to evolutionary processes. Studies of mating behavior in relation to individual reproductive success are common in many species, but studies of mating behavior in relation to genetic variation and species boundaries are less commonly conducted in socially complex species. Here we leveraged extensive observations of a wild yellow baboon (Papio cynocephalus) population that has experienced recent gene flow from a close sister taxon, the anubis baboon (Papio anubis), to examine how admixture-related genetic background affects mating behavior. We identified novel effects of genetic background on mating patterns, including an advantage accruing to anubis-like males and assortative mating among both yellow-like and anubis-like pairs. These genetic effects acted alongside social dominance rank, inbreeding avoidance, and age to produce highly nonrandom mating patterns. Our results suggest that this population may be undergoing admixture-related evolutionary change, driven in part by nonrandom mating. However, the strength of the genetic effect is mediated by behavioral plasticity and social interactions, emphasizing the strong influence of social context on mating behavior in socially complex species.