Within and across species of primates, the number of males in primate groups is correlated with the number of females. This correlation may arise owing to ecological forces operating on females, with subsequent competition among males for access to groups of females. The temporal relationship between changes in male and female group membership remains unexplored in primates and other mammalian groups. We used a phylogenetic comparative method for detecting evolutionary lag to test whether evolutionary change in the number of males lags behind change in the number of females. We found that change in male membership in primate groups is positively correlated with divergence time in pairwise comparisons. This result is consistent with male numbers adjusting to female group size and highlights the importance of focusing on females when studying primate social evolution.