While the human brain is clearly large relative to body size, less is known about the timing of brain and brain component expansion within primates and the relative magnitude of volumetric increases. Using Bayesian phylogenetic comparative methods and data for both extant and fossil species, we identified that a distinct shift in brain-body scaling occurred as hominins diverged from other primates, and again as humans and Neanderthals diverged from other hominins. Within hominins, we detected a pattern of directional and accelerating evolution towards larger brains, consistent with a positive feedback process in the evolution of the human brain. Contrary to widespread assumptions, we found that the human neocortex is not exceptionally large relative to other brain structures. Instead, our analyses revealed a single increase in relative neocortex volume at the origin of haplorrhines, and an increase in relative cerebellar volume in apes.