Vocal learning is the ability to imitate and modify sounds through auditory experience, a rare trait found in only a few lineages of mammals and birds. It is a critical component of human spoken language, allowing us to verbally transmit speech repertoires and knowledge across generations. In many vocal learning species, the vocal learning trait is sexually dimorphic, where it is either limited to males or present in both sexes to different degrees. In humans, recent findings have revealed subtle sexual dimorphism in vocal learning/spoken language brain regions and some associated disorders. For songbirds, where the neural mechanisms of vocal learning have been well studied, vocal learning appears to have been present in both sexes at the origin of the lineage and was then independently lost in females of some subsequent lineages. This loss is associated with an interplay between sex chromosomes and sex steroid hormones. Even in species with little dimorphism, like humans, sex chromosomes and hormones still have some influence on learned vocalizations. Here we present a brief synthesis of these studies, in the context of sex determination broadly, and identify areas of needed investigation to further understand how sex chromosomes and sex steroid hormones help establish sexually dimorphic neural structures for vocal learning.