The APOE ε4 allele is associated with cognitive deficits and brain atrophy in older adults, but studies in younger adults are mixed. We examined APOE genotype effects on cognition and brain structure in younger adults and whether genotype effects differed by age and with presence of depression. 157 adults (32 % ε4 carriers, 46 % depressed) between 20 and 50 years of age completed neuropsychological testing, 131 of which also completed 3 T cranial MRI. We did not observe a direct effect of APOE genotype on cognitive performance or structural MRI measures. A significant genotype by age interaction was observed for executive function, where age had less of an effect on executive function in ε4 carriers. Similar interactions were observed for the entorhinal cortex, rostral and caudal anterior cingulate cortex and parahippocampal gyrus, where the effect of age on regional volumes was reduced in ε4 carriers. There were no significant interactions between APOE genotype and depression diagnosis. The ε4 allele benefits younger adults by allowing them to maintain executive function performance and volumes of cingulate and temporal cortex regions with aging, at least through age fifty years.