Diabetes and impaired brain insulin signaling are linked to the pathogenesis of Alzheimer's disease (AD). The association between diabetes and AD-associated amyloid pathology is stronger among carriers of the apolipoprotein E (APOE) ε4 gene allele, the strongest genetic risk factor for late-onset AD. Here we report that apoE4 impairs neuronal insulin signaling in human apoE-targeted replacement (TR) mice in an age-dependent manner. High-fat diet (HFD) accelerates these effects in apoE4-TR mice at middle age. In primary neurons, apoE4 interacts with insulin receptor and impairs its trafficking by trapping it in the endosomes, leading to impaired insulin signaling and insulin-stimulated mitochondrial respiration and glycolysis. In aging brains, the increased apoE4 aggregation and compromised endosomal function further exacerbate the inhibitory effects of apoE4 on insulin signaling and related functions. Together, our study provides novel mechanistic insights into the pathogenic mechanisms of apoE4 and insulin resistance in AD.