The association between dietary diversity (DD) and psychological resilience among older people is an underdeveloped area of research. This cross-sectional study explored the associations of DD with psychological resilience among 8571 community-based elderly individuals. The intake frequencies of food groups were collected, and dietary diversity was assessed based on the mean DD score. Psychological resilience was assessed using a simplified resilience score (SRS). Data were analyzed using multiple linear regression and logistic regression models. Poor DD was significantly associated with psychological resilience, with a β (95% CI) of -0.94 (-1.07, -0.81) for the SRS (p < 0.01) and an odds ratio (95% CI) of 1.83 (1.66, 2.01) for low SRS status. The interaction effects of age with DD were observed for the SRS (p < 0.001) and low SRS status (p < 0.001). Based on separate analyses by age group, the association of a low SRS with poor DD was more prominent in the younger elderly than the oldest old, with OR (95% CI) 2.32 (1.96, 2.74) and 1.61 (1.43, 1.82), respectively. Compared with younger participants with good DD, the risk of a low SRS was greater for younger participants with poor DD, the oldest old with good DD, and the oldest old with poor DD, with OR (95% CI) 2.39 (2.02, 2.81), 1.28 (1.09, 1.51), and 2.03 (1.72, 2.39), respectively. The greatest contribution to DD was from a high consumption of vegetables, fruits, and nuts. Our study suggested that poor DD was associated with a low psychological resilience among the Chinese elderly, especially the younger elderly. These findings suggest that augmentation of DD might promote psychological resilience.