The current extinction and climate change crises pressure us to predict population dynamics with ever-greater accuracy. Although predictions rest on the well-advanced theory of age-structured populations, two key issues remain poorly explored. Specifically, how the age-dependency in demographic rates and the year-to-year interactions between survival and fecundity affect stochastic population growth rates. We use inference, simulations and mathematical derivations to explore how environmental perturbations determine population growth rates for populations with different age-specific demographic rates and when ages are reduced to stages. We find that stage- vs. age-based models can produce markedly divergent stochastic population growth rates. The differences are most pronounced when there are survival-fecundity-trade-offs, which reduce the variance in the population growth rate. Finally, the expected value and variance of the stochastic growth rates of populations with different age-specific demographic rates can diverge to the extent that, while some populations may thrive, others will inevitably go extinct.