The adipose tissue stroma is a rich source of molecularly distinct stem and progenitor cell populations with diverse functions in metabolic regulation, adipogenesis, and inflammation. The ontology of these populations and the mechanisms that govern their behaviors in response to stimuli, such as overfeeding, however, are unclear. Here, we show that the developmental fates and functional properties of adipose platelet-derived growth factor receptor beta (PDGFRβ)+ progenitor subpopulations are tightly regulated by mitochondrial metabolism. Reducing the mitochondrial β-oxidative capacity of PDGFRβ+ cells via inducible expression of MitoNEET drives a pro-inflammatory phenotype in adipose progenitors and alters lineage commitment. Furthermore, disrupting mitochondrial function in PDGFRβ+ cells rapidly induces alterations in immune cell composition in lean mice and impacts expansion of adipose tissue in diet-induced obesity. The adverse effects on adipose tissue remodeling can be reversed by restoring mitochondrial activity in progenitors, suggesting therapeutic potential for targeting energy metabolism in these cells.