In the natural world, sex prevails, despite its costs. Although much effort has been dedicated to identifying the intrinsic costs of sex (e.g., the cost of males), few studies have identified the ecological fitness consequences of sex. Furthermore, correlated biological traits that differ between sexuals and asexuals may alter these costs, or even render the typical costs of sex irrelevant. We conducted a large-scale, multisite, reciprocal transplant using multiple sexual and asexual genotypes of a native North American wildflower to show that sexual genotypes have reduced lifetime fitness, despite lower herbivory. We separated the effects of sex from those of hybridity, finding that overwinter survival is elevated in asexuals regardless of hybridity, but herbivores target hybrid asexuals more than nonhybrid asexual or sexual genotypes. Survival is lowest in homozygous sexual lineages, implicating inbreeding depression as a cost of sex. Our results show that the consequences of sex are shaped not just by sex itself, but by complex natural environments, correlated traits, and the identity and availability of mates.