Impulsivity is a complex trait associated with a range of maladaptive behaviors, including many forms of psychopathology. Previous research has implicated multiple neural circuits and neurotransmitter systems in impulsive behavior, but the relationship between impulsivity and organization of whole-brain networks has not yet been explored. Using graph theory analyses, we characterized the relationship between impulsivity and the functional segregation (``modularity'') of the whole-brain network architecture derived from resting-state functional magnetic resonance imaging (fMRI) data. These analyses revealed remarkable differences in network organization across the impulsivity spectrum. Specifically, in highly impulsive individuals, regulatory structures including medial and lateral regions of the prefrontal cortex were isolated from subcortical structures associated with appetitive drive, whereas these brain areas clustered together within the same module in less impulsive individuals. Further exploration of the modular organization of whole-brain networks revealed novel shifts in the functional connectivity between visual, sensorimotor, cortical, and subcortical structures across the impulsivity spectrum. The current findings highlight the utility of graph theory analyses of resting-state fMRI data in furthering our understanding of the neurobiological architecture of complex behaviors.