Language delay and impairment are salient features of autism. More specifically, there is evidence of atypical semantic organization in autism, but the functional brain correlates are not well understood. The current study used functional MRI to examine activation associated with semantic category decision. Ten high-functioning men with autism spectrum disorder and 10 healthy control subjects matched for gender, handedness, age, and nonverbal IQ were studied. Participants indicated via button press response whether visually presented words belonged to a target category (tools, colors, feelings). The control condition required target letter detection in unpronounceable letter strings. Significant activation for semantic decision in the left inferior frontal gyrus (Brodmann areas 44 and 45) was found in the control group. Corresponding activation in the autism group was more limited, with smaller clusters in left inferior frontal areas 45 and 47. Autistic participants, however, showed significantly greater activation compared to controls in extrastriate visual cortex bilaterally (areas 18 and 19), which correlated with greater number of errors on the semantic task. Our findings suggest an important role of perceptual components (possibly visual imagery) during semantic decision, consistent with previous evidence of atypical lexicosemantic performance in autism. In the context of similar findings from younger typically developing children, our results suggest an immature pattern associated with inefficient processing, presumably due to atypical experiential embedding of word acquisition in autism.