Functional magnetic resonance imaging (fMRI) studies have demonstrated that the blood oxygenation level-dependent (BOLD) hemodynamic response (HDR) to a stimulus is reduced by the previous presentation of a similar stimulus. We investigated the dependence of this refractory effect upon stimulus characteristics using a novel adaptation paradigm while scanning subjects using fMRI at 4 T. The stimuli were composed of horizontal stripes that scrolled up, scrolled down, or remained static, randomly presented for 1-s duration with stimulus-onset asynchronies (SOAs) of 2-7 s. We identified regions of interest (ROI) in lateral temporal--occipital cortex that were activated by motion stimuli, regardless of direction or SOA. We found strong evidence for direction specificity in motion-sensitive lateral temporal-occipital (LTO) cortex. For stimuli whose direction of motion reprised that of the previous stimulus (e.g., up preceded by up), the fMRI response was attenuated at short SOAs (2-4 s) compared to long SOAs (5-7 s). However, for stimuli whose direction of motion was opposite that of the previous stimulus (e.g., up preceded by down), little or no refractory effect was observed. Additionally, examination of activity in pericalcarine cortex indicated a similar pattern. We conclude that the fMRI refractory effect predominantly reflects local stimulus-specific neuronal or neurovascular adaptation and is unlikely to be a nonspecific response of large vessels that support broad functional regions.