Analysis of the neural code for sensory-motor latency in smooth pursuit eye movements reveals general principles of neural variation and the specific origin of motor latency. The trial-by-trial variation in neural latency in MT comprises a shared component expressed as neuron-neuron latency correlations and an independent component that is local to each neuron. The independent component arises heavily from fluctuations in the underlying probability of spiking, with an unexpectedly small contribution from the stochastic nature of spiking itself. The shared component causes the latency of single-neuron responses in MT to be weakly predictive of the behavioral latency of pursuit. Neural latency deeper in the motor system is more strongly predictive of behavioral latency. A model reproduces both the variance of behavioral latency and the neuron-behavior latency correlations in MT if it includes realistic neural latency variation, neuron-neuron latency correlations in MT, and noisy gain control downstream of MT.