The responses of visual neurons in experimental animals have been extensively characterized. To ask whether these responses are consistent with a wholly empirical concept of visual perception, we optimized simple neural networks that responded according to the cumulative frequency of occurrence of local luminance patterns in retinal images. Based on this estimation of accumulated experience, the neuron responses showed classical center-surround receptive fields, luminance gain control and contrast gain control, the key properties of early level visual neurons determined in animal experiments. These results imply that a major purpose of pre-cortical neuronal circuitry is to contend with the inherently uncertain significance of luminance values in natural stimuli.