The same target luminance in different contexts can elicit markedly different perceptions of brightness, a fact that has long puzzled vision scientists. Here we test the proposal that the visual system encodes not luminance as such but rather the statistical relationship of a particular luminance to all possible luminance values experienced in natural contexts during evolution. This statistical conception of vision was validated by using a database of natural scenes in which we could determine the probability distribution functions of co-occurring target and contextual luminance values. The distribution functions obtained in this way predict target brightness in response to a variety of challenging stimuli, thus explaining these otherwise puzzling percepts. That brightness is determined by the statistics of natural light patterns implies that the relevant neural circuitry is specifically organized to generate these probabilistic responses.