One of the most intriguing of the many discrepancies between perceived spatial relationships and the physical structure of visual stimuli is the Poggendorff illusion, when an obliquely oriented line that is interrupted no longer appears collinear. Although many different theories have been proposed to explain this effect, there has been no consensus about its cause. Here, we use a database of range images (i.e., images that include the distance from the image plane of every pixel in the scene) to show that the probability distribution of the possible locations of line segments across an interval in natural environments can fully account for all of the behavior of this otherwise puzzling phenomenon.