GOAL: Cataracts are a clouding of the lens and the leading cause of blindness worldwide. Assessing the presence and severity of cataracts is essential for diagnosis and progression monitoring, as well as to facilitate clinical research and management of the disease. METHODS: Existing automatic methods for cataract grading utilize a predefined set of image features that may provide an incomplete, redundant, or even noisy representation. In this study, we propose a system to automatically learn features for grading the severity of nuclear cataracts from slit-lamp images. Local filters are first acquired through clustering of image patches from lenses within the same grading class. The learned filters are fed into a convolutional neural network, followed by a set of recursive neural networks, to further extract higher order features. With these features, support vector regression is applied to determine the cataract grade. RESULTS: The proposed system is validated on a large population-based dataset of [Formula: see text] images, where it outperforms the state of the art by yielding with respect to clinical grading a mean absolute error ( ε) of 0.304, a 70.7% exact integral agreement ratio ( R0), an 88.4% decimal grading error ≤ 0.5 ( Re0.5 ), and a 99.0% decimal grading error ≤ 1.0 ( Re1.0 ). SIGNIFICANCE: The proposed method is useful for assisting and improving clinical management of the disease in the context of large-population screening and has the potential to be applied to other eye diseases.