Sparse representation theory has been increasingly used in signal processing and machine learning. In this paper we introduce a hierarchical sparse modeling approach which integrates information from the image patch level to derive a mid-level invariant image and pattern representation. The proposed framework is based on a hierarchical architecture of dictionary learning for sparse coding in a cortical (log-polar) space, combined with a novel pooling operator which incorporates the Rapid transform and max pooling to attain rotation and scale invariance. The invariant sparse representation of patterns here presented - can be used in different object recognition tasks. Promising results are obtained for three applications - 2D shapes classification, texture recognition and object detection. © 2011 IEEE.