This paper addresses the challenge of tracking small unmanned aerial systems in urban terrain. Tracking is made up of two parts, state estimation and data association. For low, slow, small targets in the presence of other non-Target movers with similar dynamics, data association is arguably the greatest challenge. In traditional trackers, measurements are associated with existing tracks based on hand-crafted dynamical models. In this paper, we present recurrent neural network (RNN) based tracker which uses both learned target dynamic models and classification scores to associate measurements with target tracks. Classification scores for each radar measurement are generated by a neural-network trained using in situ 'pattern-of-life' radar data. These scores together with location estimates are then passed to RNN trackers with long-short-Term-memory (LSTM) trained for each object class. In this paper, we present simulated and real data results indicating that classification scores significantly improves small UAS track holding times in urban environments.