Online Action Change Detection for Automatic Vision-based Ground Control of Aircraft
Classical action recognition algorithms require the user to pre-select the time window for the action by clipping the video or choosing the initial and final time frames. Recently, new deep learning algorithms have been developed to detect a key action from an untrimmed video. However, they are unsuitable for temporally segmenting continuous action sequences and are too computationally expensive for implementation on autonomous systems. This paper presents a fast and accurate online action change detection framework. Given a streaming RGB video, the algorithm is used to detect any changes in the sequence. If a new action is discovered, the action recognition module will be applied to classify the action. Compared to existing methods, this two-stage approach reduces computational cost by not applying the recognition algorithm on every time window and improves classification accuracy by locating the starting time step of each action. In a simulated airport environment created using Unreal Engine ™, the framework is demonstrated and validated by detecting and recognizing sequential gestures from a ground crew, who sends gesture commands to control the movement of an autonomous aircraft. A hybrid optimal controller is developed to combine the visual information obtained from the framework and prior information, such as airport map and reference lines, to control the aircraft to safely navigate to a terminal gate.
