Analyzing animal behavior via classifying each video frame using convolutional neural networks.

Journal Article (Journal Article)

High-throughput analysis of animal behavior requires software to analyze videos. Such software analyzes each frame individually, detecting animals' body parts. But the image analysis rarely attempts to recognize "behavioral states"-e.g., actions or facial expressions-directly from the image instead of using the detected body parts. Here, we show that convolutional neural networks (CNNs)-a machine learning approach that recently became the leading technique for object recognition, human pose estimation, and human action recognition-were able to recognize directly from images whether Drosophila were "on" (standing or walking) or "off" (not in physical contact with) egg-laying substrates for each frame of our videos. We used multiple nets and image transformations to optimize accuracy for our classification task, achieving a surprisingly low error rate of just 0.072%. Classifying one of our 8 h videos took less than 3 h using a fast GPU. The approach enabled uncovering a novel egg-laying-induced behavior modification in Drosophila. Furthermore, it should be readily applicable to other behavior analysis tasks.

Full Text

Duke Authors

Cited Authors

  • Stern, U; He, R; Yang, C-H

Published Date

  • September 23, 2015

Published In

Volume / Issue

  • 5 /

Start / End Page

  • 14351 -

PubMed ID

  • 26394695

Pubmed Central ID

  • PMC4585819

Electronic International Standard Serial Number (EISSN)

  • 2045-2322

Digital Object Identifier (DOI)

  • 10.1038/srep14351


  • eng

Conference Location

  • England