Geometric deep learning enables 3D kinematic profiling across species and environments.

Journal Article (Journal Article)

Comprehensive descriptions of animal behavior require precise three-dimensional (3D) measurements of whole-body movements. Although two-dimensional approaches can track visible landmarks in restrictive environments, performance drops in freely moving animals, due to occlusions and appearance changes. Therefore, we designed DANNCE to robustly track anatomical landmarks in 3D across species and behaviors. DANNCE uses projective geometry to construct inputs to a convolutional neural network that leverages learned 3D geometric reasoning. We trained and benchmarked DANNCE using a dataset of nearly seven million frames that relates color videos and rodent 3D poses. In rats and mice, DANNCE robustly tracked dozens of landmarks on the head, trunk, and limbs of freely moving animals in naturalistic settings. We extended DANNCE to datasets from rat pups, marmosets, and chickadees, and demonstrate quantitative profiling of behavioral lineage during development.

Full Text

Duke Authors

Cited Authors

  • Dunn, TW; Marshall, JD; Severson, KS; Aldarondo, DE; Hildebrand, DGC; Chettih, SN; Wang, WL; Gellis, AJ; Carlson, DE; Aronov, D; Freiwald, WA; Wang, F; Ölveczky, BP

Published Date

  • May 1, 2021

Published In

Volume / Issue

  • 18 / 5

Start / End Page

  • 564 - 573

PubMed ID

  • 33875887

Pubmed Central ID

  • PMC8530226

Electronic International Standard Serial Number (EISSN)

  • 1548-7105

Digital Object Identifier (DOI)

  • 10.1038/s41592-021-01106-6

Language

  • eng

Conference Location

  • United States