Robust motion tracking and classification of zebrafish with deep learning

Zebrafish serve as a key pre-clinical model for disease research. Studying disease progression requires long-term tracking of fish motion, particularly when considering the function of the musculoskeletal apparatus. While vision-based tracking has been studied, current approaches typically require substantial fine tuning and often lack robustness, especially when multiple fish need to be tracked and motion occurs along all three spatial dimensions. We propose a simplified setup and a 3D tracking method using only two cameras and two convolutional neural networks (CNN) to localize and track multiple fish robustly. We separate the localization of fish and their classification as an individual. This allows for simple training data acquisition by placing fish into the basin individually, without the need for manual annotation. We evaluate the approach on sets of two, five, and ten fish resulting in multiple object tracking accuracies (MOTA) of 0.997, 0.994, and 0.966, respectively. The mean MOTA over all scenarios is 0.99 compared to a mean MOTA of 0.89 for a state-of-the-art tracking algorithm (idtracker.ai). Finally, we present comprehensive data on the motion activity of individual fish in a 30-minute video sequence, demonstrating robust tracking and unique activity profiles of each fish.

Subjects

Deep learning

fish tracking

fish trajectory dataset

locomotion pattern

physical activity

DDC Class

006.3: Artificial Intelligence

570: Life Sciences, Biology

Funding(s)

GRK 3144: Multiskalen-Analytik und biomedizinische Bildgebung an Grenzflächen für die Skelettfunktion

More Funding Information

This work was supported in part by the Interdisciplinary Competence Center for Interface Research (ICCIR) through Hamburg University of Technology (TUHH) and University Medical Center Hamburg-Eppendorf (UKE), TUHH i3 Initiative, the RTG 3144 ‘‘Multiscale Imaging and Analytics of Interfaces in Musculoskeletal Health’’ funded by Deutsche Forschungsgemeinschaft (DFG) under Grant 547468385, in part by the Forschungszentrum Medizintechnik Hamburg (FMTHH) Funding Program under Grant 03fmthh2018, and in part by the Open Access Publishing Program of TUHH.

Lizenz

https://creativecommons.org/licenses/by/4.0/

Publication version

publishedVersion

Name

Robust_Motion_Tracking_and_Classification_of_Zebrafish_With_Deep_Learning.pdf

Type

Main Article

Size

3.44 MB

Format

Adobe PDF

Options

Robust motion tracking and classification of zebrafish with deep learning