A Caltech Library Service

Computational Methods for Behavior Analysis


Eyjolfsdottir, Eyrun-Arna (2017) Computational Methods for Behavior Analysis. Dissertation (Ph.D.), California Institute of Technology. doi:10.7907/Z9445JJH.


Behavioral scientists strive to decode the functional relationship between sensory input and motor output of the brain, which requires quantitive measurement of animal behavior. Artificial intelligence researchers aim to build intelligent systems, capable of understanding, predicting, and generating behavior. Our research lies on the intersection of the two fields; our goal is to automate measurement of animal behavior and to model their sensory-motor relationship using machine learning.

We have developed a tool that tracks the pose of multiple fruit flies and aims to maintain their identity throughout a video. It outputs motion trajectories that can be used to quantify behavioral differences between individuals, for example by comparing histograms of velocities and wing angles. We show that the tool also works well on non-fly-like animals such as zebrafish larvae.

Embedded in these motion trajectories are temporal patterns that constitute actions. We developed two supervised learning frameworks for action detection: a sliding window framework and a structured output framework. Both frameworks learn to classify actions from motion trajectories and expert annotated action intervals. Our results show that the simpler sliding window framework achievers better results in spite of being much faster to train, reaching 90% of human performance.

Supervised learning requires a lot of training data which involves time consuming and painstaking annotation. To alleviate that we have built a semi-supervised neural network framework that, in addition to classifying actions, learns to predict how an animal will move next given its motion and sensory inputs so far. Our model archives as good results as its supervised counterpart with only half of the expert labels. In addition, we show that motion prediction can be used to generate convincing simulations of fruit fly behavior and handwritten text, and that our model learns to represent high level information, such as identity, when trained unsupervised.

Although developed for animal behavior, our methods are general and could be applied to other motion data. We hope that this thesis demonstrates the value of studying animal behavior for the development of artificial intelligence.

Item Type:Thesis (Dissertation (Ph.D.))
Subject Keywords:machine learning; computer vision; fly tracking; action detection; behavior analysis
Degree Grantor:California Institute of Technology
Division:Engineering and Applied Science
Major Option:Computer Science
Thesis Availability:Public (worldwide access)
Research Advisor(s):
  • Perona, Pietro
Thesis Committee:
  • Perona, Pietro (chair)
  • Yue, Yisong
  • Anderson, David J.
  • Fowlkes, Charless C.
Defense Date:16 September 2016
Record Number:CaltechTHESIS:06052017-121818057
Persistent URL:
Related URLs:
URLURL TypeDescription from Chapter 2 used in this article. from Chapter 2 used in this article. adapted for Chapter 3. adapted for Chapter 4.
Default Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:10281
Deposited By: Eyrun Eyjolfsdottir
Deposited On:07 Jun 2017 21:08
Last Modified:08 Nov 2023 00:44

Thesis Files

PDF - Final Version
See Usage Policy.


Repository Staff Only: item control page