A Caltech Library Service

New Algorithms for Programmatic Deep Learning with Applications to Behavior Modeling


Zhan, Eric (2022) New Algorithms for Programmatic Deep Learning with Applications to Behavior Modeling. Dissertation (Ph.D.), California Institute of Technology. doi:10.7907/5n5q-x203.


Raw behavioral data is becoming increasingly more abundant and more easily obtainable in spatiotemporal domains such as sports, video games, navigation & driving, motion capture, and animal science. How can we best use this data to advance their respective domains forward? For instance, researchers for self-driving vehicles would like to identify the key features of the environment state that impact decision-making the most; game developers would like to populate their games with characters that have unique and diverse behaviors to create a more immersive gaming experience; and behavioral neuroscientists would like to uncover the underlying mechanisms that drive learning in animals. Machine learning, the science of developing models and algorithms to identify and leverage patterns in data, is well-equipped to aid in these endeavors. But how do we integrate machine learning with these spatiotemporal domains in a principled way? In this dissertation, we develop and introduce new algorithms in programmatic deep learning that tackle some of the new challenges encountered in behavior modeling.

Our work in programmatic deep learning comprises two main themes: in the first, we show how to use expert-written programs as sources of weak labels in domains where manually-annotated expert labels are scarce; in the second, we explore programs as a flexible function class with human-interpretable structure and show how to learn them via neurosymbolic program learning. Augmenting deep learning with programmatic structure allows domain experts to easily incorporate domain knowledge into machine learning models; we show that this results in significant improvements in many behavior modeling applications like imitation learning, controllable generation, counterfactual analysis, and unsupervised clustering.

Item Type:Thesis (Dissertation (Ph.D.))
Subject Keywords:Machine Learning, Deep Learning, Behavior Modeling, Neurosymbolic Program Learning, Program Synthesis, Imitation Learning, Controllable Generation, Representation Learning, Unsupervised Clustering, Generative Modeling
Degree Grantor:California Institute of Technology
Division:Engineering and Applied Science
Major Option:Computing and Mathematical Sciences
Thesis Availability:Public (worldwide access)
Research Advisor(s):
  • Yue, Yisong
Thesis Committee:
  • Murray, Richard M. (chair)
  • Perona, Pietro
  • Chaudhuri, Swarat
  • Yue, Yisong
Defense Date:16 November 2021
Non-Caltech Author Email:zhaneric94 (AT)
Funding AgencyGrant Number
Northrop GrummanUNSPECIFIED
Record Number:CaltechTHESIS:11302021-224628633
Persistent URL:
Related URLs:
URLURL TypeDescription adapted for Ch. 2 adapted for Ch. 3 adapted for Ch. 4 adapted for Ch. 5
Zhan, Eric0000-0003-0521-5329
Default Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:14436
Deposited By: Eric Zhan
Deposited On:07 Dec 2021 19:05
Last Modified:14 Dec 2021 17:58

Thesis Files

[img] PDF - Final Version
See Usage Policy.


Repository Staff Only: item control page