CaltechTHESIS
  A Caltech Library Service

Optimal Data Distributions in Machine Learning

Citation

González Palacios, Carlos Roberto (2015) Optimal Data Distributions in Machine Learning. Dissertation (Ph.D.), California Institute of Technology. doi:10.7907/Z9DR2SD5. https://resolver.caltech.edu/CaltechTHESIS:05262015-094933189

Abstract

In the first part of the thesis we explore three fundamental questions that arise naturally when we conceive a machine learning scenario where the training and test distributions can differ. Contrary to conventional wisdom, we show that in fact mismatched training and test distribution can yield better out-of-sample performance. This optimal performance can be obtained by training with the dual distribution. This optimal training distribution depends on the test distribution set by the problem, but not on the target function that we want to learn. We show how to obtain this distribution in both discrete and continuous input spaces, as well as how to approximate it in a practical scenario. Benefits of using this distribution are exemplified in both synthetic and real data sets.

In order to apply the dual distribution in the supervised learning scenario where the training data set is fixed, it is necessary to use weights to make the sample appear as if it came from the dual distribution. We explore the negative effect that weighting a sample can have. The theoretical decomposition of the use of weights regarding its effect on the out-of-sample error is easy to understand but not actionable in practice, as the quantities involved cannot be computed. Hence, we propose the Targeted Weighting algorithm that determines if, for a given set of weights, the out-of-sample performance will improve or not in a practical setting. This is necessary as the setting assumes there are no labeled points distributed according to the test distribution, only unlabeled samples.

Finally, we propose a new class of matching algorithms that can be used to match the training set to a desired distribution, such as the dual distribution (or the test distribution). These algorithms can be applied to very large datasets, and we show how they lead to improved performance in a large real dataset such as the Netflix dataset. Their computational complexity is the main reason for their advantage over previous algorithms proposed in the covariate shift literature.

In the second part of the thesis we apply Machine Learning to the problem of behavior recognition. We develop a specific behavior classifier to study fly aggression, and we develop a system that allows analyzing behavior in videos of animals, with minimal supervision. The system, which we call CUBA (Caltech Unsupervised Behavior Analysis), allows detecting movemes, actions, and stories from time series describing the position of animals in videos. The method summarizes the data, as well as it provides biologists with a mathematical tool to test new hypotheses. Other benefits of CUBA include finding classifiers for specific behaviors without the need for annotation, as well as providing means to discriminate groups of animals, for example, according to their genetic line.

Item Type:Thesis (Dissertation (Ph.D.))
Subject Keywords:dual distribution, covariate shift, domain adaptation, supervised learning, targeted matching, soft matching, behavior analysis
Degree Grantor:California Institute of Technology
Division:Engineering and Applied Science
Major Option:Electrical Engineering
Awards:Charles And Ellen Wilts Prize, 2015
Thesis Availability:Public (worldwide access)
Research Advisor(s):
  • Abu-Mostafa, Yaser S. (advisor)
  • Perona, Pietro (co-advisor)
Thesis Committee:
  • Abu-Mostafa, Yaser S. (chair)
  • Perona, Pietro (co-chair)
  • Vaidyanathan, P. P.
  • Bruck, Jehoshua
  • Yue, Yisong
Defense Date:22 May 2015
Non-Caltech Author Email:carlos.r.gonzalez (AT) gmail.com
Record Number:CaltechTHESIS:05262015-094933189
Persistent URL:https://resolver.caltech.edu/CaltechTHESIS:05262015-094933189
DOI:10.7907/Z9DR2SD5
Default Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:8888
Collection:CaltechTHESIS
Deposited By: Carlos Gonzalez
Deposited On:02 Jun 2015 15:15
Last Modified:08 Nov 2023 00:44

Thesis Files

[img]
Preview
PDF (Full Thesis) - Final Version
See Usage Policy.

1MB
[img]
Preview
PDF (Cover-Acknowledgements-Abstract) - Final Version
See Usage Policy.

117kB
[img]
Preview
PDF (Table of Contents) - Final Version
See Usage Policy.

115kB
[img]
Preview
PDF (Chapter 1) - Final Version
See Usage Policy.

174kB
[img]
Preview
PDF (Chapter 2) - Final Version
See Usage Policy.

290kB
[img]
Preview
PDF (Chapter 3) - Final Version
See Usage Policy.

464kB
[img]
Preview
PDF (Chapter 4) - Final Version
See Usage Policy.

388kB
[img]
Preview
PDF (Chapter 5) - Final Version
See Usage Policy.

318kB
[img]
Preview
PDF (Chapter 6) - Final Version
See Usage Policy.

256kB
[img]
Preview
PDF (Chapter 7) - Final Version
See Usage Policy.

858kB
[img]
Preview
PDF (Conclusion) - Final Version
See Usage Policy.

54kB
[img]
Preview
PDF (Appendix) - Final Version
See Usage Policy.

181kB
[img]
Preview
PDF (Bibliography) - Final Version
See Usage Policy.

90kB

Repository Staff Only: item control page