A Caltech Library Service

Learning Patterns with Kernels and Learning Kernels from Patterns


Yoo, Gene Ryan (2021) Learning Patterns with Kernels and Learning Kernels from Patterns. Dissertation (Ph.D.), California Institute of Technology. doi:10.7907/c5fn-ac81.


A major technique in learning involves the identification of patterns and their use to make predictions. In this work, we examine the symbiotic relationship between patterns and Gaussian process regression (GPR), which is mathematically equivalent to kernel interpolation. We introduce techniques where GPR can be used to learn patterns in denoising and mode (signal) decomposition. Additionally, we present the kernel flow (KF) algorithm which learns a kernels from patterns in the data with methodology inspired by cross validation. We further show how the KF algorithm can be applied to artificial neural networks (ANNs) to make improvements to learning patterns in images.

In our denoising and mode decomposition examples, we show how kernels can be constructed to estimate patterns that may be hidden due to data corruption. In other words, we demonstrate how to learn patterns with kernels. Donoho and Johnstone proposed a near-minimax method for reconstructing an unknown smooth function u from noisy data u + ζ by translating the empirical wavelet coefficients of u + ζ towards zero. We consider the situation where the prior information on the unknown function u may not be the regularity of u, but that of ℒu where ℒ is a linear operator, such as a partial differential equation (PDE) or a graph Laplacian. We show that a near-minimax approximation of u can be obtained by truncating the ℒ-gamblet (operator-adapted wavelet) coefficients of u + ζ. The recovery of u can be seen to be precisely a Gaussian conditioning of u + ζ on measurement functions with length scale dependent on the signal-to-noise ratio.

We next introduce kernel mode decomposition (KMD), which has been designed to learn the modes vi = ai(t)yi(θi(t)) of a (possibly noisy) signal Σivi when the amplitudes ai, instantaneous phases θi, and periodic waveforms yi may all be unknown. GPR with Gabor wavelet-inspired kernels is used to estimate ai, θi, and yi. We show near machine precision recovery under regularity and separation assumptions on the instantaneous amplitudes ai and frequencies ˙θi.

GPR and kernel interpolation require the selection of an appropriate kernel modeling the data. We present the KF algorithm, which is a numerical-approximation approach to this selection. The main principle the method utilizes is that a "good" kernel is able to make accurate predictions with small subsets of a training set. In this way, we learn a kernel from patterns. In image classification, we show that the learned kernels are able to classify accurately using only one training image per class and show signs of unsupervised learning. Furthermore, we introduce the combination of the KF algorithm with conventional neural-network training. This combination is able to train the intermediate-layer outputs of the network simultaneously with the final-layer output. We test the proposed method on Convolutional Neural Networks (CNNs) and Wide Residual Networks (WRNs) without alteration of their structure or their output classifier. We report reduced test errors, decreased generalization gaps, and increased robustness to distribution shift without significant increase in computational complexity relative to standard CNN and WRN training (with Drop Out and Batch Normalization).

As a whole, this work highlights the interplay between kernel techniques with pattern recognition and numerical approximation.

Item Type:Thesis (Dissertation (Ph.D.))
Subject Keywords:Gaussian Process Regression, Kernels, Pattern Learning, Denoising, Mode Decomposition, Image Classification, Machine Learning, Artificial Intelligence
Degree Grantor:California Institute of Technology
Division:Physics, Mathematics and Astronomy
Major Option:Mathematics
Thesis Availability:Public (worldwide access)
Research Advisor(s):
  • Owhadi, Houman
Thesis Committee:
  • Stuart, Andrew M. (chair)
  • Owhadi, Houman
  • Makarov, Nikolai G.
  • Schroeder, Peter
  • Tamuz, Omer
Defense Date:2 September 2020
Record Number:CaltechTHESIS:09062020-172055989
Persistent URL:
Related URLs:
URLURL TypeDescription adapted for Chapter II. 8-10 in article adapted for Chapter IV and V. adapted for Chapter VI. adapted for Chapter VII.
Yoo, Gene Ryan0000-0002-5319-5599
Default Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:13868
Deposited By: Gene Yoo
Deposited On:21 Sep 2020 15:35
Last Modified:28 Oct 2021 18:44

Thesis Files

PDF - Final Version
See Usage Policy.


Repository Staff Only: item control page