Learning Patterns with Kernels and Learning Kernels from Patterns

Citation

Yoo, Gene Ryan (2021) Learning Patterns with Kernels and Learning Kernels from Patterns. Dissertation (Ph.D.), California Institute of Technology. doi:10.7907/c5fn-ac81. https://resolver.caltech.edu/CaltechTHESIS:09062020-172055989

Abstract

A major technique in learning involves the identification of patterns and their use to make predictions. In this work, we examine the symbiotic relationship between patterns and Gaussian process regression (GPR), which is mathematically equivalent to kernel interpolation. We introduce techniques where GPR can be used to learn patterns in denoising and mode (signal) decomposition. Additionally, we present the kernel flow (KF) algorithm which learns a kernels from patterns in the data with methodology inspired by cross validation. We further show how the KF algorithm can be applied to artificial neural networks (ANNs) to make improvements to learning patterns in images.

In our denoising and mode decomposition examples, we show how kernels can be constructed to estimate patterns that may be hidden due to data corruption. In other words, we demonstrate how to learn patterns with kernels. Donoho and Johnstone proposed a near-minimax method for reconstructing an unknown smooth function u from noisy data u + ζ by translating the empirical wavelet coefficients of u + ζ towards zero. We consider the situation where the prior information on the unknown function u may not be the regularity of u, but that of ℒu where ℒ is a linear operator, such as a partial differential equation (PDE) or a graph Laplacian. We show that a near-minimax approximation of u can be obtained by truncating the ℒ-gamblet (operator-adapted wavelet) coefficients of u + ζ. The recovery of u can be seen to be precisely a Gaussian conditioning of u + ζ on measurement functions with length scale dependent on the signal-to-noise ratio.

We next introduce kernel mode decomposition (KMD), which has been designed to learn the modes v_i = a_i(t)y_i(θ_i(t)) of a (possibly noisy) signal Σ_iv_i when the amplitudes a_i, instantaneous phases θ_i, and periodic waveforms y_i may all be unknown. GPR with Gabor wavelet-inspired kernels is used to estimate a_i, θ_i, and y_i. We show near machine precision recovery under regularity and separation assumptions on the instantaneous amplitudes a_i and frequencies ˙θ_i.

GPR and kernel interpolation require the selection of an appropriate kernel modeling the data. We present the KF algorithm, which is a numerical-approximation approach to this selection. The main principle the method utilizes is that a "good" kernel is able to make accurate predictions with small subsets of a training set. In this way, we learn a kernel from patterns. In image classification, we show that the learned kernels are able to classify accurately using only one training image per class and show signs of unsupervised learning. Furthermore, we introduce the combination of the KF algorithm with conventional neural-network training. This combination is able to train the intermediate-layer outputs of the network simultaneously with the final-layer output. We test the proposed method on Convolutional Neural Networks (CNNs) and Wide Residual Networks (WRNs) without alteration of their structure or their output classifier. We report reduced test errors, decreased generalization gaps, and increased robustness to distribution shift without significant increase in computational complexity relative to standard CNN and WRN training (with Drop Out and Batch Normalization).

As a whole, this work highlights the interplay between kernel techniques with pattern recognition and numerical approximation.

Thesis Files