CaltechTHESIS
  A Caltech Library Service

Structured Signal Recovery from Nonlinear Measurements with Applications in Phase Retrieval and Linear Classification

Citation

Salehi, Fariborz (2021) Structured Signal Recovery from Nonlinear Measurements with Applications in Phase Retrieval and Linear Classification. Dissertation (Ph.D.), California Institute of Technology. doi:10.7907/1c69-wq71. https://resolver.caltech.edu/CaltechTHESIS:05172021-044724906

Abstract

Nonlinear models are widely used in signal processing, statistics, and machine learning to model real-world applications. A popular class of such models is the single-index model where the response variable is related to a linear combination of dependent variables through a link function. In other words, if x ∈ Rp denotes the input signal, the posterior mean of the generated output y has the form, E[y|x] = ρ(xTw), where ρ :R → R is a known function (referred to as the link function), and w ∈ Rp is the vector of unknown parameters. When ρ(•) is invertible, this class of models is called generalized linear models (GLMs). GLMs are commonly used in statistics and are often viewed as flexible generalizations of linear regression. Given n measurements (samples) from this model, D = {(xi, yi) | 1 ≤q i ≤ n}, the goal is to estimate the parameter vector w. While the model parameters are assumed to be unknown, in many applications these parameters follow certain structures (sparse, low-rank, group-sparse, etc.) The knowledge on this structure can be used to form more accurate estimators.

The main contribution of this thesis is to provide a precise performance analysis for convex optimization programs that are used for parameter estimation in two important classes of single-index models. These classes are: (1) phase retrieval in signal processing, and (2) binary classification in statistical learning.

The first class of models studied in this thesis is the phase retrieval problem, where the goal is to recover a discrete complex-valued signal from amplitudes of its linear combinations. Methods based on convex optimization have recently gained significant attentions in the literature. The conventional convex-optimization-based methods resort to the idea of lifting which makes them computationally inefficient. In addition to providing an analysis of the recovery threshold for the semidefinite-programming-based methods, this thesis studies the performance of a new convex relaxation for the phase retrieval problem, known as phasemax, which is computationally more efficient as it does not lift the signal to higher dimensions. Furthermore, to address the case of structured signals, regularized phasemax is introduced along with a precise characterization of the conditions for its perfect recovery in the asymptotic regime.

The next important application studied in this thesis is the binary classification in statistical learning. While classification models have been studied in the literature since 1950's, the understanding of their performance has been incomplete until very recently. Inspired by the maximum likelihood (ML) estimator in logistic models, we analyze a class of optimization programs that attempts to find the model parameters by minimizing an objective that consists of a loss function (which is often inspired by the ML estimator) and an additive regularization term that enforces our knowledge on the structure. There are two operating regimes for this problem depending on the separability of the training data set D. In the asymptotic regime, where the number of samples and the number of parameters grow to infinity, a phase transition phenomenon is demonstrated that happens at a certain over-parameterization ratio. We compute this phase transition for the setting where the underlying data is drawn from a Gaussian distribution.

In the case where the data is non-separable, the ML estimator is well-defined, and its attributes have been studied in the classical statistics. However, these classical results fail to provide reasonable estimate in the regime where the number of data points is proportional to the number of samples. One contribution of this thesis is to provide an exact analysis on the performance of the regularized logistic regression when the number of training data is proportional to the number of samples. When the data is separable (a.k.a. the interpolating regime), there exist multiple linear classifiers that perfectly fit the training data. In this regime, we introduce and analyze the performance of "extended margin maximizers" (EMMs). Inspired by the max-margin classifier, EMM classifiers simultaneously consider maximizing the margin and the structure of the parameter. Lastly, we discuss another generalization to the max-margin classifier, referred to as the robust max-margin classifier, that takes into account the perturbations by an adversary. It is shown that for a broad class of loss functions, gradient descent iterates (with proper step sizes) converge to the robust max-margin classifier.

Item Type:Thesis (Dissertation (Ph.D.))
Subject Keywords:Optimization, Convexity, High-dimensional Models, Phase Retrieval, PhaseMax, Linear Classification, Logistic Regression, Robust Classifiers
Degree Grantor:California Institute of Technology
Division:Engineering and Applied Science
Major Option:Electrical Engineering
Minor Option:Applied And Computational Mathematics
Thesis Availability:Public (worldwide access)
Research Advisor(s):
  • Hassibi, Babak
Thesis Committee:
  • Chandrasekaran, Venkat (chair)
  • Hassibi, Babak
  • Vaidyanathan, P. P.
  • Kostina, Victoria
Defense Date:15 March 2021
Funders:
Funding AgencyGrant Number
National Science Foundation (NSF)CNS-0932428
QualcommQualcomm Innovation Fellowship
National Science Foundation (NSF)CCF-1018927
National Science Foundation (NSF)CCF-1423663
Record Number:CaltechTHESIS:05172021-044724906
Persistent URL:https://resolver.caltech.edu/CaltechTHESIS:05172021-044724906
DOI:10.7907/1c69-wq71
Related URLs:
URLURL TypeDescription
https://doi.org/10.1109/IEEECONF44664.2019.9049040DOIArticle adapted for Ch. 2
https://doi.org/10.1109/ICASSP.2017.7952897DOIArticle adapted for Ch. 3
https://proceedings.neurips.cc/paper/2019/hash/dffbb6efd376d8dbb22cdf491e481edc-Abstract.htmlPublisherArticle adapted for Ch. 4
https://doi.org/10.1109/ISIT.2018.8437494DOIArticle adapted for Ch. 5
https://proceedings.neurips.cc/paper/2018/hash/b91f4f4d36fa98a94ac5584af95594a0-Abstract.htmlPublisherArticle adapted for Ch. 6
https://proceedings.neurips.cc/paper/2019/hash/ab49ef78e2877bfd2c2bfa738e459bf0-Abstract.htmlPublisherArticle adapted for Ch. 8
http://proceedings.mlr.press/v119/salehi20a.htmlPublisherArticle adapted for Ch. 9
https://arxiv.org/abs/2010.15391arXivArticle adapted for Ch. 10
ORCID:
AuthorORCID
Salehi, Fariborz0000-0002-9679-1016
Default Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:14150
Collection:CaltechTHESIS
Deposited By: Fariborz Salehi
Deposited On:27 May 2021 15:38
Last Modified:03 Jun 2021 15:54

Thesis Files

[img] PDF (PhD Thesis) - Final Version
See Usage Policy.

3MB

Repository Staff Only: item control page