Guaranteed Policy Performance in Reinforcement Learning

Citation

Voloshin, Cameron (2024) Guaranteed Policy Performance in Reinforcement Learning. Dissertation (Ph.D.), California Institute of Technology. doi:10.7907/n2fg-e554. https://resolver.caltech.edu/CaltechTHESIS:06062024-061120491

Abstract

Decision-making is ubiquitous in everyday life. Increasingly, researchers are seeking answers on how to optimally solve sequential decision-making tasks. Thanks to recent availability of computation, advances in deep learning, and released open-sourced code, it has become easy to train a computational agent to make decisions in many domains. Nevertheless, in realistic scenarios where the consequences of failure are high, running a trained computational agent in the wild poses substantial risk.

The goal of this thesis is to develop and advance techniques that guarantee a learned agent does what we expect it to do. The thesis tackles two central questions:

1) Given an agent, how can we predict if it will perform desirably?

2) Can we structure the learning process to guarantee desirable post-learning performance?

On the former question, this thesis proposes multiple algorithms to evaluate such agents, finds factors that have high influence on the success of agent evaluation, and open-sources benchmarks for further development in the space.

On the latter question, this thesis formulates desirable agent behavior as a constrained optimization with varying types of constraints depending on the structure afforded to the practitioner. Constraining the search space over the learning process ensures post-learning behaviors will, by definition, perform as desired.

Thesis Files