Manuscript of my PhD thesis.

Abstract

In Reinforcement Learning (RL), an AI (agent) must learn an optimal sequence of decisions (actions) to maximize the sum of discounted rewards it gathers. This occurs through a trial-and-error loop, where the agent tries different actions in different scenarios and observes the resulting consequences and reward from the environment it evolves in. Traditional RL methods and algorithms focus uniquely on the reward provided by the environment, which leads to a highly specialized agent that can only solve the reward function it has been provided. This is problematic in several scenarios: first, if the reward function was not aligned with the actual need, and the learned behavior is not what we desired, training needs to start over from scratch. Next, if the reward function changes over time, or if we switch to a new reward after the first task was completed, we again need to start the learning process over from scratch. In general, we would like the agent to be able to learn how its environment works irrespective of reward, and use this knowledge to complete any task. Therefore, this dissertation is interested in learning representations and behaviors that are useful for the vast majority of tasks in a given environment, and which can be used to quickly obtain a behavior that performs greatly for any new task. I start by discussing the problem of directed exploration in RL, in which the agent should explicitly seek out unknown and interesting interactions with the environment. This is useful and necessary regardless of the task that should be accomplished. To maintain knowledge throughout tasks and time, I propose an algorithm based on the decoupling of exploration and exploitation. Next, I discuss the representation and behaviors the agent should learn to obtain reward-agnostic knowledge about its environment, which will be used to quickly produce strong behaviors on new tasks. I argue for Successor Features and propose a novel choice of features based on a partition of the state space. Finally, I propose to augment such learned behaviors and representations with decision-time planning, in other words, to enable the agent to explicitly plan several steps forward in the future. Such methods provide strong short-term benefits but are difficult to extend to long-term. I argue that this makes them the exact counterpart to learned representations such as Successor Features, and show that both work great in tandem. All the methods and claims in this dissertation are supported by quantitative and qualitative evidence through experiments on a wide variety of environments and methods. This ranges from simple tabular environments and approximators to complex problems such as video-games and robotics simulations, addressed using function approximation with deep neural networks.