University of Illinois Urbana-Champaign

Learning value functions from undirected state-only experience

Gupta, Arjun

Content Files
GUPTA-THESIS-2022.pdf

Permalink

https://hdl.handle.net/2142/117822

Description

Title
Learning value functions from undirected state-only experience
Author(s)
Gupta, Arjun
Issue Date
2022-12-06
Director of Research (if dissertation) or Advisor (if thesis)
Gupta, Saurabh
Department of Study
Electrical & Computer Eng
Discipline
Electrical & Computer Engr
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
M.S.
Degree Level
Thesis
Keyword(s)
  • Reinforcement Learning
  • Offline RL
  • Offline RL without actions
Abstract
This thesis tackles the problem of learning value functions from undirected state-only experience (state transitions without action labels i.e. (s,s′,r) tuples). We first theoretically characterize the applicability of Q-learning in this setting. We show that tabular Q-learning in discrete Markov decision processes (MDPs) learns the same value function under any arbitrary refinement of the action space. This theoretical result motivates the design of Latent Action Q-learning or LAQ, an offline RL method that can learn effective value functions from state-only experience. Latent Action Q-learning (LAQ) learns value functions using Q-learning on discrete latent actions obtained through a latent-variable future prediction model. We show that LAQ can recover value functions that have high correlation with value functions learned using ground truth actions. Value functions learned using LAQ lead to sample efficient acquisition of goal-directed behavior, can be used with domain-specific low-level controllers, and facilitate transfer across embodiments. Our experiments in 5 environments ranging from 2D grid world to 3D visual navigation in realistic environments demonstrate the benefits of LAQ over simpler alternatives, imitation learning oracles, and competing methods.
Graduation Semester
2022-12
Type of Resource
Thesis
Handle URL
https://hdl.handle.net/2142/117822
Copyright and License Information
Copyright 2022 Arjun Gupta

Owning Collections

Graduate Dissertations and Theses at Illinois PRIMARY
Graduate Theses and Dissertations at Illinois