University of Illinois Urbana-Champaign

Robot learning from videos

Chang, Matthew

Content Files
CHANG-DISSERTATION-2024.pdf

Permalink

https://hdl.handle.net/2142/124193

Description

Title
Robot learning from videos
Author(s)
Chang, Matthew
Issue Date
2024-03-08
Director of Research (if dissertation) or Advisor (if thesis)
Gupta, Saurabh
Doctoral Committee Chair(s)
Gupta, Saurabh
Committee Member(s)
  • Forsyth, David
  • Lazebnik, Svetlana
  • Chaplot, Devendra
Department of Study
Computer Science
Discipline
Computer Science
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
Ph.D.
Degree Level
Dissertation
Keyword(s)
  • computer vision
  • robotics
  • robot learning
Abstract
State-of-the-art machine learning models are extremely powerful and are finally breaking through into commercial products for computer vision and natural language processing. One common factor among these successful models is, they all require massive datasets for training. Following this trend, large-scale learning-based methods present a promising way forward for robotics research. This line of thinking naturally raises two questions: from where can we collect the appropriate data? and, how can it be leveraged to create effective robotic systems? Fortunately, a vast amount of data already exists, showcasing the complexity of real-world environments and interactions that robots need to understand, in the form of videos. However, these video sources of data cannot be directly used with the techniques traditionally applied for robot learning. Videos may lack explicit action or goal labels, often depict suboptimal trajectories, and present a significant embodiment gap, both visually and in dynamics. These challenges underscore the need for new robot learning methods that can overcome these obstacles. In this work, we present our efforts to realize the goal of robot learning at scale using in-the-wild videos, developing methods to address each of the challenges that limit robot learning from videos. We introduce techniques for inferring actions and goals in unlabelled video data, learning optimal behavior from sub-optimal data, and tackling the embodiment gap by leveraging factored representations. Overall, this dissertation lays the foundations for how video data can be leveraged for robot learning at scale. We hope this work can serve as a step towards general robotic agents that can make significant positive impacts in the world.
Graduation Semester
2024-05
Type of Resource
Thesis
Handle URL
https://hdl.handle.net/2142/124193
Copyright and License Information
Copyright 2024 Matthew Chang

Owning Collections

Graduate Dissertations and Theses at Illinois PRIMARY
Graduate Theses and Dissertations at Illinois