Robot learning from videos

Chang, Matthew

Permalink

https://hdl.handle.net/2142/124193 Copy

Description

Title

Robot learning from videos

Author(s)

Chang, Matthew

Issue Date

2024-03-08

Director of Research (if dissertation) or Advisor (if thesis)

Gupta, Saurabh

Doctoral Committee Chair(s)

Gupta, Saurabh

Committee Member(s)

• Forsyth, David
• Lazebnik, Svetlana
• Chaplot, Devendra

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• computer vision
• robotics
• robot learning

Abstract

State-of-the-art machine learning models are extremely powerful and are finally breaking through into commercial products for computer vision and natural language processing. One common factor among these successful models is, they all require massive datasets for training. Following this trend, large-scale learning-based methods present a promising way forward for robotics research. This line of thinking naturally raises two questions: from where can we collect the appropriate data? and, how can it be leveraged to create effective robotic systems? Fortunately, a vast amount of data already exists, showcasing the complexity of real-world environments and interactions that robots need to understand, in the form of videos. However, these video sources of data cannot be directly used with the techniques traditionally applied for robot learning. Videos may lack explicit action or goal labels, often depict suboptimal trajectories, and present a significant embodiment gap, both visually and in dynamics. These challenges underscore the need for new robot learning methods that can overcome these obstacles. In this work, we present our efforts to realize the goal of robot learning at scale using in-the-wild videos, developing methods to address each of the challenges that limit robot learning from videos. We introduce techniques for inferring actions and goals in unlabelled video data, learning optimal behavior from sub-optimal data, and tackling the embodiment gap by leveraging factored representations. Overall, this dissertation lays the foundations for how video data can be leveraged for robot learning at scale. We hope this work can serve as a step towards general robotic agents that can make significant positive impacts in the world.

Graduation Semester

2024-05

Type of Resource

Thesis

Copyright and License Information

Copyright 2024 Matthew Chang

Owning Collections