Collaborative embodied agents

Jain, Unnat

Permalink

https://hdl.handle.net/2142/115410 Copy

Description

Title

Collaborative embodied agents

Author(s)

Jain, Unnat

Issue Date

2022-04-20

Director of Research (if dissertation) or Advisor (if thesis)

• Schwing, Alexander
• Lazebnik, Svetlana

Doctoral Committee Chair(s)

• Schwing, Alexander
• Lazebnik, Svetlana

Committee Member(s)

• Hoiem, Derek
• Jiang, Nan
• Grauman, Kristen

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• embodied agents
• visual navigation
• multi-agent systems
• multi-agent RL
• imitation learning
• simulation
• virtual environments
• embodied AI

Abstract

In recent years, research in computer vision has increasingly focused on solving realistic tasks in an interactive or embodied setting. Research in embodied agents lies at the intersection of computer vision, reinforcement learning, and language understanding. Prior research and tasks (such as navigation, question answering, language grounded navigation) have equipping agents with intelligent skills with a focus on a single agent. Going forward, we believe multi-agent learning can facilitate solving increasingly complex tasks such as driving, playing sports, or moving heavy objects, or rearranging them inside the house. Taking first steps in this direction, we build AI Agents that can collaborate and communicate in virtual visual worlds. Particularly, we'll discuss collaborative learning within both homogeneous and heterogeneous sets of agents. For collaborative homogeneous agents, we include (1) formulation of collaborative tasks and effect of 'explicit' and 'implicit' communication and (2) rich 'mixture-of-marginals' policies that overcome restrictions of existing decentralized multi-agent policies. For collaborative heterogeneous agents, we include research that (3) allow learning of embodied agents from free supervision from simplistic gridworlds via a 'GridToPix' methodology and (4) how to collaborate and learn from teachers that enjoy more privilege than the student. Overall, this dissertation lays the foundations of how visual AI agents can develop skills outside the silo of learning by themselves i.e. social learning. Going forward, it would be exciting to see how to learn from other agents in your surroundings from in-the-wild videos and test sim-to-real transfer of embodied ideas and results to robots.

Graduation Semester

2022-05

Type of Resource

Thesis

Copyright and License Information

Copyright 2022 Unnat Jain

Owning Collections