Introducing builder perspective in minecraft architect utterance generation

Athani, Amit A

Permalink

https://hdl.handle.net/2142/120121 Copy

Description

Title

Introducing builder perspective in minecraft architect utterance generation

Author(s)

Athani, Amit A

Issue Date

2023-04-27

Director of Research (if dissertation) or Advisor (if thesis)

Hockenmaier, Julia

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

M.S.

Degree Level

Thesis

Keyword(s)

• NLP
• Natural Language Processing
• CNN
• ROI

Abstract

The task of Collaborative Dialogue in Minecraft involves developing interactive agents that can communicate with humans to collaboratively solve tasks in grounded scenarios. Previous attempts to solve this task have treated the tasks of Architect Utterance and Builder Action Prediction as separate problems, and independent models have been developed to solve each. However, a human architect would typically think about how to build the target structure before giving verbal instructions to the Builder. Therefore, in this work, we focus on the Architect Utterance generation task and explore how the Builder Action Prediction model can be used to improve utterance generation. We achieve this by using a customized builder that can view the Minecraft world through the Architect’s perspective and generate action sequences. These action sequences represent a simulation of what the Architect would do if they were acting as the Builder, allowing the Architect to think in terms of Builder actions. We leverage this Builder-For-Architect model to introduce action sequences and their effects on the world state into the Architect’s decision-making process.

Graduation Semester

2023-05

Type of Resource

Thesis

Copyright and License Information

Copyright 2023 Amit Athani

Owning Collections