NATURAL LANGUAGE GROUNDING FOR AGRICULTURAL ROBOTICS

Viloset, Michael

Permalink

https://hdl.handle.net/2142/124800 Copy

Description

Title

NATURAL LANGUAGE GROUNDING FOR AGRICULTURAL ROBOTICS

Author(s)

Viloset, Michael

Issue Date

2023-05-01

Keyword(s)

natural language processing; grounding; large language models; robotics; data engineering

Abstract

The purpose of this study is to utilize the generic knowledge of pre-trained Large Language Models (LLM) to enable robot interaction through natural language instruction. To demonstrate this, I have studied a plant planning and placement task implemented on a CNC based robot known as FarmBot [3]. The benefits are twofold: to interact with robots after little-to-no technical training, and to compress the amount of data required to convey complex ideas to a robot. If implemented, the reduction in learning curve would benefit the accessibility of the FarmBot. Natural Language (NL) is often too vague for a program to understand as it uses knowledge about the robot’s environment and previous human experience expressed in the form of abstract natural language descriptions. Therefore, the concept of abstraction is key in compressing a complex sequence of actions. Huge LLMs such as GPT-3 have been trained on billions of examples which enable them to associate any sentence with the nearest possible item in a database [1]. However, smaller open-source models still require fine-tuning to ground complex natural language commands that involve use of absolute or relative spatial descriptions such as, “weed plant X in the top-right corner and the opposite corner.” In this experiment, I attempt to teach a relatively small LLM model the semantics behind a mathematical constraint which represents a natural language abstraction. During inferencing, the model can place plants inside defined constraints with respect to the robot’s environment from natural language instruction. However, more general models such as GPT-4 might be the answer for more generalized constraint applications.

Type of Resource

text

Language

eng

Owning Collections