TranDynaMo: A comparative study of transformer and GRU performance in modeling dynamical systems

Pandey, Amogh

Permalink

https://hdl.handle.net/2142/122269 Copy

Description

Title

TranDynaMo: A comparative study of transformer and GRU performance in modeling dynamical systems

Author(s)

Pandey, Amogh

Issue Date

2023-12-06

Director of Research (if dissertation) or Advisor (if thesis)

Mehr, Negar

Department of Study

Electrical & Computer Eng

Discipline

Electrical & Computer Engr

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

M.S.

Degree Level

Thesis

Keyword(s)

• Transformers, Large Language Models
• ChatGPT
• GPT
• LLM
• Dynamics Learning
• Dynamics Modeling
• RNN
• GRU
• Control System Dynamics
• Dynamical Models

Abstract

Dynamics modeling is critical to many robotics and control tasks and applications, such as motion planning and trajectory optimization. However, obtaining a model through current methods is difficult and expensive, so instead, there have been many efforts to come up with data-driven methods to learn dynamics models. With the popularity and success of Large Language Models (LLMs) such as ChatGPT, we were inspired to find a way to leverage transformers, the building blocks of ChatGPT, to solve the dynamics learning problem. In this thesis, we study the viability of utilizing transformers for dynamics modeling. We do this by first introducing transformers and posing dynamics learning as a sequence-to-sequence modeling task. We then create a GPT2-based dynamics learning framework called TranDynaMo and test its dynamics modeling performance for various dynamical systems, such as chaotic systems and systems with limit cycles, by giving it an initial condition and seeing how well it can predict the trajectory. We then compare TranDynaMo against a GRU-based framework. We find that when the dynamics learning framework is required to predict parts of a trajectory significantly beyond the horizon of the training trajectories it was given, TranDynaMo beats the GRU-based framework; however, if the training and testing trajectories are of almost the same horizon, both perform satisfactorily, but the GRU-based framework outperforms TranDynaMo, thus indicating that the generalization of transformers is better.

Graduation Semester

2023-12

Type of Resource

Thesis

Copyright and License Information

Copyright 2023 Amogh Pandey

Owning Collections