An exploration of advancing efficiency in learning-based systems

Wang, Yite

Permalink

https://hdl.handle.net/2142/124371 Copy

Description

Title

An exploration of advancing efficiency in learning-based systems

Author(s)

Wang, Yite

Issue Date

2024-04-24

Director of Research (if dissertation) or Advisor (if thesis)

• Hovakimyan, Naira
• Sun, Ruoyu

Doctoral Committee Chair(s)

Hovakimyan, Naira

Committee Member(s)

• Etesami, Seyed
• Li, Yingying

Department of Study

Mechanical Sci & Engineering

Discipline

Mechanical Engineering

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Efficient Learning
• Optimization
• Statistical Learning
• Learning-based System
• Data Science
• Deep Learning

Abstract

Over the past decade, learning-based systems like deep neural networks (DNNs) have demonstrated remarkable success across a range of applications, such as machine vision, natural language processing, computational physics, and robotics. However, these modern DNNs often have a large number of parameters, leading to high computational demands in both training and deployment. This issue is particularly acute in the context of large-scale models like the generative pretrained transformer, e.g., GPT-3, where training expenses can cost millions of dollars. Moreover, the deployment of these models in resource-constrained environments, such as mobile robotics, drones, and intelligent driving systems, further exacerbates the challenge, hindering the full exploitation of DNNs' potential. In my Ph.D. research, I have dedicated my efforts to addressing these computational burdens, focusing on both the training and inference stages. From large to small. The initial part of my work focuses on developing more efficient neural networks through neural architecture search (NAS) and pruning, tailored to various optimization problems. We start with the application for classical minimization problems, leveraging the Neural Tangent Kernel (NTK) theory in Chapter 3 to introduce a novel pruning-at-initialization method tailored for image classification tasks, thereby optimizing computational efficiency. Progressing to bi-level optimization problem, i.e., meta-learning ('learning to learn'), in chapter 4, we explore NAS to automate the process of finding efficient neural network architecture, based on the theoretical insights provided by the NTK theory. Finally, we study the min-max problem associated with generative adversarial neural networks (GANs) in chapter 5. We investigate pruning-during-training method to reduce both training and inference costs of sparse GANs. From small to large. The latter part of my research focuses on transferring knowledge from small networks to larger networks for efficient training. In chapter 6, we present a novel approach to initialize large neural networks by leveraging pre-trained smaller networks, thereby substantially reducing the computational demands of the training process. The effectiveness of these methodologies is empirically demonstrated across a spectrum of applications. Our proposed methods have shown remarkable performance in tasks such as image generation, classification, and language understanding, setting new benchmarks for computational efficiency. Overall, this body of work not only contributes to the practicality and sustainability of DNNs in resource-constrained scenarios but also hopes to inspire future advancements in the field, paving the way for more efficient and accessible learning-based systems.

Graduation Semester

2024-05

Type of Resource

Thesis

Copyright and License Information

Copyright 2024 Yite Wang

Owning Collections