University of Illinois Urbana-Champaign

An exploration of advancing efficiency in learning-based systems

Wang, Yite

Content Files
WANG-DISSERTATION-2024.pdf

Permalink

https://hdl.handle.net/2142/124371

Description

Title
An exploration of advancing efficiency in learning-based systems
Author(s)
Wang, Yite
Issue Date
2024-04-24
Director of Research (if dissertation) or Advisor (if thesis)
  • Hovakimyan, Naira
  • Sun, Ruoyu
Doctoral Committee Chair(s)
Hovakimyan, Naira
Committee Member(s)
  • Etesami, Seyed
  • Li, Yingying
Department of Study
Mechanical Sci & Engineering
Discipline
Mechanical Engineering
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
Ph.D.
Degree Level
Dissertation
Keyword(s)
  • Efficient Learning
  • Optimization
  • Statistical Learning
  • Learning-based System
  • Data Science
  • Deep Learning
Abstract
Over the past decade, learning-based systems like deep neural networks (DNNs) have demonstrated remarkable success across a range of applications, such as machine vision, natural language processing, computational physics, and robotics. However, these modern DNNs often have a large number of parameters, leading to high computational demands in both training and deployment. This issue is particularly acute in the context of large-scale models like the generative pretrained transformer, e.g., GPT-3, where training expenses can cost millions of dollars. Moreover, the deployment of these models in resource-constrained environments, such as mobile robotics, drones, and intelligent driving systems, further exacerbates the challenge, hindering the full exploitation of DNNs' potential. In my Ph.D. research, I have dedicated my efforts to addressing these computational burdens, focusing on both the training and inference stages. From large to small. The initial part of my work focuses on developing more efficient neural networks through neural architecture search (NAS) and pruning, tailored to various optimization problems. We start with the application for classical minimization problems, leveraging the Neural Tangent Kernel (NTK) theory in Chapter 3 to introduce a novel pruning-at-initialization method tailored for image classification tasks, thereby optimizing computational efficiency. Progressing to bi-level optimization problem, i.e., meta-learning ('learning to learn'), in chapter 4, we explore NAS to automate the process of finding efficient neural network architecture, based on the theoretical insights provided by the NTK theory. Finally, we study the min-max problem associated with generative adversarial neural networks (GANs) in chapter 5. We investigate pruning-during-training method to reduce both training and inference costs of sparse GANs. From small to large. The latter part of my research focuses on transferring knowledge from small networks to larger networks for efficient training. In chapter 6, we present a novel approach to initialize large neural networks by leveraging pre-trained smaller networks, thereby substantially reducing the computational demands of the training process. The effectiveness of these methodologies is empirically demonstrated across a spectrum of applications. Our proposed methods have shown remarkable performance in tasks such as image generation, classification, and language understanding, setting new benchmarks for computational efficiency. Overall, this body of work not only contributes to the practicality and sustainability of DNNs in resource-constrained scenarios but also hopes to inspire future advancements in the field, paving the way for more efficient and accessible learning-based systems.
Graduation Semester
2024-05
Type of Resource
Thesis
Handle URL
https://hdl.handle.net/2142/124371
Copyright and License Information
Copyright 2024 Yite Wang

Owning Collections

Graduate Dissertations and Theses at Illinois PRIMARY
Graduate Theses and Dissertations at Illinois