Reliable and efficient machine learning under distribution shifts

Wang, Haoxiang

Permalink

https://hdl.handle.net/2142/127492 Copy

Description

Title

Reliable and efficient machine learning under distribution shifts

Author(s)

Wang, Haoxiang

Issue Date

2024-12-06

Director of Research (if dissertation) or Advisor (if thesis)

• Zhao, Han
• Li, Bo

Doctoral Committee Chair(s)

• Zhao, Han
• Li, Bo

Committee Member(s)

• Schwing, Alexander Gerhard
• Raginsky, Maxim

Department of Study

Electrical & Computer Eng

Discipline

Electrical & Computer Engr

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Machine Learning
• Deep Learning
• Distribution Shift

Abstract

Over the past decade, machine learning has emerged as a transformative technology, driving advancements across various domains including computer vision, natural language processing, biology, autonomous driving, and recommendation systems. These innovations have been fueled by the development of increasingly sophisticated algorithms, the availability of large-scale datasets, and the growth of computational power. However, a crucial challenge persists in applying machine learning models to real-world scenarios: their performance tends to degrade significantly when confronted with distribution shifts. This dissertation primarily focuses on two types of distribution shifts: task distribution shifts and data distribution shifts. In addressing task distribution shifts, we first develop a theoretical framework to analyze gradient-based meta-learning algorithms from an optimization perspective, leveraging the Neural Tangent Kernel (NTK) as a theoretical tool. We then bridge multi-task learning (MTL) and meta-learning within this NTK-based framework, providing a unified understanding of these learning paradigms. Finally, we tackle a real-world challenge of aligning large language models with diverse user preferences by developing a novel MTL algorithm called Directional Preference Alignment. Regarding data distribution shifts, we first investigate the paradigm of gradual domain adaptation (GDA) through theoretical analysis. This leads to a deeper understanding of GDA and inspires a novel algorithm: Generative Gradual Domain Adaptation with Optimal Transport. Further, we consider the domain generalization setup and develop a new class of provable algorithms, Invariant-feature Subspace Recovery, which can be used to mitigate spurious correlations learned by machine learning models. Finally, we introduce a previously unexplored challenge of data distribution shifts, which we term compositional generalization in the context of multi-domain multi-class classification. To address this challenge, we design a theory-principled fine-tuning algorithm, Compositional Feature Alignment, aimed at enhancing the compositional generalization ability of pre-trained models. Through these contributions, this dissertation advances the field of machine learning by improving model reliability and efficiency under task and data distribution shifts, and bridges the gap between theoretical frameworks and practical applications in various real-world application scenarios.

Graduation Semester

2024-12

Type of Resource

Thesis

Handle URL

https://hdl.handle.net/2142/127492

Copyright and License Information

Copyright 2024 Haoxiang Wang

Owning Collections