Efficient invariant feature subspace recovery for domain generalization

Balasubramaniam, Gargi

Permalink

https://hdl.handle.net/2142/120084 Copy

Description

Title

Efficient invariant feature subspace recovery for domain generalization

Author(s)

Balasubramaniam, Gargi

Issue Date

2023-04-24

Director of Research (if dissertation) or Advisor (if thesis)

Zhao, Han

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

M.S.

Degree Level

Thesis

Keyword(s)

• Domain Generalization
• Out-of-domain (OOD) Generalization
• Invariant Feature Learning
• Spurious Correlations

Abstract

Domain generalization aims to learn a machine learning model over multiple training environments to generalize well to unseen test environments. With the advent of large pretrained models, recent work focuses on efficient domain generalization to maximize performance under minimal data and model fine-tuning requirements. Recently, Wang et al. proposed Invariant Feature Subspace Recovery (ISR-Mean), an efficient domain generalization algorithm which uses means of the class-conditional data distributions to provably identify the domain-invariant feature subspace under a given causal model. However, ISR-Mean is not applicable in the realistic setting of multi-class classification as it only utilizes information from a single class, failing to account for information across multiple classes. Further, it is unclear how this notion of invariant subspace recovery can be applied to regression, when there are no class labels. Motivated by the need to enable efficient robustness via invariant feature subspace recovery in these scenarios, this work extends ISR to two novel settings of multi-class classification and regression. First, in multi-class classification, a more general causal model is proposed under which the ISR-Multiclass algorithm is introduced: ISR-Multiclass can provably recover the invariant feature subspace in $\lceil d_{spu}/k \rceil + 1$ environments where $d_{spu}$ is the dimensionality of spurious features and $k$ is the number of classes. Thus, ISR-Multiclass leverages class information to \textit{improve} the environment complexity by a factor of $k$ as compared to the original ISR-Mean, which requires $d_{spu} + 1$ environments. Next, in the setting of regression, ISR-Regression is introduced as a provable recovery algorithm under a new causal model for regression. ISR-Regression can identify the invariant feature subspace in $d_{spu} + 1$ environments, matching that of the original ISR algorithm. Empirically, ISR-Multiclass and ISR-Regression demonstrate superior performance across new synthetic linear benchmarks (in line with the theoretically claimed environment complexity) and significantly improve the robustness of neural networks trained with various methods (such as ERM, IRM) across synthetic and real-life datasets encoding strong spurious correlations - thus uncovering the generality, accuracy and efficiency of this framework.

Graduation Semester

2023-05

Type of Resource

Thesis

Copyright and License Information

Copyright 2023 Gargi Balasubramaniam

Owning Collections