University of Illinois Urbana-Champaign

ISR: Invariant Subspace Recovery

Si, Haozhe

This item is only available for download by members of the University of Illinois community. Students, faculty, and staff at the U of I may log in with your NetID and password to view the item. If you are trying to access an Illinois-restricted dissertation or thesis, you can request a copy through your library's Inter-Library Loan office or purchase a copy directly from ProQuest.

Permalink

https://hdl.handle.net/2142/113503

Description

Title
ISR: Invariant Subspace Recovery
Author(s)
Si, Haozhe
Contributor(s)
Li, Bo
Issue Date
2021-12
Keyword(s)
  • transfer learning
  • domain generalization
  • representation learning
Abstract
Domain generalization asks for models trained on a set of training environments to perform well on unseen test environments. Recently, a series of algorithms such as Invariant Risk Minimization (IRM) and its follow-up works have been proposed for domain generalization. These algorithms are empirically successful, however, under certain assumptions. The Risk of Invariant Risk Minimization (Rosenfeld et al. 2021) shows that IRM and its alternatives cannot generalize to unseen environments with O(ds) training environments, where ds is the dimension of the spurious feature space. Meanwhile, these algorithms are computationally costly given their complex optimization objectives. In this paper, we propose a novel algorithm, Invariant Subspace Recovery (ISR), that can achieve a provable domain generalization under Gaussian data models with O(ds) training environments. Notably, unlike IRM and its alternatives, our algorithm has a global convergence guarantee without any non-convexity issue. By making assumptions on the second-order moment of data distribution, we further proposed an algorithm that can work with O(1) training environments. Our experiment results on both synthesized and real-world image data show that applying ISR on features as a post-processing method can increase the accuracy of neural models in unseen test domains.
Type of Resource
text
Language
en
Permalink
http://hdl.handle.net/2142/113503

Owning Collections

Senior Theses - Electrical and Computer Engineering PRIMARY
The best of ECE undergraduate research