Learning structured representations with hyperbolic embeddings

Sinha, Aditya

Permalink

https://hdl.handle.net/2142/124722 Copy

Description

Title

Learning structured representations with hyperbolic embeddings

Author(s)

Sinha, Aditya

Issue Date

2024-05-01

Director of Research (if dissertation) or Advisor (if thesis)

Zhao, Han

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

M.S.

Degree Level

Thesis

Keyword(s)

• representation learning
• hyperbolic geometry
• hierarchical labels
• OOD detection

Abstract

Most real-world data consists of a natural hierarchy or an inherent label structure that is either already available or can be constructed/inferred cheaply. However, majority of the existing models for representation learning either completely ignore this hierarchy, treating the labels as permutation invariant, or attempt to utilize this information using less desirable distance metrics with bounded dimensionality. This leads to distortion of the semantic context in the label hierarchy, and also adversely affect it's performance on the in-distribution (ID) classification task. In addition, in the context of real-world machine learning systems, Out-of-distribution (OOD) detection is an even more challenging and critical task to ensure reliability of the deployed models. However, the current distance-based approaches do not consider any structured knowledge and rely on a distance measurement from the ID cluster-centroids, learnt in a label invariant fashion. To approach these challenges, in this thesis, we propose using hyperbolic geometry for incorporating this rich structured hierarchy about the label space into the representation learning. We demonstrate that accurately embedding the label information can lead to more fine-grained learning of structure-informed features, which are discriminative and helpful for a variety of tasks. For this purpose, we propose a novel method HypCPCC: Hyperbolic Cophenetic Correlation Coefficient to embed the label hierarchy into features using a powerful hyperbolic geometry based tree regularization objective. Our proposed objective can easily be combined and optimized with any classification loss for improving representation learning. We also empirically demonstrate that HypCPCC accurately embeds the hierarchical relationships between the labels in web-scale real-world vision datasets and leads to learning semantically rich features that result in simultaneous improvements in the performance of both in-distribution (ID) classification tasks (upto 1%) and AUROC on Out-of-Distribution (OOD) detection tasks (upto 4%). Motivated by the expressiveness of hyperbolic geometry in embedding the tree-based structures, we also propose two principled non-parametric hyperbolic-distance based OOD detection scores: HypKNN+ and HypDist-O. We demonstrate that using these scores can lead to improvements in OOD detection FPR95 upto 2%. Finally, we also empirically show that the learnt features from our proposed methodology are geometrically and semantically more interpretable using hyperbolic visualizations, paving the way for explainable feature learning using hierarchical label information.

Graduation Semester

2024-05

Type of Resource

Thesis

Copyright and License Information

Copyright 2024 Aditya Sinha

Owning Collections