Efficient and effective learning of text representations

Meng, Yu

Permalink

https://hdl.handle.net/2142/121956 Copy

Description

Title

Efficient and effective learning of text representations

Author(s)

Meng, Yu

Issue Date

2023-10-13

Director of Research (if dissertation) or Advisor (if thesis)

Han, Jiawei

Doctoral Committee Chair(s)

Han, Jiawei

Committee Member(s)

• Abdelzaher, Tarek
• Tong, Hanghang
• Zettlemoyer, Luke

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Representation Learning
• Natural Language Processing
• Text Mining

Abstract

Text representation learning has played a pivotal role in enabling a wide range of Natural Language Processing (NLP) tasks that involve the processing, analysis, and utilization of human-generated textual data. These representations are typically obtained by transforming raw texts into vectors via deep neural networks (e.g., Transformers). Recent advances in Large Language Models (LLMs) have demonstrated the great potential of learning generic text representations applicable across a wide spectrum of applications. This success is underpinned by two critical factors: (1) the utilization of extensive text data for training LLMs in both pretraining and fine-tuning, and (2) the scaling up of LLMs to encompass tens or even hundreds of billions of parameters. Consequently, training LLMs entails substantial costs, including the acquisition of massive labeled data and the infrastructure required to support these large models. Motivated by these challenges, my doctoral research seeks to develop efficient and effective methods for learning text representations, spanning several key subtopics: 1. Leveraging the spherical space for text representation learning (Chapter 2). While the conventional choice for representation space is Euclidean, the non-Euclidean spherical space exhibits superior abilities to capture semantic correlations through directional similarity. My work focuses on self-supervised techniques that harness the spherical representation space for text representation learning. 2. Discovering topical structures with spherical text representations (Chapter 3). Based on the text representations learned in the spherical space, I have developed methods that automatically discover topical structures from a given corpus by jointly modeling topical and textual semantics. 3. Generating training data with LLMs for label-efficient natural language understanding (NLU) (Chapter 4). Achieving robust performance on NLU tasks typically necessitates a substantial number of human-annotated training samples for fine-tuning pretrained text representations. To mitigate the demands of human labeling, I have developed a new paradigm that employs LLMs as training data generators to replace the human annotation process. These endeavors collectively contribute to the more efficient and effective learning of text representations, addressing the challenges posed by the resource-intensive nature of training and using LLMs.

Graduation Semester

2023-12

Type of Resource

Thesis

Copyright and License Information

Copyright 2023 Yu Meng

Owning Collections