University of Illinois Urbana-Champaign

Efficient and effective learning of text representations

Meng, Yu

Content Files
MENG-DISSERTATION-2023.pdf

Permalink

https://hdl.handle.net/2142/121956

Description

Title
Efficient and effective learning of text representations
Author(s)
Meng, Yu
Issue Date
2023-10-13
Director of Research (if dissertation) or Advisor (if thesis)
Han, Jiawei
Doctoral Committee Chair(s)
Han, Jiawei
Committee Member(s)
  • Abdelzaher, Tarek
  • Tong, Hanghang
  • Zettlemoyer, Luke
Department of Study
Computer Science
Discipline
Computer Science
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
Ph.D.
Degree Level
Dissertation
Keyword(s)
  • Representation Learning
  • Natural Language Processing
  • Text Mining
Abstract
Text representation learning has played a pivotal role in enabling a wide range of Natural Language Processing (NLP) tasks that involve the processing, analysis, and utilization of human-generated textual data. These representations are typically obtained by transforming raw texts into vectors via deep neural networks (e.g., Transformers). Recent advances in Large Language Models (LLMs) have demonstrated the great potential of learning generic text representations applicable across a wide spectrum of applications. This success is underpinned by two critical factors: (1) the utilization of extensive text data for training LLMs in both pretraining and fine-tuning, and (2) the scaling up of LLMs to encompass tens or even hundreds of billions of parameters. Consequently, training LLMs entails substantial costs, including the acquisition of massive labeled data and the infrastructure required to support these large models. Motivated by these challenges, my doctoral research seeks to develop efficient and effective methods for learning text representations, spanning several key subtopics: 1. Leveraging the spherical space for text representation learning (Chapter 2). While the conventional choice for representation space is Euclidean, the non-Euclidean spherical space exhibits superior abilities to capture semantic correlations through directional similarity. My work focuses on self-supervised techniques that harness the spherical representation space for text representation learning. 2. Discovering topical structures with spherical text representations (Chapter 3). Based on the text representations learned in the spherical space, I have developed methods that automatically discover topical structures from a given corpus by jointly modeling topical and textual semantics. 3. Generating training data with LLMs for label-efficient natural language understanding (NLU) (Chapter 4). Achieving robust performance on NLU tasks typically necessitates a substantial number of human-annotated training samples for fine-tuning pretrained text representations. To mitigate the demands of human labeling, I have developed a new paradigm that employs LLMs as training data generators to replace the human annotation process. These endeavors collectively contribute to the more efficient and effective learning of text representations, addressing the challenges posed by the resource-intensive nature of training and using LLMs.
Graduation Semester
2023-12
Type of Resource
Thesis
Handle URL
https://hdl.handle.net/2142/121956
Copyright and License Information
Copyright 2023 Yu Meng

Owning Collections

Graduate Dissertations and Theses at Illinois PRIMARY
Graduate Theses and Dissertations at Illinois