University of Illinois Urbana-Champaign

Towards image-based node embedding: A novel approach for enriching multimodal representation

Seo, Seung Byum

Content Files
SEO-THESIS-2022.pdf

Permalink

https://hdl.handle.net/2142/116119

Description

Title
Towards image-based node embedding: A novel approach for enriching multimodal representation
Author(s)
Seo, Seung Byum
Issue Date
2022-07-21
Director of Research (if dissertation) or Advisor (if thesis)
Delgosha, Payam
Department of Study
Mathematics
Discipline
Applied Mathematics
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
M.S.
Degree Level
Thesis
Keyword(s)
  • multimodal learning
  • representation learning
  • graph neural network
  • natural language processing
Abstract
While there have been advances in Natural Language Processing (NLP), their success is mainly gained by applying a self-attention mechanism into single or multi-modalities. While this approach has brought significant improvements in multiple downstream tasks, it fails to capture the interaction between different entities. Therefore, we propose MM-GATBT, a multimodal graph representation learning model that captures not only the relational semantics within one modality but also the interactions between different modalities. Specifically, the proposed method constructs image-based node embedding which contains relational semantics of entities. Our empirical results show that MM-GATBT achieves state-of-the-art results among all published papers on the MM-IMDb dataset.
Graduation Semester
2022-08
Type of Resource
Thesis
Handle URL
https://hdl.handle.net/2142/116119
Copyright and License Information
Copyright 2022 Seung Byum Seo

Owning Collections

Graduate Dissertations and Theses at Illinois PRIMARY
Graduate Theses and Dissertations at Illinois