Towards image-based node embedding: A novel approach for enriching multimodal representation
Seo, Seung Byum
Loading…
Permalink
https://hdl.handle.net/2142/116119
Description
Title
Towards image-based node embedding: A novel approach for enriching multimodal representation
Author(s)
Seo, Seung Byum
Issue Date
2022-07-21
Director of Research (if dissertation) or Advisor (if thesis)
Delgosha, Payam
Department of Study
Mathematics
Discipline
Applied Mathematics
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
M.S.
Degree Level
Thesis
Keyword(s)
multimodal learning
representation learning
graph neural network
natural language processing
Abstract
While there have been advances in Natural Language Processing (NLP), their success is mainly gained by applying a self-attention mechanism into single or multi-modalities. While this approach has brought significant improvements in multiple downstream tasks, it fails to capture the interaction between different entities. Therefore, we propose MM-GATBT, a multimodal graph representation learning model that captures not only the relational semantics within one modality but also the interactions between different modalities. Specifically, the proposed method constructs image-based node embedding which contains relational semantics of entities. Our empirical results show that MM-GATBT achieves state-of-the-art results among all published papers on the MM-IMDb dataset.
Use this login method if you
don't
have an
@illinois.edu
email address.
(Oops, I do have one)
IDEALS migrated to a new platform on June 23, 2022. If you created
your account prior to this date, you will have to reset your password
using the forgot-password link below.
