University of Illinois Urbana-Champaign

Natural language image description: data, models, and evaluation

Hodosh, Micah A

Permalink

https://hdl.handle.net/2142/89008

Description

Title
Natural language image description: data, models, and evaluation
Author(s)
Hodosh, Micah A
Issue Date
2015-11-25
Director of Research (if dissertation) or Advisor (if thesis)
Hockenmaier, Julia
Doctoral Committee Chair(s)
Hockenmaier, Julia
Committee Member(s)
  • Dolan, Bill
  • Forsyth, David
  • Roth, Dan
Department of Study
Computer Science
Discipline
Computer Science
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
Ph.D.
Degree Level
Dissertation
Keyword(s)
  • Computer Vision
  • Natural Language Processing
  • Image Description
  • Neural Networks
  • computer vision (CV)
  • natural language processing (NLP)
  • Machine Learning
  • Machine Learning Applications
  • Image Captioning
Abstract
Automatically describing an image with a concise natural language description is an ambitious and emerging task bringing together the Natural Language and Computer Vision communities. With any emerging task, the necessary groundwork developing appropriate datasets, strong baseline models, and evaluation frameworks is key. In this thesis, we introduce the rst large datasets speci cally designed with image description in mind, focusing on concrete descriptions that can be gleaned from the image alone. Furthermore, we develop strong baseline models that show the need to model language beyond a simple bag-of-words approach to increase performance. Most importantly, we introduce a ranking based framework for comparing image description models. We show that this framework is more reliable and accurate than the conventional wisdom of evaluating on novel model generated text. As this task has gained popularity recently, we further analyze the drawbacks of current evaluation methods, and put forth concrete extensions to our ranking framework that will guide progress towards modeling the association of natural language and the images the language describes.
Graduation Semester
2015-12
Type of Resource
text
Permalink
http://hdl.handle.net/2142/89008
Copyright and License Information
Copyright 2015 Micah A Hodosh

Owning Collections

Graduate Dissertations and Theses at Illinois PRIMARY
Graduate Theses and Dissertations at Illinois
Dissertations and Theses - Computer Science
Dissertations and Theses from the Dept. of Computer Science