University of Illinois Urbana-Champaign

Studies of lighting for dense prediction and generation in computer vision

Soole, James

Content Files
SOOLE-THESIS-2024.pdf

Permalink

https://hdl.handle.net/2142/124287

Description

Title
Studies of lighting for dense prediction and generation in computer vision
Author(s)
Soole, James
Issue Date
2024-04-25
Director of Research (if dissertation) or Advisor (if thesis)
Forsyth, David A
Department of Study
Computer Science
Discipline
Computer Science
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
M.S.
Degree Level
Thesis
Keyword(s)
  • Computer Vision
  • Artificial Intelligence
  • Deep Learning
  • Generative AI
  • Intrinsics
  • Dense Prediction
Abstract
Modern dense prediction models do exceedingly well on benchmarks for the standard computer vision tasks of depth and normal estimation, object detection, and semantic segmentation. However, we show that the lighting in a scene has a significant effect on predictions, often producing inconsistent results for relit versions of the exact same scene. For surface normal prediction, we demonstrate that fine-tuning to enforce consistency under various lightings can mitigate this problem without sacrificing base accuracy of the pretrained model. Yet, such fine-tuning requires a dataset of relit scenes, which exist in limited quantity and are burdensome to produce. We therefore explore existing generative methods to create a synthetic relighting dataset, and propose our new method StyLitGAN. Based on StyleGAN architecture and the use of latent stylecode directions, StyLitGAN can realistically relight complex scenes without the need for labeled data. Fine-tuning a dense predictor with StyLitGAN images results in improvements comparable with that obtained by fine-tuning with true multi-illuminant images. We continue to investigate the effect of stylecode directions across different scenes, and show their use in producing desired results from reference images. We explore stylecode applications in explicitly-controllable scene lighting and uncover hints at their internal representation.
Graduation Semester
2024-05
Type of Resource
Thesis
Handle URL
https://hdl.handle.net/2142/124287
Copyright and License Information
Copyright 2024 James Soole

Owning Collections

Graduate Dissertations and Theses at Illinois PRIMARY
Graduate Theses and Dissertations at Illinois