Studies of lighting for dense prediction and generation in computer vision

Soole, James

Permalink

https://hdl.handle.net/2142/124287 Copy

Description

Title

Studies of lighting for dense prediction and generation in computer vision

Author(s)

Soole, James

Issue Date

2024-04-25

Director of Research (if dissertation) or Advisor (if thesis)

Forsyth, David A

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

M.S.

Degree Level

Thesis

Keyword(s)

• Computer Vision
• Artificial Intelligence
• Deep Learning
• Generative AI
• Intrinsics
• Dense Prediction

Abstract

Modern dense prediction models do exceedingly well on benchmarks for the standard computer vision tasks of depth and normal estimation, object detection, and semantic segmentation. However, we show that the lighting in a scene has a significant effect on predictions, often producing inconsistent results for relit versions of the exact same scene. For surface normal prediction, we demonstrate that fine-tuning to enforce consistency under various lightings can mitigate this problem without sacrificing base accuracy of the pretrained model. Yet, such fine-tuning requires a dataset of relit scenes, which exist in limited quantity and are burdensome to produce. We therefore explore existing generative methods to create a synthetic relighting dataset, and propose our new method StyLitGAN. Based on StyleGAN architecture and the use of latent stylecode directions, StyLitGAN can realistically relight complex scenes without the need for labeled data. Fine-tuning a dense predictor with StyLitGAN images results in improvements comparable with that obtained by fine-tuning with true multi-illuminant images. We continue to investigate the effect of stylecode directions across different scenes, and show their use in producing desired results from reference images. We explore stylecode applications in explicitly-controllable scene lighting and uncover hints at their internal representation.

Graduation Semester

2024-05

Type of Resource

Thesis

Copyright and License Information

Copyright 2024 James Soole

Owning Collections