Toward robust and efficient 3D reconstruction

Lee, Jae Yong

Permalink

https://hdl.handle.net/2142/124292 Copy

Description

Title

Toward robust and efficient 3D reconstruction

Author(s)

Lee, Jae Yong

Issue Date

2024-04-17

Director of Research (if dissertation) or Advisor (if thesis)

Hoiem, Derek

Doctoral Committee Chair(s)

Hoiem, Derek

Committee Member(s)

• Forsyth, David
• Wang, Shenlong
• Müller, Thomas

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• computer vision
• 3D reconstruction
• multi-view stereo
• neural rendering
• machine-learning

Abstract

In the post-smartphone era, everyone carries a personal camera, making 3D reconstruction more affordable than ever. This study aims to further enhance such affordability by exploring ways to make 3D reconstruction as efficient as possible. Our work is divided into three branches: (1) learning-based 3D geometry inference for sparse-view 3D reconstruction, (2) applying and improving geometric priors for sparse-view 3D reconstruction, and (3) designing efficient sampling and scene representation for 3D reconstruction. Initially, we focus on generating accurate 3D geometry with a small number of images captured sparsely. This step is crucial in efficient 3D reconstruction since dense camera coverage leads to longer processing times for both camera pose estimation in Structure-from-Motion (SfM) and Multi-View Stereo (MVS) 3D reconstruction. Unlike existing learning-based MVS systems that utilize a cost volume to target densely sampled views, we design an end-to-end learning-based MVS using PatchMatch, which efficiently solves for sparsely sampled views. After the introduction of Neural Radiance Fields (NeRF) as a high-quality, re-renderable 3D reconstruction representation, we focus on solving NeRF for sparse view inputs using geometric priors. These priors are necessary to overcome the ill-posedness in the sparse-view setup for NeRF, which often falls into local optima. We find that monocular priors clearly help, while multi-view geometry further improves the results. Given that sparse multi-view geometry is often available, we investigate further improvements in geometric priors from sparse geometry. Optimizing and rendering with NeRF is known to be a time-consuming process. Recent advancements in making NeRF faster to train and render in real-time have been successful, yet the resulting models are large in size. We design a method to compress NeRF to an extreme, enabling the optimization to complete in minutes and providing a real-time, re-renderable 3D model in just a few hundred kilobytes in size. We show that up to 12 resulting models can be rendered on many lightweight devices, such as mobile phones, using WebGL.

Graduation Semester

2024-05

Type of Resource

Thesis

Copyright and License Information

Copyright 2024 Jae Yong Lee

Owning Collections