Towards controllable and consistent 3D scene editing

Dong, Jiahua

Permalink

https://hdl.handle.net/2142/124714 Copy

Description

Title

Towards controllable and consistent 3D scene editing

Author(s)

Dong, Jiahua

Issue Date

2024-05-01

Director of Research (if dissertation) or Advisor (if thesis)

Wang, Yuxiong

Department of Study

Computer Science

Discipline

Computer Science

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

M.S.

Degree Level

Thesis

Keyword(s)

• 3D Editing
• Diffusion Model
• Computer Vision

Abstract

Recent advancements in diffusion-based generative models have unlocked the potential for transformative 3D content creation. However, existing approaches for 3D scene editing face challenges in efficiency, consistency, and controllability. In this thesis, we introduce two novel methods that address these limitations and provide more effective means for editing 3D scenes. Our first exploration focuses on global style editing with text instructions. We present ViCA-NeRF, the first view-consistency-aware method for 3D editing with text instructions. By leveraging depth information derived from neural radiance fields (NeRF) and aligning latent codes in the 2D diffusion model, ViCA-NeRF ensures multi-view consistency through geometric and learned regularization. Our two-stage approach, involving edit blending and refinement, results in more flexible, efficient, and detailed editing compared to state-of-the-art methods. Secondly, we introduce DragGaussian, an interactive framework for fine-grained 3D scene editing using intuitive drag manipulation. Our method ensures 3D consistency through Gaussian deformation-based geometric guidance and an efficient inverse-free drag editing approach. By fine-tuning 2D diffusion models with a history-aware tuning strategy, DragGaussian enables high-quality drag-based editing of geometry and appearance, supporting various manipulations such as object movement, pose adjustment, and shape modification. Experimental results demonstrate the effectiveness and versatility of our proposed methods. ViCA-NeRF achieves a 3x speedup in editing time while maintaining higher levels of consistency and detail. DragGaussian enables diverse edits within a 10-20 minute timeframe on a single GPU, showcasing its efficiency. By combining text-based instructions and interactive point-based manipulation, our methods significantly advance the field of 3D scene editing, providing more efficient, consistent, and controllable tools for content creation.

Graduation Semester

2024-05

Type of Resource

Thesis

Copyright and License Information

Copyright 2024 Jiahua Dong

Owning Collections