Geometry-based video prediction with camera motion for mobile robotics

Liang, Weihang

Permalink

https://hdl.handle.net/2142/115799 Copy

Description

Title

Geometry-based video prediction with camera motion for mobile robotics

Author(s)

Liang, Weihang

Issue Date

2022-04-29

Director of Research (if dissertation) or Advisor (if thesis)

Driggs-Campbell, Katherine Rose

Department of Study

Electrical & Computer Eng

Discipline

Electrical & Computer Engr

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

M.S.

Degree Level

Thesis

Keyword(s)

• Video Prediction
• Computer Vision
• Robotics
• Machine Learning

Abstract

Video prediction is one of the fundamental research problems in computer vision, and it has a wide range of applications in planning and control for robotics. Recent learning-based approaches show promising results on various video datasets, and some have seen successful applications in planning robot arm motion. However, predicting future observations in a sequence given a set of past images remains a challenging task in mobile robotics, especially when the camera is in motion. Early works in this area use deterministic approaches, which often yield visually unintuitive results due to the intrinsic variability of motion in the future. More recent works have adopted stochastic models to generate sharper future frames. However, most methods do not account for camera motion and perform poorly in scenarios with moving cameras when they are deployed on vehicles and mobile robots. To solve the challenging task of video prediction on mobile platforms, we propose a geometry-based prediction framework that combines visual odometry prediction and view synthesis. Based on a sequence of observed frames, our method first predicts future camera poses and extracts the 3D geometry of the world, which are then jointly used to generate predicted future frames. Specifically, we train a recurrent visual odometry prediction model conditioned on raw RGB images with ground-truth pose labels. In addition, we train SynSin, a view synthesis method to generate 2D images from novel viewpoints using a 3D world representation. By combining these approaches, we demonstrate that our hierarchical deterministic approach outperforms previous stochastic works on the KITTI dataset.

Graduation Semester

2022-05

Type of Resource

Thesis

Copyright and License Information

Copyright 2022 Weihang Liang

Owning Collections