University of Illinois Urbana-Champaign

Robust online video instance segmentation with track queries

Zhan, Zitong

This item's files can only be accessed by the Administrator group.

Permalink

https://hdl.handle.net/2142/121199

Description

Title
Robust online video instance segmentation with track queries
Author(s)
Zhan, Zitong
Issue Date
2023-07-17
Director of Research (if dissertation) or Advisor (if thesis)
Lazebnik, Svetlana
Department of Study
Computer Science
Discipline
Computer Science
Degree Granting Institution
University of Illinois at Urbana-Champaign
Degree Name
M.S.
Degree Level
Thesis
Keyword(s)
  • computer vision
  • video tracking
Abstract
With support of data and maturity of model, deep learning has been successful in high level understanding on videos. Recently, transformer-based methods have achieved impressive results on Video Instance Segmentation (VIS). However, most of these top-performing methods run in an offline manner by processing the entire video clip at once to predict instance mask volumes. This makes them incapable of handling the long videos that appear in challenging new video instance segmentation datasets like Unidentified Video Objects (UVO) and Occluded Video Instance Segmentation (OVIS). We propose a fully online transformer-based video instance segmentation model that performs comparably to top offline methods on the YouTube-VIS 2019 benchmark and considerably outperforms them on UVO and OVIS. This method, called Robust Online Video Segmentation (ROVIS), augments the Mask2Former image instance segmentation model with track queries, a lightweight mechanism for carrying track information from frame to frame, originally introduced by the TrackFormer method for multi-object tracking. We show that, when combined with a strong enough image segmentation architecture, track queries can exhibit impressive accuracy while not being constrained to short videos. Lastly, we extend ROVIS to the Refer-VOS task which is highly relevant to VIS, such that ROVIS can make segmentation predictions based on language input, and present our preliminary results.
Graduation Semester
2023-08
Type of Resource
Thesis
Copyright and License Information
Copyright 2023 Zitong Zhan

Owning Collections

Graduate Dissertations and Theses at Illinois PRIMARY
Graduate Theses and Dissertations at Illinois