DUT: learning video stabilization by simply watching unstable videos

Xu, Y. and Zhang, J. and Maybank, Stephen and Tao, D. (2022) DUT: learning video stabilization by simply watching unstable videos. IEEE Transactions on Image Processing 31 , pp. 4306-4320. ISSN 1057-7149.

Abstract

Previous deep learning-based video stabilizers require a large scale of paired unstable and stable videos for training, which are difficult to collect. Traditional trajectory-based stabilizers, on the other hand, divide the task into several sub-tasks and tackle them subsequently, which are fragile in textureless and occluded regions regarding the usage of hand-crafted features. In this paper, we attempt to tackle the video stabilization problem in a deep unsupervised learning manner, which borrows the divide-and-conquer idea from traditional stabilizers while leveraging the representation power of DNNs to handle the challenges in real-world scenarios. Technically, DUT is composed of a trajectory estimation stage and a trajectory smoothing stage. In the trajectory estimation stage, we first estimate the motion of keypoints, initialise and refine the motion of grids via a novel multi-homography estimation strategry and a motion refinement network, respectively, and get the grid-based trajectories via temporal association. In the trajectory smoothing phase we devise a novel network to predict dynamic smoothing kernels for trajectory smoothing, which can well adapt to trajectories with different dynamic patterns. We exploit the spatial and temporal coherence of keypoints and grid vertices to formulate the training objectives, resulting in an unsupervised training scheme. Experiment results on public benchmarks show that DUT outperforms state-of-the-art methods both qualitatively and quantitatively. The source code is available at https://github.com/Annbless/DUTCode.

Metadata

Statistics

DownloadsShow export options

Export as XMLJSONCSV

Activity Overview

6 month trend

210Downloads

6 month trend

93Hits

Additional statistics are available via IRStats2.

Archive Staff Only (login required)

Edit/View Item