Regenerating Arbitrary Video Sequences with

Distillation Path-Finding

Thi-Ngoc-Hanh Le, Shang-Yi Yao, Chun-Te Wu, and Tong-Yee Lee, Senior Member, IEEE

CGVSL - National Cheng-Kung University

Home Demo video Comparison Evaluation results More results Ablation Study Failure results Cite this Paper Source code

Abstract

If the video has long been mentioned as a widespread visualization form, the animation sequence in the video is mentioned as storytelling for people. Producing an animation requires intensive human labor from skilled professional artists to obtain plausible animation in both content and motion direction, incredibly for animations with complex content, multiple moving objects, and dense movement. This paper presents an interactive framework to generate new sequences according to the users' preference on the starting frame. The critical contrast of our approach versus prior work and existing commercial applications is that novel sequences with arbitrary starting frame are produced by our system with a consistent degree in both content and motion direction. To achieve this effectively, we first learn the feature correlation on the frameset of the given video through a proposed network called RSFNet. Then, we develop a novel path-finding algorithm, SDPF, which formulates the knowledge of motion directions of the source video to estimate the smooth and plausible sequences. The extensive experiments show that our framework can produce new animations on the cartoon and natural scenes and advance prior works and commercial applications to enable users to obtain more predictable results.

Keywords -- animation, sequencing, RSFNet, distillation, SDPF

BibTeX

@ARTICLE{10018537, author={Le, Thi-Ngoc-Hanh and Yao, Sheng-Yi and Wu, Chun-Te and Lee, Tong-Yee},
journal={IEEE Transactions on Visualization and Computer Graphics},
title={Regenerating Arbitrary Video Sequences with Distillation Path-Finding},
year={2023},
volume={},
number={},
pages={1-14},
doi={10.1109/TVCG.2023.3237739}}
}

Grant

This work was supported in part by the National Science and Technology Council (under nos. 111-2221-E-006 -112 -MY3 and 110-2221-E-006-135-MY3), Republic of China (ROC), Taiwan.

Demo video

Comparisons

River flow (the first row) and Chinese ink (the second row) are mentioned as the limitation in [14]. In both cases, there exists long linear motion segments. The flip-flop phenomenon occurs in River flow sequence in [14]'s result. With Chinese ink, [14] generate abnormal motion directions and enventually leads to unrealistic resultant video. Reversely, our system generates smooth and realistic motions in both cases.


Input video

Manifold sequence [14]

By ours

Our system can generate different sequences at arbitrary starting frames. Here, we demonstrate the results with starting at the first frame (i.e., frame 0), the middle frames (i.e., frame 156 and 214), and the last frame (i.e., frame 313).


Source sequence starting at frame 0

Source sequence starting at frame 156

Source sequence starting at frame 214

This is the final frame

Generated sequence starting at frame 0

Generated sequence starting at frame 156

Generated sequence starting at frame 214

Generated sequence starting at the final frame 313

Starting at the same frame (i.e., frame 0) and resulting different smooth sequences

Sequence of Hippo funk

Resultant Sequence 1 with starting frame 0

Resultant Sequence 2 with starting frame 0

Evaluation Results

Here we show the results of 12 videos that we used in our evaluation. They are denoted as A to L in Figure 6 of our manuscript.

A - Love bird

Input clip

Output sequence at frame 426

B - Daffy Duck

Input clip

Output sequence at frame 214

C - Hippo funk

Input clip

Output sequence at frame 0

D - Frog dance

Input clip

Output sequence at frame 102

E - Michigan

Input clip

Output sequence at frame 37

F - Umbrella

Input clip

Output sequence at frame 0

G - Little doctor

Input clip

Output sequence at frame 67

H - Basketball match

Input clip

Output sequence at frame 0

I - River flow

Input clip

Output sequence at frame 0

J - Harry Porter

Input clip

Output sequence at frame 0

K - Chinese ink

Input clip

Output sequence at frame 0

L - Movie scene

Input clip

Output sequence at frame 306

More Results

Input sequence

Starting frame 48 (i.e., final frame)

Output sequence at frame 48

Input sequence


Starting frame 100

Output sequence at frame 100

Input sequence


Starting frame 126

Output sequence at frame 116
>

Input sequence


Starting frame 79

Output sequence at frame 79

Input sequence


Starting frame 97

Output sequence at frame 97

Buffalo input sequence


Starting frame 0

Buffalo output sequence