Motion Inpainting on Dynamic Backgrounds with Large Object Removal


Kanoksak Wattanachote, Timothy K. Shih, Shwu-Huey Yen, and Shih-Jung Wu



Abstract—Video inpainting technologies use special and temporal properties to recover lost portion of continuous area. Example applications include object removal for special effects, defect recovering in aged movies, and the recent 3D inpainting techniques for 3D videos. One of the difficult problems not yet completely solved is to remove objects or to recover lost defects in dynamic backgrounds such as fire, smoke, or waterfall. The difficult issue is to maintain a visual continuity, by using motion gestures. This article demonstrates a motion inpainting mechanism, to recover the lost of areas/objects in dynamic background, with visual pleasant results as well as quantitative evaluations. The mechanism needs to search for video clips from a set of irregular shaped frames, called patch streams. If the duration of a found patch stream is shorter than a target background video, the patch stream will be extended by our algorithm inspired by seam carving. Finally, the inpainted video is evaluated with a newly proposed motion coherence metric.


Index Terms—Motion Coherence, Motion Inpainting, Object Removal, Video Inpainting


——————————   <<>>   ——————————


(a) S1: original video

(b) S1: inpainting result with waterfall added

(c) S2: original video

(d) S2: inpainting result with bear removed

(e) S3: original video

(f) S3: inpainting result with additional smoke on the right

(g) S4: original video

(h) S4: inpainting result with a bigger eruption

(i) S5: original video

(j) S5: inpainting result to remove the person

(k) S6: original video

(l) S6: inpainting result with a larger waterfall

(m) S7: original video

(n) S7: inpainting result with a tree branch removed

(o) S8: original video

(p) S8: inpainting result with additional waterfall

(q) S9: original video

(r) S9: inpainting result with additional waterfall

(s) S10: original video

(t) S10: inpainting result with more fire on the left

(u) S11: original video

(v) S11: inpainting result with additional smoke

(w) S12: original video

(x) S12: inpainting result with additional smoke

(y) S13: original video

(z) S13: inpainting result with one more window on fire

(α) S14: original video

(β) S14: inpainting result with a bigger eruption

Motion coherence evaluation's compared between [38] and our inpainting methods

The experimental result using [38]'s method is demonstrated in the right column. The motion vectors angle variation and standard deviation are observed from this video and compared with the result video in (d) where the rock was removed by [38]'s inpainting method.

High resolution video results (1280×720)

(a) H1: original video

(b) H1: inpainting result with additional waterfall

(a) H2: original video

(b) H2: inpainting result with additional explosion

(a) H3: original video

(b) H3: inpainting result with additional waterfall

Limitation: An Example of Non-Stationary Video Inpainting

(a) original video

(b) a non-stationary video inpainting result with mask