Deep learning-based Importance Map for Content-Aware Media Retargeting

Thi-Ngoc-Hanh Le 1, Tong-Yee Lee 1, Senior Member, IEEE, Shih-Syun Lin 2, Weiming Dong 3, Senior Member, IEEE

(1) National Cheng-Kung University, Taiwan; (2) National Taiwan Ocean University, Taiwan; (3) Institute of Automation, CAS, China

This work has been accepted for publication on Journal of Multimedia Tools and Applications (MTAP), January 20, 2024


We present a deep learning-based framework to generate the importance map adaptable to the existing image and video retargeting operators. A conventional retargeting algorithm uses a heuristic way to seek an off-the-self approach that is not originally designed for retargeting applications that could be integrated into their retargeting system. The extracted importance map of the image does not match the characteristics of the input image; therefore, it affects the retargeting results and limits the performance of the retargeting method. Our designed framework attempts to minimize the artifacts/distortions caused by inappropriate energy, e.g., the shrunk phenomenon in warping-based results and carving-through-object distortion in the seam carving-based approach. Our proposed framework focuses on capturing sensitive-to-distortion regions and activating their energy to solve this challenge. We verify the effectiveness of our proposed scheme by plugging it into three typical retargeting methods: seam carving-based, warping-based for image, and video retargeting. Besides, we test it on various challenging and low retargetability input images, compare it with prior works and evaluate it on a benchmark dataset. The experimental results and evaluation demonstrate that our importance map substantially outperforms the previous works in terms of visual quality.

Keywords -- retargeting, imaportance map, seam carving, warping, A2R-Map

Seam carving - based Image Retargeting Results

In this session, we plug our A2R-Map to seam carving operator and compare the retargeting results against the use of the Gradient map.

All results in this session are reduced to 75% of the original width.

Input image

SC + Gradient map

SC + Our A2R-Map

Warping-based Image Retargeting Results

In this session, we intergrate our A2R-Map with warping operators, then let our A2R-Map compete with Saliency map in this manner.

All results in this session are reduced to 50% of the original width.

Input image

Warp + Saliency map

Warp + A2R-Map

Warping-based Video Retargeting Results

In the below video, we highlight the effectiveness of our A2R-Map against the saliency map in Video Retargeting application.

Each row below is the case by case of the video results. All the results in this session are reduced to 50% of the original width.

Input video

Warp + Saliency map

Warp + A2R-Map

Back to homepage