Deep learning-based Importance Map for Content-Aware Media Retargeting

We present a deep learning-based framework to generate the importance map adaptable to the existing image and video retargeting operators. A conventional retargeting algorithm uses a heuristic way to seek an off-the-self approach that is not originally designed for retargeting applications that could be integrated into their retargeting system. The extracted importance map of the image does not match the characteristics of the input image; therefore, it affects the retargeting results and limits the performance of the retargeting method. Our designed framework attempts to minimize the artifacts/distortions caused by inappropriate energy, e.g., the shrunk phenomenon in warping-based results and carving-through-object distortion in the seam carving-based approach. Our proposed framework focuses on capturing sensitive-to-distortion regions and activating their energy to solve this challenge. We verify the effectiveness of our proposed scheme by plugging it into three typical retargeting methods: seam carving-based, warping-based for image, and video retargeting. Besides, we test it on various challenging and low retargetability input images, compare it with prior works and evaluate it on a benchmark dataset. The experimental results and evaluation demonstrate that our importance map substantially outperforms the previous works in terms of visual quality.

Keywords -- retargeting, imaportance map, seam carving, warping, A2R-Map

Seam carving - based Image Retargeting Results

In this session, we plug our A2R-Map to seam carving operator and compare the retargeting results against the use of the Gradient map.

All results in this session are reduced to 75% of the original width.

Warping-based Image Retargeting Results

In this session, we intergrate our A2R-Map with warping operators, then let our A2R-Map compete with Saliency map in this manner.

All results in this session are reduced to 50% of the original width.

Warping-based Video Retargeting Results

In the below video, we highlight the effectiveness of our A2R-Map against the saliency map in Video Retargeting application.

Each row below is the case by case of the video results. All the results in this session are reduced to 50% of the original width.