Saboya Yang

Sparse representation based super resolution using saliency and edge information

Sparse representation provides effective prior information for single-frame super resolution reconstruction. The diversified training samples of the general dictionary lead to the difficulty of recovering fine grained details due to the negligence of redundant structural characteristics. Thus, the dictionary which is adaptive to local structures is needed. Considering the highly structured information of saliency and edge regions, we present a novel sparse representation based super resolution approach. Salient regions are segmented to train the saliency dictionary. The same is true for edge regions. Thus, more adaptive dictionaries are acquired. When reconstructing the input image, dictionaries are chosen adaptively and then more clear details are achieved. Objective quality evaluation shows that our proposed algorithm achieves highest PSNR results comparing with the state-of-the-art methods. And subjective results demonstrate the proposed method reduces artifacts and preserves more details.

Image super resolution using saliency-modulated context-aware sparse decomposition

This paper presents a novel saliency-modulated sparse representation algorithm for image super resolution. In images, regions salient to human eyes appear to be more organized and structured. This property is utilized in both the dictionary learning and the sparse coding process to capture more structural details for the reconstructed image. Apart from a general dictionary, example patches from the salient regions are extracted to train a salient dictionary. We also incorporate context-aware sparse decomposition to model dependencies between dictionary atoms of adjacent patches, especially in the salient regions. Experiments show the proposed method outperforms state-of-the-art methods with the highest PSNR gain. Subjective results demonstrate the proposed method reduces artifacts and preserves more details.

Joint image denoising using self-similarity based low-rank approximations

The observed images are usually noisy due to data acquisition and transmission process. Therefore, image denoising is a necessary procedure prior to post-processing applications. The proposed algorithm exploits the self-similarity based low rank technique to approximate the real-world image in the multivariate analysis sense. It consists of two successive steps: adaptive dimensionality reduction of similar patch groups, and the collaborative filtering. For each target patch, the singular value decomposition (SVD) is used to factorize the similar patch group collected in a local search window by block-matching. Parallel analysis automatically selects the principal signal components by discarding the nonsignificant singular values. After the inverse SVD transform, the denoised image is reconstructed by the weighted averaging approach. Finally, the collaborative Wiener filtering is applied to further remove the noise. Experimental results show that the proposed algorithm surpasses the state-of-the-art methods in most cases.

Image super-resolution via nonlocal similarity and group structured sparse representation

Sparse prior provides an effective tool for the image reconstruction. However, the sparse coding for independent patches leads to the unstable sparse decomposition. In this paper, we propose a group structured sparse representation model by considering the nonlocal similarity. The nonlocal similar patches are collected and classified into groups. Patches in the same group are reconstructed based the same basis of dictionaries. The dictionary is organized as the combination of many orthogonal sub-dictionaries. To provide the redundancy, the dictionary used for the sparse coding is generated online with several sub-dictionaries, thus it is over-complete. We apply the proposed model into a gradual SR framework. The framework enlarges LR to HR by a patch enhancement and an alternative sparse reconstruction on the patch and group. Objective quality evaluation shows that our proposed SR method achieves highest PSNR results comparing with the state-of-the-art methods. And subjective results demonstrate the proposed method reduces artifacts and preserves more details.

Automatic portrait oil painter: joint domain stylization for portrait images

Everyone has the dream of being in the center of famous art paintings, admired by numerous future generations. However, the dream came true at a huge cost of the painter’s commission in old days. In our paper, another practical choice is provided for everyone to achieve that dream – an automatic portrait oil painter transferring some artistic styles from one single reference painting. To address this issue, we propose a joint-domain image stylization approach, particularly for portrait oil paintings. From the view of artistic appreciation, we analyze an amount of oil painting art works and summarize three critical factors to depict the figure, i.e. color, structure and texture. Based on this point, we separate and represent an artistic work into these three domains. Then, considering their intrinsic properties and following an art creation route, we propose the corresponding approaches to jointly model and transfer the features in these domains. First, a swatch-based color adjustment is proposed to recolor the tone of the input image based on semantic regions corresponding to the references. Second, the main structures of the input image is maintained by sparse reconstruction. Third, a coarse-to-fine texture synthesis is used to enhance the detail oil painting patterns. Extensive experimental results demonstrate that the proposed method achieves desirable results compared with state-of-the-art methods in not only transferring the styles from references but also keeping consistent contents with the given portrait.

Joint-domain unsupervised stylization for portraits

People wish to own a portrait painting of themselves by Da Vinci. Unfortunately, it is impossible to make this dream come true; nevertheless, it may give us an opportunity by transferring some artistic features from one single reference painting. To address this issue, we propose a joint-domain image stylization approach, particularly for portrait oil paintings. From the view of artistic appreciation, we analyze an amount of oil painting artworks and summarize three critical factors to depict the figure, i.e. color, structure and texture. First, the tone of the input image is recolored based on semantic regions corresponding to the reference. Those semantic regions are segmented automatically via the color swatch, by considering the constraints of colors and positions. Then, we exploit sparse representation to reconstruct the layout by acquiring the structure from the reference. The paired training set for sparse dictionary learning is built with the guidance of edge features. Third, considering that texture is usually locally stochastic but regularly repetitive in global, a coarse-to-fine texture synthesis is used to enhance the detail pattern. Subjective results demonstrate the proposed method achieves desirable results compared with state-of-art methods while keeping consistent with artists style.

Hierarchical oil painting stylization with limited reference via sparse representation

Traditional image stylization is enforced by learning the mappings with an external paired training set. But in practice, people usually encounter a specific stylish image and want to transfer its style to their own pictures without the external dataset. Thus, we propose a hierarchical stylization model with limited reference particularly for oil paintings. First, the edge patch based dictionary is trained to build connections between images and limited reference, then reconstruct the structure layer. Due to the highly structured property of saliency regions, the saliency mask is extracted to integrate the structure layer and the texture layer with different weights. Hence, the advantages of both sparse representation based methods and example based methods are integrated. Moreover, the color layer and the surface layer are considered to make the output more consistent with the artists individual oil painting style. Subjective results demonstrate the proposed method produces desirable results with state-of-art methods while keeping consistent with the artists oil painting style.

Scale-compensated nonlocal mean super resolution

In this paper, we propose a novel algorithm for multi-frame super resolution (SR) with consideration of scale changing between frames. First, we detect the scale of each frame by scale-detector. Based on the scale gap between adjacent frames, we extract patches and modify them from different scales into the same scale to obtain more redundant information. Finally, a reconstruction approach based on patch matching is applied to generate a high resolution (HR) frame. Compared to original Nonlocal Means SR (NLM SR), the proposed Scale-Compensated NLM finds more potential similar patches in different scales which are easily neglected in NLM SR. Experimental results demonstrate better performance of the proposed algorithm in both objective measurement and subjective perception.