Ryosuke Furuta

Fast Volume Seam Carving With Multipass Dynamic Programming

In volume seam carving, i.e., seam carving for 3D cost volume, an optimal seam surface can be derived by graph cuts, resulting from sophisticated graph construction. To date, the graph-cut algorithm is the only solution for volume seam carving. However, it is not suitable for practical use because it incurs a heavy computational load. We propose a multipass dynamic programming (DP)-based approach for volume seam carving, which reduces computation time and memory consumption while maintaining a similar image quality as that of graph cuts. Our multipass DP scheme is achieved by conducting DP in two directions to accumulate the cost in a 3D volume and then tracing back to find the best seam. In our multipass DP, a suboptimal seam surface is created instead of a global optimal one, and it has been experimentally confirmed by more than 198 crowdsourced workers that such suboptimal seams are good enough for image processing. The proposed scheme offers two options: a continuous method that ensures the connectivity of seam surfaces and a discontinuous method that ensures the connectivity in only one direction. We applied the proposed volume seam carving method based on multipass DP to conventional video retargeting and tone mapping. These two applications are completely different; however, the volume seam carving method can be applied similarly by changing the axes of the cost volume. Even though the results obtained using our methods were similar to those obtained by graph cuts, our computation time was approximately 90 times faster that of graph cuts and the memory usage was eight times smaller than that of graph cuts. We also extend the idea of tone mapping to the contrast enhancement method based on volume seam carving.

Fooling Neural Networks in Face Attractiveness Evaluation: Adversarial Examples with High Attractiveness Score But Low Subjective Score

People are fond of taking and sharing photos in their social life, and a large part of it is face images, especially selfies. A lot of researchers are interested in analyzing attractiveness of face images. Benefited from deep neural networks (DNNs) and training data, researchers have been developing deep learning models that can evaluate facial attractiveness of photos. However, recent development on DNNs showed that they could be easily fooled even when they are trained on a large dataset. In this paper, we used two approaches to generate adversarial examples that have high attractiveness scores but low subjective scores for face attractiveness evaluation on DNNs. In the first approach, experimental results using the SCUT-FBP dataset showed that we could increase attractiveness score of 20 test images from 2.67 to 4.99 on average (score range: [1, 5]) without noticeably changing the images. In the second approach, we could generate similar images from noise image with any target attractiveness score. Results show by using this approach, a part of attractiveness information could be manipulated artificially.

Coarse-to-fine strategy for efficient cost-volume filtering

Cost-volume filtering is one of the most widely known techniques to solve general multi-label problems, however it is problematically inefficient when the label space size is extremely large. This paper presents a coarse-to-fine strategy of the cost-volume filtering that handles efficiently and accurately multi-label problems with a large label space size. Based upon the observation that true labels at the same image coordinate of different scales are highly correlated, we truncate unimportant labels for the cost-volume filtering by leveraging the labeling output of lower scales. Experimental results show that our algorithm achieves much higher efficiency than the original cost-volume filtering while enjoying the comparable accuracy to it.

Efficient and Interactive Spatial-Semantic Image Retrieval

This paper proposes an efficient image retrieval system. When users wish to retrieve images with semantic and spatial constraints (e.g., a horse is located at the center of the image, and a person is riding on the horse), it is difficult for conventional text-based retrieval systems to retrieve such images exactly. In contrast, the proposed system can consider both semantic and spatial information, because it is based on semantic segmentation using fully convolutional networks (FCN). The proposed system can accept three types of images as queries: a segmentation map sketched by the user, a natural image, or a combination of the two. The distance between the query and each image in the database is calculated based on the output probability maps from the FCN. In order to make the system efficient in terms of both the computation time and memory usage, we employ the product quantization technique (PQ). The experimental results show that the PQ is compatible with the FCN-based image retrieval system, and that the quantization process results in little information loss. It is also shown that our method outperforms a conventional text-based search system.

Efficiency-enhanced cost-volume filtering featuring coarse-to-fine strategy

Cost-volume filtering (CVF) is one of the most widely used techniques for solving general multi-labeling problems based on a Markov random field (MRF). However it is inefficient when the label space size (i.e., the number of labels) is large. This paper presents a coarse-to-fine strategy for cost-volume filtering that efficiently and accurately addresses multi-labeling problems with a large label space size. Based on the observation that true labels at the same coordinates in images of different scales are highly correlated, we truncate unimportant labels for cost-volume filtering by leveraging the labeling output of lower scales. Experimental results show that our algorithm achieves much higher efficiency than the original CVF method while maintaining a comparable level of accuracy. Although we performed experiments that deal with only stereo matching and optical flow estimation, the proposed method can be employed in many other applications because of the applicability of CVF to general discrete pixel-labeling problems based on an MRF.

Object detection refinement using Markov random field based pruning and learning based rescoring

Contextual information such as the co-occurrence of objects and the location of objects has played an important role in object detection. We present candidate pruning and object rescoring methods that leverage contextual information and that can improve the state-of-the-art CNN-based object detection methods such as Fast R-CNN and Faster R-CNN. In our pruning method, we formulate candidate reduction as a Markov random field optimization problem. In our rescoring method, we employ a machine learning technique to reconsider the detection scores of candidate windows. We experimentally demonstrate improvements in R-CNN-based object detection methods using two datasets. Moreover, we apply our model to the structured retrieval task to show the potential applications of our model.

Fast volume seam carving with multi-pass dynamic programming

In volume seam carving, seam carving for three-dimensional (3D) cost volume, an optimal seam surface can be derived by graph cuts, resulting from sophisticated graph construction. However, the graph cuts algorithm is not suitable for practical use because it incurs a heavy computational load. We propose a multi-pass dynamic programming (DP) based approach for volume seam carving that reduces computation time to 60 times faster and memory consumption to 10 times smaller than those of graph cuts, while maintaining a similar image quality as that of graph cuts. In our multi-pass DP, a suboptimal seam surface is created instead of a globally optimal one, but it has been experimentally confirmed by more than 198 crowd workers that such suboptimal seams are good enough for image processing.

Towards Online Impression Prediction of Oral Presentations Using Soft Coding

We have been developing impression prediction techniques for oral presentations. The contribution of this paper is two folds. First, we introduce soft code assignment for the bag-of-features (BoF) representation to improve the prediction accuracy. Second, we discuss towards online impression prediction aiming at real-time feedback to the speaker. Experimental results using over 1,600 TED presentation videos show that about 3% accuracy improvement can be achieved by the soft-coding and half amount of presentation is need to achieve comparable prediction accuracy to using the whole presentation.