Ning Xie

IR2s: interactive real photo to Sumi-e

We propose an interactive sketch-based system for rendering oriental brush strokes on complex shapes. We introduce a contour-driven approach; the user inputs contours to represent complex shapes, the system estimates automatically the optimal trajectory of the brush, and then renders them into oriental ink paintings. Unlike previous work where the brush trajectory is explicitly provided as input, we automatically estimate this trajectory from the outline of the shapes to paint using a three-stages algorithm; first complex shapes are decomposed into elementary shapes that can be rendered with a single brush stroke. Second, we formulate the optimal brush trajectory estimation as the minimization of an energy function that measures the quality of the trajectory constrained by the variation along a stroke of the painting process parameters, such as the footprint position, size, orientation, and angular velocity. Finally, the estimated trajectories are rendered into brush strokes by mapping footprint textures scanned from real images. We combine the proposed framework with an interactive segmentation in order to convert real images into Oriental ink paintings. Experiments on complex shapes show that the proposed contour-based approach produces a large variety of strokes compared to trajectory-based approaches. It is particularly suitable for converting real images into Oriental ink paintings with minimum interaction.

Hashing with Angular Reconstructive Embeddings

Large-scale search methods are increasingly critical for many content-based visual analysis applications, among which hashing-based approximate nearest neighbor search techniques have attracted broad interests due to their high efficiency in storage and retrieval. However, existing hashing works are commonly designed for measuring data similarity by the Euclidean distances. In this paper, we focus on the problem of learning compact binary codes using the cosine similarity. Specifically, we proposed novel angular reconstructive embeddings (ARE) method, which aims at learning binary codes by minimizing the reconstruction error between the cosine similarities computed by original features and the resulting binary embeddings. Furthermore, we devise two efficient algorithms for optimizing our ARE in continuous and discrete manners, respectively. We extensively evaluate the proposed ARE on several large-scale image benchmarks. The results demonstrate that ARE outperforms several state-of-the-art methods.Large-scale search methods are increasingly critical for many content-based visual analysis applications, among which hashing-based approximate nearest neighbor search techniques have attracted broad interests due to their high efficiency in storage and retrieval. However, existing hashing works are commonly designed for measuring data similarity by the Euclidean distances. In this paper, we focus on the problem of learning compact binary codes using the cosine similarity. Specifically, we proposed novel angular reconstructive embeddings (ARE) method, which aims at learning binary codes by minimizing the reconstruction error between the cosine similarities computed by original features and the resulting binary embeddings. Furthermore, we devise two efficient algorithms for optimizing our ARE in continuous and discrete manners, respectively. We extensively evaluate the proposed ARE on several large-scale image benchmarks. The results demonstrate that ARE outperforms several state-of-the-art methods.

Artist Agent: A Reinforcement Learning Approach to Automatic Stroke Generation in Oriental Ink Painting

Oriental ink painting, called Sumi-e, is one of the most appealing painting styles that has attracted artists around the world. Major challenges in computer-based Sumi-e simulation are to abstract complex scene information and draw smooth and natural brush strokes. To automatically generate such strokes, we propose to model a brush as a reinforcement learning agent, and learn desired brush-trajectories by maximizing the sum of rewards in the policy search framework. We also elaborate on the design of actions, states, and rewards tailored for a Sumi-e agent. The effectiveness of our proposed approach is demonstrated through simulated Sumi-e experiments.

Lightweighting for Web3D visualization of large-scale BIM scenes in real-time

Semantics-guided lightweighting is proposed to reduce the amount of data processing in the front end by removing the redundant data and creating an IFC Lightweight Scene Graph (IFC_LSG).Double-Layered Sparse Voxel (DLSV) is proposed for data indexing to improve the access efficiency of real-time Web3D building visualization.Incremental Frustum of Interest (I-FOI) is proposed to manage the scene by combining the rendering pipeline and the current scene index. Display Omitted As a result of informatization in construction, Building Information Modeling (BIM) has now become a core technology for smart construction. We present a Web3D-based lightweighting solution for real-time visualization of large-scale BIM scenes, considering the redundancy, semantics, and the parameterization of BIM data under the limited resources of network bandwidth and web browsers. Taking the Industry Foundation Classes (IFC) as the input data format, we firstly conduct a semantics-guided lightweighting operation on the raw BIM scenes by removing the repetitive objects and parameterizing the swept surfaces. Secondly, we extract the exterior products from the raw BIM buildings for visibility culling and construct a Double-Layered Sparse Voxel (DLSV) index based on sparse voxelization. Thirdly, we integrate the above two together into a new data structure named Incremental Frustum of Interest (I-FOI) to manage the scene data in real-time. Our experiments demonstrate that: (1) with the semantics information, our method is able to significantly reduce the redundancy of raw large-scale BIM scenes; (2) the DLSV structure supports progressive data loading and facilitates the indoor/outdoor visibility culling efficiently; and (3) the I-FOI introduces a frustum incremental-driven mechanism into progressive data loading or unloading to improve the efficiency of resource consumption.

Fast accessing Web3D contents using lightweight progressive meshes

Accessing Web3D contents is relatively slow through Internet under limited bandwidth. Preprocessing of 3D models can certainly alleviate the problem, such as 3D compression and progressive meshes (PM). But none of them considers the similarity between components of a 3D model, so that we could take advantage of this to further improve the efficiency. This paper proposes a similarity‐aware data reduction method together with PM, called lightweight progressive meshes (LPM). LPM aims to excavate similar components in a 3D model, generates PM representation of each component left after removing redundant components, and organizes all the processed data using a structure called lightweight scene graph. The proposed LPM possesses four significant advantages. First, it can minimize the file size of 3D model dramatically without almost any precision loss. Because of this, minimal data is delivered. Second, PM enables the delivery to be progressive, so called streaming. Third, when rendering at client side, due to lightweight scene graph, decompression is not necessary and instanced rendering is fully exerted. Fourth, it is extremely efficient and effective under very limited bandwidth, especially when delivering large 3D scenes. Performance on real data justifies the effectiveness of our LPM, which improves the state‐of‐the‐art in accessing Web3D contents. Copyright

Conditional density estimation with dimensionality reduction via squared-loss conditional entropy minimization

Regression aims at estimating the conditional mean of output given input. However, regression is not informative enough if the conditional density is multimodal, heteroskedastic, and asymmetric. In such a case, estimating the conditional density itself is preferable, but conditional density estimation (CDE) is challenging in high-dimensional space. A naive approach to coping with high dimensionality is to first perform dimensionality reduction (DR) and then execute CDE. However, a two-step process does not perform well in practice because the error incurred in the first DR step can be magnified in the second CDE step. In this letter, we propose a novel single-shot procedure that performs CDE and DR simultaneously in an integrated way. Our key idea is to formulate DR as the problem of minimizing a squared-loss variant of conditional entropy, and this is solved using CDE. Thus, an additional CDE step is not needed after DR. We demonstrate the usefulness of the proposed method through extensive experiments on various data sets, including humanoid robot transition and computer art.

3D tree skeletonization from multiple images based on PyrLK optical flow

We introduce a novel 3D tree skeletonization method from multiple images.Affine transformation and iterative tracking are added to PyrLK optical flow.Parallel voxel flooding is proposed for determining point cloud of single branch.L-System extraction algorithm for tree skeletons is proposed. Display Omitted 3D tree models are widely applied to construct large-scale virtual scenes. However, converting real trees into computer representation faces two main problems. One is the low quality of reconstructed point cloud. The other is that the skeletonization produces inaccurate results due to the complex structure of trees. We propose a novel pipeline to reconstruct automatically accurate 3D tree skeletons from sequential tree images. It involves three steps: (1) Pre-processing: an optical flow based feature-matching algorithm is proposed to acquire high-quality point clouds. (2) Shape-based parallel skeletonization: tree shape is taken into consideration to detect specific structure features of trees during parallel flooding. (3) Post-processing: an L-System extraction algorithm is applied on the skeletons and the lightweight representation is obtained to meet various requirements. The effectiveness of our approach is demonstrated through experiments.

The Shape Space of 3D Botanical Tree Models

We propose an algorithm for generating novel 3D tree model variations from existing ones via geometric and structural blending. Our approach is to treat botanical trees as elements of a tree-shape space equipped with a proper metric that quantifies geometric and structural deformations. Geodesics, or shortest paths under the metric, between two points in the tree-shape space correspond to optimal deformations that align one tree onto another, including the possibility of expanding, adding, or removing branches and parts. Central to our approach is a mechanism for computing correspondences between trees that have different structures and a different number of branches. The ability to compute geodesics and their lengths enables us to compute continuous blending between botanical trees, which, in turn, facilitates statistical analysis, such as the computation of averages of tree structures. We show a variety of 3D tree models generated with our approach from 3D trees exhibiting complex geometric and structural differences. We also demonstrate the application of the framework in reflection symmetry analysis and symmetrization of botanical trees.

Zero-shot learning via discriminative representation extraction

Zero-shot learning (ZSL) aims to recognize classes whose samples did not appear during training. Existing research focuses on mapping deep visual feature to semantic embedding space explicitly or implicitly. However, ZSL improvements led by discriminative feature transformation is not well studied. In this paper, we propose a ZSL framework that maps semantic embeddings to a discriminative representation space, which are learned in two different ways: Kernelized Linear Discriminant Analysis (KLDA) and Central-loss based Network (CLN). KLDA and CLN can both force samples to be intra-class aggregation and inter-class separation. With the learned discriminative representations, we map class embeddings to representation space using Kernelized Ridge Regression (KRR). Our experiments show that both KLDA+KRR and CLN+KRR surpass state-of-art approaches in both recognition and retrieval task.

Client-Driven Strategy of Large-Scale Scene Streaming

Different from the strategy of virtual scene in the stand-alone application, web version may have larger scale scene and more users online at same time. However, because of the network delay (latency) and limitation of computational power of the servers, it causes that users unable to interactively access the virtual scene fluently. Meanwhile, the large scale virtual scene cannot be successfully loaded in the entry-level client machine. In this paper, we propose a client-driven strategy of scene streaming in order to solve large-scale data transmission. The experiment demonstrates that our method enables clients to enter the scene roaming and reduce the server network load. Meanwhile, it also adapts different network architectures.