Sebastian Rieger

SMULGRAS: a platform for smart multicodal graphics search

In this paper we describe our SMULGRAS platform for smart multicodal graphics search, which aims at fusing web-based content creation tools and content-based search interfaces. Our framework provides an easy to use web frontend that integrates a 3D editor capable of creating, editing, and presenting 3D scenes as well as intuitive interfaces for image/graphics search, which can serve queries by both, 3D models or camera-captured footage. Correspondingly, our proposed backend pipeline allows a similarity search both by image / shape and for images / shapes. To do so, we employ state-of-the-art deep learning techniques based on convolutional neural networks, which employ to both, shape and image representations. Our framework not only provides good retrieval accuracy as well as scalability with training and retrieval, but it also gives the user more control over the (iterative) search process by directly integrating fully interactive, web-based editing tools. This makes the approach also suitable for usage within large-scale community-based modeling applications.