Image Recommendation for Automatic Report Generation using Semantic Similarity

Abstract

Automatic report generation is a technology that automatically generates documents in the form of report by summarizing various materials according to a specific topic in time sequence or subject. Although the main content of the report is text, insertion of appropriate images can improve the completeness of the report. In this paper, we propose an image recommendation method for automatically selecting and inserting appropriate images corresponding to a specific part of a report. In our proposed method, reevaluation of the candidate images is performed based on the semantic similarity between query and the contents of the images. In order to transform semantic information of text query and image into one vector space, we extracted semantic information from image as a set of tags form using deep learning based object detection module. Also, we extracted tags from the given title of the image so that the proposed system can evaluate the candidate images even in the case that the given query includes specific keywords or proper nouns which were not learned by object detection and recognition module in advance. In this paper, we conducted experiments on eight queries related to recent events to verify the applicability of our proposed image recommendation system and evaluate the image selection accuracy.