An Advert Creation System for Next-Gen Publicity
Atul Nautiyal, Killian McCabe, Murhaf Hossari, Soumyabrata Dev, Matthew Nicholson, Clare Conran, Declan McKibben, Jian Tang, Xu Wei, Francois Pitie
AAn Advert Creation System forNext-Gen Publicity
Atul Nautiyal (cid:63) , Killian McCabe (cid:63) , Murhaf Hossari (cid:63) , Soumyabrata Dev (cid:63) ,Matthew Nicholson , Clare Conran , Declan McKibben , Jian Tang , XuWei , and Fran¸cois Piti´e , The ADAPT SFI Research Centre, Trinity College Dublin Department of Electronic & Electrical Engineering, Trinity College Dublin Huawei Ireland Research Center, Dublin
Abstract.
With the rapid proliferation of multimedia data in the inter-net, there has been a fast rise in the creation of videos for the viewers.This enables the viewers to skip the advertisement breaks in the videos,using ad blockers and ‘skip ad’ buttons – bringing online marketing andpublicity to a stall. In this paper, we demonstrate a system that caneffectively integrate a new advertisement into a video sequence. We usestate-of-the-art techniques from deep learning and computational pho-togrammetry, for effective detection of existing adverts, and seamlessintegration of new adverts into video sequences. This is helpful for tar-geted advertisement, paving the path for next-gen publicity.
Keywords: advertisement · online content · deep learning. With the ubiquity of multimedia videos, there has been a massive interest fromthe advertisement and marketing agencies to provide targeted advertisementsfor the customers. Such targeted advertisements are useful, both from the per-spectives of marketing agents and end users. The advertisement agencies canuse a powerful media for marketing and publicity; and the users can interact viaa personalized consumer experience. In this paper, we attempt to solve this bydesigning an online advert creation system for next-gen publicity. We developand implement an end-to-end system for automatically detecting and seamlesslychanging an existing billboard in a video by inserting a new advert. This systemwill be helpful for online marketers and content developers, to develop videocontents for targeted audience.Figure 1 illustrates our system. Our system automatically detects the pres-ence of a billboard in an image frame from the video sequence. Post billboarddetection, our system also localizes its position in the image frame. The user isgiven an opportunity to manually adjust and refine the detected four cornersof the billboard. Finally, a new advertisement is integrated into the image, andtracked across all frames of the video sequence. Thereby, we generate a newcomposite video with the integrated advert. (cid:63)
Authors contributed equally and arranged alphabetically. a r X i v : . [ c s . MM ] A ug A. Nautiyal, K. McCabe, M. Hossari, S. Dev et al.
Fig. 1: New advert integrated into the scene at the place of an existing billboard.Currently, there are no such existing framework available in the literaturethat aid the marketing agents to seamlessly integrate a new advertisement,into an original video sequence. However, a few companies viz. Mirriad [1] usespatented advertisement plantation technique to integrate 3D objects in a videosequence.
The backbone of our advert creation system is based on state-of-the-art tech-niques from deep learning and image processing. In this section, we briefly de-scribe the underlying techniques used in the various components of the demosystem. The different modules of our system are: advert- recognition, localiza-tion, and integration.
The first module of our advert creation system is used for the recognition ofbillboard – does an image frame from the video sequence contain billboard?This helps the system user to automatically detect the presence of billboard inan image frame of the video. We use a deep neural network (DNN) as a bi-nary classifier where classes represent presence and absence of billboard in videoframe respectively. We use a VGG-based network [4] for billboard detection. Weuse transfer learning with pre-trained ImageNet weights. We freeze the corre-sponding weights of all layers apart from last 5 layers. We add 3 fully connectedlayers with a softmax layer as the output layer. We train this deep network onour annotated dataset, containing both billboard and non-billboard images, andachieve good accuracy on billboard recognition. The second module of our advert creation system is used for localizing the po-sition of recognized billboard – where is the billboard located in image frame?We use a encoder-decoder based deep neural network that localizes the billboardposition in an image. We train this model on our billboard dataset comprisinginput images (cf. Fig. 2(a)) and corresponding binary ground truth image (cf.Fig. 2(b)). We train the model for several thousands of epochs. The localizedbillboard is a probabilistic image, that denotes the probability of an image pixelto belong to billboard class. We generate the binary threshold image from our In this paper, we interchangeably use both the terms, billboard and advert to indicatea candidate object for new advertisement integration in an image frame.n Advert Creation System for Next-Gen Publicity 3 computed heatmap using thresholding, and detect the various closed contourson the binary image. Finally, we select the contour with the largest area as ourlocalized billboard position. We thereby compute the initial four corners fromthe binary image by circumscribing a rectangle on the selected contour withminimum bounding area. The localized advert is shown in Fig. 2. (a) Input Image (b) Ground Truth (c) Detected Advert (d) Localized Advert
Fig. 2: Localization of billboard using our advert creation system. We localizethe advert from the probabilistic heatmap, by circumscribing a rectangle withminimum bounding area.
The third and final module of our system is advert integration – how to integratea new advert in the video? In this stage, the localized billboard is replaced witha new advert in a seamless and temporally consistent manner. We use Poissonimage editing [3] on the new advert, to achieve similar local illumination and localcolor tone, as the original video sequence. Furthermore, the relative motion ofthe billboard within the scene is tracked using Kanade-Lucas-Tomasi (KLT) [2]tracking technique.
We have designed an online system to demonstrate the functionalities of thevarious modules . The web UI interface is designed in Vue.js - the progressiveJavaScript Framework. The back end is supported via
Express - Node.js webapplication framework. The deep neural networks for advert recognition and lo-calization is designed in pure python , and the advert integration is implementedin
C++ . The web service to support advert detection is performed in pythonflask . The integration of a new advert into the existing video in the web serveris executed via
C++ binary.Figure 3 illustrates a sample snapshot of our developed web-based tool. Theweb interface consists of primarily three sections:
Home , Demo and
Images . Thepage
Home provides an overview of the system. The next page
Demo describes theentire working prototype of our system. The user selects a sample video fromthe list, runs the billboard detection module to accurately localize the billboard A demonstration video of our advert creation system can be accessed via https://youtu.be/zaKpJZhBVL4
A. Nautiyal, K. McCabe, M. Hossari, S. Dev et al. at sample image frames of the video. The detection module estimates the fourcorners of the billboard. However, the user also gets an option to refine thefour corners manually, if the detected four corners are not completely accurate.The refined four corners of the billboard are subsequently used for tracking andintegration of a new advertisement into the video sequence. The third and finalweb page
Images contains the list of all candidate adverts that can be integratedinto the selected video sequence.Fig. 3: Interface of the demo for advert detection and integration.Finally, our system integrates the new advertisement into the detected bill-board position, and generates a new composite video with the implanted adver-tisement.
In this paper, we have presented an online advert creation system on multimediavideos for a personalized and targeted advertisement. We use techniques fromdeep neural networks and image processing, for a seamless integration of newadverts into existing videos. Our system is trained on datasets that comprisesoutdoor scenes and views. Our future work involve further refining the perfor-mance of the system, and also generalizing it to other video sequence types.
Acknowledgement
The ADAPT Centre for Digital Content Technology is funded under the SFI ResearchCentres Programme (Grant 13/RC/2106) and is co-funded under the European Re-gional Development Fund.
References
1. Mirriad: Scalable, effective campaigns (Accessed 7-May-2018 2018),
2. Lucas, B.D., Kanade, T., et al.: An iterative image registration technique with anapplication to stereo vision (1981)n Advert Creation System for Next-Gen Publicity 53. P´erez, P., Gangnet, M., Blake, A.: Poisson image editing. ACM Transactions ongraphics (TOG) (3)4. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale imagerecognition. CoRR abs/1409.1556abs/1409.1556