Atanu R. Sinha

Summarizing Multimedia Content

Today multimedia content comprising both text and images is growing at a rapid pace. There has been a body of work to summarize text content, but to the best of our knowledge, no method has been developed to summarize multimedia content. We propose two methods for summarizing multimedia content. Our novel approach explicitly recognizes two desirable, normative characteristics of a summary - good coverage and diversity of the respective text and images, and that text and images should be coherent with each other. Two methods are examined - graph based and a modification to the submodular approach. Moreover, we propose a metric to measure the quality of a multimedia summary which captures coverage and diversity of text and images as well as coherence between the text and images in the summary. We experimentally demonstrate that the proposed methods achieve good quality multimedia summaries.

Measurement of Users' Experience on Online Platforms from Their Behavior Logs.

Explicit measurement of experience, as mostly practiced, takes the form of satisfaction scores obtained by asking questions to users. Obtaining response from every user is not feasible, the responses are conditioned on the questions, and provide only a snapshot, while experience is a journey. Instead, we measure experience values from users’ click actions (events), thereby measuring for every user and for every event. The experience values are obtained without-asking-questions, by combining a recurrent neural network (RNN) with value elicitation from event-sequence. The platform environment is modeled using an RNN, recognizing that a user’s sequence of actions has a temporal dependence structure. We then elicit value of a user’s experience as a latent construct in this environment. We offer two methods: one based on rules crafted from consumer behavior theories, and another data-driven approach using fixed point iteration, similar to that used in model-based reinforcement learning. Evaluation and comparison with baseline show that experience values by themselves provide a good basis for predicting conversion behavior, without feature engineering.

[POSTER] Enhanced Personalized Targeting Using Augmented Reality

Augmented Reality (AR) based applications have existed for some time; however, their true potential in digital marketing remains unexploited. To bridge this gap we create a novel consumer targeting system. First, we analyze consumer interactions on AR-based retail apps to identify her preferred purchase viewpoint during the session. We then target the consumer through a personalized catalog, created by embedding recommended products in her viewpoint visual. The color and style of the embedded product are matched with the viewpoint to create recommendations, and personalized text content is created using visual cues from the AR data. Evaluation with user studies show that our system is able to identify the viewpoint, our recommendations are better than tag-based recommendations, and targeting using the viewpoint is better than that of usual product catalogs.