Yung-Ju Chang

Deep 360 Pilot: Learning a Deep Agent for Piloting through 360° Sports Videos

Watching a 360° sports video requires a viewer to continuously select a viewing angle, either through a sequence of mouse clicks or head movements. To relieve the viewer from this 360 piloting task, we propose deep 360 pilot - a deep learning-based agent for piloting through 360° sports videos automatically. At each frame, the agent observes a panoramic image and has the knowledge of previously selected viewing angles. The task of the agent is to shift the current viewing angle (i.e. action) to the next preferred one (i.e., goal). We propose to directly learn an online policy of the agent from data. Specifically, we leverage a state-of-the-art object detector to propose a few candidate objects of interest (yellow boxes in Fig. 1). Then, a recurrent neural network is used to select the main object (green dash boxes in Fig. 1). Given the main object and previously selected viewing angles, our method regresses a shift in viewing angle to move to the next one. We use the policy gradient technique to jointly train our pipeline, by minimizing: (1) a regression loss measuring the distance between the selected and ground truth viewing angles, (2) a smoothness loss encouraging smooth transition in viewing angle, and (3) maximizing an expected reward of focusing on a foreground object. To evaluate our method, we built a new 360-Sports video dataset consisting of five sports domains. We trained domain-specific agents and achieved the best performance on viewing angle selection accuracy and users preference compared to [53] and other baselines.

A field study comparing approaches to collecting annotated activity data in real-world settings

Collecting ground-truth annotations for contextual data is vital to context-aware system development. However, current research lacks a systematic analysis of different approaches to collecting such data. We present a field experiment comparing three approaches: Participatory, Context-Triggered In Situ, and Context-Triggered Post Hoc, which involved users in recording and annotating activity data in real-world settings. We compared the quantity and quality of collected data using each approach, as well as the participant experience. We found Context-Triggered approaches produced more recordings, whereas the Participatory approach produced a greater amount of data with higher completeness and precision. Moreover, while participants appreciated automated recording and reminders for convenience, they highly valued having control over what and when to record and annotate. We conclude that user burden and user control are key aspects to consider when collecting and annotating contextual data with participants, and suggest features for a future tool focused on these two aspects.

An investigation of using mobile and situated crowdsourcing to collect annotated travel activity data in real-word settings

Collecting annotated activity data is vital to many forms of context-aware system development. Leveraging a crowd of smartphone users to collect annotated activity data in the wild is a promising direction because the data being collected are realistic and diverse. However, current research lacks a systematic analysis comparing different approaches for collecting such data and investigating how users use these approaches to collect activity data in real world settings. In this paper, we report results from a field study investigating the use of mobile crowdsourcing to collect annotated travel activity data through three approaches: Participatory, Context-Triggered In Situ, and Context-Triggered Post Hoc. In particular, we conducted two phases of analysis. In Phase One, we analyzed and compared the resulting data collected via the three approaches and user experience. In Phase Two, we analyzed users recording and annotation behavior as well as the annotation content in using each approach in the field. Our results suggested that although Context-Triggered approaches produced a larger number of recordings, they did not necessarily lead to a larger quantity of data than the Participatory approach. It was because many of the recordings were either not labeled, incomplete, and/or fragmented due to the imperfect context detection. In addition, recordings collected by the Participatory approach tended to be more complete and contain less noise. In terms of user experience, while users appreciated automated recording and reminders because of their convenience, they highly valued having the control over what and when to record and annotate that the Participatory approach provided. Finally, we showed that activity type (Driver, Riding as Passenger, Walking) influenced users behaviors in recording and annotating their activity data. It influenced the timing of recording and annotating using the Participatory approach, users receptivity using the Context-Triggered In Situ approach, and the characteristics of the content of annotations. Based on these findings, we provide design and methodological recommendations for future work that aims to leverage mobile crowdsourcing to collect annotated activity data. The Participatory approach produced high-quality annotated activity data.User burden and control are two crucial aspects for sustaining user compliance.Activity affects recording and annotation timing and characteristics of annotations.Activity affects users receptivity when using the Context-Triggered approach.We offer suggestions on the approaches, tools, and instructions to collect activity.

HEED: situated and distributed interactive devices for self-reporting

In situ self-reporting is a widely used technique in HCI, ubiquitous computing, especially for assessment and intervention in health and wellness. Although, smartphones are widely used for self-reporting, there is an opportunity to design dedicated, unobtrusive and distributed self-reporting devices that improve the coverage of sampled experiences. We designed self-reporting devices for two scenarios of reporting-Activities and Stress/Sleepiness. The devices were placed by the users in their surroundings for ease of access. We show that the devices are useful especially in certain situations such as when the user is engaged in focus work. Moreover, we show that the preference of phone or devices to self-report varied between users based on multiple factors such as their engagement with phone and their preferences about being surrounded by multiple devices.

Heed: Exploring the Design of Situated Self-Reporting Devices

In-situ self-reporting is a widely used data collection technique for understanding peoples behavior in context. Characteristics of smartphones such as their high proliferation, close proximity to their users, and heavy use have made them a popular choice for applications that require frequent self-reporting. Newer device categories such as wearables and voice assistants offer their own advantages, providing an opportunity to explore a wider range of self-reporting approaches. In this paper, we focus on exploring the design space of Situated Self-Reporting (SSR) devices. We present the Heed system, consisting of simple, low-cost, and low-power SSR devices that are distributed in the environment of the user and can be appropriated for reporting measures such as stress, sleepiness, and activities. In two real-world studies with 10 and 7 users, we compared and analyzed the use of smartphone and Heed devices to uncover differences in their use due to the influence of factors such as situational and social context, notification types, and physical design. Our findings show that Heed devices complemented smartphones in the coverage of activities, locations and interaction preferences. While the advantage of Heed was its single-purpose and dedicated location, smartphones provided mobility and flexibility of use.

Making Local Information More Accessible: A Diary Study of Information Channel Selection of Mobile Users

The mobile Web has been a dominant channel for mobile users to fulfill information needs. Mobile users, however, also use other information channels such as the environment and personal sources. Through a 14-week exploratory diary study with ten mobile users, we found uncertainty with the accessibility of target information in an information channel to be a significant barrier for mobile users in using that channel to address an information need. This barrier is especially apparent when the information to be sought is for an upcoming activity or plan. In addition, we found that local infrastructural information such as information of services, resources, and directions within a specific point of interest was often perceived inaccessible or difficult to access on the Web, resulting in mobile users often preferring to use other information channels to obtain the information. We provide an explanation of the impact of such uncertainty using the cognitive maps framework from environmental cognition, and provide design implications on how Ubicomp systems can make local infrastructural information more accessible to mobile users.

Understanding how trace segmentation impacts transportation mode detection

Transportation mode (TM) detection is one of the activity recognition tasks in ubiquitous computing. A number of previous studies have compared the performance of various classifiers for TM detection. However, the current study is the first work aiming to understand how TM detection performance is impacted by how the recorded location traces are segmented into data segments for training a classifier. In our preliminary experiments we examine three trace segmentation (TS) methods---Uniform Duration (UniDur), Uniform Number of Location Points (UniNP), and Uniform Distance (UniDis)---and compare their performance on detecting different transportation modes. The results indicate that while driving can be more accurately detected by using UniDis method, walking and bus can be more accurately detected by using UniDur method. This suggests that choosing a right TS method for training a TM classifier is an important step to accurately detect particular transportation modes.