Olufisayo Omojokun

From devices to tasks: automatic task prediction for personalized appliance control

One of the driving applications of ubiquitous computing is universal appliance interaction: the ability to use arbitrary mobile devices to interact with arbitrary appliances, such as TVs, printers, and lights. Because of limited screen real estate and the plethora of devices and commands available to the user, a central problem in achieving this vision is predicting which appliances and devices the user wishes to use next in order to make interfaces for those devices available. We believe that universal appliance interaction is best supported through the deployment of appliance user interfaces (UIs) that are personalized to a user’s habits and information needs. In this paper, we suggest that, in a truly ubiquitous computing environment, the user will not necessarily think of devices as separate entities; therefore, rather than focus on which device the user may want to use next, we present a method for automatically discovering the user’s common tasks (e.g., watching a movie, or surfing TV channels), predicting the task that the user wishes to engage in, and generating an appropriate interface that spans multiple devices. We have several results. We show that it is possible to discover and cluster collections of commands that represent tasks and to use history to predict the next task reliably. In fact, we show that moving from devices to tasks is not only a useful way of representing our core problem, but that it is, in fact, an easier problem to solve. Finally, we show that tasks can vary from user to user.

Comparing end-user and intelligent remote control interface generation

Traditional remote controls typically allow users to activate functionality of a single device. Given that users activate a subset of functionality across devices to accomplish a particular task, it is attractive to consider a remote control directly supporting this behavior. We present qualitative and quantitative results from a study of two promising approaches creating such a remote control: end-user programming and machine learning. In general, results show that each approach possesses advantages and disadvantages, and that neither is optimal.

Automatic Generation of Device User-Interfaces?

One of the visions of pervasive computing is using mobile computers to interact with networked devices. A question raised by this vision is: should the user-interfaces of these devices be handcrafted manually or generated automatically? Based on experience within the domain of desktop computing, the answer seems to be that automatic generation is not flexible enough to support a significant number of useful interfaces but requires substantially less coding effort for the interfaces it can create. We show that the answer is much more complicated when we consider networking of traditional appliances such as stereos and TVs. Using qualitative arguments and quantitative experimental data, we show that the manual vs. generated issue must be resolved based on: (a) not only user-interface programming and flexibility but also several other metrics such as space and time costs, binding time, and reliability (b) whether it is a graphical or speech based user-interface, (c) the size of the device user-interface, (d) whether the manually written user-interface code is available at the mobile computer or at a remote machine, and (e) the network bandwidth between the mobile computer and remote factory

Experiments with mobile computing middleware for deploying appliance UIs

An important component of the vision of ubiquitous computing is universal appliance-interaction-the ability to use mobile and wearable computers such as cell phones and palmtops to interact with arbitrary devices and appliances distributed over a network. This raises the user interface (UI) deployment issue, which involves implementing middleware for deploying compatible UIs of arbitrary appliances on different kinds of mobile computers. In this paper, we provide a preliminary evaluation of different approaches to implementing such middleware consisting of experimental results and a qualitative analysis. In this evaluation, we compare (a) generating vs. installing predefined UIs, (b) installing local vs. remote predefined UIs, (c) deployment of speech vs. graphical UIs, (d) deployment of UIs of simple vs. complex appliances, (e) deployment over a wireless and wired network, and (f) deployment on laptop and palmtop computers. We consider storage costs, deployment times, and operation invocation times in our experiments.

Impact of user context on song selection

The rise of digital music has led to a parallel rise in the need to manage music collections of several thousands of songs on a single device. Manual selection of songs for a music listening experience is a cumbersome task. In this paper, we present an initial exploration of the feasibility of using song signal properties and user context information to assist in automatic song selection. Users listened to music over the course of a month while their context and song selections were tracked. Initial results suggest the use of context information can improve automated song selection when patterns are learned for each individual.

Directions in ubiquitous computing: a position statement

One of the visions of ubiquitous computing is the ability to use arbitrary interactive devices such as cell phones and handheld computers to interact with arbitrary remote networked appliances such as TVs, printers, and EKG machines. There are many potential practical benefits of controlling a network appliance using a remote user interface on a mobile computer rather than directly using the native, physical user interface offered by it. Users can control appliances from arbitrary locations, for instance, people on vacation can set their home water sprinklers. Moreover, an interactive mobile device can be used as a true universal control, accessing arbitrary devices such as TVs, thermostats, and light switches. Furthermore, a mobile device can offer user interfaces that are more sophisticated than the physical user interfaces offered by the appliances directly. For instance, it can offer a single command to shut off all lights in a building. User interface generation is a promising research direction in ubiquitous computing because it allows a mobile device to interact with an arbitrary appliance without incurring the problems of downloading and classifying user interface code. On the other hand, the generation approach offers limited presentation styles. We believe this is not a serious drawback, for three reasons which are outlined.

User modeling for personalized universal appliance interaction

One of the driving applications of ubiquitous computing is universal appliance interaction. It is the ability to use arbitrary mobile devices-some of which we traditionally think of as computers (e.g. handhelds and wearables), and some of which we do not (e.g. cell phones)-to interact with arbitrary appliances such as TVs, printers, and lights. We believe that universal appliance interaction is best supported through the deployment of appliance user-interfaces (UIs) that are personalized to a users habits and information needs. We are building a UI deployment system for universal appliance interaction to support various personalization features based on predicting a users behavior. It is our belief that we can achieve these features in our system by modeling user actions using machine learning (ML) algorithms. The initial step in building such a system that relies on ML for prediction is to show that there are patterns in user appliance interaction. In this paper, our goal is to present evidence demonstrating these patterns.

History-based device graphical user-interfaces

Due to limited screen space on mobile computers, device GUIs can span multiple screens-requiring tedious scrolling and tabbing for commands. History-based device GUIs can significantly reduce required space by only presenting the commands a user typically needs based on the users behavior over a short training period. Moreover, history-based UIs and model-based UI generation are symbiotic. Generation relieves programmers from the overhead of logging and interpreting the interaction histories. Conversely, history-based user-interaction noticeably lowers inherent UI generation time by omitting unneeded commands.

Efficient Retargeting of Generated Device User-Interfaces

Many pervasive computing systems have been built for using mobile computers to interact with networked devices. To deploy a devices user-interface, several systems dynamically generate the user-interface on a mobile computer. While this approach has several advantages, empirical results from different generators show that it takes a relatively long time for a mobile computer to create a user-interface from scratch. This paper shows that it is possible to overcome this limitation by efficiently mapping or retargeting a previously generated user-interface of one (source) device to another (target) device of the same or different type. Using the implementation of an existing generator and a set of real-world scenarios, we show that user-interface retargeting can yield deployment times that are often as good as or noticeably better than the approach of locally loading pre-existing manually-written user-interface code.

Partial signal extraction for mobile media players

Audio signal properties can provide a media player with highly descriptive feature sets in order to intelligently select similar songs for a music stream. A well-known problem among researchers in music information retrieval, however, is that extracting signal properties requires a significant amount of computational resources, thus making it impractical for even the most advanced mobile media players. Although other approaches to retrieving data are possible, local extraction still has unique benefits. Using a combination of machine learning and profiling techniques, this paper presents an initial evaluation of partial signal extraction, which reduces resource requirements by locally collecting signals from parts of a song rather than all. Our preliminary experiments suggest that this idea can offer significantly lower resource requirements while losing marginal song information.