Joshua M. Hailpern

TEAM STORM: demonstrating an interaction model for working with multiple ideas during creative group work

Informal design tools can provide immense value during the creative stages of the design process, e.g., by transforming sketches into interactive simulations. Two key limitations of informal and many other design tools are that they do not promote working with multiple design ideas in parallel or collaboration. In this paper, we present a new interaction model that allows a team of designers to work efficiently with multiple ideas in parallel. The model is grounded in theories of creativity and collaboration, and was further informed by observations of creative group work practice. Our interaction model is fully demonstrated within a new system called TEAM STORM. Results from an initial evaluation indicate that design teams are able to effectively utilize our system to create, organize, and share multiple design ideas during creative group work. The benefit of our model is that it demonstrates how many existing single-user design tools can be extended to support working efficiently with multiple ideas in parallel and co-located collaboration.

VCode and VData: illustrating a new framework for supporting the video annotation workflow

Digital tools for annotation of video have the promise to provide immense value to researchers in disciplines ranging from psychology to ethnography to computer science. With traditional methods for annotation being cumbersome, time-consuming, and frustrating, technological solutions are situated to aid in video annotation by increasing reliability, repeatability, and workflow optimizations. Three notable limitations of existing video annotation tools are lack of support for the annotation workflow, poor representation of data on a timeline, and poor interaction techniques with video, data, and annotations. This paper details a set of design requirements intended to enhance video annotation. Our framework is grounded in existing literature, interviews with experienced coders, and ongoing discussions with researchers in multiple disciplines. Our model is demonstrated in a new system called VCode and VData. The benefit of our system is that is directly addresses the workflow and needs of both researchers and video coders.

Web 2.0: blind to an accessible new world

With the advent of Web 2.0 technologies, websites have evolved from static pages to dynamic, interactive Web-based applications with the ability to replicate common desktop functionality. However, for blind and visually impaired individuals who rely upon screen readers, Web 2.0 applications force them to adapt to an inaccessible use model. Many technologies, including WAI-ARIA, AJAX, and improved screen reader support, are rapidly evolving to improve this situation. However, simply combining them does not solve the problems of screen reader users. The main contributions of this paper are two models of interaction for screen reader users, for both traditional websites and Web 2.0 applications. Further contributions are a discussion of accessibility difficulties screen reader users encounter when interacting with Web 2.0 applications, a user workflow design model for improving Web 2.0 accessibility, and a set of design requirements for developers to ease the users burden and increase accessibility. These models, accessibility difficulties, and design implications are based directly on responses and lessons learned from usability research focusing on Web 2.0 usage and screen reader users. Without the conscious effort of Web engineers and designers, most blind and visually impaired users will shy away from using new Web 2.0 technology in favor of desktop based applications.

Creating a spoken impact: encouraging vocalization through audio visual feedback in children with ASD

One hallmark difficulty of children with Autism Spectrum Disorder (ASD) centers on communication and speech. Research into computer visualizations of voice has been shown to influence conversational patterns and allow users to reflect upon their speech. In this paper, we present the Spoken Impact Project (SIP), an effort to examine the effect of audio and visual feedback on vocalizations in low-functioning children with ASD by providing them with additional means of understanding and exploring their voice. This research spans over 12 months, including the creation of multiple software packages and detailed analysis of more than 20 hours of experimental video. SIP demonstrates the potential of computer generated audio and visual feedback to encourage vocalizations of children with ASD.

A3: HCI Coding Guideline for Research Using Video Annotation to Assess Behavior of Nonverbal Subjects with Computer-Based Intervention

HCI studies assessing nonverbal individuals (especially those who do not communicate through traditional linguistic means: spoken, written, or sign) are a daunting undertaking. Without the use of directed tasks, interviews, questionnaires, or question-answer sessions, researchers must rely fully upon observation of behavior, and the categorization and quantification of the participant’s actions. This problem is compounded further by the lack of metrics to quantify the behavior of nonverbal subjects in computer-based intervention contexts. We present a set of dependent variables called A3 (pronounced A-Cubed) or Annotation for ASD Analysis, to assess the behavior of this demographic of users, specifically focusing on engagement and vocalization. This paper demonstrates how theory from multiple disciplines can be brought together to create a set of dependent variables, as well as demonstration of these variables, in an experimental context. Through an examination of the existing literature, and a detailed analysis of the current state of computer vision and speech detection, we present how computer automation may be integrated with the A3 guidelines to reduce coding time and potentially increase accuracy. We conclude by presenting how and where these variables can be used in multiple research areas and with varied target populations.

Designing visualizations to facilitate multisyllabic speech with children with autism and speech delays

The ability of children to combine syllables represents an important developmental milestone. This ability is often delayed or impaired in a variety of clinical groups, including children with autism spectrum disorders (ASD) and speech delays (SPD). Prior work has demonstrated successful use of computer-based voice visualizations to facilitate speech production and vocalization in children with and without ASD/SPD. While prior work has focused on increasing frequency of speech-like vocalizations or accuracy of speech sound production, we believe that there is a potential new direction of research: exploration of real-time visualizations to shape multisyllabic speech. Over two years we developed VocSyl, a real-time voice visualization system. Rather than building visualizations based on what adult clinicians and software designers may think is needed, we designed VocSyl using the Task Centered User Interface Design (TCUID) methodology throughout the design process. Children with ASD and SPD, targeted users of the software, were directly involved in the development process, allowing us to focus on what these children demonstrate they require. This paper presents the results of our TCUID design cycle of VocSyl, as well as design guidelines for future work with children with ASD and SPD.

Encouraging speech and vocalization in children with autistic spectrum disorder

Technology can improve the life of those with Autistic Spectrum Disorder (ASD). Specifically, some children with ASD are not fortunate enough to acquire the ability to communicate with language on their own. With language being an important method of communication, socialization, and interacting with the world, these children need researchers to develop new solutions to help teach vocalization and speech. Without speech, these children will have difficulty communicating their needs, wants, and emotions, as well as being able to function within society at large. This paper examines the existing HCI research on ASD, as well as proposes a new direction for investigation.

WINACS: construction and analysis of web-based computer science information networks

WINACS (Web-based Information Network Analysis for Computer Science) is a project that incorporates many recent, exciting developments in data sciences to construct a Web-based computer science information network and to discover, retrieve, rank, cluster, and analyze such an information network. With the rapid development of the Web, huge amounts of information are available in the form of Web documents, structures, and links. It has been a dream of the database and Web communities to harvest such information and reconcile the unstructured nature of the Web with the neat, semi-structured schemas of the database paradigm. Taking computer science as a dedicated domain, WINACS first discovers related Web entity structures, and then constructs a heterogeneous computer science information network in order to rank, cluster and analyze this network and support intelligent and analytical queries.

Visualizations: speech, language & autistic spectrum disorder

Without speech, we can have great difficulty communicating wants, emotions, needs, and interacting with society at large. During typical child development, an infant acquires language skills without explicit teaching. However, some children, including those with Autistic Spectrum Disorder (ASD) have explicit difficulty developing these skills in the context of everyday interactions. HCI is situated to help by developing technology and techniques to teach speech and language skills to children with ASD through the use of visual and auditory feedback. This paper examines preliminary results from a study, as well as describes new directions of research.

To print or not to print: hybrid learning with METIS learning platform

As part of the explosion in educational software, online tools, and open educational resources there has been a rapid devaluation of printed textbooks. While digital texts have advantages, printed textbooks still provide irreplaceable value over online media. Therefore technology should enhance, rather than eliminate printed text. To this end, this paper presents METIS, a hybrid learning software/service platform that is designed to support active reading. METIS provides easy digital-to-print-to-digital usage, simple creation of Cheat Sheets & FlexNotes for personal note taking and organization, and a custom flexible rendering & publishing engine for education called Aero. METIS was designed based on lessons learned from a formative study of 523 students at SJSU, and validated through focus groups involving 32 educators and students at both high school and college levels.