Matt Willis

Crowdsourcing Participatory Evaluation of Medical Pictograms Using Amazon Mechanical Turk

Background Consumer and patient participation proved to be an effective approach for medical pictogram design, but it can be costly and time-consuming. We proposed and evaluated an inexpensive approach that crowdsourced the pictogram evaluation task to Amazon Mechanical Turk (MTurk) workers, who are usually referred to as the “turkers”. Objective To answer two research questions: (1) Is the turkers’ collective effort effective for identifying design problems in medical pictograms? and (2) Do the turkers’ demographic characteristics affect their performance in medical pictogram comprehension? Methods We designed a Web-based survey (open-ended tests) to ask 100 US turkers to type in their guesses of the meaning of 20 US pharmacopeial pictograms. Two judges independently coded the turkers’ guesses into four categories: correct, partially correct, wrong, and completely wrong. The comprehensibility of a pictogram was measured by the percentage of correct guesses, with each partially correct guess counted as 0.5 correct. We then conducted a content analysis on the turkers’ interpretations to identify misunderstandings and assess whether the misunderstandings were common. We also conducted a statistical analysis to examine the relationship between turkers’ demographic characteristics and their pictogram comprehension performance. Results The survey was completed within 3 days of our posting the task to the MTurk, and the collected data are publicly available in the multimedia appendix for download. The comprehensibility for the 20 tested pictograms ranged from 45% to 98%, with an average of 72.5%. The comprehensibility scores of 10 pictograms were strongly correlated to the scores of the same pictograms reported in another study that used oral response–based open-ended testing with local people. The turkers’ misinterpretations shared common errors that exposed design problems in the pictograms. Participant performance was positively correlated with their educational level. Conclusions The results confirmed that crowdsourcing can be used as an effective and inexpensive approach for participatory evaluation of medical pictograms. Through Web-based open-ended testing, the crowd can effectively identify problems in pictogram designs. The results also confirmed that education has a significant effect on the comprehension of medical pictograms. Since low-literate people are underrepresented in the turker population, further investigation is needed to examine to what extent turkers’ misunderstandings overlap with those elicited from low-literate people.

Healthcare utilization and symptom variation among veterans using Behavioral Telehealth Center services.

Substance use and mental health problems are often underdiagnosed and undertreated in primary care. Veterans affairs facilities are using the Behavioral Telehealth Center (BTC) to provide evidence-based assessments for primary care patients via telephone. Whether participation in BTC services is associated with (1) increases in healthcare utilization and (2) decreases in symptoms based on behavioral health screening instruments, post-BTC services compared with pre-BTC services were investigated. Retrospective data were extracted for 1,820 patients who were referred to the BTC. Differences in utilization rates and symptom scores pre- and post-BTC services were tested using repeated measures analysis of covariance while controlling for relevant sociodemographic variables. Participants (1) utilized significantly more substance use and mental health treatment services and (2) had significantly lower alcohol and depression screening scores post-BTC services compared with pre-BTC services. This initial evaluation provides support that BTC services are associated with increased healthcare utilization and decreased alcohol and depressive symptoms.

Documenting work: From participant observation to participant tracing

In this chapter, we argue that combining different qualitative research methods can facilitate the study of collective cognition in organizations, thus compensating the limitations of more traditional approaches. Using our own research experience in studying how designers develop new ideas, we explain how the combined use of ethnography, grounded theory and visual narrative analysis allowed us to gain a deep understanding of how material practices influence collective cognitive sensemaking in organizations. In particular, we show (1) how ethnography allowed us to map and unpack the material practices designers engage in when developing new ideas, (2) how interviews and grounded theory helped us articulate informants’ interpretations of these practices and reveal the underlying cognitive processes, and, finally, (3) how visual narrative analysis was useful to systematically track changes in the evolving collective interpretations, and by doing so to link together practices and processes in a longitudinal fashion.In this chapter we discuss a sampling technique that has been employed in recent works, but has yet to be delineated as a distinct methodology: “structural sampling.” Structural sampling allows the investigator to illuminate the inner-workings of a social system by interviewing actors in a variety of roles and making comparisons across multiple levels of analysis. We describe the technique of structural sampling and its purpose, elucidate the benefits and challenges of structural sampling, provide several examples to illustrate potential uses of this technique, and situate structural sampling in the context of extant qualitative research methodologies.

Document Practice as Insight to Digital Infrastructures of Distributed, Collaborative Social Scientists

The objective of this study is to understand scholarly research practice in virtual, distributed collaborations by focusing on the flow of documents among the participants and to advance design guidance for supporting improved document practice across distributed collaborative platforms. To do so, we develop a theoretical framework on document practice highlighting the sociotechnical role of documents in digital infrastructure. This mixed-methods study will first conduct semi-structured interviews to understand document practices. The second phase of the study will collect trace data of documents as a way to understand how they change over time. In this poster, we report on the analysis of twelve interviews from social scientists working in virtual collaborations. Initial findings show that social scientists organize their documents and scholarly work on emergent digital infrastructures. Although not ideal, emergent digital infrastructures provide stability for collaboration across time and space.

Towards a method of documentary practices for personal health information management

Many intellectual fields study patients’ use of health information systems, yet there is little innovation in the variety and richness of methodologies employed to investigate the complex socio-technical relationship between health consumers who use technical systems to engage in the practice of personal health information management. This is, in part, due to the methodological challenges of studying patients’ documentation practices. In this note, we present the early development of a document practice approach to address these gaps. The documentary practice approach is rooted in the socio-technical perspective that social systems and technology are intricately connected and recursive. Documentary practice methodologies are mindful of this entanglement in studying the work practices of actors by connecting practice to the use of documents, both physical and digital, and mapping the patient’s field of documents and technical infrastructure within which health management occurs.

Work that Enables Care: Understanding Tasks, Automation, and the National Health Service.

Automation of jobs is discussed as a threat to many job occupations, but in the UK healthcare sector many view technology and automation as a way to save a threatened system. However, existing quantitative models that rely on occupation-level measures of the likelihood of automation suggest that few healthcare occupations are susceptible to automation. In order to improve these quantitative models, we focus on the potential impacts of task-level automation on health work, using qualitative ethnographic research to understand the mundane information work in general practices. By understanding the detailed tasks and variations of information work, we are building a more complete and accurate understanding of how healthcare staff work and interact with technology and with each other, often mediated by technology.

Real-time individualized training vectors for experiential learning.

Military training utilizing serious games or virtual worlds potentially generate data that can be mined to better understand how trainees learn in experiential exercises. Few data mining approaches for deployed military training games exist. Opportunities exist to collect and analyze these data, as well as to construct a full-history learner model. Outcomes discussed in the present document include results from a quasi-experimental research study on military game-based experiential learning, the deployment of an online game for training evidence collection, and results from a proof-of-concept pilot study on the development of individualized training vectors. This Lab Directed Research & Development (LDRD) project leveraged products within projects, such as Titan (Network Grand Challenge), Real-Time Feedback and Evaluation System, (Americas Army Adaptive Thinking and Leadership, DARWARS Ambush! NK), and Dynamic Bayesian Networks to investigate whether machine learning capabilities could perform real-time, in-game similarity vectors of learner performance, toward adaptation of content delivery, and quantitative measurement of experiential learning.

Leveraging Mobile Devices to Develop Intercultural Competency for Digital Students

Mobile devices can help digital students reach out across cultures to develop intercultural competence, improve learning, and provide course support for a variety of course topics. Intercultural competence is expressed through openness, cognitive adaptability, and behavioral flexibility toward unfamiliar cultures. Digital students demonstrate a behavioral flexibility toward technology use that can be leveraged to encourage students to embrace cultures different from their own. This paper explores the feasibility of using mobile devices as viable options for course support by utilizing traditional learning styles and cultural learning styles. From the conducted survey preferred networks are identified for creating a community to support mobile learning.