Fred B. Wood

A practical approach to e-government Web evaluation

Does your Web site function smoothly enough to deliver government services? Combining evaluation techniques gives you a multidimensional answer. The article discusses usability testing, and the use of user feedback, usage data, and Web and Internet performance data.

Web Evaluation at the US National Institutes of Health: Use of the American Customer Satisfaction Index Online Customer Survey

Background The National Institutes of Health (NIH), US Department of Health and Human Services (HHS), realized the need to better understand its Web users in order to help assure that websites are user friendly and well designed for effective information dissemination. A trans-NIH group proposed a trans-NIH project to implement an online customer survey, known as the American Customer Satisfaction Index (ACSI) survey, on a large number of NIH websites—the first “enterprise-wide” ACSI application, and probably the largest enterprise Web evaluation of any kind, in the US government. The proposal was funded by the NIH Evaluation Set-Aside Program for two years at a cost of US

Evaluating Internet End-to-end Performance: Overview of Test Methodology and Results

1.5 million (US

Big Data Science: Opportunities and Challenges to Address Minority Health and Health Disparities in the 21st Century

1.275 million for survey licenses for 60 websites at US

Interactive Science Publishing: A joint OSA-NLM project

18,000 per website; US

Program Management and Policy Issues in Information Outreach

225,000 for a project evaluation contractor). Objective The overall project objectives were to assess the value added to the participating NIH websites of using the ACSI online survey, identify any NIH-wide benefits (and limitations) of the ACSI, ascertain any new understanding about the NIH Web presence based on ACSI survey results, and evaluate the effectiveness of a trans-NIH approach to Web evaluation. This was not an experimental study and was not intended to evaluate the ACSI survey methodology, per se, or the impacts of its use on customer satisfaction with NIH websites. Methods The evaluation methodology included baseline pre-project websites profiles; before and after email surveys of participating website teams; interviews with a representative cross-section of website staff; observations of debriefing meetings with website teams; observations at quarterly trans-NIH Web staff meetings and biweekly trans-NIH leadership team meetings; and review and analysis of secondary data. Results Of the original 60 NIH websites signed up, 55 implemented the ACSI survey, 42 generated sufficient data for formal reporting of survey results for their sites, and 51 completed the final project survey. A broad cross-section of websites participated, and a majority reported significant benefits and new knowledge gained from the ACSI survey results. NIH websites as a group scored consistently higher on overall customer satisfaction relative to US government-wide and private sector benchmarks. Conclusions Overall, the enterprise-wide experiment was successful. On the level of individual websites, the project confirmed the value of online customer surveys as a Web evaluation method. The evaluation results indicated that successful use of the ACSI, whether site-by-site or enterprise-wide, depends in large part on strong staff and management support and adequate funding and time for the use of such evaluative methods. In the age of Web-based e-government, a broad commitment to Web evaluation may well be needed. This commitment would help assure that the potential of the Web and other information technologies to improve customer and citizen satisfaction is fully realized.

Public library consumer health information pilot project: results of a National Library of Medicine evaluation

Objective: An evaluation of Internet end-to-end performance was conducted for the purpose of better understanding the overall performance of Internet pathways typical of those used to access information in National Library of Medicine (NLM) databases and, by extension, other Internet-based biomedical information resources. Design: The evaluation used a three-level test strategy: 1) user testing to collect empirical data on Internet performance as perceived by users when accessing NLM Web-based databases, 2) technical testing to analyze the Internet paths between the NLM and the users desktop computer terminal, and 3) technical testing between the NLM and the World Wide Web (“Web”) server computer at the users institution to help characterize the relative performance of Internet pathways. Measurements: Time to download the front pages of NLM Web sites and conduct standardized searches of NLM databases, data transmission capacity between NLM and remote locations (known as the bulk transfer capacity [BTC], “ping” round-trip time as an indication of the latency of the network pathways, and the network routing of the data transmissions (number and sequencing of hops). Results: Based on 347 user tests spread over 16 locations, the median time per location to download the main NLM home page ranged from 2 to 59 seconds, and 1 to 24 seconds for the other NLM Web sites tested. The median time to conduct standardized searches and get search results ranged from 2 to 14 seconds for PubMed and 4 to 18 seconds for Internet Grateful Med. The overall problem rate was about 1 percent; that is, on the average, users experienced a problem once every 100 test measurements. The user terminal tests at five locations and Web host tests at 13 locations provided profiles of BTC, RTT, and network routing for both dial-up and fixed Internet connections. Conclusion: The evaluation framework provided a profile of typical Internet performance and insights into network performance and time-of-day/day-of-week variability. This profile should serve as a frame of reference to help identify and diagnose connectivity problems and should contribute to the evolving concept of Internet quality of service.

MedlinePlus and the challenge of low health literacy: findings from the Colonias project

Addressing minority health and health disparities has been a missing piece of the puzzle in Big Data science. This article focuses on three priority opportunities that Big Data science may offer to the reduction of health and health care disparities. One opportunity is to incorporate standardized information on demographic and social determinants in electronic health records in order to target ways to improve quality of care for the most disadvantaged populations over time. A second opportunity is to enhance public health surveillance by linking geographical variables and social determinants of health for geographically defined populations to clinical data and health outcomes. Third and most importantly, Big Data science may lead to a better understanding of the etiology of health disparities and understanding of minority health in order to guide intervention development. However, the promise of Big Data needs to be considered in light of significant challenges that threaten to widen health disparities. Care must be taken to incorporate diverse populations to realize the potential benefits. Specific recommendations include investing in data collection on small sample populations, building a diverse workforce pipeline for data science, actively seeking to reduce digital divides, developing novel ways to assure digital data privacy for small populations, and promoting widespread data sharing to benefit under-resourced minority-serving institutions and minority researchers. With deliberate efforts, Big Data presents a dramatic opportunity for reducing health disparities but without active engagement, it risks further widening them.

Measuring the Difference: Guide to Planning and Evaluating Health Information Outreach

Interactive Scientific Publishing (ISP) has been developed by the Optical Society of America with support from the National Library of Medicine at NIH. It allows authors to publish papers which are linked to the referenced 2D and 3D original image datasets that can then be viewed and analyzed interactively by the reader. ISP provides the software for authors to organize and publish source data while offering readers the viewing and analysis tools. The ultimate goal of ISP is to improve learning and understanding of the information being presented. The results reported here are intermediate in reaching the ultimate goals. Goals: Implement and evaluate online user access to interactive datasets that complement online scientific papers. Methods: Software development; website evaluation, including web log data monitoring, online user survey. Results: An expert group of respondents found the interactive data component to have positive impacts on understanding and discovery. However a significant minority reported that the software interface, instructions, and/or download presented a barrier to effective use of the interactive data functionality. Conclusions: This exploratory study indicates that interactive datasets offer potential benefits as a complement to published papers, but that further software improvements and more in-depth evaluation are needed to best discern future directions.

Use of Internet Audience Measurement Data to Gauge Market Share for Online Health Information Services

Abstract With the advent of the Internet, American Indian/Alaska Native (AI/AN) communities in the Pacific Northwest have new opportunities to access high quality and relevant health information. The Pacific Northwest Regional Medical Library (PNRML), regional headquarters of the National Network of Libraries of Medicine, a program sponsored by the National Library of Medicine, sought to facilitate that access and worked with a selected group of sixteen tribes and native village consortia. The steps were: (1) work with AI/AN communities to arrive at mutually-agreeable health information connectivity objectives and long-term solutions, (2) provide funding to AI/AN communities to ensure Internet connectivity and the presence of Internet workstations for health workers and for the public, and (3) train in effective health information seeking. Community-based approaches helped the PNRML adjust policies and practice for improved information outreach to AI/AN communities in the region. The project participants, collaborating with our staff, successfully carried out many of the community goals and, at the same time, we gained insight about the variables that were barriers or facilitators of success. While we are coming at outreach from a library perspective, the policy and method lessons we learned could apply to a broad variety of outreach endeavors.