Elliot R. Siegel

A practical approach to e-government Web evaluation

Does your Web site function smoothly enough to deliver government services? Combining evaluation techniques gives you a multidimensional answer. The article discusses usability testing, and the use of user feedback, usage data, and Web and Internet performance data.

Web Evaluation at the US National Institutes of Health: Use of the American Customer Satisfaction Index Online Customer Survey

Background The National Institutes of Health (NIH), US Department of Health and Human Services (HHS), realized the need to better understand its Web users in order to help assure that websites are user friendly and well designed for effective information dissemination. A trans-NIH group proposed a trans-NIH project to implement an online customer survey, known as the American Customer Satisfaction Index (ACSI) survey, on a large number of NIH websites—the first “enterprise-wide” ACSI application, and probably the largest enterprise Web evaluation of any kind, in the US government. The proposal was funded by the NIH Evaluation Set-Aside Program for two years at a cost of US

Evaluating Internet End-to-end Performance: Overview of Test Methodology and Results

1.5 million (US

Interactive Science Publishing: A joint OSA-NLM project

1.275 million for survey licenses for 60 websites at US

Empowering patients and community online: Evaluation of the AIDS community information outreach program

18,000 per website; US

Enhancing communications and connectivity in Africa: the multilateral initiative on malaria (MIM) model.

225,000 for a project evaluation contractor). Objective The overall project objectives were to assess the value added to the participating NIH websites of using the ACSI online survey, identify any NIH-wide benefits (and limitations) of the ACSI, ascertain any new understanding about the NIH Web presence based on ACSI survey results, and evaluate the effectiveness of a trans-NIH approach to Web evaluation. This was not an experimental study and was not intended to evaluate the ACSI survey methodology, per se, or the impacts of its use on customer satisfaction with NIH websites. Methods The evaluation methodology included baseline pre-project websites profiles; before and after email surveys of participating website teams; interviews with a representative cross-section of website staff; observations of debriefing meetings with website teams; observations at quarterly trans-NIH Web staff meetings and biweekly trans-NIH leadership team meetings; and review and analysis of secondary data. Results Of the original 60 NIH websites signed up, 55 implemented the ACSI survey, 42 generated sufficient data for formal reporting of survey results for their sites, and 51 completed the final project survey. A broad cross-section of websites participated, and a majority reported significant benefits and new knowledge gained from the ACSI survey results. NIH websites as a group scored consistently higher on overall customer satisfaction relative to US government-wide and private sector benchmarks. Conclusions Overall, the enterprise-wide experiment was successful. On the level of individual websites, the project confirmed the value of online customer surveys as a Web evaluation method. The evaluation results indicated that successful use of the ACSI, whether site-by-site or enterprise-wide, depends in large part on strong staff and management support and adequate funding and time for the use of such evaluative methods. In the age of Web-based e-government, a broad commitment to Web evaluation may well be needed. This commitment would help assure that the potential of the Web and other information technologies to improve customer and citizen satisfaction is fully realized.

E-learning in a virtual science camp for urban youth

Objective: An evaluation of Internet end-to-end performance was conducted for the purpose of better understanding the overall performance of Internet pathways typical of those used to access information in National Library of Medicine (NLM) databases and, by extension, other Internet-based biomedical information resources. Design: The evaluation used a three-level test strategy: 1) user testing to collect empirical data on Internet performance as perceived by users when accessing NLM Web-based databases, 2) technical testing to analyze the Internet paths between the NLM and the users desktop computer terminal, and 3) technical testing between the NLM and the World Wide Web (“Web”) server computer at the users institution to help characterize the relative performance of Internet pathways. Measurements: Time to download the front pages of NLM Web sites and conduct standardized searches of NLM databases, data transmission capacity between NLM and remote locations (known as the bulk transfer capacity [BTC], “ping” round-trip time as an indication of the latency of the network pathways, and the network routing of the data transmissions (number and sequencing of hops). Results: Based on 347 user tests spread over 16 locations, the median time per location to download the main NLM home page ranged from 2 to 59 seconds, and 1 to 24 seconds for the other NLM Web sites tested. The median time to conduct standardized searches and get search results ranged from 2 to 14 seconds for PubMed and 4 to 18 seconds for Internet Grateful Med. The overall problem rate was about 1 percent; that is, on the average, users experienced a problem once every 100 test measurements. The user terminal tests at five locations and Web host tests at 13 locations provided profiles of BTC, RTT, and network routing for both dial-up and fixed Internet connections. Conclusion: The evaluation framework provided a profile of typical Internet performance and insights into network performance and time-of-day/day-of-week variability. This profile should serve as a frame of reference to help identify and diagnose connectivity problems and should contribute to the evolving concept of Internet quality of service.

Long-term outcomes of the ¡VIVA! Peer Tutor Project: use of MedlinePlus by former peer tutors and the adults they taught

Interactive Scientific Publishing (ISP) has been developed by the Optical Society of America with support from the National Library of Medicine at NIH. It allows authors to publish papers which are linked to the referenced 2D and 3D original image datasets that can then be viewed and analyzed interactively by the reader. ISP provides the software for authors to organize and publish source data while offering readers the viewing and analysis tools. The ultimate goal of ISP is to improve learning and understanding of the information being presented. The results reported here are intermediate in reaching the ultimate goals. Goals: Implement and evaluate online user access to interactive datasets that complement online scientific papers. Methods: Software development; website evaluation, including web log data monitoring, online user survey. Results: An expert group of respondents found the interactive data component to have positive impacts on understanding and discovery. However a significant minority reported that the software interface, instructions, and/or download presented a barrier to effective use of the interactive data functionality. Conclusions: This exploratory study indicates that interactive datasets offer potential benefits as a complement to published papers, but that further software improvements and more in-depth evaluation are needed to best discern future directions.

Program Management and Policy Issues in Information Outreach

The AIDS Community Information Outreach Program (ACIOP) was created in 1994 to assist the affected community in utilizing electronic HIV/AIDS information resources. Nearly 300 competitive awards have been made to mostly community-based organizations. A formal evaluation was undertaken to determine the performance and impact of the ACIOP. A mixed methods design combined quantitative abstractions and summarization of 47 awardee final reports from 44 organizations, and qualitative telephone interviews with 17 individuals representing 20 projects. Findings revealed that project objectives were mostly met; high-risk populations were reached; low resource organizations were funded; community partnerships were significant; projects built on existing efforts; information resources and training were tailored to local needs; and most projects overcame barriers experienced. Needed modifications centered on: 1) enhancing evaluation capacity at the individual project level and 2) revising project reporting requirements to increase the amount of information available to assess the ACIOP; both have been implemented.

NLM Workshop Marks 20 Years of Community Outreach and Capacity Building in HIV/AIDS

The U.S. National Library of Medicine, working in concert with the Multilateral Initiative on Malaria (MIM), has developed and implemented a unique organizational and technical strategy to connect malaria research sites to the Internet for purposes of facilitating North-South scientific communications and access to electronic information resources on the Web. The model employs microwave and VSAT technologies, and shares bandwidth and costs among participating malaria research sites and their respective research funders in Mali, Kenya, Ghana, Tanzania and other sub-Saharan locations affiliated with MIM. The concept of institutional partnership is an essential element of this information technology capacity building effort, which may find applicability in other developing regions of the world with similar communications and research networking needs and capabilities.