Mark J. Bourne

The development of an integrated performance category system for supervisory jobs in the U.S. Navy

When a single job is the target, established job analysis approaches provide relatively straightforward methods for identifying performance requirements. However, when multiple jobs are involved, the identification of a single set of relevant performance dimensions can be more daunting. In the application here, there was also a compelling requirement to develop a dimension set that conformed with U.S. Navy fleet personnel perceptions of the performance domain. Accordingly, the behavioral performance constructs that experienced Navy officers believed differentiate effective from ineffective supervisory performance were gathered using a personal construct theory protocol. We then used a methodology suggested by Borman and Brush (1993) to integrate and summarize these personal work constructs, to provide—through a series of qualitative and quantitative strategies—a dimension set targeted toward supervisors in all Navy communities. The resulting dimensions revealed certain themes that might not have emerged if traditional job analysis strategies had been used. The dimension set and the resulting behaviorally anchored rating scales appear widely relevant to Navy supervisor jobs, and the performance appraisal system that employs the scales is nearing implementation. In addition, performance feedback and development tools were developed to complement the performance appraisal system.

Multi-Iteration Usability Testing of the U.S. Navy's Performance Management System

The U.S. Navy has undertaken the second of two iterations of usability testing for the Navys Human Performance Feedback and Development (HPFD) and ePerformance system. This second of two iterations included 34 officer and enlisted supervisors and nonsupervisors in usability testing conducted at three Navy locations—Naval Meteorology and Oceanographic Center (NAVMETOCCEN) Norfolk in Norfolk, Virginia; USS HOWARD (DDG 83) in San Diego, California; and the Bureau of Naval Personnel (BUPERS) in Arlington, Virginia. Each participant completed a usability test scenario in addition to pretest and posttest surveys designed to obtain Navy personnels subjective impressions of the HPFD and ePerformance systems. Results from analyses comparing data from Iteration 1 (reported in Schwerin, Dean, Robbins, Bourne, & Reed, 2006) and Iteration 2 (reported in Dean, Aspinwall, Schwerin, & Kendrick, 2006) show an overall reduction in user burden (e.g., fewer errors and less time to complete usability tasks) and increased user satisfaction (e.g., more satisfied with the professionalism, efficiency, and overall effectiveness). These findings indicate that the Navy HPFD and ePerformance systems are easier to use but, more generally, results support the value and effectiveness of usability in human systems integration (HSI) and usability testing. Recommendations for system refinement, policy development, and implementation planning are discussed.

Subjective and Objective Results of Usability Testing of a Prototype Performance Management System Among U.S. Navy Personnel

The U.S. Navy conducted usability testing for the Navy’s Human Performance Feedback and Development (HPFD) and ePerformance system to capture both quantitative and objective data as well as qualitative and subjective data from participants to identify potential sources of error and user burden. A total of 57 officer and enlisted supervisors and nonsupervisors took part in usability testing conducted at 3 Navy test sites. Each participant completed a usability test scenario and took part in focus group interview debriefings of their usability testing session to elicit recommendations for improvement and cultural factors that might affect system implementation. Results identified various usability errors, their frequency, and duration, as well as recommendations for improvement. In addition, user ratings of the system and qualitative data from focus group interviews indicate that system users believe the systems worked well. Focus group interview results indicate that the frequency and severity of connectivity problems had a negative effect on user perceptions of the HPFD and ePerformance systems, and there were several cultural factors that were key for the acceptance of a new performance appraisal system. Implications of the findings, study limitations, and recommendations for future research are discussed.