Timothy John Mavin

Exploring the Use of Categories in the Assessment of Airline Pilots’ Performance as a Potential Source of Examiners’ Disagreement

It is a current trend in aviation to use categories of technical (e.g., knowledge) and nontechnical skills (e.g., situation awareness) to assess airline pilots’ performance. Several studies have revealed large disagreement between assessors when airline professionals use these categories to assess the performance of their peers. The aim of the present study is to investigate whether the categories themselves are at the source of disagreement. We explore the reasoning of flight examiners who assess an engine fire scenario in pairs. The results provide insight into the overlap of topics that constitute certain categories. Implications are drawn in regards to the use of assessment categories and their influence on pilot performance assessment.

Peer Assessment of Aviation Performance: Inconsistent for Good Reasons

Research into expertise is relatively common in cognitive science concerning expertise existing across many domains. However, much less research has examined how experts within the same domain assess the performance of their peer experts. We report the results of a modified think-aloud study conducted with 18 pilots (6 first officers, 6 captains, and 6 flight examiners). Pairs of same-ranked pilots were asked to rate the performance of a captain flying in a critical pre-recorded simulator scenario. Findings reveal (a) considerable variance within performance categories, (b) differences in the process used as evidence in support of a performance rating, (c) different numbers and types of facts (cues) identified, and (d) differences in how specific performance events affect choice of performance category and gravity of performance assessment. Such variance is consistent with low inter-rater reliability. Because raters exhibited good, albeit imprecise, reasons and facts, a fuzzy mathematical model of performance rating was developed. The model provides good agreement with observed variations.

A Holistic View of Cockpit Performance: An Analysis of the Assessment Discourse of Flight Examiners

What pilots do on the job frequently is analyzed in terms of individual skills and human factors. Performances often do not consist of separable skills but of a holistic event, which can be analyzed into irreducible, mutually constitutive moments. A discursive psychology approach was used to analyze the discourse of flight examiners, based on 7 extended interviews about performance aspects. This study shows that in the discourse of flight examiners, cockpit performance is presented holistically, even though it manifests itself in different ways. Six main discourse repertoires are identified in examiners’ discourse about flight deck performance, each of which has between 3 and 5 identifiable subdimensions. Case studies show the connectedness and interdetermination of the 6 main repertoires for talking about what pilots do and how they do it.

How a cockpit forgets speeds (and speed-related events): toward a kinetic description of joint cognitive systems

Two seminal papers investigating the cognition involved in navigating modern, multi-crew aircraft developed a joint cognitive system approach while focusing on how a cockpit calculates and remembers speed (Henriqson et al. in Cogn Technol Work 13:217–231, 2011; Hutchins in Cogn Sci 19(3):265–288, 1995b). Although the joint cognitive systems approach constituted an advance over more traditional approaches focusing on individual pilot performances, both fall short of describing the cockpit joint cognitive system. Based on extensive ethnographic data—including recorded modified think-aloud protocols, debriefing sessions following (recorded) simulator biannual simulator assessments, and simulator performances followed by stimulated recall using simulator recordings—we show that a considerable amount of flying involves bodily and embodied knowledge sequentially and temporally organized in flows (kinetic melodies). These become apparent when pilots are asked to fly multiple aircraft (mixed-fleet flying), where the flow patterns from one cockpit are triggered in a physically and cognitively different cockpit. Focusing on embodied flows (kinetic melodies) allows us to highlight how cockpits forget speeds and how they fail to calculate the required speeds and speed-related events.

The Development of Airline Pilot Skills through Simulated Practice

Pilot education and training has historically been centred on knowledge of aircraft systems and flying skills. Additions to pilot training syllabi over the years have generally occurred due to advances in technology that have required new skills and knowledge to be taught. However, research into aviation accidents over the last 30 years has identified disconnects between current syllabi and pilot needs in the work place. Despite the new perspectives of required pilot skills, traditional habits within training are still embedded in modern training syllabi and, more importantly, the current practice-based methods. Changes are needed in pilot training programmes to improve areas such as team skills, decision making, and communication. These changes are likely to see an increasing use of simulators of varying levels of sophistication to allow authentic practice-based training activities for pilots. Importantly, there is also a need to improve assessment methods for practice-based activities.

The theory-practice gap: epistemology, identity, and education

Purpose – The purpose of this paper is to theorize the theory-practice gap and to provide examples of how it currently expresses itself and how it might be addressed to better integrate between the worlds of thought and praxis. Design/methodology/approach – Two empirical examples exemplify how the theory-practice gap is an institutionally embodied social reality. Cultural-historical activity theory is described as a means for theorizing the inevitable gap. An example from the airline industry shows how the gap may be dealt with in, and integrated into, practice. Findings – Cultural-historical activity theory suggests different forms of consciousness to exist in different activity systems because of the different object/motives in the world in which we think and the practical world in which we live. A brief case study of the efforts of one airline to integrate reflection on practice (i.e. theory) into their on-the-job training shows how the world in which pilots think about what they do is made part of the...

Between reflection on practice and the practice of reflection: a case study from aviation

Reflection on practice continues to gain increasing support, if not a requirement, within vocational and professional fields. As a method of instruction, it can be used to develop increased awareness of individual performance and support lifelong learning. However, whereas research generally focuses on how individuals become proficient practitioners through reflection, it is less concerned, if at all, with how individuals learn to become proficient as practitioners of reflection. This paper turns to an example from aviation to describe a modification of practice that makes reflection an integral part of practice. In our partner airline, all pilots not only engage in reflection on practice for improving practice but also in activities that improve the practice of their reflections. Training pilots to assess video recorded sessions of other pilots – via benchmark training – is viewed as an important step in improving pilots’ ability to review their own performance during their biannual performance assessment.

Models for and Practice of Continuous Professional Development for Airline Pilots: What We Can Learn from One Regional Airline

Most pilots have extensive flying experience prior to joining a new airline. Yet the time taken for these pilots to transfer onto a new aircraft type and be inducted into the airline can take as long as four to 6 months. Even when a pilot completes this training and is assessed as skilled enough to fly as an operational airline pilot, there still remains a continuing education and training program culminating with performance assessment throughout their career. During this recurrent training program numerous modes of instruction are available to an airline. Such training and assessment methods include classroom-based tuition, computer-based training, simulator-oriented flight instruction and real aircraft flight training. Although each mode has both strengths and limitations, there has been a tendency in airline training systems to limit change even when mode limitations have been identified. The aim of this chapter is to describe how one airline and a university-based research team have changed different modes of teaching/learning focusing more on reflective practice. I exemplify the change that arises from the collaboration in the context of two training methods and outline the strengths and weakness of each.

Can Competency-Based Training Fly?: An Overview of Key Issues for "Ab Initio" Pilot Training.

Abstract Competency-based training (CBT) for pilots was formally introduced in 1999 by the Civil Aviation Safety Authority (CASA) for training leading to the issue of aeroplane private and commercial pilot licences. This initiative followed the Australian government’s introduction of CBT policy for vocational and workplace training in the late 1980’s. Since then CBT has been criticised for supporting the teaching and assessment of complex skills by breaking them down into sets of simple skills or sub-routines. This paper argues that in the case of aviation in Australia, codifying flying skills for the purpose of standardising and regulating flying instruction and assessment in early flying lessons has resulted in unintended consequences for pilot training policy and practice. It proposes that while CBT may be used appropriately for initial development of physical flying skills, its application is limited in areas of pilot training which require complex decision-making and critical judgement. The paper considers alternative approaches to pilot training that may be more suitable for teaching and assessing complex flying skills, whilst also addressing the identified limitations inherent in CBT.

Experiencing Resilience via Video Games: A Content Analysis of the PlayStation Blog

The purpose of this study was to investigate the influence of games, where characters must overcome adversity, on player’s perceptions of their psychological resilience. Located on the PlayStation blog (blog.us.playstation.com), the online PlayStation Network (PSN) community group focuses on video gamers unique stories and experiences. Using a qualitative and exploratory design, blogs posted between March 2012 and January 2013 were analyzed for content describing experiences via gameplay that members reported made them feel more resilient. Both social and emotional aspects of resilience were discussed with players reporting game experiences had helped them feel more confident in their abilities. Many also associated themselves with the same resilient traits as their characters display in games. A range of popular off-the-shelf video games were reported as helpful in providing players with the opportunity to feel confident under pressure and, importantly, some players reported transferring these positive psychological effects to their real-world lives.