M.F. van der Schaaf

Interpretation of radiological images: towards a framework of knowledge and skills

The knowledge and skills that are required for radiological image interpretation are not well documented, even though medical imaging is gaining importance. This study aims to develop a comprehensive framework of knowledge and skills, required for two-dimensional and multiplanar image interpretation in radiology. A mixed-method study approach was applied. First, a literature search was performed to identify knowledge and skills that are important for image interpretation. Three databases, PubMed, PsycINFO and Embase, were searched for studies using synonyms of image interpretation skills or visual expertise combined with synonyms of radiology. Empirical or review studies concerning knowledge and skills for medical image interpretation were included and relevant knowledge and skill items were extracted. Second, a preliminary framework was built and discussed with nine selective experts in individual semi-structured interviews. The expert team consisted of four radiologists, one radiology resident, two education scientists, one cognitive psychologist and one neuropsychologist. The framework was optimised based on the experts comments. Finally, the framework was applied to empirical data, derived from verbal protocols of ten clerks interpreting two-dimensional and multiplanar radiological images. In consensus meetings adjustments were made to resolve discrepancies of the framework with the verbal protocol data. We designed a framework with three main components of image interpretation: perception, analysis and synthesis. The literature study provided four knowledge and twelve skill items. As a result of the expert interviews, one skill item was added and formulations of existing items were adjusted. The think-aloud experiment showed that all knowledge items and three of the skill items were applied within all three main components of the image interpretation process. The remaining framework items were apparent only within one of the main components. After combining two knowledge items, we finally identified three knowledge items and thirteen skills, essential for image interpretation by trainees. The framework can serve as a guideline for education and assessment of two- and three-dimensional image interpretation. Further validation of the framework in larger study groups with different levels of expertise is needed.

How Visual Search Relates to Visual Diagnostic Performance: A Narrative Systematic Review of Eye-Tracking Research in Radiology.

Eye tracking research has been conducted for decades to gain understanding of visual diagnosis such as in radiology. For educational purposes, it is important to identify visual search patterns that are related to high perceptual performance and to identify effective teaching strategies. This review of eye-tracking literature in the radiology domain aims to identify visual search patterns associated with high perceptual performance. Databases PubMed, EMBASE, ERIC, PsycINFO, Scopus and Web of Science were searched using ‘visual perception’ OR ‘eye tracking’ AND ‘radiology’ and synonyms. Two authors independently screened search results and included eye tracking studies concerning visual skills in radiology published between January 1, 1994 and July 31, 2015. Two authors independently assessed study quality with the Medical Education Research Study Quality Instrument, and extracted study data with respect to design, participant and task characteristics, and variables. A thematic analysis was conducted to extract and arrange study results, and a textual narrative synthesis was applied for data integration and interpretation. The search resulted in 22 relevant full-text articles. Thematic analysis resulted in six themes that informed the relation between visual search and level of expertise: (1) time on task, (2) eye movement characteristics of experts, (3) differences in visual attention, (4) visual search patterns, (5) search patterns in cross sectional stack imaging, and (6) teaching visual search strategies. Expert search was found to be characterized by a global-focal search pattern, which represents an initial global impression, followed by a detailed, focal search-to-find mode. Specific task-related search patterns, like drilling through CT scans and systematic search in chest X-rays, were found to be related to high expert levels. One study investigated teaching of visual search strategies, and did not find a significant effect on perceptual performance. Eye tracking literature in radiology indicates several search patterns are related to high levels of expertise, but teaching novices to search as an expert may not be effective. Experimental research is needed to find out which search strategies can improve image perception in learners.

Developing and Validating a Design for Teacher Portfolio Assessment.

Developing and using a design for teacher portfolio assessment is a complex process including several components: the domain to be assessed (the teacher competences), the content standards or criteria, the portfolio format, the completion of the format (by teachers) with content, and the scoring of the portfolios (by raters). For a portfolio assessment to be valid, these components should be systematically linked. Using a chain model of the assessment process two links were verified: the link between content standards and portfolio format, and the link between content standards and raters’ scoring. Eight experts evaluated an initial portfolio assessment design. They greatly supported the design, including the link between standards and format. Subsequently, using the format 18 teachers developed a portfolio. Six raters scored these portfolios according to eight content standards. Their scoring was substantially based on the content standards. Implications of the results and suggestions for research are discussed.

An argument-based approach to the validation of UHTRUST: can we measure how recent graduates can be trusted with unfamiliar tasks?

There is a need for valid methods to assess the readiness for clinical practice of medical graduates. This study evaluates the validity of Utrecht Hamburg Trainee Responsibility for Unfamiliar Situations Test (UHTRUST), an authentic simulation procedure to assess whether medical trainees are ready to be entrusted with unfamiliar clinical tasks near the highest level of Miller’s pyramid. This assessment, in which candidates were judged by clinicians, nurses and standardized patients, addresses the question: can this trainee be trusted with unfamiliar clinical tasks? The aim of this paper is to provide a validity argument for this assessment procedure. We collected data from various sources during preparation and administration of a UHTRUST-assessment. In total, 60 candidates (30 from the Netherlands and 30 from Germany) participated. To provide a validity argument for the UHTRUST-assessment, we followed Kane’s argument-based approach for validation. All available data were used to design a coherent and plausible argument. Considerable data was collected during the development of the assessment procedure. In addition, a generalizability study was conducted to evaluate the reliability of the scores given by assessors and to determine the proportion of variance accounted by candidates and assessors. It was found that most of Kane’s validity assumptions were defendable with accurate and often parallel lines of backing. UHTRUST can be used to compare the readiness for clinical practice of medical graduates. Further exploration of the procedures for entrustment decisions is recommended.

Secondary-school students’ motivation for portfolio reflection

Several studies concluded that deep reflection is infrequently reached in student portfolios. An explanation for these disappointing conclusions might be that motivation for portfolio reflection determines the quality of reflection. This study aimed to examine the relationship between motivation for using digital portfolios and reflection. Participants were 156 eleventh-grade students in secondary education, whose motivation for composing a digital portfolio was measured by the motivation part of the Motivated Strategies for Learning Questionnaire. Portfolios of 37 of the 156 students were examined in terms of the amount and nature of reflection by means of a coding scheme based on Mezirows model of transformative learning. On average, one-fifth (19.5%) of the paragraphs in a portfolio contained reflection, and paragraphs with deep reflection were hardly found (0.8%). It was concluded that motivation for composing a portfolio was fair, but not related to the amount and nature of reflection. This exploratory study gives rise to further research into factors that might influence the quality of portfolio reflection.

Practical Implementation of Innovative Image Testing

The testing of image interpretation skills within the profession of Radiology (often paper- pencil) lags behind practice. To increase the authenticity of assessment of image interpretation skills, the Dutch national progress test for medical specialists in training to become radiologists, is digitized using the program VQuest. This programme makes it possible to administer a test with 2D and 3D images, in which images can be viewed and processed as they can in practice. During implementation, the entire assessment cycle from test design to assessment analysis and evaluation has been run through twice. Excluding some small improvements, both trainee specialist and organizational members were satisfied with the digitized assessment. Amongst other things, the trainee specialist feel that this application of digital testing is more consistent with the situation in practice than the conventional testing method.