Marjolein Heijne-Penninga

Necessary steps in factor analysis : Enhancing validation studies of educational instruments. The PHEEM applied to clerks as an example

Background: The validation of educational instruments, in particular the employment of factor analysis, can be improved in many instances. Aims: To demonstrate the superiority of a sophisticated method of factor analysis, implying an integration of recommendations described in the factor analysis literature, over often employed limited applications of factor analysis. We demonstrate the essential steps, focusing on the Postgraduate Hospital Educational Environment Measure (PHEEM). Method: The PHEEM was completed by 279 clerks. We performed Principal Component Analysis (PCA) with varimax rotation. A combination of three psychometric criteria was applied: scree plot, eigenvalues >1.5 and a minimum percentage of additionally explained variance of approximately 5%. Furthermore, four interpretability criteria were used. Confirmatory factor analysis was performed to verify the original scale structure. Results: Our method yielded three interpretable and practically useful dimensions: learning content and coaching, beneficial affective climate and external regulation. Additionally, combining several criteria reduced the risk of overfactoring and underfactoring. Furthermore, the resulting dimensions corresponded with three learning functions essential to high-quality learning, thus strengthening our findings. Confirmatory factor analysis disproved the original scale structure. Conclusions: Our sophisticated approach yielded several advantages over methods applied in previous validation studies. Therefore, we recommend this method in validation studies to achieve best practice.

Assessment of professional behaviour in undergraduate medical education: peer assessment enhances performance

Objectivesu2002 To examine whether peer assessment can enhance scores on professional behaviour, with the expectation that students who assess peers score more highly on professional behaviour than students who do not assess peers.

Influence of open- and closed-book tests on medical students' learning approaches

Contextu2002 Two learning approaches are consistently distinguished in the literature: deep and surface learning. The deep learning approach is considered preferable. Open‐book tests are expected to stimulate deep learning and to offer a possible way of handling the substantial growth in medical knowledge. In this study we test the hypothesis that open‐book tests stimulate deep learning more than closed‐book tests.

Influences of deep learning, need for cognition and preparation time on open- and closed-book test performance

Medical Education 2010: 44: 884–891

Influence of PBL with open-book tests on knowledge retention measured with progress tests

The influence of problem-based learning (PBL) and open-book tests on long-term knowledge retention is unclear and subject of discussion. Hypotheses were that PBL as well as open-book tests positively affect long-term knowledge retention. Four progress test results of fifth and sixth-year medical students (nxa0=xa01,648) of three medical schools were analyzed. Two schools had PBL driven curricula, and the third one had a traditional curriculum (TC). One of the PBL schools (PBLob) used a combination of open-book (assessing backup knowledge) and closed-book tests (assessing core knowledge); the other two schools (TC and PBLcb) only used closed-book tests. The items of the progress tests were divided into core and backup knowledge. T tests (with Bonferroni correction) were used to analyze differences between curricula. PBL students performed significantly better than TC students on core knowledge (average effect size (av ES)xa0=xa00.37–0.74) and PBL students tested with open-book tests scored somewhat higher than PBL students tested without such tests (av ESxa0=xa00.23–0.30). Concerning backup knowledge, no differences were found between the scores of the three curricula. Students of the two PBL curricula showed a substantially better long-term knowledge retention than TC students. PBLob students performed somewhat better on core knowledge than PBLcb students. These outcomes suggest that a problem-based instructional approach in particular can stimulate long-term knowledge retention. Distinguishing knowledge into core and backup knowledge and using open-book tests alongside closed-book tests could enhance long-term core knowledge retention.

Directing students to profound open-book test preparation: The relationship between deep learning and open-book test time

Background: Considering the growing amount of medical knowledge and the focus of medical education on acquiring competences, using open-book tests seems inevitable. A possible disadvantage of these tests is that students underestimate test preparation. Aims: We examined whether students who used a deep learning approach needed less open-book test time, and how students performed on open-book questions asked in a closed-book setting. Method: Second- (Nu2009=u2009491) and third-year students (Nu2009=u2009325) prepared half of the subject matter to be tested closed-book and half to be tested open-book. In agreement with the Board of Examiners, some questions in the closed-book test concerned open-book subject matter, and vice versa. Data were gathered about test time, deep learning and preparation time. Repeated measurement analysis, t-tests and partial correlations were used to analyse the data. Results: We found a negative relationship between deep learning and open-book test time for second-year students. Students scored the lowest on closed-book questions about open-book subject matter. Conclusions: Reduction of the available test time might force students to prepare longer and deeper for open-book tests. Further research is needed to identify variables that influence open-book test time and to determine how restrictive this time should be.

The transferability of Western concepts to other cultures: Validation of the Zuckerman–Kuhlman Personality Questionnaire in a Saudi Arabic context

Abstract Background: The importance of medical education research in Saudi Arabia has been acknowledged increasingly and a lot of concepts used have been derived from the Western world. The question arises, however, whether Western concepts and questionnaires are transferable to societies with different cultures. The aim of this study was to investigate the instrument structure and the reliability of the Arabic version of the Zuckerman–Kuhlman Personality Questionnaire-medium (ZKPQ-m). Methods: Three statistical methods with decreased amount of strictness were used to analyse our data: Confirmatory Factor Analysis, Procrustes rotation and Principal Component Analysis. Results: Our outcomes did not confirm the original instrument structure. Instead, we found four interpretable components: Emotional Instability, Impulse-seeking, Activeness and Self-Control. However, the amount of explained variance was not very high and the internal consistencies ranged from unsatisfactory to only moderate. The data showed a high percentage of respondents agreeing with more than three items of the Infrequency scale, which may be attributable to the collectivistic culture in Saudi Arabia. Conclusion: We did not succeed in replicating the ZKPQ structure in the Arabic context. Social desirability, a common characteristic in collectivistic cultures, may have reduced the replicability of the internal structure of the ZKPQ-m. Different methods to measure concepts in collectivistic cultures may help to get round social desirability.

Open-book tests: Search behaviour, time used and test scores

Background: Because of the increasing medical knowledge and the focus of medical education on acquiring competencies, the use of open-book tests seems inevitable. Dealing with a large body of information, indicating which kind of information is needed to solve a problem, and finding and understanding that knowledge at the right moment are behaviours that cannot be assessed during closed-book tests. Aims: To examine whether there is a relationship between students’ search behaviour – using references or not when answering a question – during open-book tests and their test scores. Method: Second- (Nu2009=u2009491) and third-year medical students (Nu2009=u2009325) participated in this study. Search behaviour was operationalized as the number of questions for which students consulted their references. Furthermore, we collected data on the time students spent on answering all open-book questions and their test scores. To determine the relations, we calculated Spearmans and Pearsons correlations. Results: Second- and third-year students consulted their references for 87% and 73% of the questions and spent 5.0 and 4.3u2009min on answering an open-book question, respectively. We did not find significant correlations between search behaviour and test scores. Conclusion: Both ‘well’ and ‘poorer performing’ students often consulted their references. Spending almost 5u2009min per open-book question in multiple choice format seems to be too much. More research is needed to establish optimal open-book test time and to explore how ‘well performing’ students use their references during open-book tests.