Olga Zlatkin-Troitschanskaia

The international state of research on measurement of competency in higher education

With the Program for International Student Assessment and Trends in International Mathematics and Science Study surveys, competency assessment became an important policy instrument in the school sector; only recently has international competency measurement gained attention in higher education with the Assessment of Higher Education Learning Outcomes (AHELO) feasibility study. As AHELO showed, measurement of competencies in higher education is a complex and multidimensional task, which poses great methodological challenges. These challenges arise out of the high diversity of degree courses, study programs, and institutions of higher education. Nevertheless, we need to address these challenges immediately if we aim to create evidence-based high-quality educational systems. This paper overviews the field of international research on competency measurement in higher education. Our analyses revealed a substantial lack of research in this area. Nonetheless, existing studies and assessment practices in various countries provide orientation on how to model competencies based on curricular and professional requirements, design assessments following the assessment triangle, and validate them comprehensively.

Effects of prior economic education, native language, and gender on economic knowledge of first-year students in higher education. A comparative study between Germany and the USA

The assessment of university students’ economic knowledge has become an increasingly important research area within and across countries. Particularly, the different influences of prior education, native language, and gender as some of the main prerequisites on students’ economic knowledge have been highlighted since long. However, the findings are often only available within countries and focus on students who are at different levels of their studies or graduates. To remedy this research deficit, the goal of our article is to analyze the status of economic knowledge of students at the beginning of their course of studies and compare the effects of prior economic education, gender, and native language between the USA and Germany. Therefore, we used a translated and adapted version of the fourth edition of the Council for Economic Educations Test of Understanding in College Economics (TUCE). The TUCE is an international measuring instrument that has been validated and widely been used in several countries to assess the economic knowledge of students in higher education. Since the curricular structure as well as higher education studies/programs in both countries is quite comparable, no tremendous general between-country effect has been expected. But as our results show, if micro- and macroeconomics are analyzed separately, divergent effects on the students’ economic knowledge were detected showing that prior education has a positive significant effect merely on micro test scores, while gender and native language are purely significant on macro test scores in both countries. In our conclusion, we discuss several reasons that might have caused these different effects.

Modeling and Measuring Competencies in Higher Education

Measuring competencies acquired in higher education has to be regarded as a widely neglected research field. The progress made in empirical research on the school system since the 1990s – for example, through large-scale assessments such as the Trends in International Mathematics and Science Study (TIMSS) and the Programme for International Student Assessment (PISA) and through a massive expansion of instructional research in general – has revealed that nothing comparable exists at the higher education level.

Gender Effects in Assessment of Economic Knowledge and Understanding: Differences Among Undergraduate Business and Economics Students in Germany, Japan, and the United States

Gender effects in large-scale assessments have become an increasingly important research area within and across countries. Yet few studies have linked differences in assessment results of male and female students in higher education to construct-relevant features of the target construct. This paper examines gender effects on students’ economic content knowledge with a focus on construct-relevant explanations. Moreover, we compare gender effects cross-nationally between Germany, Japan, and the United States. To assess economic content knowledge of higher education students, we used translated, adapted, and validated versions of the Test of Understanding in College Economics (TUCE, 4th ed.), an instrument that is commonly used internationally. We found gender effects on test scores in all three countries; effects were larger in Germany and the United States than in Japan. Gender effects were generally more pronounced on the numeracy subscale than on the literacy subscale, that is, male students had a greater edge over female students when items required calculations. In our conclusion, we discuss how numeracy and literacy items may tap different abilities.

Kompetenz und ihre Erfassung – das neue „Theorie-Empirie- Problem“ der empirischen Bildungsforschung?

Die Bildungsforschung in Deutschland erlebt in den letzten Dekaden eine „empirische Wende“, die oft auch als kognitive Wende bezeichnet wird. Dies ist u.a. darauf zuruckzufuhren, dass eine Vielzahl von Theorien und (Mess-)Methoden aus dem Bereich der (kognitiv orientierten) Psychologie zur Beschreibung und Erklarung des Handelns der Bildungsakteure (Lehrende und Lernende) herangezogen wird.

Assessing student learning outcomes in higher education: challenges and international perspectives

Profound structural and programmatic changes in higher education, due in part to globalisation and the harmonisation of educational structures and study models (Lucena et al. 2008; Brennan, Patel, and Tang 2009), have raised questions regarding the quality of and the individual and societal returns on higher education. At the international level, modelling and measuring academic learning outcomes have become increasingly important in recent years (Coates 2014). To this end, there is an urgent need to develop instruments that enable fair and valid assessment of student competencies, the key learning outcomes of higher education (Zlatkin-Troitschanskaia, Shavelson, and Kuhn 2015). Competencies are broadly defined as a combination of cognitive, affectivemotivational, volitional, and social dispositions that form the basis for performance (Shavelson 2013). Competencies are understood as trait dispositions that are relatively stable over time and across situations, while changes are induced by dynamic components. In developmental terms, competencies can increase through learning or decrease through forgetting. Competencies acquired in higher education are assumed to be multidimensional and specific to a field of study or at least to higher education (e.g., general research skills), which distinguishes them from intelligence and general cognitive abilities. National and international qualification frameworks for describing learning outcomes and competencies, such as Lumina’s Degree Qualification Profile (Lumina Foundation 2015), Assessment and Teaching of Twenty-first Century Skills (Binkley et al. 2012), the Framework for Learning and Development Outcomes (CAS 2009) and the European Qualifications Framework (EC 2015), have proven to be not differentiated enough for use in higher education practice. They require additional empirical approaches to further differentiate and operationalise learning outcomes and competencies. To meet those challenges and close the international research gap in this field, more research is being conducted at the national level in several countries (Coates 2014). In Germany, for instance, the Federal Ministry of Education and Research has established a national research programme on Modeling and Measuring Competencies in Higher Education (KoKoHs). The first funding phase (2011–2015) involved 24 collaborative projects comprising approximately 70 individual projects conducted by almost 220 researchers, focusing on modelling and measuring domainspecific and generic competencies in higher education. The 40 competency models and more than 100 measuring instruments developed by the KoKoHs research teams were administered to almost 50,000 students in selected domains at more than 220 higher education institutions throughout Germany to gain well-founded indications

Macroeconomic knowledge of higher education students in Germany and Japan – a multilevel analysis of contextual and personal effects

Recent trends towards harmonising and internationalising business and economics studies in higher education are affecting the structure and content of programmes and courses, and necessitate more transparent and comparable information on students’ economic knowledge and skills. In this study, we examine by linear multilevel regression modelling the current state of macroeconomic knowledge of higher education students in Germany and Japan, while controlling for the effects of key study-related aspects such as study progress and completion of economics courses. We assess macroeconomic knowledge using the internationally established Test of Understanding in College Economics, which has been adapted and validated for use in Germany and Japan. We found a significant positive correlation between students’ level of knowledge and study progress in both countries, as well as varied gender-related and country-specific differences. Implications for assessment practices and future research are discussed.

Modeling and Measuring University Students’ Subject-Specific Competencies in the Domain of Business and Economics – The ILLEV Project

Current research provides very little empirical evidence regarding the influence of academic higher education on the development of professionalism among students. Over the course of the reform of the higher education systems in Europe (Bologna declaration) this issue has become increasingly important.

Empirische Bildungsforschung – ein Überblick aus interdisziplinärer Perspektive

Hatte man Klaus Beck vor dreisig Jahren den Entwicklungsstand der Empirischen Bildungsforschung im Jahr 2011 vorhergesagt, hatte er folgende Fakten wahrscheinlich kaum geglaubt: 1. Die Sektion „Empirische Bildungsforschung“ in der Deutschen Gesellschaft fur Erziehungswissenschaft hat mittlerweile uber 500 Mitglieder. Zur Erinnerung: Die „Arbeitsgruppe Empirische Padagogische Forschung“ als grosere der beiden Kommissionen der Empirischen Bildungsforschung wurde im Jahr 1965 mit 17 Mitgliedern gegrundet. 2. Sowohl die Deutsche Forschungsgemeinschaft als auch das Bundesministerium fur Bildung und Forschung haben in den letzten zehn Jahren gros angelegte Forderprogramme aufgelegt. Diese Fordermasnahmen zielen zum einen darauf ab, konkrete Forschungsprojekte zu unterstutzen, um mehr empirisch begrundetes Wissen zu Bildungsthemen zu generieren. Zum anderen sollen diese Masnahmen dazu beitragen, die strukturellen Rahmenbedingungen fur die Empirische Bildungsforschung zu starken (Deutsche Forschungsgemeinschaft 2005), beispielsweise durch die Einrichtung neuer Professuren in der Forderinitiative „Empirische Bildungsforschung“ der DFG oder durch die systematische Forderung des wissenschaftlichen Nachwuchses. Insgesamt ist im Bereich der Empirischen Bildungsforschung seit der Jahrtausendwende ein deutlicher Zuwachs an Forschungsprojekten zu bemerken (Zedler & Dobert 2010). 3. Mittlerweile bestehen an zahlreichen Universitaten Professuren, deren Lehr- und Forschungsgebiet der Empirischen Bildungsforschung zuzurechnen ist. Dem „Datenreport Erziehungswissenschaft“ von 2008 zufolge wurden allein in den Jahren von 2003 bis 2006 an deutschen Universitaten 34 Professuren im Bereich der Empirischen Bildungsforschung ausgeschrieben und besetzt (Kruger, Schnorr & Weishaupt 2008). Summiert mit den zuvor bereits bestehenden und danach ausgeschriebenen Professuren kann man die Anzahl auf ca. 100 schatzen. 4. Die Empirische Bildungsforschung erhalt in den Medien und in der bildungspolitischen Diskussion grose Aufmerksamkeit. Besonders hoch ist die Resonanz bei der Veroffentlichung der Ergebnisse der internationalen und nationalen Leistungsvergleiche PISA und IGLU (z.B. Baumert et al. 2003; Bos et al. 2008; Klieme et al. 2010; Prenzel et al. 2008). Aber auch Studien zu anderen Themen, beispielsweise zur Leistungsfahigkeit der sechsjahrigen Grundschule oder der fruhkindlichen Bildung werden von einer breiten Offentlichkeit wahrgenommen und diskutiert.

Assessing the previous economic knowledge of beginning students in Germany: Implications for teaching economics in basic courses

Study-related prior knowledge plays a decisive role in business and economics degree courses. Prior knowledge has a significant influence on knowledge acquisition in higher education, and teachers need information on it to plan their introductory courses accordingly. Very few studies have been conducted of first-year students’ prior economic knowledge or differences among students in such knowledge. In this article, the prior economic knowledge and the influence of personal factors on first-year students’ prior economic knowledge are examined. For this purpose, an adaptation of the fourth, revised edition of the American Test of Economic Literacy was administered, which was adapted in 2014 according to the Test and Adaptation Guidelines for use in higher education in Germany. We present findings based from an assessment conducted in higher education in Germany in the summer term of 2014; the subsample for this study comprises first-year students of business and economics. Findings indicate that there are significant differences in prior economic knowledge among first-year students within universities. Influence factors include economic learning experiences prior to starting university, gender, and mother tongue. The article closes with implications for teaching and degree program design.