Marietjie Potgieter

Undergraduate students’ performance and confidence in procedural and conceptual mathematics

The general perception is that high school teaching of mathematics in South Africa tends to be fairly procedural and that students that enter university are better equipped to deal with procedural problems rather than conceptual. This study compares the conceptual and procedural skills of first-year calculus students in life sciences. Also investigated is students’ confidence in handling conceptual and procedural problems. The study seems to indicate that these students do not perform better in procedural problems than in conceptual problems. They are more confident of their ability to handle conceptual problems than to handle procedural problems. Furthermore the study seems to indicate that students do not have more misconceptions about conceptual mathematics than about procedural issues.

Inaccuracy of self-evaluation as additional variable for prediction of students at risk of failing first-year chemistry

Early identification of students at risk of failing first-year chemistry allows timely intervention. Cognitive factors alone are insufficient predictors for success; however, non-cognitive factors are usually difficult to measure. We have explored the use of demographic and performance variables, as well as the accuracy of self-evaluation as an indicator of metacognitive ability, as possible indicators for students at risk of failing the first semester course in General Chemistry (CMY 117) at the University of Pretoria. Variables with a strong correlation with performance in CMY 117 were used to develop a prediction model based on logistic regression. Three variables, i.e. prior performance in mathematics and in physical science, and the extent of overconfidence expressed as the ratio between expected and actual performance in a chemistry pre-test written at the start of the semester, were shown to be significant predictors for risk of failing. The highest overall accuracy of prediction (76%) was obtained for a subset of students with a C or D grade for their high school leaving examination in mathematics when high risk students were defined as those with a final mark for CMY 117 as 51% or lower. The prediction model, based on the model building data set, had a sensitivity of 92% and a specificity of 46%; whilst the sensitivity and specificity using the validation data set were 88% and 38% respectively.

Assessment of preparedness of first-year chemistry students: development and application of an instrument for diagnostic and placement purposes

Abstract Many universities in South Africa use alternative admissions tests together with results of the Grade 12 examinations for access or placement. These tests focus on academic literacy and mathematical skills and do not provide information about proficiencies in disciplines other than mathematics. The implementation of a new curriculum for Grades 10–12 in South Africa generates a need to monitor preparedness of first-year students to align first-year curricula as closely as possible to the content of the new syllabi. The need for a tailor-made instrument to assess preparedness for chemistry at the tertiary level was prompted by the lack of suitable tests with the appropriate focus, depth and coverage for application in the South African context. This paper describes the development and evaluation of a test instrument designed to assess and monitor baseline conceptual understanding in chemistry at the secondary-tertiary interface. The study was carried out with mainstream students at the universities of Pretoria and Cape Town (N = 513 and 258, respectively). Data analysed using the Rasch statistical method confirmed that the majority of students from both cohorts was reasonably well prepared for chemistry at the tertiary level. The use of the instrument for diagnostic purposes was demonstrated. It was possible to identify gaps in the assumed pre-knowledge of students who have qualified for admission to chemistry at a tertiary level at their respective institutions. Inadequate pre-knowledge was most noticeable in the areas of chemical reactions and electrochemistry. In addition, the instrument also shows promise as a placement tool within programmes at a particular institution, i.e. in either mainstream or academic development programmes. Its predictive ability compares well with non-South African placement tests and with Grade 12 mathematics performance, the latter is widely used for this purpose in South Africa.

Preparedness for tertiary chemistry: multiple applications of the chemistry competence test for diagnostic and prediction purposes

The development of the Chemistry Competence Test was prompted by the extensive curriculum changes in the South African school system after democracy was established in 1994. As chemists, we were concerned that there might be a lack of articulation between secondary and tertiary levels, since we anticipated that curriculum changes would have an impact on the knowledge base and skills development of prospective students. A diagnostic test developed initially to document proficiencies of first-time entering students to South African universities has proved to be a versatile instrument for multiple uses. Apart from monitoring levels of preparedness for tertiary chemistry during a period of systemic change, it has also been used to evaluate institutional placement policies, to identify specific conceptual problems and procedural deficiencies, and to measure conceptual gains over the course of the first year at university. In addition, its application for the prediction of risk of failure in first-year chemistry based on cognitive and non-cognitive variables was demonstrated. All these findings are valuable resources to inform lecturers who are concerned about minimizing the conceptual gap between secondary and tertiary chemistry.

Confidence versus performance as an indicator of the presence of alternative conceptions and inadequate problem-solving skills in mechanics

This study investigated the use of performance–confidence relationships to signal the presence of alternative conceptions and inadequate problem‐solving skills in mechanics. A group of 33 students entering physics at a South African university participated in the project. The test instrument consisted of 20 items derived from existing standardised tests from literature, each of which was followed by a self‐reported measure of confidence of students in the correctness of their answers. Data collected for this study included students’ responses to multiple‐choice questions and open‐ended explanations for their chosen answers. Fixed response physics and confidence data were logarithmically transformed according to the Rasch model to linear measures of performance and confidence. The free response explanations were carefully analysed for accuracy of conceptual understanding. Comparison of these results with raw score data and transformed measures of performance and confidence allowed a re‐evaluation of the model developed by Hasan, Bagayoko, and Kelley in 1999 for the detection of alternative conceptions in mechanics. Application of this model to raw score data leads to inaccurate conclusions. However, application of the Hasan hypothesis to transformed measures of performance and confidence resulted in the accurate identification of items plagued by alternative conceptions. This approach also holds promise for the differentiation between over‐confidence due to alternative conceptions or due to inadequate problem‐solving skills. It could become a valuable tool for instructional design in mechanics.

Metacognitive monitoring and learning gain in foundation chemistry

The ability to make realistic judgements of ones performance is a demonstration of the possession of strong metacognitive skills. In this study we investigate the relationship between accuracy of self-evaluation as an expression of metacognitive skill, and learning gain in stoichiometry. The context is an academic development programme at a South African University offered for under-prepared students enrolled for science and engineering. These students generally exhibit unrealistically high levels of confidence in performance and this could potentially place them at risk by negatively affecting decisions regarding time management and self-regulation. We investigated whether overconfidence before instruction is corrected upon exposure to teaching. A three-tier stoichiometry test was used to collect qualitative and quantitative data before and after instruction. Findings indicate that the majority of the students were overconfident in the evaluation of their performance in both the pre- and posttests. Overconfidence was not a debilitating disposition when demonstrated in the pretest provided that it was corrected during teaching and learning. The most vulnerable students were those that judged their performance or lack thereof realistically in the pretest but became overconfident during the teaching and learning of stoichiometry. Our results suggest that under-prepared students are slow in developing accurate metacognitive monitoring skills within a classroom environment that did not include instruction focused on the development of such skills. We recommend a proactive and constructive response by educators which may reduce the incidence of failure and preserve the positive contribution of confidence, albeit excessively positive.

Chemical concepts inventory of grade 12 learners and UP foundation year students

Abstract Incoming chemistry students at tertiary institutions have a variety of academic backgrounds that influence their prospects of success at first-year level. The proficiencies of incoming students are currently changing due to the introduction of outcomes-based education and new syllabi for physical science in secondary schools. In order to ensure a smooth transition from secondary to tertiary education, university lecturers should be well informed about the content knowledge, conceptual understanding and skills development of prospective first-year students. This study evaluates the proficiencies of Grade 12 learners in physical science in terms of a number of clearly identified problem areas: conceptual understanding, logical scientific reasoning, basic mathematical ability, knowledge of subject content and scientific process skills. A test instrument was developed that consisted mainly of conceptual questions rather than recall or algorithmic items. Paired questions (two-tier methodology) and pictorial representations were used extensively. A follow-up question about certainty of response was included for all fixed-response items in order to evaluate the influence of guessing on response distributions. The test instrument was administered at the end of the third term to Grade 12 learners taking Physical science at three schools in privileged environments (1 English and 2 Afrikaans medium) and four township schools, and to all University of Pretoria Foundation Year (UPFY) students. Analysis of results highlighted the generally poor performance of students from township schools and the significant improvement in performance after one year of intensive instruction of UPFY students, who generally came from similar or more impoverished backgrounds. The poor performance for all cohorts on basic concepts, such as the mole concept, stoichiometry and the limiting reagent, as well as on several special topics, indicates that students lack a sound basis for tertiary chemistry. Of real concern is the evidence of over-confidence obtained from the certainty of response analyses. This result indicates that respondents failed to judge the complexity and level of difficulty of questions accurately. Learners/students from all groups displayed weak understanding of events at molecular level. In order to address this situation, lecturers at tertiary level will have to actively promote conceptual understanding of all basic concepts in chemistry and resist the temptation to teach and assess mainly procedural fluency. Analysis of certainty of response data showed that the guess factor was less serious a complication than anticipated.

Use of the Rasch measurement model to explore the relationship between content knowledge and topic-specific pedagogical content knowledge for organic chemistry

ABSTRACT Research has shown that a high level of content knowledge (CK) is necessary but not sufficient to develop the special knowledge base of expert teachers known as pedagogical content knowledge (PCK). This study contributes towards research to quantify the relationship between CK and PCK in science. In order to determine the proportion of the variance in PCK accounted for by the variance in CK, instruments are required which are valid and reliable as well as being unidimensional to measure person abilities for CK and PCK. An instrument consisting of two paper-and-pencil tests was designed to assess Grade 12 teachers CK and PCK in organic chemistry. We used the Rasch measurement model to convert raw score data into interval measures and to provide empirical evidence for the validity, reliability and unidimensionality of the tests. The correlation between CK and PCK was estimated as r = .66 (p < .001). We found evidence to suggest that while topic-specific PCK (TSPCK) develops with increasing teaching experience, high levels of CK can be acquired with limited teaching experience. These findings support the hypothesis that CK is a requirement for the development of TSPCK; proficiency in CK is, however, not necessarily associated with high levels of TSPCK.

Evaluating the Success of a Science Academic Development Programme at a Research-intensive University

Academic development (AD) programmes for students not complying with the entrance requirements of mainstream programmes in science have been running at a number of universities in South Africa. In this study we contribute to the debate on criteria for the success of AD programmes, specifically in the context of research-intensive universities in South Africa. We identify five quantitative criteria for evaluating such a programme: retention, completion rate, migration to other faculties, comparison with other institutions and enrolment in graduate studies. Most of these criteria are relatively common in evaluating similar programmes whereas the extent of migration to other faculties is particular to this study. We illustrate the use of these criteria evaluating the BSc Four Year Programme at the University of Pretoria based on the group of students who registered for the programme in 2008. We found that around 15% of this group had completed their studies in the minimum time while the eventual completion rate of students getting degrees is expected to be in the order of 50%. The attrition rate across the four year science degree programme was high (43%), but the majority of students who terminated their studies did so during the first year. Of those still enrolled in their fourth year a high percentage of students remained in science-related programmes (79%). The fact that more than half of first graduates had enrolled for postgraduate programmes in 2012 is a criterion for success that is especially important for a research-intensive university.

The Usefulness of the Rasch Model for the Refinement of Likert Scale Questionnaires

In this paper the use of the Rasch model is explored as a transparent, systematic and theoretically underpinned response to quality issues that are widely recognised as problematic in the refinement of Likert scale questionnaires. Key issues are the choice of length of scale, the pursuit of a favourable estimate of Cronbachs alpha at the possible expense of construct validity, and the fact that total raw scores arise from ordinal data but are used and interpreted as if measurement had occurred. We use a questionnaire under development for the measurement of perceptions of first-year chemistry students on demonstrator effectiveness to illustrate the process of Rasch analysis and instrument refinement. This process involves investigation of fit of the data to the model, possible violations of the assumption of local independence, and several aspects of item functioning. We identified disordered response categories as the probable reason for misfit in this data set and propose strategies for modification of items so that they can be retained rather than rejected.