Multiple choice homework as a cost-effective and efficient tool for student self-training
MMultiple choice homework as a cost-effective andefficient tool for student self-training
G.L. Lippi
Universit´e Cˆote d’Azur, Institut de Physique de Nice, UMR 7010 CNRS1361 Route des Lucioles, F-06560 Valbonne, FranceE-mail: [email protected]
Abstract.
A self-training scheme geared at inducing students to improve theirskills through independent homework is presented. The motivation is to identifyan inexpensive, yet effective tool for raising the competence level of students inthe Fundamental Sciences (in particular Physics). Since globally existing financialrestrictions do not allow for extensive supervised work, a scheme is devised wherethe additional personal training is rewarded through bonuses in the grade, whilesafeguarding against the danger of cheating. Overburdening the instructors isavoided through the use of computer-based grading of homework, while a carefullychosen bonus plan, weighted by the grades obtained in supervised tests, countersthe effects of potential cheating.
Submitted to:
Eur. J. Phys. a r X i v : . [ phy s i c s . e d - ph ] S e p tudent self-training with multiple choice tests
1. Introduction
As French Universities are not allowed to preselect students ‡ through entrance examsor through the assessment of previous performance (e.g., high-school results) theaverage level of classes is often rather poor. The prohibition of any form of selectionbecomes an additional reason for the better students to run away from fundamentalsciences in favour of applied ones (engineering in this case) where admission is based onmerit and where the teaching resources and personalized attention are more generous.A paradoxical situation arises, where the students who would need more follow-up– due to their poorer initial background competence – are relegated to curricula whichdo not receive sufficient resources to meet those needs. Students spend a nonnegligibletime in problem sessions, which should be expected to raise their competence level, butas the class size is at least 30 – and often approaches 40 –, any form of the direly neededpersonalized help is impossible. Nearly no provisions are made for closer follow-up,for instance by hiring Teaching Assistants, while the permanent personnel is assignedto large student groups.The aim of this paper is to discuss a training technique devised to palliate thisproblem through a scheme which has proven effective with Physics students in theunderprivileged environment outlined above. The following discussion is thereforerelevant to addressing the issue of how to raise the level of students who – in average– start their university studies with an insufficient background (in particular inmathematics) with the foreseeable consequences on the acquisition of the materialas the curriculum progresses. As such, the ideas I present will be most interesting tocolleagues teaching in institutions where either no selection on merit is performed orwhich receive their students from the less favoured milieus.One of the central points of the discussion is to seek a way of motivating thestudents and of providing easily accessible material for individual, independent (athome) training. Due to lack of personnel/hour which can be dedicated to followingthe students’ work, the aim is to find a way of efficiently improving the performancewhile keeping the time invested by the teaching personnel reasonably low. Indeed,this time is not explicitely accounted for in the instructor’s duties and comes frompersonal free time or from research time.It is important to remark that due to this overall setting, a nonnegligible fractionof the enrolled students fails courses (at least at the first try), thus it is common tosee class averages which are well below average. While the consequent re-enrolmentfor the same course amounts to a waste of time (and a reduction in motivation forsome students), the University fees are only nominal and do not place an additionalfinancial burden on the student § . Nonetheless, the situation is highly unsatisfactoryand, in spite of new measures proposed every year, the problem persists. The trainingthrough the use of multiple choice tests appears to bring a small positive contributionto improving students’ competence, as explained in the following. ‡ Professional curricula, such as medicine, engineering, etc. are exempt from this and accept only alimited number of students each year on the basis of merit. § The costs of tuition amount to approximately a couple hundred euros per year – including somebenefits such as access to sport facilities, reductions in transportation costs, health, etc. For thisreason Universities have to base their offer on funding provided almost exclusively by the governmentand cannot afford to offer teaching more adapted to the students’ needs. Discussions about raising thetuitions in special cases are under way and encounter strong resistance. However, this is a politicalissue that does not enter into the topic of this paper, aimed at finding means of improving the currentsituation. tudent self-training with multiple choice tests (cid:107) .The Student Self-Training (SST) scheme discussed below aims at improving thetechnical skills necessary for the quantitative analysis of the concepts acquired in thecourse of the Physics curriculum. It does not represent a pedagogical technique foreffectively presenting and acquiring concepts and as such is not in competition withthe techniques (e.g., inverted classroom) that are being explored and tested for thatpurpose. For the moment SST has been developed and used for the improvement ofstudents’ mathematical skills, but it is envisageable to extend it to complement thestandard, weekly class homework directly pertaining to the topic of the course.
2. Self-training scheme
The idea of the SST scheme is to motivate students to perform additional work athome to improve their mathematical skills, which are normally the first and foremost– though not unique – stumbling block in their path towards the successful acquisitionof expertise in Physics. ¶ . This homework, geared at improving mathematicalskills through exercises specifically matching the kinds of calculations appearing incourse–related problems, is added to the required one, to laboratory work, to exampreparation, etc. Thus, it is necessary to provide incentives which may induce thestudents to accept the challenge. The unsupervised nature of the work performed athome limits the way credit can be assigned for this additional effort because of thepotential for cheating, which has to be dealt with in a creative way.The homework consists of a set of problems (same for all students) placed ona web page freely available. The interested students are required to download thetext, print the answer matrix, fill-in the correct answers and return the papers to theinstructor before the mid-term exam (at the latest just before the beginning of thetest).The conditions under which the SST is conducted are the following: • The homework is unsupervised and the student is allowed to consult books,references and web resources in order to learn; • Discussion with fellow students are acceptable, but simply copying the result fromsomeone else (or from an available solution) is not; • No enforcement of rules is reasonably possible, thus it is impossible to detectcheating, if present.The last point represents the potentially weak link in the scheme. However,cheating is rendered useless by the way the incentives for students are offered: (cid:107)
University-level French grades are given over 20 points, where 10 represents the passing grade. ¶ Additional support material is provided to the students in the form of “reminder” of techniquesand fundamentals which are considered as prerequisite – and which should have been acquired priorto enrolling in the course. These materials include, in addition to specific writeups, web resourceswhich the student can (and hopefully will) consult independently. tudent self-training with multiple choice tests • While the homework is voluntary, in addition to the obvious advantage comingfrom additional training, students derive a direct benefit from their efforts fromthe attribution of ( conditional ) bonus-points, as described below; • The points acquired in the homework are added to the mid-term grade accordingto a weighting function C ( G m ) (sigmoid): F m = G m + C ( G m ) × G h , (1)where F m stands for the final mid-term grade (comprehensive of the bonus), G m for the grade obtained in the written mid-term test, C ( G m ) is the weight and G h is the grade obtained in the homework.The sigmoidal function C ( G m ) used in this scheme is C ( G m ) = 1 + tanh( αG m )2 (2)where α is a coefficient controlling the slope of the hyperbolic tangent + (cf.Fig. 1); • As clear from the weight coefficient C , students who gain 50% of the points in themid-term ( G m = 0 .
5) cumulate 50% of the homework points ( C (0 .
5) = 0 . G m = 0 .
25 (where practically no homeworkcredit is accumulated) and G m = 0 .
75 (where most of the homework credit iscumulated) encourages students to strongly improve their performance. Theobjective of the exercises is exactly to achieve this goal. • Students who may underperform in the mid-term, in spite of homework training,will anyway benefit from the practice on the longer term since it will help themin the final exam (as witnessed in the real setting). • Students who may have cheated in the homework by copying the answers do notdraw any benefits from the grade G h they have obtained, since it is extremelyunlikely that they will be able to perform well in the exam (leading to C ≈
3. Homework preparation and time investment
The homework set is constructed on the basis of a program (Auto-Multiple-Choice,AMC) designed for tests based on multiple choice questions and briefly describedbelow. The reason for this choice is the automatization of the grading of this additionalhomework which transforms an impossibly lengthy task for instructors, with theconstraints outlined above (lack of personnel), into a reasonable one. Given that,as previously discussed, the homework set can be reused from one year to another, thedevelopment of the set of questions becomes an investment worth the required time.AMC is not the only suitable program, but it is the one used for these tests and theone for which some useful information can be given here. The program offers, amongmany other options which are not relevant for this discussion, automatic grading of the + Depending on the choice of value for α – cf. caption of Fig. 1 – the accessible range of weightsdoes not cover the entire interval [0 , C ( G m = 0) and C ( G m = 1) are close enough to 0 and 1, respectively, to render the error negligibly small. tudent self-training with multiple choice tests Figure 1.
Weight coefficient C ( G m ) for cumulating the benefit of the homeworkgrade, G h , onto the mid-term grade G m . The full credit ( G h,max ) assigned to G h is a free parameter which can be adjusted as a function of the quantity ofexercises given, their difficulty, the amount of time allotted and the maximumbenefit to be given to the students. In the examples which follow G h,max = 0 . C ( G m ) is plotted, and used, with α = 5. assignment from a scanned image of the matrix of answers returned by the student,with the boxes corresponding to the correct results blackened out in pen ink.The AMC programme is based on a graphical interface which uses LaTeX asunderlying language, but with specific commands and functions specifically developedfor the purpose. No knowledge of LaTeX is required, as AMC is provided also with itsown set of commands (the user chooses between those or LaTeX commands). AMC isnot specifically developed for the sciences, thus it is quite versatile and offers a largepalette of options. As such, it is not always easy to use and the fairly extensive manualrequires, on occasion, some effort to interpret the meaning of specific functionalities.AMC is free and can be installed on Windows-based systems, as well as on most Linuxreleases. For more information, the potentially interested user should look at the AMCweb page [1].The AMC option chosen to prepare the homework is the one where multiplecorrect answers are allowed, which automatically implies the possible absence of anycorrect answer among the proposed set. Other existing options, such as answerscrambling or randomization, are not used here.In the implementation I have set up for self-training each question proposes acertain number of answers, with a minimum of at least 6 (more if possible) to renderguessing too unlikely to be useful. An important, and time-consuming, part of thework is preparing incorrect answers in sufficiently large numbers for each question. tudent self-training with multiple choice tests Table 1.
Number of students, N e , enrolled in the Waves class (3 rd semester) inthe last four academic years. For the 2017-18 class, N is the number of studentshanding in the homework, while N = N e − N stands for the students who havechosen not to do so.Year N e N N Typically, I offer at least two groups of answers: one containing the correct answer(s),with the addition of a few variations consisting of small changes (a coefficient, forinstance) for the purpose of testing the student’s ability to perform the full calculationcorrectly, down to the final detail; at least a second group of answers (more ifpossible) based on common mistakes (even gross ones) with, again, some variations.Occasionally, one question is proposed without any correct answer to keep the studentfrom assuming that there must be a right one and therefore trying to “bend” thecalculation to try and find it. This choice is made to reinforce the student’s confidencein personal work, rather than in external suggestions.The AMC option of attributing negative points to incorrect answers is used todiscourage the blackening of random boxes in the hope of hitting on the correct answer.Since the number of points (both positive and negative) is attributed to each individualanswer by the instructor at the time of generating the code, it is possible to sanctionentirely wrong answers more strongly than answers which are a slight variation of thecorrect one (e.g., due to an incorrect coefficient but correct functional dependence).A very important point, in light of a possible interest in using the technique inother settings, is an estimate of the time to be invested into the project to set it upthe first time and to run it. This estimate is not an easy task, as the details dependcrucially on the available facilities (scanner speed and quality, connection, computerspeed and memory, etc.), but is offered in the hope of providing at least an idea of therequired effort and help potentially interested colleagues in making their choice.First, one needs to invest into learning how to use AMC. A good mastery of theprogram can be acquired by someone without previous experience in about a week; lessfor users familiar with LaTeX and intending to just learn the basics for this project.However, this is a one-off investment which can also be rendered profitable through itsuse in other settings (ex. preparation of true in-class tests, even in randomized form).The preparation of questions and of answers (mostly wrong ones) is somethingthat only each instructor can evaluate, since it depends on personal factors and onclass topics. This time would be nearly the same if the homework were prepared forstandard (i.e., by hand) correction – up to the preparation of the wrong answers.The typesetting, including debugging which again depends on the preparer’s skillswith the program, may take approximately one day for 30 questions. This part is alsoa one-time investment, since one can later reuse the same homework for the years tocome. Placing the questions on a stable web site ensures the durability of the postingwhile the time required for this operation can be considered negligible, compared tothe rest.The recurrent time investment comes from collecting the homework, scanning the tudent self-training with multiple choice tests Table 2.
Number of students, N e , enrolled in the Waves class (3 rd semester).For the 2017-18 class, the first line contains the information for all students, thesecond for those of group 1 (i.e., those having handed in their homework), whilethe last one corresponds to group 2 (no homework). The second column reportsthe normalized grade for the level test administered at the beginning of the course,the third the final normalized grade, and the last the progression.Group G lt W f W f − G lt All 0.33 0.36 +0.031 0.39 0.52 +0.132 0.31 0.29 -0.02 sheets, grading and returning the results to students (preferably by email, since thecorrection is electronic). From experience, one can expect to scan, grade and return thefiles by email in approximately 30 minutes for every 15 students without using the mailserver which AMC offers (useful mostly when large number of students are involved,as in introductory courses). This is an estimate based on personal experience anddepends on the actual material (scanner, computer, etc.), but it conveys the order ofmagnitude of the required time. Compared to the benefits, the investment is profitableand sustainable.
4. Analysis
The SST scheme was implemented with the class 2017-18 of the Waves course, heldby the author for 3 rd semester Physics students at the Universit´e de Cˆote d’Azur. Ofthe 68 students enrolled in the class, only 21 chose to return the homework.In Table 1, the global class results are compared, to test for variations in theaverage grade, to those of three previous years (from 2014-15 to 2016-17) when noform of additional optional homework was offered. Details on number of students aregiven in Table 1.Fig. 2 graphically shows similar information: the class average for the differentyears, accompanied by its standard deviation. The results show that the variationsfrom one year to the next are rather small, thus we can exclude a bias coming froma better-prepared cohort of students, or from a different degree of difficulty in tests.Indeed, the overall performance of the 2017-18 class was even somewhat poorer.The third column in Table 2 gives the normalized average grade for all students(first row), for the students who returned the homework (group 1, second row) and forthose who did not do the homework (group 2, third row). The same data is graphicallyreported, with error bars, in Fig. 3.In 2017-18, in addition to the SST option, a 30-minutes-long multiple-choice testwas administered to probe the mathematical competence on the needed basic concepts;its scope was to allow for a quantification of the students’ progress.Students who later chose to do the homework scored sensibly higher (Fig. 3 –values from Table 2) in this initial test ( G lt = 0 .
39) than their remaining colleagues( G lt = 0 . .
13 points, while the other one lost 0 . tudent self-training with multiple choice tests Figure 2.
Final grade W f with standard deviation for the students enrolled inthe course for the academic years indicated in the abscissa (only the enrollmentyear is marked, for simplicity). Notice that the average grade fluctuations fromone cohort to the next are rather small and that, overall, the last group performedcomparably to (even somewhat less well than) the previous years. It is important to remark that the average contribution of the homework gradedirectly onto the final grade (i.e., the average of term C ( G m ) × G h in equation 1) is 0 . .
1) can be reasonably interpretedas representing the average student progression coming from the additional training.Since this amounts to 10% of the maximum grade, the progress is substantial.
5. Conclusions
While in an ideal setting one would have students arriving at University with asufficiently strong background and skills to independently fill the (hopefully small)gaps which may possibly remain in their previous education, the current situationproves the contrary. Even in the presence of a web environment which provides a broadself-training offer, it has become necessary to convince and guide students to improvetheir technical knowledge, in order to complete a degree in science. The currentglobal economic situation renders this potentially very expensive action unfeasible,thus alternative schemes for self-training must be sought.The challenge is to motivate students in doing what has always been known tobe the necessary step for progress (exercise) without incurring into potential cheatingproblems or into very high costs, which cannot be met by the Institutions. The SST tudent self-training with multiple choice tests Figure 3.
Level test grade ( G lt , left) and final grade ( W f , right) displayed withits standard deviation for: N e – all students; N – students not having returnedthe homework; N student who returned the homework. The dashed vertical linesseparate the grades by group, as indicated in the labels on the abscissa. scheme proposed in this paper, based on computer-graded multiple choice answers,fulfills the needs of both sides: a low-cost investment for instructors (at zero-cost forthe Universities) and an effective, cheat-proof progression tool designed to encouragestudents to engage in the process.It would be very interesting to extend this experiment to different Institutionsbelonging to different countries and including various specialties. Colleagues interestedin such ideas are welcome to contact the author for discussions and for potential futurecollaborations. Acknowledgments