Don't answer the question! How on-line Moodle-based 'Data Retrieval Tests' encourage good record-keeping and a divergent experimental mindset for undergraduate physics students
DDont answer the question! How on-lineMoodle-based ‘Data Retrieval Tests’ encouragegood record-keeping and a divergentexperimental mindset for undergraduatephysics students
P. A. BartlettMay 31, 2019
Abstract
The use of Data Retrieval Tests (DRTs), as an alternative tophysics laboratory notebook marking, is discussed. The implemen-tation of a Moodle-based, on-line DRT for 1st year physics studentsis described. The advantages of using such a methodology are high-lighted and student comments shown. The paper also describes howstudents change their behaviour as a consequence of having an end ofmodule DRT via ‘bootstrapping’, both singly and in peer groups.
In many higher education courses, particularly Science, Technology, Engi-neering and Mathematics (STEM) ones, students are required to keep lab-oratory notebooks. These notebooks tend to be the core of the laboratoryexperiment or field trip process and a data-recording methodology. Typ-ically, these courses are designed so that students can be trained to keepgood laboratory notebooks that comply with a subject areas best practice.It can be quite difficult to encourage students to comply with this and so itis common for these laboratory notebooks to be assessed by academic staff,laboratory demonstrators or field-trip leaders.In the past, in University College London’s (UCL’s) physics teaching lab-oratories this assessment was based upon formally marking the laboratorynotebook against competency rubrics. However, experience indicated that1 a r X i v : . [ phy s i c s . e d - ph ] M a y tudents seemed only interested in the grades associated with this process.Many students did not read the supporting feedback comments that indi-cated how they could improve their reporting in future. In addition, thismarking process was undertaken in such a way that every page of a labora-tory notebook was formally assessed. This was a laborious task that couldtake many hours for markers to complete, with no guarantee that the mark-ers comments would be heeded by students. In addition, at UCL the classnumbers in the first-year teaching laboratory went from 80 in 2008 to 120in 2013 (180 in 2018). This imposed a significant load on this old markingsystem. Due to these issues, a survey of the literature was undertaken in 2013 to iden-tify if there were any other assessment approaches that could be implementedinstead of the more traditional methods. This resulted in the discovery of apaper [1] that introduced the concept of the Data Retrieval Test. This paperstated: “The ‘Data Retrieval Test (DRT)’ is an open book test in which thenotebook is taken into the exam room and questions are asked on its con-tents. The questions are based on what the tutor knows should have beenincluded. A survey of third year degree students showed that this type oftest is seen as a fair method of assessment. It is also popular with staff as itcan reduce the marking time by up to 80%”.UCL students are trained on how to keep a laboratory notebook andare told that they will undertake an ‘open book’ DRT at the end of themodule. During this test they can only use their laboratory notebook andno pens, pencils, mobile phones or internet resources can be used. At UCL,students are given one practice experiment at the start of their first term atthe university that is supported by training and a guide-document on howlaboratory notebooks should be completed during an experiment.To ensure that students are following this guidance, each of them hasa one-to-one oral assessment of their laboratory notebook twice during theterm. This gives students advice on how they are doing with their recordingpractice. It also helps them to fine tune their recording techniques and givesthem confidence that they will be able to undertake the DRT successfully.It was considered that only having a DRT at the end of term would betoo retrospective on its own and so these mini-orals were put in place togive students in flight corrections to their laboratory practice. In addition,it allowed students to ask questions of experienced practitioners that theymight not ask in the normal conduct of an experiment.2he DRTs initially undertaken at Edge Hill University [1] and UCL werepaper based. Students would fill in the blank fields on question sheets andthese could be marked manually by staff. At UCL the DRT comprised ofmultiple choice, numerical and written answers that would require a completelaboratory notebook record to complete.One thing that was immediately noticeable upon introduction of the DRTin 2013 was how it focused the minds of the students. They knew that theDRT was coming at the end of term and because it represented a significantcontribution to their course marks, they were highly motivated to make surethat they kept a good record that complied with best practice.It was noticed that students would come to the laboratory to work ontheir notebook outside allotted sessions and that they would form peer-groups to improve their practice through mutual support. That had notbeen observed before the implementation of the DRTs. This was a signif-icant improvement as students, rather than just relying on staff to guidethem, were creating communities of interest amongst their peers. The self-motivated self-improvement that the looming presence of the end of termDRT causes can be considered to be the students being made to ‘pull themselves up by their own bootstraps (bootstrapping). This is done individuallyand in these self-help groups.This is behaviour that UCL encourages through its Connected Curricu-lum ethos where it is thought to be possible to bring university research andstudent education into a more connected, more symbiotic relationshipwhereit is a shared framework for thinking about how curriculum is designed, andhow students can become partners in both research and educational devel-opment [2].
However, we at UCL have deviated away from this paper-based DRT as-sessment method in favour of a on-line Moodle-based approach. Moodleis a commonly used open-source learning management system. It containsthe ability to host on-line quizzes that can comprise of a range of multiplechoice, select the right word, numerical and written answers. In addition, itis possible for students to use other packages (Word or Excel, for example)to upload answers as files, into their Moodle quiz submission. Our teachinglaboratories have sufficient computers so that there is one for each studentconducting the on-line DRT. Usually these tests comprise 15 to 20 questionsand the students are given 3hrs to complete it. Examples of a multiple choiceand written style questions are shown In Fig 1 and 2.3igure 1: Example of a Data Retrieval Test multiple choice question tocheck that students are recording important information in their laboratorynotebooksFigure 2: Example of a Data Retrieval Test question that requires a writtenresponse. It tests that students have recorded information in accordance withlocal laboratory best practice.For written questions, students must give the page numbers used fromtheir notebooks, so that markers can cross reference what was written inthe DRT with what is in them. In practice, this is done by comparison viastatistical sampling, but few students seem to offer written answers that donot reflect what they have in their notebooks and, occasionally, apologize fortheir recording deficiencies in their answer submissions (see Fig. 3).In the past it was noted was that the written questions posed in ear-lier, paper-based, iterations of the DRTs could ask students to comment onsomething they didnt or couldnt do in the laboratory. This might be becausethe experiment was not a success (from their point of view) due to equip-ment malfunction or unexpected results or because the students deviated4igure 3: Example of a Data Retrieval Test question that required a writtenanswer from a student. It shows a typical response when a student does nothave the required information recorded in their laboratory notebook. It alsoshows pre-written automated feedback (yellow field) and a comment from amarker (green field).from the laboratory script to investigate another feature of the physics underinvestigation.
In the UCL physics laboratories, we encourage our students to modify whatthey do so they can use their creativity and enthusiasm to create their ownexperiments with the laboratory script just acting as a starting point. Thatmeans that requesting that the students describe the results of an experimentthat they could not do or deviated from would throw them onto turmoil asthey felt they would be penalized in the DRT for not following the safescripted path. To resolve this, we allow the students to NOT answer thequestion that is asked in the written questions. This helps to develop theskills needed to be able to defend their work and decisions to others. Inthis case, those marking the DRT. The instructions the students get on theMoodle Quiz page are as follows:“In some cases, you may find it difficult to answer a written question ifyou experiment did not go as planned. In that case, explain what you did,what you found and what you think about it. You are being assessed onwhat is in your laboratory notebook, so any difficulties should be recordedthere”.Students are told that the DRT is the only time where they can answer aquestion that is not on in the test if they can justify this. This seems to be aunique feature of this assessment system. Our students have the freedom to5hange what they do in the laboratory and have ’permission to fail’ as longas they have recorded what they did well. They know that they can thenjustify NOT answering the question asked if they can justify answering onethat is better suited to what they did and/or experienced.The overarching emphasis is on using the laboratory notebook as a docu-ment of record rather than a repository of predetermined “correct” answers.If they can justify why they are giving a different answer that is not directlyrelated to the question, then they will be assessed on this. This gives consid-erable comfort to students that had experiments that “went wrong” becausethey can then make this the focus of their answer rather than the resultsthat the question may have requested.It must be noted that the Moodle quiz system enables the automaticmarking of everything but the written components of the students submis-sions. In addition, by having written questions that could be on any topic onany of the experiments, it is possible to assess that the laboratory notebookbest practice competencies are achieved without having to mark every partof the notebook itself. Both features of the Moodle-based DRT significantlyreduce marking workload without reducing the testing of students competen-cies. Another advantage is that markers can do the initial marking on-lineand so do not have to transport large numbers of laboratory notebooks tooffices or homes to do this.
The students were asked about what they thought of the DRT process via anend of module questionnaire. Some representative comments are reproducedhere:“DRT itself is useful, as they help students keep a well filled in lab book,preventing them from leaving out information”.“This is a good method which I think really works, and I appreciate thefact that it is dependent almost solely on the lab book and not revision”.This comment is important as it also highlight that this is a test thatrequires no revision.“That (it) probably helps, (because) there are some really detailed ques-tions like what’s the brand of the apparatus or on which page the referenceuse in the book etc”.“(A) good way of testing if you have sufficient detail in your lab book”.“Data Retrieval Test is a great idea! However, the weightage is ratherdisproportionate”.This reflects the high marks that are assigned to this test as part of the6ourse grade. It is kept high (70%) to focus students minds regarding theimportance of laboratory notebooks“I think it is helped realise how much info I need to put in my lab book,thus making me more competent in the process”.“I think it’s quite a good idea - after all, the way the DRT is designed it’sless of a test that you conduct in those three hours and more of one that youconduct across the entire term, and in that respect, I think it has achievedits objective”.“Knowing that the Data Retrieval Test was at the end of term made methink more about writing key details down in my lab book as I know that itcould be in the test’.These comments agree with those that have been obtained through dis-cussions with students. They tend to think it is a good way to encouragestudents to keep a good laboratory notebook, but they do wish that it did notrepresent a significant proportion of the marks. Which seems odd as most oftheir other lecture courses have an end of year exam that would represent alarge proportion of the marks for that module.
In conclusion, it is considered that Moodle-based DRTs are an importantmethod for assessing student notebooks.Some of the benefits of using Moodle-based DRTs are as follows: • Labour intensive laboratory notebook marking can be eliminated. • Non-written answer questions on Moodle can be automatically markedby Moodle itself. This reduces the markers’ marking load. • Students seek to improve their own performance throughout the modulevia ’bootstrapping’. • If students cannot answer a question (due to equipment failure or themchoosing to move away from the laboratory script), they can justify thisin the Moodle test and so NOT answer the question that was asked.This applies only to written answer questions. • It helps students to have the confidence to justify their work and deci-sions to other people (in this case, the markers). • Pre-determined feedback can be given automatically by Moodle. Thisis supported by comments given by markers for the written answerquestions. 7
Markers can mark on-line and so do not need to transport large numbersof notebooks to their offices or homes to mark them. • It is easier for second marking to take place as the written answers forthe students’ DRT submissions are in one database that is accessibleanywhere.The introduction of Moodle-based Data Retrieval Tests (based on a paper-based process created by Edge Hill University in the UK) has significantlyreduced the marking load of staff within UCL’s 1st year physics teachinglaboratories. The responsibility for keeping a good laboratory notebook hasbeen placed squarely on the shoulders of the students. They have respondedwith more professional recording behaviours. It is recommenced for otherpractical physics courses and for any course in a STEM practical environ-ment.
References [1] H. Bedford, A. Bedford, J. Thomas, and P. Ashton. An alternative ap-proach to assessing laboratory and field notebooks: The data retrievaltest’.
Bioscience Education , 16:1–5, 2010.[2] D. Fung.