QuizCram: A Quiz-Driven Lecture Viewing Interface
QQuizCram: A Quiz-Driven Lecture Viewing Interface
Geza Kovacs
Stanford UniversityStanford, [email protected]
Darren Edge
Microsoft Research AsiaBeijing, [email protected]
ABSTRACT
QuizCram is an interface for navigating lecture videos thatuses quizzes to help users determine what they should view.We developed it in response to observing peaks in video seek-ing behaviors centered around Coursera’s in-video quizzes.QuizCram shows users a question to answer, with an associ-ated video segment. Users can use these questions to navigatethrough video segments, and find video segments they need toreview. We also allow users to review using a timeline of pre-viously answered questions and videos. To encourage users toreview the material, QuizCram keeps track of their question-answering and video-watching history and schedules sectionsthey likely have not mastered for review. QuizCram-formatmaterials can be generated from existing lectures with in-video quizzes. Our user study comparing QuizCram to in-video quizzes found that users practice answering and review-ing questions more when using QuizCram, and are better ableto remember answers to questions they encountered.
Author Keywords video flashcards; lecture viewing; in-video quizzes
ACM Classification Keywords
H.5.2. User Interfaces: Graphical user interfaces (GUI)
INTRODUCTION
Lectures on platforms such as Coursera use in-video quizzes to test learners on material while they watch videos. Althoughonline courses also have problem sets and exams, many learn-ers only watch lectures [1] [5]. For these students, in-videoquizzes are an important opportunity to test themselves on thematerial, which is critical for long-term retention [11].While analyzing viewing logs of the Machine Learningcourse on Coursera, we observed that in-video quizzes playan important role in video navigation. Specifically, we ob-served that users often seek backward from in-video quizzesto review the preceding section, and forward to in-videoquizzes to look at the upcoming question. We also observedthat users rarely review lecture videos. Based on these ob-servations, we wished to develop a video viewer that would
A demo is available at http://quizcram.herokuapp.com/
Please view it in Chrome.
Figure 1. The Quizcram interface shows questions on the left, and cor-responding video segments on the right. The scrollable timeline displaysthe past videos and associated questions, to help users review parts theyhad trouble with. better support quiz-centric navigation strategies and encour-age reviewing.Our system, Quiz-driven Video Cramming (QuizCram), usesquizzes to help users navigate the course and guide their re-view process. It includes the following features: • QuizCram shows questions while users watch the video, toserve as a preview of the video content, and to guide theirfocus towards key concepts. • QuizCram keeps track of which video portions users havealready seen, as well as their past performance on ques-tions, in order to suggest which videos and questions theuser should review. • QuizCram facilitates adding questions to videos by allow-ing questions to depend on multiple video segments ratherthan just the immediately preceding one. This enables agreater density of questions to be presented in QuizCram.We used a user study with a within-subjects design to com-pare QuizCram to the in-video quiz format. We found that: • Users remember answers to in-video questions signifi-cantly better when studying using QuizCram. • Users practice answering and reviewing questions more of-ten when studying using QuizCram • We can improve the recall of particular facts from the videoby adding extra questions in QuizCram. a r X i v : . [ c s . H C ] F e b igure 2. Seek sources and destinations in a lecture with 2 in-video quizzes. Each point at (x,y) represents a seek from time x to y. Most seeks do notcross over in-video quizzes. There are many seeks to in-video quizzes from the start of the video, the previous section, and between quizzes.Figure 3. Sources and destinations of seek chains in the Machine Learn-ing course on Coursera, which uses in-video quizzes. Users tend to seekbackward from in-video quizzes (55x higher than baseline backward-seek rate), and forward to in-video quizzes and the 10 seconds immedi-ately preceding them (4x higher than baseline forward-seek rate) MOTIVATION: COURSERA’S IN-VIDEO QUIZZES
This work was motivated by interesting patterns of seek-ing activity around in-video quizzes which we uncoveredwhile analyzing viewing logs of Coursera’s Machine Learn-ing course. We observed that there are large peaks in seekingactivity around in-video quizzes, which is likely due to userspreviewing the questions and trying to find answers to them.Since users may seek several times while trying to reach theirtarget, our analysis groups together seek events that occurwithin 5 seconds of each other into a seek chain , so we canbetter observe users’ intended seek targets. Details on thismethodology can be found in the supplement.There are many backward seeks starting from in-videoquizzes. As shown in Figure 3, 8.6% of all backward seekchains (or 3.8% of total seek chains) start from in-videoquizzes – which is 55x more seeking per in-video quiz thanwe’d expect from a second of video in the course. This peakin backward-seeking from in-video quizzes is likely due tousers searching for answers in the preceding section.
Figure 4. Portions of the video that are skipped over by seek chains inthe Machine Learning course on Coursera. Users do not tend to seekforward across in-video quizzes.
We also observe that there are many forward seeks that endup at or immediately preceding the quiz. As shown in Fig-ure 3, 6.6% of all forward seek chains (or 3.7% of total seekchains) end up either at the in-video quiz or within 10 sec-onds preceding it. These forward seeks are likely generatedby users attempting to view the in-video quiz – as Coursera’sinterface does not provide an option to jump directly to in-video quizzes, users must seek to directly before the in-videoquiz in order to view it.Most seek chains (93%) do not cross in-video quiz bound-aries. As shown in Figure 4, users are 0.4x less likely toskip forward across an in-video quiz, than across a secondof video. Figure 2 visualizes seek sources and destinations ina single lecture video with 2 in-video quizzes: there are manyforward seeks to quizzes, and backward seeks from quizzes.Users also rarely rewatch lecture videos: only 11% of userswho finished watching a lecture will ever open it again.Based on these findings, we aimed to develop a video viewerthat would better support quiz-centric navigation strategiesand encourage reviewing.
ELATED WORKTesting and Pre-Testing Effects
The testing effect shows that repeated testing combined withfast, informative feedback helps students remember mate-rial [11]. QuizCram’s emphasis on answering and reviewingquestions is designed to exploit this effect.The pre-testing effect shows that having users try answeringa question before they actually study the material enhanceslong-term retention [10]. QuizCram exploits the pre-testingeffect by allowing users to preview the question before watch-ing the associated video.
Spaced repetition
Spaced repetition is a technique designed to help learners re-tain information by having them review items at regular in-tervals [3]. A class of applications that exploit this are flash-cards, which split information into independent chunks thatare scheduled for review based on factors such as mastery andrecency of review. There have been a number of algorithmsand models designed for optimizing learners’ retention of thematerial via spaced repetition [9] [2]. However, they tend tobe designed for flashcard-like content, such as isolated factsor vocabulary, rather than lecture videos.A key difference between flashcard-like content and lecturevideos is that lecture videos are typically presented in se-quence, and a given video may build upon concepts intro-duced in a previous video. Additionally, there are differencesin the costs of testing and reviewing. With flashcards, bothtesting and reviewing can be done in seconds. In contrast, thecost of reviewing a video is much greater than the cost of test-ing – we can test a user’s knowledge of a video segment witha question that takes seconds to answer, but viewing a videomay require several minutes. These additional constraints arereflected in QuizCram’s modified scheduling algorithm thattakes into account the order of videos, as well as its increasedemphasis on testing via questions.
Advance Organizers
Advance organizers are information presented prior to learn-ing, that help the learner process the material that is about tobe presented [12]. QuizCram’s questions can be thought ofas an advance organizer for the video segment – the questionprovides a preview of the content that is to be covered in thevideo.
Interfaces for Navigating Lecture Videos
Video Digests is a system that uses textual summaries ofvideo clips to help users navigate through the video [8]. Lec-tureScape uses other users’ aggregated viewing logs to helpidentify points of interest in the video [4]. Panopticon usesa visual display of all video segments to help users find seg-ments of interest [7]. Similar to these systems, QuizCramaims to help users navigate through lecture videos. However,rather than relying on external annotations, QuizCram insteaduses questions extracted from existing in-video quizzes as anavigational aid.
Figure 5. The QuizCram interface, showing the focus question on theleft, and the associated video on the right. The progress bar highlightsthe relevant portion of the video in yellow. Segments that have alreadybeen watched are highlighted in blue (segments from previous parts) andgreen (segments from current part).
SYSTEM DESIGN PROCESS
Based on our observations that users tend to engage within-video quizzes but rarely ever revisit MOOC lecture con-tent (see supplement), as well as the importance of testingand review for retention, our goal was to build a system thatwould test users’ knowledge of lecture materials and encour-age them to review materials using spaced repetition.Our initial design was to treat video segments as flashcards,and schedule them using a spaced repetition algorithm. By as-sociating each video segment with a question, we could easilytest users’s knowledge of each segment. However, schedul-ing videos with a standard spaced-repetition algorithm wouldoften result in the user being asked to review older materialbefore they completed all of the video segments, which wefound that users were unaccustomed to. Hence, we also en-abled users to freely review videos on their own, and onlystarted scheduling older videos for review once they had at-tempted an initial pass through the videos.
QUIZCRAM INTERFACE FEATURES
QuizCram’s interface displays a question and associatedvideo segment, as shown in Figure 5. It also shows a timelineof previous questions below the current question, as shownin Figure 1. Once the user has made an initial pass throughthe questions, we suggest questions that they should review,based on past performance. We use the video progress bar toindicate the section of the video that is relevant to the currentquestion, and portions that the user has previously seen. Ex-isting courses with in-video quizzes can easily be transformedinto the QuizCram format.
Question-Directed Video Viewing
Each video section is associated with a question. We can ex-tract these question-video pairs automatically from existingvideos with in-video quizzes, by associating the in-video quizsection with the immediately preceding video segment. Forvideo segments that do not have an associated in-video quiz,we can either automatically insert a generic “How well didyou understand this video” question, or manually write a newquestion.The question is designed to help users decide whether theyshould watch the video. If the user knows the answer, theycan answer the question and move to the next section. Forsers who do not know the answer, reading the question pro-vides a preview of the key points they will see in the video.
Timeline of Previous Questions and Videos
The timeline feature is designed to encourage review by mak-ing it easy to refer back to previously answered questions andvideo segments. Whenever a question is correctly answered,we insert the next question and associated video segment atthe top of the interface, and push the existing questions down.This results in a scrollable visual history of previously an-swered questions, as shown in Figure 1. The timeline dis-plays the question, its answer, and a miniaturized version ofthe video which can be clicked to enlarge it to full size andplay it. The miniaturized video displays the frame the userleft off at, so it serves both as a visual summary, and alsoallows users to easily resume watching previous videos.The timeline gives users the option to use a more self-directedreviewing strategy, in contrast to the flashcard-style reviewingthat our question scheduling algorithm encourages. By orga-nizing the list of previous video segments according to theassociated question that users answered, this allows users toscan video segments with a more salient summary than justthe video title. Furthermore, re-reading the previously an-swered questions can help trigger the users’ memory of theassociated video clips
Scheduling Questions and Video Sections for Review
We want users to spend their study time focusing on mate-rial that they have not yet mastered. Hence, we assign eachquestion a mastery score , which represents how well the usercurrently knows the material, and show users the questionsfor which they have low mastery score. The question’s mas-tery score is based on the following 3 factors: • Past performance on question : This element of the scoreencourages users to review questions they answered incor-rectly. Each time a user tries answering a question, we givethem a score equal to the fraction of checkboxes they cor-rectly checked (the questions used in our study were allmultiple-check questions). We then take a weighted-meanof historic scores, weighing recent answers more heavily. • Fraction of associated video segment watched : This ele-ment of the score encourages users to view video segmentsthey have not seen. For each second of video, we keeptrack of whether the user has ever seen it. This score is thefraction of the video segment that has been seen. • Recency of review : This element of the score ensures thatusers review old questions, but are not shown same ques-tions repeatedly. For simplicity, we use a score that is in-versely proportional to how recently the question was lastanswered. Ideally, one would instead use a more advancedspaced-repetition algorithm like MemReflex [2].Once the user has seen all the questions in the unit, QuizCramencourages them to review questions and sections for whichthey have low mastery scores, by showing them at the top ofthe video timeline.
Figure 6. The in-video quiz format that served as our baseline. Locationsof quizzes are indicated in red on the progress bar.
Directing Attention to Unseen Parts of Videos
To help users review videos and resume where they left off,QuizCram keeps track of which parts have been watched. Ithighlights on the progress bar the portions that have alreadybeen seen. If the user is viewing a section they have alreadywatched, they can skip to the unseen portion by clicking a but-ton, as shown in Figure 5. This technique for visualizing theviewing history has previously been shown in the literature[6] [4], though our system adds the novel feature of allowingusers to skip to the next unseen portion.
EVALUATION
Our study used a within-subjects design to compare users’studying behavior with QuizCram against an in-video quizinterface that mimcs the format used on Coursera, as shownin Figure 6. We used the videos, in-video quizzes, and unitexam from the Neurobiology course on Coursera. We wishedto answer the questions: • Does QuizCram help users better remember answers to theoriginal in-video questions? • Does QuizCram help users score higher on exams? • Do users find QuizCram helpful for studying videos? • How do users interact with questions and videos when us-ing QuizCram?
Participants
We recruited 18 students by posting on university mailinglists. 12 were female, 6 male. Their average age was 21.7( σ =4.91, min=18, max=37). All had native-level English pro-ficiency. None had prior exposure to neuroscience. They re-ceived $60 for participating. Materials
The videos, in-video quizzes, and unit exams were from Unit1 of the Neurobiology course on Coursera. There were 9questions and 5 videos in each 25-minute section. We gener-ated the initial QuizCram materials directly from the course.Not all of the segments of the lecture videos had in-videoquizzes immediately following them. For such segments,QuizCram would normally show a generic “How well did youunderstand this video” question as the focus question. How-ever, in pilot studies, users indicated that they found theseself-assessment questions less helpful than regular questions,as they did not provide a preview of what the section woulde about. Furthermore, we believe that the QuizCram for-mat is best-suited towards a more question-heavy viewing ex-perience than in-video quizzes currently provide. Hence, tosimulate what content that was designed for the QuizCramformat would look like, we added our own extra questionsfor video segments which lacked associated in-video quizzes.This doubled the total number of questions per section in theQuizCram condition. The extra questions were in the samemultiple-checkbox format as the original questions. We madesure that the extra questions did not depend on the same factsas the unit exam or original in-video quizzes, to ensure thatthey would not help users learn the other material by givingthem an extra testing opportunity.We also wrote a set of free-response questions, with one cor-responding to each of the extra questions. We used thesefree-response questions to test whether users had learned thematerial tested by in-video questions well enough to recall it(rather than recognizing it).
Procedure
The study was conducted online over 2 days. Before usersstarted the study, we informed them that they would be given2 sets of videos, they would study them for 40 minutes apiece,and they would be given an exam the next day. We did not tellthem about the content of the exam in advance.On day 1, users studied the first section with one tool for 40minutes, and answered a survey about the tool. Then, theystudied the second section with the other tool for 40 minutes,and answered a survey about the tool. The order of tools wasrandomized.On day 2, users took the following exams:1. Extra free-response questions2. Original in-video questions from Coursera3. Original unit exam from Coursera4. Extra multiple-checkbox questionsParts 2-4 were automatically graded. Free-response questionswere graded blindly according to the formula: correct examples givenMaximum ( examples requested , examples given ) RESULTSExam Results
Exam results are shown in Figure 7. QuizCram users per-formed significantly better on the original in-video ques-tions, which had been shown in both conditions. They alsoperformed better at both types of extra questions. Thus,QuizCram improves retention of the original in-video ques-tions, and we can use added questions to improve retention ofparticular facts from the video. However, there was no signif-icant improvement in scores on the original unit exam.
Survey Results
Survey responses after using each tool are shown in Figure 8.61% said would prefer to use QuizCram if they wanted toremember material long-term or were preparing for an exam.These improvements were not statistically significant.
Figure 7. Average exam scores for each conditionFigure 8. Survey responses showed slight preferences in favor ofQuizCram, but they were not statistically significant
Survey feedback showed that users thought QuizCram’s focusquestions helpful for reviewing videos:
I liked that it picked out the key information I should retainby asking me questions. It helped me decide what to focus onas I watched the video. The chunks were very manageable aswell. I liked how it was broken up.
However, some users thought that the prominent display ofquestions distracted them from watching the video.
I did not like the fact that you could answer questions whilethe video was playing. It made me more focused on answeringthe questions rather than watching and learning the material.
Analysis of Users’ Video Interaction Logs
To compare how users interacted with the two tools, welogged the users’ interactions as they studied the lectures, asshown in Figure 9.We found that users practiced answering each question moretimes when using QuizCram. They also tended to answerquestions correctly a higher percentage of the time, per-haps because they had been able to preview the question be-fore watching the video. Users also reviewed previously-answered questions more often when using QuizCram. Thisincrease in practice and reviewing helps explain the increasedexam scores on the original in-video questions.Users seeked less on average when using QuizCram, whichmay partly be because they did not have to seek to and fromin-video quizzes. However, this difference was not statisti-cally significant.
Figure 9. Average number of events logged per user in each condition
ONCLUSION
We have presented QuizCram, a system that guides users’video viewing using questions. QuizCram aims to: • Encourage users to answer and review questions while theywatch videos • Enable users to easily follow question-driven video navi-gation strategies (which we currently observe some usersalready using on Coursera)QuizCram breaks the video into segments associated withquestions, and shows a focus question alongside the video.This question serves as an advance organizer that guidesthe user’s attention towards the key points in the video.QuizCram also encourages reviewing based on questions: itdisplays a timeline of questions previously answered and theirassociated videos. It keeps track of users’ progress throughquestions and videos, and suggests questions for users to re-view. Courses in the QuizCram format can be generated fromexisting videos with in-video quizzes.Our user study found that QuizCram increases retention ofquestions – when the in-video questions were tested a daylater, QuizCram users remembered them better than if theywere presented as in-video quizzes. Users practiced answer-ing and reviewing questions more when using QuizCram.Our user study has focused on a cramming scenario – wherethe user is trying to memorize a small amount of material toprepare for an imminent exam. However, another potentialuse case for QuizCram-like systems is for long-term reten-tion – where the user is attempting to remember the contentof multiple courses over multiple months. Given the successof spaced repetition systems in helping users’ long-term re-tention of flashcards and vocabulary, we expect that having asystem schedule quizzes that review course contents shouldsimilarly be helpful for helping users’ long-term retentionof course materials. Studying how question-driven lecture-reviewing systems can scale to entire courses and longerstudy periods is potential future work.We designed QuizCram to address the needs of users whowish to complete the MOOC and master the entire material.Hence, the system tests users’ knowledge of video segments,and schedules reviews to ensure that users remember the ma-terial. That said, learning the complete course material is notthe objective of many learners – many users are only inter-ested in a subset of the material, and do not complete the restof the course [5] [1]. Although addressing the needs of usersinterested in only a subset of the material was not an objectiveof QuizCram, it is potential future work.Current online courses rely on external problem sets and ex-ams to test understanding of content in more depth than thein-video quizzes. However, many MOOC participants inter-act primarily with videos and do not take exams or do prob-lem sets [5] [1]. Thus, moving more of the course contentout of problem sets and making the video more interactiveand question-oriented provides a way to benefit these viewerswithout removing them from the scaffolding of videos. Webelieve that QuizCram is a logical step from in-video quizzestowards more interactive, question-driven study experiences.
REFERENCES
1. Anderson, A., Huttenlocher, D., Kleinberg, J., andLeskovec, J. Engaging with massive online courses. In
Proceedings of the 23rd international conference onWorld wide web , International World Wide WebConferences Steering Committee (2014), 687–698.2. Edge, D., Fitchett, S., Whitney, M., and Landay, J.Memreflex: adaptive flashcards for mobilemicrolearning. In
Proceedings of the 14th internationalconference on Human-computer interaction with mobiledevices and services , ACM (2012), 431–440.3. Karpicke, J. D., and Bauernschmidt, A. Spaced retrieval:absolute spacing enhances learning regardless of relativespacing.
Journal of Experimental Psychology: Learning,Memory, and Cognition 37 , 5 (2011), 1250.4. Kim, J., Guo, P. J., Cai, C. J., Li, S.-W. D., Gajos, K. Z.,and Miller, R. C. Data-driven interaction techniques forimproving navigation of educational videos. In
Proceedings of the 27th annual ACM symposium onUser interface software and technology , ACM (2014),563–572.5. Kizilcec, R. F., Piech, C., and Schneider, E.Deconstructing disengagement: analyzing learnersubpopulations in massive open online courses. In
Proceedings of the third international conference onlearning analytics and knowledge , ACM (2013),170–179.6. Mertens, R., Farzan, R., and Brusilovsky, P. Socialnavigation in web lectures. In
Proceedings of theseventeenth conference on Hypertext and hypermedia ,ACM (2006), 41–44.7. Nicholson, J., Huber, M., Jackson, D., and Olivier, P.Panopticon as an elearning support search tool. In
Proceedings of the 32nd annual ACM conference onHuman factors in computing systems , ACM (2014),1221–1224.8. Pavel, A., Reed, C., Hartmann, B., and Agrawala, M.Video digests: A browsable, skimmable format forinformational lecture videos. In
Proceedings of the 27thannual ACM symposium on User interface software andtechnology , ACM (2014).9. Pavlik, P. I., and Anderson, J. R. Using a model tocompute the optimal schedule of practice.
Journal ofExperimental Psychology: Applied 14 , 2 (2008), 101.10. Richland, L. E., Kornell, N., and Kao, L. S. Thepretesting effect: Do unsuccessful retrieval attemptsenhance learning?
Journal of Experimental Psychology:Applied 15 , 3 (2009), 243.11. Roediger III, H. L., and Butler, A. C. The critical role ofretrieval practice in long-term retention.
Trends incognitive sciences 15 , 1 (2011), 20–27.12. Stone, C. L. A meta-analysis of advance organizerstudies.