Designing Narrative-Focused Role-Playing Games for Visualization Literacy in Young Children
Elaine Huynh, Angela Nyhout, Patricia Ganea, Fanny Chevalier
TTo appear in IEEE Transactions on Visualization and Computer Graphics.
Designing Narrative-Focused Role-Playing Gamesfor Visualization Literacy in Young Children
Elaine Huynh, Angela Nyhout, Patricia Ganea, and Fanny Chevalier a b c
Fig. 1. Sample screenshots from our educational role-playing game (RPG) to support visualization literacy education at ages 11 - 13.Our game encompasses data-related problems involving visualizations that young learners are required to solve to make progressin the game. Players must first indicate the right graph to solve the problem (a), and then interpret the selected graph to answer aquestion (b). In this work, we investigate the effect of incorporating narrative elements on learning and engagement (c).
Abstract —Building on game design and education research, this paper introduces narrative-focused role-playing games as a wayto promote visualization literacy in young children. Visualization literacy skills are vital in understanding the world around us andconstructing meaningful visualizations, yet, how to better develop these skills at an early age remains largely overlooked andunderstudied. Only recently has the visualization community started to fill this gap, resulting in preliminary studies and development ofeducational tools for use in early education. We add to these efforts through the exploration of gamification to support learning, andidentify an opportunity to apply role-playing game-based designs by leveraging the presence of narratives in data-related problemsinvolving visualizations. We study the effects of including narrative elements on learning through a technology probe, grounded in aset of design considerations stemming from visualization, game design and education science. We create two versions of a gameone with narrative elements and one without and evaluate our instances on 33 child participants between 11- to 13-years old using abetween-subjects study design. Despite participants requiring double the amount of time to complete their game due to additionalnarrative elements, the inclusion of such elements were found to improve engagement without sacrificing learning; our results indicateno significant differences in development of graph-reading skills, but significant differences in engagement and overall enjoyment of thegame. We report observations and qualitative feedback collected, and note areas for improvement and room for future work.
Index Terms —Visualization Literacy; Educational technology; Gamification; Narrative
NTRODUCTION
From global-scale events such as climate change and public healthcrises, to societal topics such as economics, education, culture, or nutri-tion, many infographics, tables, and charts are created and circulateddaily to help the general population better understand what is happeningand what actions need to be taken. Graphical representations of datacan exert considerable influence on the readership: the mere presenceof a chart – regardless of triviality – is capable of convincing readers ofthe scientific credibility of an article [84]. An important caveat is that,when designing data visualizations, parties may present informationin ways that mislead or deceive readers [65], whether intentionally ornot. As such, the ability to understand and appropriately handle datavisualizations – or more succinctly, visualization literacy [26, 31] – is avital skill. Yet, despite its importance, visualization literacy amongstthe general public remains low [25].Recently, the visualization community has begun to acknowledgeand address this deficiency. Children are exposed to charts and otherdata management topics as early as grade 1 [8], and previous researchhas looked at ways to support visualization literacy education at theelementary school level [15, 21, 31]. Research in this area is still in • Elaine Huynh is with the Department of Computer Science at the Universityof Toronto. E-mail: [email protected].• Angela Nyhout and Patricia Ganea are with the Ontario Institute for Studiesin Education, University of Toronto. E-mail: { patricia.ganea,angela.nyhout } @utoronto.ca• Fanny Chevalier is with the Department of Computer Science and StatisticalSciences at the University of Toronto. E-mail: [email protected]. its infancy, and a range of methods to support learning, especially at ayoung age, have yet to be investigated. With our work, we add to theseefforts through the exploration and application of a form of educationalsupport not yet formally studied in visualization: gamification.Gamification seeks to provide an interactive and engaging way ofteaching players about new concepts or skills. We see gamificationbeing applied to a broad range of subjects including music [60] andphysics [61] in an attempt to enrich the learning experience. Differ-ences between the designs of educational games exist in order to bettertarget difficulties and challenges offered by each topic. For example,gamification of basic arithmetic concepts such as addition or multi-plication may result in a design that encourages players to repeatedlysolve problems in a short period of time. While this approach maywork for some topics, this rote learning style of gamification has beencriticized for its focus on training rather than actual learning [36, 42],and may not lend itself well to tasks where memorization or recall arenot key factors in understanding the material – as is in the case withvisualization literacy.A review of publicly-available visualization problems posed to el-ementary school children (e.g., [4, 5, 8]) shows that stories are oftenused to provide context and motivation for questions related to chartinterpretation. By leveraging the presence of such stories, we see anopportunity to apply a more narrative-focused approach to game de-sign, such as those offered by role-playing games (“RPGs”). Our mainresearch questions then become: (RQ1) What are the key factors to consider for an RPG-style educa-tional game focused on visualization literacy? (RQ2)
How does the inclusion of narrative elements affect learningand engagement, in the context of a visualization literacy game?1 a r X i v : . [ c s . H C ] A ug n this work, we introduce the first story-based game to supportvisualization literacy learning. We describe our own instance of a game,complete with a set of exercises, overarching narrative, and mechan-ics to support narrative elements (§3), built as a technology probe toevaluate our approach. To gauge the impact of narrative elements ingames focused on visualization literacy, we design a between-subjectsstudy to evaluate two variations of our game: one with narrative ele-ments, and one without (§4). Our results (§5) show that the inclusion ofnarrative elements yields a significantly positive impact on children’sengagement and enjoyment, though it requires players to spend moretime interacting with added elements. Interview sessions reveal thatchildren in the with-narrative condition are generally pleased with thestory and associated interactions, but individual differences exist.Using the qualitative feedback and observations from the study, weidentify areas of improvement and discuss directions for future works(§§6-10). Specifically, we contribute a set of design considerations toguide future work in creating RPG-style visualization games, groundedin the literature from both education and game design fields (§7), andreflect on lessons learned (§8). ELATED W ORK
Our work builds on prior literature in visualization literacy, educationalgames, and the role of narratives in games.
As studies surfaced the difficulties faced by the general populationin interpreting and creating visual representations of data [25, 46, 63], visualization literacy has emerged as an increasingly pressing con-cern. Visualization literacy, or visual information literacy [85], is theability to critically read and construct data visualizations [24, 26, 31].Lacking such skill can be detrimental to individuals, resulting in poordecision-making through misinterpretation of charts. There is substan-tial anecdotal evidence of chart misuse, influencing the way people buyproducts, vote, or respond to a pandemic [28, 33].In response to this problem, visualization researchers have startedto discuss a research agenda to promote visualization literacy, via aseries of workshops [52, 74], lessons learned from observational stud-ies [31], or most recently, synthesis of accumulated knowledge in thedomain [24]. In an effort to develop standardized tools for measur-ing visualization literacy levels in people, Boy et al. [26] introducedvisualization literacy tests grounded in Item Response Theory, andLee et al. [54] proposed the Visualization Literacy Assessment Test(VLAT). Building the theoretical ground for guiding the design of amuch needed visualization curricula has also been outlined as critical.Brner et al. [24] contributed an initial proposal of data visualizationliteracy framework (DVL-FW), covering terminology and typology ofcore concepts in visualization, as well as a model to guide the devel-opment of curricula and assessment of visualization literacy. Theseresearch initiatives contribute to the goal of establishing structured,informed foundations, methods, models, and standardized instrumentsfor assessing, teaching and learning visualization.One important concentration of effort in the area has been the docu-mentation and research surrounding teaching and learning of visualiza-tion in young children. Alper et al. [15] suggest that we can addressvisualization illiteracy in generations to come by improving the way weteach visualization as early as elementary school. To this end, they stud-ied what materials are currently used in classrooms to support learningof visualization concepts through a review of textbooks and interviewswith teachers. They designed “C’est la vis”, a technology-mediatedintervention to support teachers as they teach pictographs and bar chartsto students in grades K-4, aligned with current practices in education.Building on Huron et al.’s work on constructive visualizations [48, 49],Bishop et al. [21] studied the process of constructing visualization inchildren aged 5-12 using “Construct-a-Vis”, a tablet-based multi-userapplication where children can collaboratively or independently mapvisual variables to data dimensions in a free-form, creative manner. Wecontinue these efforts focused on development of visualization literacyeducation in young children through the exploration of another form ofintervention: gamification.
While gamification long predates computer games [50], the usage ofeducational video games (also referred to as “serious games”) is par-ticularly appealing to younger generations. For such audiences, gameshave been found to be more engaging than passive media (e.g., videos)due to the presence of active control [43, 70]. Through games, playersare able to engage in subject matter beyond simply reading and respond-ing to problems, leading to increased performance when compared totraditional learning methods [23, 57, 67]. Games enable curiosity andexperimentation through freedom to explore, allowing users to learnthrough experiential play [88]. In addition to the incorporation of activecontrol, digital games take advantage of the auditory and visual streamsenabled by electronic devices, and the combination of visualisation andexploration has been shown to enhance learning [17, 19, 67].Interest in gamification has spawned a variety of educational games,covering broad range of topics such as physics [61] and music [60]. Invisualization, there exists educational games focused on bar charts [40]and scatter plots [86], as well as non-educational games inspired byinfographics [6]. Yet, despite the variety of genres and mechanics seenin commercial games, we observe that many of these educational gamestend to focus on repetitive gameplay: players may be asked to completea set of mathematical computations within a specified amount of time,or need to come up with a configuration to satisfy constraints for aparticular level. Some educational games integrate narrative elementsto an extent (e.g., with short video clips or quests [1, 2, 7]), but unlikethe puzzle and action-based designs, there is a lack of research on theefficacy of such other genre and elements they offer, especially RPGsand their focus on storytelling, guidelines for designing such games,and information on how they affect learning and engagement.
In commercial games, the presence of a story is credited with increasedfeeling of involvement in the gameplay and immersion in the environ-ment [76]. The presence of narratives in role-playing games helps tocreate memorable and pleasurable experiences, with players likeninggames in this genre to interactive story books and films [35]. Sto-ries also act as intrinsic motivators, encouraging players to overcomeobstacles and challenges in order to see what happens next in thestory [42, 58, 88].In online multi-player role-playing communities, there are oftengroups of players who engage specifically in role-playing activities [35].These players create avatars in the game world, craft rich lore andpersonal stories for these characters, and interact with fellow role-players or roam the game space while assuming the identity of theircreations. As such, it is clear that narratives are very important to atleast a subset of players, and this should not be ignored when designingeducational games.In traditional education settings, the usage of story elements hasbeen shown to have positive effects on learner memory, motivation, andengagement [20], though the degree of these effects may vary betweengroups based on level of prior knowledge [90]. In teaching higher-level math and computer science concepts, incorporating narrativesand storytelling has been suggested in order to generate interest in thesubject matter [66]. There has also been an attempt at interleavingstorytelling and learning in the context of a textbook [37], which hasbeen generally well-received by the public and praised for its engagingpresentation of educational content.When looking at the effects of narratives in serious games specif-ically, there seems to be conflicting information with respect to howconducive they are to learning. Adams et al. experimented with theeffects of narratives in two games and concluded that, compared tosimply viewing a slideshow presentation, playing a narrative game doesnot aid in learning [62]. In contrast, Bittick reported that narrativesincreased self-perceived immersion in serious games and that genderplays a role in the effectiveness of narratives on learning outcomes [22].Another study [38] found that narration quality had positive effectson learning effectiveness, as well as enjoyment. Further work lookingat narratives in serious games is required before making claims as towhether or not they are largely beneficial or detrimental to learning.2 o appear in IEEE Transactions on Visualization and Computer Graphics.
Fig. 2.
Activity View . This view consists of a question display box (a), instructions (b), choices to select from (c), a feedback box (d), and charactersprites (e, f). Follow-up “interpret the chart” questions include a picture of the previously-chosen chart (g), and choices of written answers (h).
ECHNOLOGY P ROBE : A G
AME TO L EARN V ISUALIZATION
We first briefly describe our own example of an educational RPG, thedesign decisions we made, and its implementation. This technologyprobe was created with the intent of studying the impact of narrativeelements in a visualization-focused educational game on learning andengagement (RQ2; see §4). We share further reflections on designconsiderations to account for when designing such games in §7.
For our game and subsequent study, we targeted children between 11-to 13-years old, as we felt they would be comfortable with the amountof reading required in an educational RPG. This age range also includesthe intersection of children who are most likely to read for fun in theirspare time, as well as children who already express familiarity withelectronic games and applications [13].
We first looked at the provincial curriculum [8] to understand whichtopics our participants should be familiar with and decided our educa-tional activities would focus on pie charts (i.e. “circle graphs” [8]) andhistograms. Pie charts were chosen because children in this age rangeshould be mostly familiar with them but still find them challenging,testing our game as a supplementary tool for practice of previously-taught topics. Histograms would likely be new to most participants,giving us insight on the game as a tool for teaching new concepts.Educational activities are presented as multiple-choice questions,consisting of choose a chart question (i.e. which chart is best suitedand/or the correct one to solve the problem), followed by an interpretthe chart question. To further streamline the process through whichplayers would be exposed to questions, we presented these questiontypes in pairs (e.g. Figure 1a, then Figure 1b).We designed 10 story-neutral question templates (5 pairs) for eachof the charts tested, resulting in a total pool of 20 questions. Thesetemplates captured the learning objective/topic to be tested (e.g., un-derstanding intervals in histograms) in order to mimic the structure ofquestions found in typical textbook exercises, and omitted the pres-ence of a story or scenario, which would be added after the narrativedesign was complete. For each question template, we created threepossible options/responses, along with textual feedback to be displayedwhenever an option is selected as a response. The feedback providedwas intended to provide hints and educate players on why their answermay be incorrect, as well as provide further explanation or reinforceunderstanding for cases where players chose the correct answer. Two el-ementary school educators teaching in Canada and the US, respectively,helped to review and refine the generated exercises and feedback.
To add narrative context to the activities, players are further immersedin a game world that they can explore and find characters to interactwith through dialogues . We opted for a linear narrative to allow forstronger control over how players interact with the story in our study (see §7.4). A simple story with a dramatic arc was used and situatedin the fantasy genre to mimic those seen in generic RPGs. Players arefirst introduced to the main character – a student at a magic school –and assume her role as she meets other characters, helping them solvetheir problems. We used the narrative to set up context for each of theeducational activities, and reiterated the problem in the question textto be displayed when solving a problem to promote fairness betweenconditions. In linking the story and educational content, we looselytied the difficulty of problems with parts of the dramatic arc: histogramquestions, which we believed would be more difficult, were presentedonly in the latter part of the game, during the rising action and climax ofthe dramatic arc. We hoped that, while players may be discouraged bythe difficulty of the questions, they might be motivated and encouragedto persevere to see the rest of the story, as the plot becomes moreinteresting and intense.
Our proof of concept was made using the Unity game engine and com-piled for use on 10.5” iPad Pro. The game consists of three main views:the
Activity View (Figure 2) presenting the puzzles, and the
DialogueView (Figure 1c) and
Exploration View (Figure 3) to support the nar-rative dimension of the game. Through the Activity View, players arepresented with the chart-related questions (§3.2). Upon selection of aresponse, a feedback box (Figure 2d) appears in the bottom-left cornerof the screen with a predefined message associated with that choice.Players are permitted (without penalty) to make choices until theychoose the correct answer, after which the Activity View ends and thenext view is loaded. The Activity View is typical of many educationalgames [40,60,61,86], where puzzles are presented in sequence withouta narrative connecting them together.In our RPG-like game, players are also able to interact with theDialogue and Exploration Views. Progression is relatively streamlined:players engage with the narrative through character dialogue (via Dia-logue View) and go through a phase of exploration until they interactwith the character who will advance the story for them and present thequestion in the Activity View. Once finished, players are sent back tothe Dialogue View to wrap up the conversation and be given the nextobjective. This repeats until the player fulfills all objectives. Across allviews, interaction is done through simple tapping.
OMPARATIVE S TUDY : T HE R OLE OF N ARRATIVE
Using a between-subjects study design, we evaluated (RQ2): the impactof incorporating narrative elements (i.e., exploration and dialogues) ineducational RPGs on learning and engagement on children aged 11-13,using our design probe (§3). Participants were tested in independentsessions conducted in a lab setting with only the participant and investi-gator present, for a maximum of one hour. Participants were informedthat they could stop at any time without penalty, and were permitted toask questions if they needed any assistance. The study consisted of fourphases: Pre-Test, Play Time, Post-Test, and Interview. In Play Time,participants were all presented with the same activities and were ran-3 ig. 3.
Exploration View . Players are able to explore the game world bytapping on arrows (hotspots) connecting to adjacent maps. Characterinteractions are triggered by tapping on a character’s respective sprite.
Chart Question Type Scenario
Q1. Pie Choose a Chart Non-FantasyQ2. Pie Choose a Chart FantasyQ3. Pie Interpret the Chart Non-FantasyQ4. Pie Interpret the Chart FantasyQ5. Histogram Choose a Chart Non-FantasyQ6. Histogram Choose a Chart FantasyQ7. Histogram Interpret the Chart Non-FantasyQ8. Histogram Interpret the Chart Fantasy
Table 1. Order and classification of chart questions used in the tests. domly assigned to either the without-narrative (i.e., activities only,presented in sequence without further context), or with-narrative (i.e., activities, exploration, and dialogues) condition. Children werecompensated with their choice of small toys; parents received monetarycompensation for travel expenses.
Participants.
Participants were recruited through a database offamilies who had previously expressed an interest in child studies andcollected over several years using mailing lists, public events, and wordof mouth. Parents of children between 11 to 13 were contacted byphone or by e-mail, and given a short description of the study. Dueto the amount of reading involved in the study, we requested that onlychildren who could read at a grade 6 level or higher participate inthe study. A total of 33 participants (16 female, mean age of 12.09)participated in the study. All recruited participants were native Englishspeakers, with two having attended a French immersion school.
Pre-Test.
Following introductions, the child participant was givena test to gauge their prior understanding of pie charts and histograms.We co-designed the pre- and post-tests with a professional educator, anditeratively refined the instrument to take into account the amount of timeit would take for participants to complete each question, ease of grading,as well as learning objectives. In particular, the educator suggested toinclude basic arithmetic and reading comprehension questions to gaugecompetence. Questions using both fantasy and non-fantasy contextswere included to address transfer of knowledge between domains. Thefinal version of the pre-test distributed to participants consisted ofone page of questions gauging basic math and reading comprehensionskills (4 addition, 4 subtraction, 3 multiplication, and 2 simple wordproblems), followed by 6 pages of chart problems (8 multiple-choicequestions). Table 1 outlines the order and types of chart questions used.
Play Time.
Participants were asked to play through one version ofthe game ( with-narrative or without-narrative ). This phase wasvideo- and screen-recorded for further analysis. Children were told theywould have approximately 30 minutes to complete the game, but therewould be no penalty or reward associated with speed of completion.We quietly observed participants, and made notes about whether theyappeared to think about the questions or brute-forced solutions, if theyread the feedback box, and how they interacted with the story (e.g.,if they appeared to read through all dialogue or skipped text). Theinvestigator only intervened when the participant appeared to be stuckor if they requested clarification or assistance. Grade 8Grade 5Grade 6
311 1
Grade 7
Female W i t hou t n a rr a t i ve
11 years12 years13 years W i t h n a rr a t i ve Male
Fig. 4. Demographics of participants per condition.
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8051015 without-narrative (pre-test)without-narrative (post-test) with-narrative (pre-test)with-narrative (post-test)
Fig. 5. Total number of participants answering questions correctly in thepre- and post-tests and across conditions
Post-Test.
Participants were then given a test on paper to gaugechanges in their understanding of the subject matter. This post-test wasidentical to the one provided in the pre-test, but with the basic mathand reading comprehension questions removed.
Post-Study Interview.
We conducted semi-structured interviewsto gather feedback on our approach. Participants were asked to rateand share their general thoughts about different aspects of the gamesuch as the art, educational activities, engagement, fun, and overallgameplay. Plot and characters were also included for those in the with-narrative condition. In cases where participants were unre-sponsive, the investigator attempted to obtain more information byasking the participant to list what they enjoyed and did not enjoy aboutthe topic. Children in this condition were then asked whether theynoticed themselves skipping through the story or dialogue, and, if so,explain why they might have engaged in such behaviour.We then asked participants from both groups about their feelings onthe educational activities, if they engaged in brute-forcing strategies,and why they might have done so when they were comfortable withacknowledging this behaviour. Finally, thoughts on the usefulness ofthe feedback box, the educational value of the game, and suggestionson how to improve were solicited.
ESULTS
Child participants were assigned to the without-narrative condition(16 participants) or the with-narrative condition (17); see demo-graphics breakdown in Figure 4. Analysis of the basic comprehen-sion test showed that, at most, participants made a single mistake(9/33, 5 with-narrative , 4 without-narrative ). From this, we de-termined all participants were fit to grasp the study material and resultsof the study were unlikely to be affected by fundamental differencesin cognitive ability. All participants played their assigned games tocompletion, spending an average of 13min ( without-narrative ) and10min ( with-narrative ) on educational activities. Participants in the with-narrative condition spent an additional 13min on dialogue and2min on exploration, for a total of 25min. Interviews lasted about 7-8min. One participant (P25) did not complete the interview due to timeconstraints. Video and audio recordings were transcribed for analysis.
Quantitative Analysis.
Figure 5 shows the number of participantsresponding correctly to questions in the pre- and post-test across con-ditions. For each condition, we conducted a paired-samples t-test tocompare the performance in the pre-test and post-test for each ques-tion. There was a significant difference in the pre-test scores (M=0.267,SD=0.449) and post-test scores (M=0.485, SD=0.507) for one ques-tion (Table 2, Q6), in the with-narrative condition; t(16)=0.436, p =0.0093. We did not find any other significant differences, but for somequestions (e.g., Q1), there was no room for improvement as most, if notall, participants answered the question correctly in the pre-test (Fig. 5).4 o appear in IEEE Transactions on Visualization and Computer Graphics.
Q4. ArtQ1. StoryQ2. Characters
10 66 7 35 7 4
Q3. InteractionsQ5. ActivitiesQ6. EngagementQ7. Fun
Q8. Overall Gameplay
Q5. ActivitiesQ6. EngagementQ7. FunQ8. Overall Gameplay very enjoyable (Q1-Q5 & Q8) / very engaging (Q6) / very fun (Q7) not enjoyable at all (Q1-Q5 & Q8) / not engaging at all (Q6) / not fun at all (Q7) W i t h n a rr a t i ve W i t hou t Q4. Art
Fig. 6. Participant ratings from the interview phase.
For each pair of pre- and post-test questions, there were four possibleoutcomes a participant could achieve, whether they would answer thequestion correctly (C: correct) or not (I: incorrect) on the pre- thenpost-test: C § C, C § I, I § C, I § I. Focusing on the overall changes inperformance at a per-participant level, we now discuss results based onthe number of times participants could have improved (sum of I § C andI § I), showed improvement (total I § C), or did worse (total C § I). ‚ Gender.
Grouping by gender revealed that all 8 males in the without-narrative condition either improved (I § C ą
0; 6/8 males) orshowed no change between tests (I § C= 0; 2/8). Cases where males per-formed worse (C § I ą
0; 4/8) were only found in the with-narrative condition. There were no such trends seen in female participants. ‚ Age and Grade.
All 11-year-old participants either improved (6/8; 3in grade 5, 3 in grade 6) or showed no change (2/8; 1 in grade 5, 1 ingrade 6) between tests. None performed worse on any of the questions.No clear patterns were apparent in other age/grade groups. ‚ Degree of Change.
Participants who exhibited the greatest changebetween tests (3/33) were observed to belong to the with-narrative condition (I § C= 3, 2/3; C § I= 1, 1/3).
Qualitative Analysis.
When asked about the educational value ofthe games they played, most participants felt the game would be helpfulto people who struggle with charts or understanding word problems.Some specifically noted that the activities would have been better suitedfor students who are new to the covered charts, as opposed to someonewho is already familiar with and wanting to learn more about them. Thisfinding aligns with the quantitative data above regarding performancebased on age, as children in the 11-year-old group were less likely tobe familiar with concepts compared to older participants.Regarding the feedback box — whose primary goal was to helplearners think critically when making mistakes — the majority of par-ticipants (21/32) considered the functionality to be helpful or conduciveto learning, likening it to “having a teacher beside you.” (P1) as “itgave you little recaps about how your answer was and how it thoughtyou did” (P15), which was found particularly valuable “when the ques-tion was hard” (P15). Still, a few participants (P4, P7, P12, P23) feltconflicted about the overall usefulness of the feedback because theyfound it to be not as helpful when they made a correct choice, com-menting “it was like a pat on the back” (P4). In contrast, P17 notedthat the feedback “made [her] feel happy because whenever [she]got a question right, they said ‘good job’.”
Another participant (P14)mentioned that they would “half-read” the feedback box when theiranswer was correct.Four participants did not find the feedback box to be helpful (P13) orsimply did not bother to read it (P16, P22, P27), but could not articulatewhy they felt this way when asked to elaborate. Two participantspersonally considered the feedback box to not be an important feature,but mentioned they would prefer to keep it rather than remove it, ifgiven the choice (P19), and “for some people who don’t understandcertain parts of math, it would help them” (P21).
Quantitative Analysis.
Figure 6 shows the results of the sub-jective ratings collected during the interview. Participants in the with-narrative condition appeared to be more engaged and content with the game than those in the without-narrative condition. We ranMann-Whitney’s U tests on each rated category and found significantdifferences between the two conditions for: art (U = 53, Z = -3.14, p= 0.002, r = 0.56), activities (U = 66.5, Z = -2.49, p = 0.01, r = 0.44),engagement (U = 63, Z = -2.72, p = 0.007, r = 0.48), and fun (U = 57.5,Z = -2.82, p = 0.005, r = 0.50).
Qualitative Analysis.
Thoughts on the story and character weremostly positive, with some participants praising the usage of differentfacial expressions on characters to express emotions, as opposed todisplaying a single static image per character. Two participants (P22,P29) explicitly stated they enjoyed how the story and the problemswere connected to each other, and one participant (P31) liked how theoverarching narrative provided motivation behind the problems:
P31: “They’re questions that are actually in-depth and it’s notjust like, ’This person has this, that person has that, my answeris this.’ Why would I care how much stuff Joan has? Why do Icare about this person’s problem? ... This game is like, ’I needyour help, can you tell me what this is?’ and less like ’Thisperson needs this thing, now explain what they need to do.’”
Unprompted, one participant (P10) in the without-narrative con-dition stated, “The questions didn’t really follow a story. I’d kindalike to fit it in a story.”
However, not all feedback was positive. Someparticipants (P4, P5, P14) felt that the story and interactions were tooshort and would have benefited from more detail, commenting that “alot of the characters were very shallow – like not much personality.” (P4), and that it was regrettable that “for some of the characters, youtalked to them once and never saw them again.” (P14)Finally, while most participants enjoyed the fantasy aspect of thenarrative, two participants (P7, P15) remarked that the usage of fantasycontexts – whether they are interesting or not – made the problemsmore difficult to them:
P07: “I got a bit confused about how they kept talking aboutspells because I’m just not engaged with that overall so Iwasn’t really interested in that, but I think the questions – be-cause they used spells – would both be engaging for peoplewho do enjoy it, and in a way, used in the real world.”
P15: “I felt the drawings [...] kinda engaged you and the ques-tions were really mystical and interesting types of questions. Ithink it would’ve been slightly easier, in fact, with questionsabout the real world, but I think that’s why most mathematicalgames like Prodigy [7] veer off to mythical questions – becausethey’re a bit more difficult to answer.”
There were a few cases (6/33, 5 without-narrative , 1 with-narrative ) where participants visibly appeared to try and brute-forcetheir way through some questions. P2 began brute-forcing solutionsand ignoring feedback when exposed to histogram questions, but stillread the story. For one histogram question, P9 read the question andassociated choices but appeared to be unsure about the solution andtapped on all answers without reading the feedback box. P16 wasobserved to do this for several questions, while P18 did this for mostquestions. One participant (P13, with-narrative ) would occasionallymake a random first selection just to see if the feedback box would givethem a hint. This said, all participants appeared to read and think abouteach question. There were no instances observed where participantswould immediately begin tapping on solutions.When asked about whether they engaged in brute-forcing strategies,participants were honest in self-reporting even if it happened on a singleoccasion. In our review of the sessions, we saw that in many of theseself-reported cases (5/7), participants read feedback and paused to thinkabout what to pick next after some form of minor guessing at first– in other words, children would report uncertainty and guessing asbrute-forcing. The two participants (P16, P18) who were observed toengage in this behaviour admitted to brute-forcing during the interview.P18 did not elaborate on why they did so, but P16 stated: “If I got it wrong the first time, I’d just press the other two. Ijust didn’t think the other two were right when I got it wrongand I didn’t want to go over [the question] again.”
516 also mentioned he did not notice the feedback box at all, likelycontributing to their brute-forcing behaviour. In observing how playersinteracted with the story, we saw that almost all participants took thetime to read through the dialogue, with some exceptions (P5, P16).
Participants in both conditions (6/16 without-narrative , 3/17 with-narrative ) felt that the problems were unclear and lengthy attimes, suggesting that we should shorten the question text. Manyparticipants (15/33) said they would have liked to see more variedtopics beyond charts (e.g., fractions, percentages) or the ability to tar-get specific grade levels. On story elements, 7 ( without-narrative )wanted to see more of a story in the game, while 6 ( with-narrative )would have liked a longer story line or branching narratives. Sevenparticipants wished for more interactive activities, 5 would like to addcharacter customization, and one thought it would be good to includevoiceovers.
ISCUSSION
With the results of the study, we first address (RQ2): the impact ofnarratives on learning and engagement. With respect to learning, wedetected a significant difference between the pre- and post-test scoresfor Q6 in the with-narrative condition. Both Q5 and Q6 requiredparticipants to understand that the intervals on the x-axis should beequal. Given that a handful of participants mentioned they were notfamiliar with histograms, it could be that participants focused solelyon the bars in the pre-test. After learning more about histograms anddeception, participants may have looked at the graphs more carefully inthe post-test. Beyond that, while the overall impact on learning was notstatistically significant, trends based on per-participant performanceprovide further insight on the games and their effects on learning.All children in the youngest age group either improved or exhibitedno change in performance, unlike in other age groups. This, combinedwith comments that the game would be better suited for students newto a topic and past work indicating that narrative text supports learningwhen prior knowledge of the content is low [90], provides support forthe game as a tool to support learning about new concepts. Moreover,it could be used to target children identified as struggling in a particu-lar area or slightly younger audiences, though issues associated withreading comprehension skills may arise. The imbalance in age/gradebetween conditions due to randomized group assignments could alsomean less room for improvement in the with-narrative condition.The usage of narrative elements appeared to negatively affect per-formance of male participants in our study, as well as exacerbate thedegree to which people improve or do worse between the pre- andpost-tests. The reason is unclear, but could perhaps be attributed toplayers becoming too focused on the story and not the educationalcontent, gender-based preferences [14, 29, 44, 91, 92], and/or fatigue.Participants in the with-narrative condition spent nearly double theamount of time playing their version of the game due to the additionof dialogue and exploration views, and we noticed one participant inthis condition was visibly exhausted by the end of the study. Thisparticipant was also recorded to have performed considerably worse inthe post-test. As such, the results of the post-test and interview sessionsin the with-narrative condition may have been negatively impacted.On the impact of narrative elements on engagement, we saw signifi-cant differences in how participants rated their enjoyment of the game,specifically the art, activities, engagement, and fun. Higher ratingsfor the art were likely due to the presence of portraits that appearedduring conversations, on top of the smaller sprites seen by those inthe without-narrative condition. With respect to educational activi-ties, some participants mentioned they appreciated how word problemswere linked to the story and how the story helped make them feel in-vested in the problems they were solving. Meanwhile, improvementsin engagement and fun could be attributed to the presence of explo-ration and story, giving players more to do than answering questions,supporting past findings on experiential play [88] and the benefits con-ferred through engaging with the material beyond simply reading andresponding to questions [23,57,67]. Overall, the inclusion of additional narrative elements had a positive effect on player enjoyment, while notimpeding learning, which is encouraging.
ESIGN C ONSIDERATIONS FOR E DUCATIONAL
RPG S Our game was motivated by design considerations grounded in gamedesign and education literature. We intentionally limited the scopeof our technology probe, so as to make the game amenable to aninitial exploration of our research question (RQ2) in the context of atime-constrained study with child participants. While some guidelinesfor prospective game designers exist [75], there is a lot of freedomafforded to designers with respect to what decisions they can make onvarious factors. In this section, we investigate (RQ1) and present anoverview of important factors for designers to consider when buildingan educational game for visualization literacy, the personal choices wemade, as well as some alternatives and their associated trade-offs.
Children of different ages are expected to have varying levels of read-ing comprehension, affecting the narrative elements one can present.The target audience’s prior familiarity of subject matter also plays acrucial role, identifying what concepts can be built on and possiblyconsolidated, as well as which learning objectives should be addressedin the game. Children are exposed to visualizations as early as grade1, starting with very concrete counting and sorting of physical objects,followed by pictographs, then highly abstracted visualizations such asbar charts [15], but it should be noted that differences arise based onfactors such as grade and location.A comparison of curricula shows that children in France [12], On-tario [11], and various states that have adopted the Common Core StateStandards [10] are expected to begin learning about data managementin grade 1, whereas students in Manitoba [11] begin in grade 2. Withrespect to chart types, the Ontario curriculum expects children to un-derstand histograms and scatter plots by the end of grade 8, while theFrench curriculum mentions scatter plots (“graphiques cartsiens”) andhistograms as early as grades 3 and 4, respectively. Developing an un-derstanding of how visualization is taught at various ages and locations,so as to not make false assumptions about what children already know,is essential to inform the educational content to be covered.In our design process, we validated our content with professionaleducators. To measure readability, designers could also analyze textusing the Flesch-Kincaid readability tests [81]. We designed activitiesthat would be most suitable to learners in grade 7 to see whetheryounger (grades 5 and 6) and older (grade 8) learners may have adifferent response to the built prototype, as it is common for childrento commence learning a novel concept in a grade, then revisit andfurther explore it in subsequent grades. From our study, we saw eitherimprovements or no change in younger grades, but this was not truefor older groups. This does not mean, however, that we should solelyinclude content from higher grades — some children are bound tostruggle with concepts more than others. Personalized experiences(e.g., adjustable difficulty, topic/concept selection) would be helpfulhere, though more work is required to further explore this avenue.
On top of the time spent on the educational content, designers mustalso consider how much time is needed for players to interact with thenarrative elements. In our study, there was not a significant differencebetween the time spent on educational activities between the two condi-tions, but the total time spent in the with-narrative condition broughtthe average play time to double that of the without-narrative condi-tion. This disparity cannot be understated, but the boost to engagementwithout sacrificing learning shows that the inclusion of narratives canbe beneficial. Still, designers must be considerate of the trade-off be-tween time and engagement, and whether it is “worth it” in the contextof their own games. Creative game mechanics where the narrative ispresented mostly in the form of expressive graphics (e.g., as found inthe game Florence [3]) are worth investigating to mitigate time cost.6 o appear in IEEE Transactions on Visualization and Computer Graphics.
Chart
Students are asked to create a chart to represent a given set ofdata and oftentimes also label parts of the chart.
Label
Students are given a chart or table and asked to add labels (e.g.,column names, axes, categories, intervals).
Match
Students are provided with sets of entities (e.g., a set of chartsand a set of sentences/inferences) and asked to and asked todetermine a 1:1 mapping of entities from one set to another.
Multiple Choice
Students are given a question and a set of choices, and arerequired to identify the most-correct response to the question.
Numeric
Students are given a question requiring them to read a chart ortable and report a single number to answer the question.
Plot
Students are given a chart (either incomplete or an empty tem-plate) and asked to plot data points or populate it with data.
Survey
Students are tasked with surveying others and collecting data.
Discussion
Students are given a topic and asked to verbalize their thoughtsin a classroom setting, hearing and responding to input fromfellow students and facilitators (e.g., teachers).
Short Answer
Students are asked a question about some provided data orvisualization and required to give a short textual response.
Long Answer
Similar to short answer questions, but more complete sentences(and possibly justification) are expected .
Table 2. Types of questions for data-related problems.
In school, children are given various types of problems involving datavisualizations [15, 79] (see Table 2), targeting different cognitive re-sources and requiring development of a complementary set of skills,from thinking critically about different options to choose from, to gen-erating explanations for one’s reasoning. Ideally, an educational gamefocused on visualizations should leverage all types of problems bor-rowed from current best practices and research in education, in order tosupport holistic competence development. For instance, prior researchhas shown that response format (e.g., multiple choice) is not nearly asimportant as stimulus format (i.e., context-free or context-rich) [77].We opted for multiple-choice questions due to the ease of imple-mentation, verification, and their ability to support a breadth of topics,and looked at sample exercises and worksheets [4, 5, 9] for guidance.We noticed many chart questions could be classified further into eithercreate a chart-type questions, or interpret the chart-type questions. Thelatter would be simple to translate into multiple choice questions, butthe former would present a challenge unless players are asked to make achoice for each part of the chart (e.g., “Which of the following optionsis best suited as a label for the y-axis in this chart?”). To address this,we made use of questions where players would be presented with differ-ent charts and have to choose the one that best fits a scenario, followedby a question requiring to interpret the graph, allowing learners to bringresolution to the problem at hand.Solving problems related to data visualizations encompasses a va-riety of activities, ranging from data collection, data cleaning andwrangling, data manipulations, and choice of mappings to generalconsiderations about how to best represent the same data to addressspecific questions, think critically about the generated visualization,and ultimately communicate findings. Our game started by tacklingdata-related problems directly translated and adapted from textbooks.Developing a game that encompasses all of these activities goes beyondthe scope of our paper, but we share our reflections on how the inclu-sion of different activities could guide the design of a narrative-basededucational game more generally (§8).One of the challenges with stand-alone educational games lies invalidating answers and providing appropriate guidance and feedback tothe student to support learning. The variety afforded by having humansvalidate responses in traditional settings does not translate well to games– we are restricted in how we can automatically assess answers, affectingthe types of questions we can administer to players (Table 2), as well asthe level of feedback. Responses to closed questions are very easy forcomputers to validate, as solutions are largely binary. Such questionsare suitable to visualization tasks where a clear, unique answer ispossible (e.g., retrieve a value, basic comparisons of two elements,identify an outlier) [16]. While the rigidity of these question formatsfacilitates implementation and assessment, it also acts as a catalyst forthe shortcomings. Numeric- and plot-type questions are limited in what skills they can test and are primarily used for testing basic chart- ortable-reading. Also, the answers for multiple choice and match-typequestions are always visible to learners, making them ill-suited fortopics that may require spontaneous generation of responses [77].Open-ended questions afford more nuanced, richer forms of inputthat are often necessary for the educators to understand the learner’sreasoning process. For data-related problems with visualizations, it ispossible that different analytical paths yield equally reasonable answers,especially when patterns are subtle, data is uncertain, or the problemcan be interpreted differently (i.e., in a problem asking “what makesthese shops successful?”, the concept of success could be ambiguousor hold multiple meanings). Thinking critically about how to approacha problem and use data and visual representations of this data has beenhighlighted as an essential component of visualization literacy [31].Open-ended questions, though typically requiring sophisticated meth-ods of validation, such as natural language processing techniques (e.g.,[47]) or computer vision methods (e.g., color mapping extraction [69]to be used by intelligent tutor agents (e.g., [73])) should be consideredin more advanced, comprehensive educational games. Further effortis required to create a platform that (1) supports interactions betweenstudents and teachers, (2) provides teachers with a way to assess thequality of discussions, and (3) allows for seamless transitions back tothe rest of the game, so as to not impede on player immersion.Other facets to consider include time needed to complete activities,as well as how prone different formats are to brute-forcing strategies.For instance, multiple choice questions can be very quick to answer, butare considerably more vulnerable since learners are given immediatefeedback and may exhaust all possible solutions until they pass. A goodbalance must be struck between time spent on activities compared tothe rest of the game, as well as the variety of problems. If players areconstantly required to answer open-ended questions every few minutes,they may quickly grow bored or frustrated by the game. Designerscould also explore other input methods to see if they add to playfulness(e.g., Florence [3] uses shake, drag, and draw instead of simple clicks).
Data-related problems are commonly presented with stories to providecontext and motivation for resolution of the problem. To the best of ourknowledge, while other educational games with narratives exist [1,2,7],this work is the first to leverage narratives to inform the design of anRPG-like educational game focused on visualization literacy, and eval-uate whether such an approach would be conducive to learning. Withthe exception of larger projects that span over several days and covera range of activities, individual exercises in textbooks and worksheetsare typically presented as being independent and disconnected fromone another, making it difficult for a design to translate and piece theseelements into a singular cohesive and engaging narrative. Below, weaddress a few topics related to narrative design.
There are many ways of integrating activities into a narrative. In games,narratives often follow one of three main models: linear, string ofpearls, or fully-branching [82]. Linear models are very rigid, requiringplayers to follow a specific sequence of events, which represent anappropriate choice if the activities encompassed in the game must all becompleted by the learner to acquire the targeted competencies. Becausedifferent learners may require different emphasis on certain topics, orsome more advanced activities may be provided for a motivated learnerto learn beyond the core set of exercises supported by the main storyline, additional quests could be integrated, following what the gamedesign community refer to as string of pearls. In RPGs, this is oftenaccomplished through side stories or quests that allow players to takea break from the main story line to interact with other characters andlearn more about the game world. Fully-branching games push thiseven further, allowing players to make choices that directly impact thestory and results in players being shown different events given previousdecisions. The latter is particularly valuable in instances where a learnerrequires more practice on a particular topic, as evidenced by the numberof attempts or amount of guidance required to solve the core exercises.7ur design probe employed a linear model, as we wanted to forceeach player to go through the same set of steps, making for fairer com-parisons between subjects in an empirical study, as opposed to othermodels that afford more flexibility, and therefore, possible confounds.Future work should explore how different narrative models support dif-ferent needs and preferences. On a related note, it is worth mentioningthe amount of freedom and agency offered by the different models mayalso impact the enjoyment and player experience, as making playersfeel as if their choices matter acts as a psychological motivator [51, 54].
In word problems, questions are usually presented in a way that intro-duces a character and the problem they are facing. With an overarchingnarrative, an educational RPG can spend some time presenting back-ground information for characters, giving further motivation as to whytheir problems are worth solving. In creating characters, designers mustbe aware of how character design and presence affects factors suchas impact on player immersion, attachment or identification [32, 53],cognitive and affective learning [64], resources required, and time spenton exploration and interactions. We strove for diversity of gender andpersonality, but opted for simple character stories in order to minimizetime required to meet each character. Participants noted the lack ofdepth of characters, and that some were only met once. Designers couldconsider using characters from established universes, as characters fa-miliar to the learner could reduce introductory text, and would build onan already developed attachment and empathy for these characters.
Freytag’s dramatic arc model [39] has been described as “the heart ofany good drama and any good game system” [88]. This arc consistsof five parts: the exposition/introduction, rising action, climax, fallingaction, and catastrophe/denouement). Many stories tend to incorporatethis arc, but vary in how much time they spend on each phase, as wellas the number of times they repeat a phase or set of phases. Priorstudies in education and psychology have shown that stories that followa dramatic arc are more likely to elicit empathy for characters in adultaudiences [18], and are more well-received by children when comparedto stories without dramatic arcs [45]. It should be noted that whileincluding a dramatic arc may increase engagement with the narrative,its effects on learning are unclear and remain to be explored.
Whether or not a story is interesting to an individual is largely due topersonal preferences and very difficult to account for – it is not possibleto create a story that appeals to all audiences. A survey looking atplayer motivations found that gender disparities exist when looking atgenres, e.g., females were found to be more interested in games withhigh fantasy themes as opposed to science fiction [92]. Such findingsmay be used to help guide the creation of narratives.There are many different genres of stories (e.g., science fiction, highfantasy, mystery). For brevity, we generalize and discuss two groups:fantasy and non-fantasy. Fantasy is a large part of the lives of newergenerations [70], and is considered an intrinsic motivator [56]. A studywhere children were offered the chance to interact with a game in eithera fantasy or no-fantasy context, found that children preferred to choosefantasy [68]. Moreover, RPGs generally make use of fantasy stories,so if the goal is to make an educational game similar to commercialones (e.g., perhaps to give the illusion that it is more of a game thana learning tool), it may be intuitive to make use of fantasy elements.However, there are concerns regarding its impact on education.Some literature in education suggests that fantasy hinders the abilityfor young children to transfer knowledge from fictional settings toreal-world scenarios and, instead, recommends realistic content [41, 71,72, 83, 87]. Designers must consider the trade-offs between educationand motivation when choosing a genre. Furthermore, past work hascautioned care in ensuring that usage of fantasy elements does notcompete with educational content for player attention, as increasedmotivation with the context may negatively affect learning [56].
We now discuss points of consideration in the design of the game specif-ically. In weighing options, care must be taken to ensure that the gameadequately supports and facilitates the integration and interleaving ofeducational elements with narrative elements, in order to address learn-ing objectives whilst not interrupting a player’s immersion [34, 89].
Regarding the educational aspect of the game, designers must considerhow they want to convey to players whether they have done somethingcorrectly or incorrectly, as well as how much information or feed-back to present when a player answers a question. Prior research hasshown that support features such as advice is not conducive to learning,whereas feedback and just-in-time information promotes learning [55].Designers must consider, then, how much feedback is to be presented,whether or not the feedback is meaningful, as well as how to displaythis feedback. Preset feedback messages are simple to implement, butrequire designers to anticipate reasons for why a player might haveanswered a question incorrectly.In our study, children were mostly positive about our method ofpresenting feedback, but some noted they would occasionally skim orignore feedback because they believed it would not tell them anythingthey did not already know. Designers should acknowledge that childrenmay ignore the feedback attached to correct answers, or that they willonly selectively read up to the point where they realize what they didwrong. To improve reading rates or engagement with feedback ingeneral, designers could explore the efficacy of personalized feedback,though this would require more effort to implement.
Many RPGs tend to have some sort of exploration involved, allowingplayers to move around the fictional world at their own pace. Ex-ploration is not commonly leveraged in educational games includingnarratives [1, 2, 7]. The ability to freely roam and explore creates theillusion of the game being less linear than its narrative-driven gameplaymay imply [35]. Thus, exploration should not be overlooked in thegame design. Designers should first consider how movement is handledin the game: do players simply tap on hotspots to load a connected mapand start dialogue (like in point-and-click adventures), or do they seeand control characters moving around the actual game space (like inmodern-day RPGs)? The former saves time dedicated to animationsand defining colliders, but may not feel as immersive to players sincethey are not able to explore as freely. One must also consider the sizeof the world and each area or map: a more expansive set of maps maymake the world seem more open and free, but may run the risk ofplayers becoming lost, on top of requiring more time for travel.
Combat or battles are often used in RPGs as a way of challengingand testing players, as they must improve and develop tested skills toprogress. Players are encouraged to complete side quests and defeatlower-leveled enemies in order to accumulate more experience anditems to become stronger and face harder encounters. In an educationsetting, this system could be used in a few ways: players could completeeducational activities to unlock skills or items to make battles easier,players could be asked to complete educational activities in order toinflict damage to enemies in combat, battles could be where playerscollect items [7], or be replaced by educational activities entirely.
Voice acting is commonly used in commercial games to further takeadvantage of auditory streams and add depth to characters and/or thenarrative. In an educational game, voices can be used to help childrenfollow along with the text. Younger children are used to having some-one read to them [13] and the presence of voices could help emulatethis environment, but voice acting requires additional assets, whichmay be costly depending on the level of professional quality.8 o appear in IEEE Transactions on Visualization and Computer Graphics.
Some games support multiplayer capabilities through methods such ascompetition (e.g., player vs. player, leaderboards), socialization, and/orco-operative play. Findings regarding competition in educational gamesappear to be mixed. Some found that competitive elements increasedlearning and motivation [27], while others found the opposite [30].Socialization or discussion with others has been shown to supportlearning [55, 80], but when dealing with young children, designersmust consider ways of moderating discussions, either automaticallyor by requiring a an educator or other adult to monitor the chat toensure the space remains safe for students and conducive to learning.Co-operative play could be used to support collaborative learning, buta game designed specifically for groups could run into issues whereplayers are unable to find others to play with, or simply do not enjoyconnecting with others for the sake of an educational game.For all of these methods, designers must also consider whether ornot supporting these features in the game is actually needed. In a class-room setting, teachers could emulate competition, communication, andco-operative play by setting up in-class leaderboards, holding class-room discussions, and allowing students to play together. Finally, theimpact of these features on individuals can be varied due to personalpreferences. For example, the presence of competitive elements maymotivate those who are naturally competitive, but could add unneces-sary pressure and drive away those who are not.
Games make use of various motivators (e.g., customization, achieve-ments) that we did not address. Designers may be interested in addingthese features into an educational RPG to enrich it, but the effects ofeach motivator on learning has largely yet to be explored. As personalpreferences play a large role in how effective each motivator is foran individual [91], it can be difficult to anticipate the effectivenessof including certain motivators. Unsurprisingly, our participants ex-pressed an array of opinions on all aspects of the game due to personalpreferences: some suggested adding more competitive elements, whileothers noted disdain for such features. On the educational aspect, therewere conflicting answers on how much to focus on education versusplay, as well as topics to concentrate on or cover. Such comments speakto a need for more personalized experiences and/or additional featuresthat children are not required to interact with (e.g., optional puzzles).
ESSONS L EARNED : C
ONSIDERATIONS FOR D ESIGN
Visualization learning objectives affect game design.
The design ofour probe was driven largely by the learning objectives we covered (i.e.,chart reading and interpretation). A multiple choice setup allowed us toquickly and robustly address and test a variety of topics, but considercases where designers may instead choose to focus on visualisationconstruction [21] or exploration of data; the gameplay would haveto accommodate such activities seamlessly, seeking inspiration frommechanics found in existing games where possible (e.g., in Metrico [6],player actions such as jumping and walking impact the visualization).We brainstormed several possible gameplay options before fleshing outthe learning objectives, and found that such an approach constrainsactivity design, potentially limiting their educational value – in otherwords, forcing activities to comply to game mechanics, rather thanprioritizing the quality of the educational content, could hinder thelearning experience. Instead, we recommend that designers first focuson activities independent of game mechanics and visual design, as webelieve one of the main dangers in designing educational games is tofocus more on play rather than learning.In gamification, this balance between learning and play is oftenexplored in stealth learning [78]. While this was not our focus, we dobriefly touch on ways that activity and game designs could accommo-date stealth learning through immersion and other means (§7). Futurework performing a systematic analysis of visualization learning objec-tives specifically, along with educational activity types and suitablegame mechanics would both aid future designers, as well as help organ-ise research efforts investigating what strategies best support learningof different visualization literacy skill sets.
Game design affects visualization design.
The relationship be-tween visualization and game design is, by no means, unidirectional.The design of the game also shapes the visualizations. Designers maysee a need to move away from best practices previously establishedby the visualization community in order to better assimilate chartsand other such assets within their game. For example, the usage ofsharp graphics (as opposed to sketchy graphics [59]) or certain colourschemes may clash with the aesthetic of the other art assets. The cleardistinction between “fun” assets and “serious” assets could break aplayer’s immersion in the game environment and impact their learningand engagement. In our game, we opted for sketchy rendering for thechoose-a-chart questions intentionally as the presented options wereto be seen as visualization templates, rather than representative of theunderlying data. This choice influenced the design of the interpret thegraph questions, which we decided to make consistent, though this mayimpact reading and interpreting the graphs. Future work should exploresuch trade-offs in the context of information visualization specifically.
IMITATIONS
Games are usually played at one’s own pace and players are usually ableto stop and resume playing at different periods throughout the day oracross multiple days. In our study, some participants reported playinggames for less than one hour per week, but we required participantsto focus solely on the game for upwards of 30min in a single session,leading to fatigue. A more true-to-life evaluation would perhaps allowplayers to play in shorter bursts across a longer period of time. Thiswould also address cases where participants felt characters were shallowor that a longer story would be better, as it would give us more freedomto extend the narrative to create a more rich and immersive experience.The design of our pre- and post-tests were also limited despite beingco-designed with educators. We opted for multiple-choice questions tosave time and have a test well-aligned with our games, but it is unclearwhether changes in performance between the two tests were due toactual changes in knowledge or the result of guessing. Open-endedquestions may have given us more insight into how much informationthey acquired from the game, but the time needed to answer questionswould be problematic for a one-hour study. The limited sample sizewas also an issue; studies were conducted during the active schoolterm when students and guardians have busy schedules, and concernssurrounding COVID-19 resulted in early termination of study sessions.There are also limitations caused by the design of our game. Our nar-rative used a fantasy world, but this may only appeal to some children.The implications of our narrative elements on learning and engagementmay not necessarily apply to other genres; more could be done to com-pare non-fantastical to fantastical worlds, and variations of each andhow suitable they may be to address visualization learning objectives.
10 C
ONCLUSIONS
In our work, we looked at how gamification could be used to supportvisualization literacy education and contributed what we believe to bethe first story-based game in this space. Results of our evaluation withchild participants were encouraging with regards to engagement, thoughresults on learning were mostly inconclusive. Qualitative feedbackshowed many conflicting opinions on various topics (e.g., characters,combat), echoing differences in past findings regarding narratives andlearning performance [22, 62]. We outlined considerations for narrativeand game design, and see value in further research studying the impactof each aspect (e.g., narrative genres, motivators), as well as how thesecan be best used to augment visualization teaching and learning, andhope our work will inspire future explorations in this promising area. A CKNOWLEDGMENTS
We thank A. Iannuzziello and C. Reynier for input on tests and activities,W. Szeto for participating in analyzing textbook exercises, S. Hackeekfor her help with recruitment, K.N. Truong, N. Riche, N. Sultanum andanonymous reviewers for their feedback on the manuscript. This workwas supported in part by a grant from NSERC (RGPIN-2018-05072)and University of Toronto FA&S Tri-Council Bridge Funding.9
EFERENCES [1] BrainPOP. . brainpop . com/ .[2] Flocabulary. . flocabulary . com/ .[3] Florence. http://annapurna . pictures/interactive/florence .[4] IXL Learning. https://ca . ixl . com/ .[5] Khan Academy. . khanacademy . org/ .[6] METRICO. https://metrico-game . com/ .[7] Prodigy. . prodigygame . com/ .[8] The Ontario Curriculum Grades 1-8: Mathematics . Ontario Ministry ofEducation, 2005.[9]
Targeted Implementation and Planning Supports for Revised Mathematics(TIPS4RM) . Ontario Ministry of Education, 2005.[10] Common Core State Standards. . corestandards . org/ ,2010.[11] Kindergarten to Grade 8 Mathematics: Manitoba Curriculum Frameworkof Outcomes 2013 . Government of Manitoba, 2013.[12]
Programmes d’enseignement de l’cole lmentaire et du collge . MinistreFranais de l’Education Nationale et de la Jeunesse, 2015.[13]
Kids & Family Reading Report Canadian Edition: The State of Kids &Reading in Canada . Scholastic Inc., 2017.[14]
Essential Facts About the Canadian Video Game Industry . EntertainmentSoftware Association of Canada, 2018.[15] B. Alper, N. Henry Riche, F. Chevalier, J. Boy, and T. M. Sezgin. Vi-sualization Literacy at Elementary School. In
Proceedings of the 2017CHI Conference on Human Factors in Computing Systems (CHI ’17) , pp.5485–5497. ACM, 2017. doi: 10.1145/3025453.3025877[16] R. Amar, J. Eagan, and J. Stasko. Low-level components of analyticactivity in information visualization. In
IEEE Symposium on InformationVisualization, 2005. INFOVIS 2005. , pp. 111–117. IEEE, 2005.[17] A. Amory, K. Naicker, J. Vincent, and C. Adams. The use of computergames as an educational tool: identification of appropriate game types andgame elements.
British Journal of Educational Technology , 30(4):311–321, 1999. doi: 10.1111/1467-8535.00121[18] J. A. Barraza, V. Alexander, L. E. Beavin, E. T. Terris, and P. J. Zak.The heart of the story: Peripheral physiology during narrative exposurepredicts charitable giving.
Biological Psychology , 105:138 – 143, 2015.doi: 10.1016/j.biopsycho.2015.01.008[19] J. A. Betz. Computer games: Increase learning in an interactive multi-disciplinary environment.
Journal of Educational Technology Systems ,24(2):195–205, 1995. doi: 10.2190/119M-BRMU-J8HC-XM6F[20] D. R. Bielenberg and T. Carpenter-Smith. Efficacy of story in multimediatraining.
Journal of Network and Computer Applications , 20(2):151 – 159,1997. doi: 10.1006/jnca.1997.0043[21] F. Bishop, J. Zagermann, U. Pfeil, G. Sanderson, H. Reiterer, and U. Hin-richs. Construct-a-vis: Exploring the free-form visualization processesof children.
IEEE Transactions on Visualization and Computer Graphics ,26(1):451–460, 2020. doi: 10.1109/tvcg.2019.2934804[22] S. J. Bittick and G. K. W. K. Chung. The use of narrative: Genderdifferences and implications for motivation and learning in a math game.2011.[23] M. Boeker, P. Andel, W. Vach, and A. Frankenschmidt. Game-basede-learning is more effective than a conventional instructional method: Arandomized controlled trial with third-year medical students.
PloS one ,8:e82328, 2013. doi: 10.1371/journal.pone.0082328[24] K. B¨orner, A. Bueckle, and M. Ginda. Data visualization literacy: Defini-tions, conceptual frameworks, exercises, and assessments.
Proceedingsof the National Academy of Sciences , 116(6):1857–1864, 2019. doi: 10.1073/pnas.1807180116[25] K. B¨orner, A. Maltese, R. N. Balliet, and J. Heimlich. Investigating aspectsof data visualization literacy using 20 information visualizations and 273science museum visitors.
Information Visualization , 15(3):198213, 2016.doi: 10.1177/1473871615594652[26] J. Boy, R. A. Rensink, E. Bertini, and J.-D. Fekete. A Principled Way ofAssessing Visualization Literacy.
IEEE Transactions on Visualization andComputer Graphics , 20(12):10, 2014. doi: 10.1109/TVCG.2014.2346984[27] N. E. Cagiltay, E. Ozcelik, and N. S. Ozcelik. The effect of competitionon learning in games.
Computers & Education , 87:35 – 41, 2015. doi: 10.1016/j.compedu.2015.04.001[28] A. Cairo. Graphics lies, misleading visuals. In
New Challenges for DataDesign , pp. 103–116. Springer, 2015. doi: 10.1007/978-1-4471-6596-5 5[29] D. Carr. Computer games: Text, narrative and play.
British Journal ofEducational Technology , 40(3):580–580, 2009. doi: 10.1111/j.1467-8535. 2009.00969 5.x[30] C.-H. Chen, J.-H. Liu, and W.-C. Shou. How competition in a game-basedscience learning environment influences students’ learning achievement,flow experience, and learning behavioral patterns.
Journal of EducationalTechnology & Society , 21(2):164–176, 2018.[31] F. Chevalier, N. H. Riche, B. Alper, C. Plaisant, J. Boy, and N. Elmqvist.Observations and reflections on visualization literacy in elementary school.
IEEE Computer Graphics and Applications , 38(3):21–29, 2018. doi: 10.1109/mcg.2018.032421650[32] J. Cohen. Audience identification with media characters. In
Psychologyof entertainment , pp. 201–216. Routledge, 2013.[33] M. Correll and J. Heer. Black hat visualization. 2017.[34] R. Couceiro, M. Papastergiou, M. Kordaki, and A. Veloso. Design andevaluation of a computer game for the learning of information and com-munication technologies (ICT) concepts by physical education and sportscience students.
Education and Information Technologies , 18:531–554,2013. doi: 10.1007/s10639-011-9179-3[35] C. Diane, B. Andrew, S. Gareth, and B. David. Textuality in video games.In
Proceedings of the 2003 DiGRA International Conference: Level Up(DiGRA ’03) , 2003.[36] S. Egenfeldt-Nielsen. Making sweet music: the educational use of com-puter games. 2007.[37] A. Field.
An Adventure in Statistics: The Reality Enigma . SAGE Publica-tions Ltd., 2016.[38] E. Fokides, P. Atsikpasi, P. Kaimara, and I. Deliyannis. Factors influ-encing the subjective learning effectiveness of serious games.
Journal ofInformation Technology Education: Research , 18:437–466, 2019. doi: 10.28945/4441[39] G. Freytag.
Freytag’s technique of the drama . 1863.[40] J. G¨abler, C. Winkler, N. Lengyel, W. Aigner, C. Stoiber, G. Wallner, andS. Kriglstein. Diagram safari: A visualization literacy game for youngchildren. In
Extended Abstracts on Computer-Human Interaction in Play(CHIPlay ’19) , p. 389396. Association for Computing Machinery, 2019.doi: 10.1145/3341215.3356283[41] P. A. Ganea, C. F. Canfield, K. Simons-Ghafari, and T. Chou. Do caviestalk? the effect of anthropomorphic picture books on children’s knowledgeabout animals.
Frontiers in Psychology , 5:283, 2014. doi: 10.3389/fpsyg.2014.00283[42] J. Gee. What video games have to teach us about learning and literacy.
Computers in Entertainment , 1:20, 2003. doi: 10.1145/950566.950595[43] P. M. Greenfield.
Mind and Media: The Effects of Television, Video Games,and Computers . Harvard University Press, Cambridge, MA, USA, 1984.doi: 10.4324/9781315735634[44] B. Gros. Digital games in education: The design of games-based learningenvironments. vol. 40, pp. 23–38, 2007. doi: 10.1080/15391523.2007.10782494[45] L. Guillot and P. Bloom. Dissecting ”Cinderella”: Young children’spreferences for narrative contour. Poster presented at the Society forResearch in Child Development, Seattle, WA, 2013.[46] R. L. Helen Kennedy, Andy Kirk and W. Allen. Seeing data. http://seeingdata . org/ , 2014.[47] L. Hirschman and R. Gaizauskas. Natural language question answering:the view from here. Natural Language Engineering , 7(4):275–300, 2001.doi: 10.1017/s1351324901002807[48] S. Huron, S. Carpendale, A. Thudt, A. Tang, and M. Mauerer. Construc-tive visualization. In
Proceedings of the 2014 Conference on DesigningInteractive Systems (DIS ’14) , pp. 433–442, 2014. doi: 10.1145/2598510.2598566[49] S. Huron, Y. Jansen, and S. Carpendale. Constructing visual represen-tations: Investigating the use of tangible tokens.
IEEE Transactions onVisualization and Computer Graphics , 20(12):2102–2111, 2014. doi: 10.1109/tvcg.2014.2346292[50] Y. B. Kafai. Playing and making games for learning: Instructionist andconstructionist perspectives for game studies.
Games and Culture , 1(1):36–40, 2006. doi: 10.1177/1555412005281767[51] F. Ke and T. Abras. Games for engaged learning of middle school childrenwith special learning needs.
British Journal of Educational Technology ,44, 2013. doi: 10.1111/j.1467-8535.2012.01326.x[52] S.-H. Kim, J. Boy, S. Lee, J. S. Yi, and N. Elmqvist. Towards an OpenVisualization Literacy Testing Platform. In
IEEEVIS 2014 Workshop ,2014.[53] E. A. Konijn and M. N. Bijvank. Doors to another me: Identity construc-tion through digital game play. In
Serious games: Mechanisms and effects , o appear in IEEE Transactions on Visualization and Computer Graphics. pp. 177–201. Routledge, 2009.[54] K. Lee, N. Park, and S. Jin. Narrative and interactivity in computergames , pp. 304–322. Routledge Taylor & Francis Group, 2006. doi: 10.4324/9780203873700[55] H. Leemkuil and R. De Hoog. Is support really necessary within educa-tional games? 2005.[56] M. R. Lepper and T. W. Malone. Intrinsic motivation and instructionaleffectiveness in computer-based education. 1987.[57] D. Leutner. Guided discovery learning with computer-based simulationgames: Effects of adaptive and non-adaptive instructional support.
Learn-ing and Instruction , 3(2), 1993. doi: 10.1016/0959-4752(93)90011-n[58] D. Lieberman.
What Can We Learn From Playing Interactive Games? , pp.379–397. 2006.[59] S. Liu, W. Cui, Y. Wu, and M. Liu. A survey on information visualization:recent advances and challenges.
The Visual Computer , 30(12):1373–1393,2014.[60] Y. Liu, N. Yan, and D. Hu. Chorlody: A music learning game. In
ExtendedAbstracts on Human Factors in Computing Systems (CHI ’14) , pp. 277–280. ACM, 2014. doi: 10.1145/2559206.2580098[61] N. Lyon, J. Valls-Vargas, C. Guevara, N. Shao, J. Zhu, and J. Zhu. Littlenewton: an educational physics game. In
Proceedings of the 2014 SIGCHISymposium on Computer-Human Interaction in Play (CHIPlay ’14) , pp.351–354, 2014. doi: 10.1145/2658537.2662985[62] D. M. Adams, R. Mayer, A. Macnamara, A. Koenig, and R. Wainess.Narrative games for learning: Testing the discovery and narrative hypothe-ses.
Journal of Educational Psychology , 104:235–249, 2012. doi: 10.1037/a0025595[63] A. V. Maltese, J. A. Harsh, and D. Svetina. Data visualization liter-acy: Investigating data interpretation along the noviceexpert continuum.
Journal of College Science Teaching , 45(1):84–90, 2015. doi: 10.2505/4/jcst15 045 01 84[64] A. Paiva, J. Dias, D. Sobral, R. Aylett, S. Woods, L. Hall, and C. Zoll.Learning by feeling: Evoking empathy with synthetic characters.
Ap-plied Artificial Intelligence , 19(3-4):235–266, 2005. doi: 10.1080/08839510590910165[65] A. V. Pandey, K. Rall, M. L. Satterthwaite, O. Nov, and E. Bertini. Howdeceptive are deceptive visualizations? an empirical analysis of commondistortion techniques. In
Proceedings of the 2015 CHI Conference onHuman Factors in Computing Systems (CHI ’15) , p. 14691478. ACM,2015. doi: 10.1145/2702123.2702608[66] C. H. Papadimitriou. MythematiCS: In Praise of Storytelling in the Teach-ing of Computer Science and Math.
SIGCSE Bulletin , 35(4):7–9, 2003.doi: 10.1145/960492.960494[67] S. Papert and I. Harel. Situating constructionism. In
Constructionism .Ablex Publishing Corporation, 1991.[68] L. Parker and M. Lepper. Effects of fantasy contexts on children’s learningand motivation: Making learning more fun.
Journal of Personality andSocial Psychology , 62:625–33, 1992. doi: 10.1037/0022-3514.62.4.625[69] J. Poco, A. Mayhua, and J. Heer. Extracting and retargeting color mappingsfrom bitmap images of visualizations.
IEEE Transactions on Visualizationand Computer Graphics , 24(1):637–646, 2017.[70] M. Prensky. Digital game-based learning.
McGraw-Hill, New York , 1,2001. doi: 10.1145/950566.950567[71] R. A. Richert, A. B. Shawber, R. Hoffman, and M. B. Taylor. Learningfrom fantasy and real characters in preschool and kindergarten. 2009. doi:10.1080/15248370902966594[72] R. A. Richert and E. I. Smith. Preschoolers’ quarantining of fantasy stories.
Child Development , 82(4):1106–1119, 2011. doi: 10.1111/j.1467-8624. 2011.01603.x[73] K. Rivers and K. R. Koedinger. Data-driven hint generation in vast solutionspaces: a self-improving python programming tutor.
International Journalof Artificial Intelligence in Education , 27(1):37–64, 2017. doi: 10.1007/s40593-015-0070-z[74] M. Romero, M. Velez, G. McInemy, D. Silver, and M. Chen. TowardsVisualization Literacy. In
EuroVis Workshop , 2014.[75] K. Salen and E. Zimmerman.
Rules of Play: Game Design Fundamentals .The MIT Press, 2003.[76] E. F. Schneider. Death with a story.
Human Communication Research ,30(3):361–375, 2004. doi: 10.1111/j.1468-2958.2004.tb00736.x[77] L. Schuwirth and C. Van der Vleuten. Different written assessment meth-ods: What can be said about their strengths and weakness?
Medicaleducation , 38:974–9, 2004. doi: 10.1111/j.1365-2929.2004.01916.x[78] L. A. Sharp. Stealth learning: Unexpected learning opportunities throughgames.
Journal of Instructional Research , 1:42–48, 2012.[79] T. L. Shreiner. Data literacy for social studies: Examining the role of datavisualizations in k–12 textbooks.
Theory & Research in Social Education ,46(2):194–231, 2018. doi: 10.1080/00933104.2017.1400483[80] K. Sobel, G. Kovacs, G. McQuillen, A. Cross, N. Chandrasekaran, N. H.Riche, E. Cutrell, and M. R. Morris. Edufeed: A social feed to engagepreliterate children in educational activities. In
Proceedings of the 2017ACM Conference on Computer Supported Cooperative Work and SocialComputing , pp. 491–504, 2017.[81] M. Solnyshkina, R. Zamaletdinov, L. Gorodetskaya, and A. Gabitov. Eval-uating text complexity and flesch-kincaid grade level.
Journal of SocialStudies Education Research , 8(3):238–248, 2017.[82] C. Stone. The evolution of video games as a storytelling medium, and therole of narratives in modern games.
Gamasutra , 2019.[83] G. A. Strouse, A. Nyhout, and P. A. Ganea. The role of book features inyoung children’s transfer of information from picture books to real-worldcontexts.
Frontiers in Psychology , 9:50, 2018. doi: 10.3389/fpsyg.2018.00050[84] A. Tal and B. Wansink. Blinded with science: Trivial graphs and for-mulas increase ad persuasiveness and belief in product efficacy.
Pub-lic Understanding of Science , 25(1):117–125, 2016. doi: 10.1177/0963662514549688[85] C. Taylor. New kinds of literacy, and the world of visual information.
Literacy , 2003.[86] O. Wagih. Guess the Correlation. http://guessthecorrelation . com/ , 2018.[87] C. M. Walker, A. Gopnik, and P. A. Ganea. Learning to learn from stories:Children’s developing sensitivity to the causal structure of fictional worlds. Child Development , 86(1):310–318, 2015. doi: 10.1111/cdev.12287[88] H. Wang and A. Singhal.
Entertainment-Education through Digital Games ,pp. 271–292. 2009.[89] W. Westera. Why and how serious games can become far more effective:Accommodating productive learning experiences, learner motivation andthe monitoring of learning gains. vol. 22, pp. 59–69, 2019.[90] M. W. Wolfe and J. A. Mienko. Learning and memory of factual contentfrom narrative and expository text.
The British journal of educationalpsychology , 77:541–64, 2007. doi: 10.1348/000709906x143902[91] N. Yee. 7 things we learned about primary gaming motivations from over250,000 gamers.
Quantic Foundry , 2016.[92] N. Yee. Beyond 50/50: Breaking down the percentage of female gamersby genre.
Quantic Foundry , 2017., 2017.