Automating Gamification Personalization: To the User and Beyond
Luiz Rodrigues, Armando M. Toda, Wilk Oliveira, Paula T. Palomino, Julita Vassileva, Seiji Isotani
PPREPRINT SUBMITTED TO IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES 1
Automating Gamification Personalization:To the User and Beyond
Luiz Rodrigues, Armando M. Toda, Wilk Oliveira, Paula T. Palomino, Julita Vassileva, Seiji Isotani
Abstract —Personalized gamification explores knowledge aboutthe users to tailor gamification designs to improve one-size-fits-all gamification. The tailoring process should simultaneouslyconsider user and contextual characteristics (e.g., activity to bedone and geographic location), which leads to several occasionsto tailor. Consequently, tools for automating gamification person-alization are needed. The problems that emerge are that which ofthose characteristics are relevant and how to do such tailoring areopen questions, and that the required automating tools are lack-ing. We tackled these problems in two steps. First, we conductedan exploratory study, collecting participants’ opinions on thegame elements they consider the most useful for different learningactivity types (LAT) via survey. Then, we modeled opinionsthrough conditional decision trees to address the aforementionedtailoring process. Second, as a product from the first step, weimplemented a recommender system that suggests personalizedgamification designs (which game elements to use), addressingthe problem of automating gamification personalization. Ourfindings i) present empirical evidence that LAT, geographiclocations, and other user characteristics affect users’ preferences,ii) enable defining gamification designs tailored to user andcontextual features simultaneously, and iii) provide technologicalaid for those interested in designing personalized gamification.The main implications are that demographics, game-relatedcharacteristics, geographic location, and LAT to be done, as wellas the interaction between different kinds of information (userand contextual characteristics), should be considered in defininggamification designs and that personalizing gamification designscan be improved with aid from our recommender system.
Index Terms —Gamified Learning; Personalization; Educa-tional System; Recommender Systems; Context-aware.
I. I
NTRODUCTION T O improve learning technologies capability of engag-ing and motivating users, practitioners and researchersstarted to employ gamification: the use of game elementsin contexts that are not games [1], [2], [3]. Overall resultsfrom these applications are positive, showing improvements inlearning outcomes such as academic achievement, conceptualand application-oriented knowledge, and motivation to learn[4]. However, there are situations in which gamification isineffective in impacting learning outcomes, or even negative[5], often due to poorly designed gamification [6], [7], suchas assuming that the same choices will work for all users, theone-size-fits-all approach [8], [9]. To overcome such failures,
L. Rodrigues, A. M. Toda, W. Oliveira, P. T. Palomino, and S. Isotaniare with the Laboratory of Applied Computing to Education and AdvancedSocial Technology, Institute of Mathematics and Computer Science, Univer-sity of S˜ao Paulo, S˜ao Carlos, Brazil. E-mail: { lalrodrigues, armando.toda,wilk.oliveira, paulatpalomino } @usp.br, [email protected]. J. Vassileva iswith the Multi-User Adaptive Distributed Mobile and Ubiquitous Com-puting (MADMUC) Lab, Department of Computer Science, University ofSaskatchewan, Saskatoon, Canada. E-mail: [email protected]. researchers started to investigate personalized gamification[10].Personalized gamification concerns exploring knowledgeabout the users to enable providers (e.g., an instructor or thesystem itself) to offer game elements tailored to those users[11]. For instance, a case would be a system or instructorchanging from game elements set A to game elements set Bwhen users are females because the latter is more tailoredto these users. The premise for personalizing gamificationemerged from discussions that people with, for instance,different demographic characteristics and cultural backgroundhave distinct preferences [12], [13], behaviors [14], and aremotivated differently [15] and, consequently, might experienceand respond to the same conditions in distinct ways [16], [17].The common practice for gamification is selecting which gameelements to add to the system from a list of available elements[18], [19]. Accordingly, researchers invested in providingrecommendations indicating which game elements suit betterusers of different groups to provide personalized gamification,predominantly based on their preferences (e.g., [15], [10],[20]).Mainly, those recommendations are based on users’ behav-ioral profiles [21], [22], [23]. However, the application contextis relevant for gamification’s success as well [18], [24], andgamification designs should be aligned to it [9]. Furthermore,multiple factors (e.g., users’ demographics [25], [26], [27] andthe system’s context [7], [28]) moderate users’ experience,either positive or negatively, but tailoring approaches oftenconsider a single one (e.g., [29], [30], [31]). These limitationsreflect current gaps in the field of personalized gamification,highlighted in recent literature reviews [32], [21]; the facts thati) personalization models should consider more than users’characteristics, such as encompassing the learning activitiesand geographic locations, and that ii) personalization methodsshould consider multiple aspects simultaneously, as well astheir interactions.To address these gaps, we sought to understand how to tailorgamified systems to the education domain by considering thelearning activity at hand, the user’s characteristics, and thegeographic location simultaneously, as well as the interactionsbetween all aspects taken into account. To achieve that goal,we performed an exploratory, survey-based research to cap-ture users’ preferences, a methodology that has been widelyaccepted and adopted by related research, as personalizationis often based on user preference [19], [8], [10], [33]. As thisdevelopment process is concerned with understanding whichaspects (i.e., among learning activity at hand, user’s charac-teristics, and the geographic location) affect user preference, a r X i v : . [ c s . H C ] J a n REPRINT SUBMITTED TO IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES 2 as well as the most suitable game elements for each aspectscombination, we sought to answer the following researchquestions: • RQ1 : Does users’ preferences differ depending on (a)their characteristics, (b) geographic location, and (c) thetype of the learning activity to be performed? ; • RQ2 : What is the most useful game elements set, fromusers’ preferences, according to their characteristics,geographic location, and Learning Activity Type (LAT) ? RQ1 and RQ2 are interrelated as the first informs thesecond in terms of which characteristics should be consideredwhen defining the most useful game elements set. Then,the challenge that emerges is that interactions from multiplecharacteristics lead to several combinations; for instance, fivebinary characteristics would lead to 25 combinations, thus, 25recommendations, while the number of recommendations forfive three-valued characteristics would exponentially increase.Consequently, it becomes of utmost importance providing away to automate such recommendations, which corroboratesanother challenge of personalized gamification: automatingthe personalization process [32]. Therefore, as a product ofour answers, we implement a Recommender System (RS) forpersonalized gamification [34] that can be used to be informedon the most useful game elements set, according to userspreferences, given an input of user’s characteristics and theirgeographic location along with the LAT to be performed. Thus,our main contributions are the following:1) Evidence, from users’ preferences, that can be used toinform researchers and practitioners on how to tailorGamified Educational Systems (GES) to LAT, geo-graphic location, and user characteristics;2) An RS to automate gamification personalization, whichperforms recommendations by considering multiple as-pects simultaneously (i.e., user characteristics, geo-graphic location, and LAT), enabling the implementationof gamification designs more aligned to their prefer-ences; and3) Demonstrating which user characteristics impacted theirpreferences, along with the degree of each one’s influ-ence; thus, one might decide which user characteristic toprioritize, take into account, and/or pay more attentionas, for instance, moderators of gamification’s effective-ness. II. L
ITERATURE R EVIEW
This section provides background information on the topicscovered by this paper, reasons about the literature to justifyresearch choices, and highlights the contribution our studyprovides to existing literature compared to similar works.
A. Game elements
There are many definitions and categorizations of gameelements. In the scope of this paper, we consider game ele-ments similar to the game element definition adopted by [19], In the scope of this study, a LAT is defined based on its main expectedoutcome (see Section II for further details) which are the building blocks impacting users’ experiencewith the system, that are characteristic to gameful systems [2],following the vocabulary used more often by similar research[21].Given the numerous game elements available, it has beencommon practice for each study to self-select which set ofthose elements to use. Based on a literature review, [19]presented 59 general elements. In [24], the authors reviewedthe literature to select 12 common game elements, withoutconsidering any content game element [1] due to the genericnature of their research. In both studies, game elements wereselected with no consideration for the domain application,according to their purposes. Differently, [10] explored anelement set created from gamification on education literature[35], which is composed by eight options.Given that our research focuses on a specific domain,education, this paper differs from [19], [24] by exploring ataxonomy [36] containing the most common game elements(N = 21) from GES. This taxonomy was created through arigorous, systematic process, and was validated by 19 expertsin the field of gamification and games, whereas [10] relied on asimpler, reduced game elements set, which was created basedon a literature review. Furthermore, by selecting an expert-validated taxonomy, we ensure the game elements available arewell defined, avoid using elements with the same purpose butdifferent names, and prevent possible bias from the selectionprocess. On top of that, the selected taxonomy also providesguidance on how the elements are expected to affect users[37], another advantage to those using it [19].
B. Personalized Gamification
Personalizing information systems is an important aspectthat should be available to enhance these systems’ relevanceto users [9]. Within the scope of gamified systems, a commonpractice to achieve personalization has been to tailor the gam-ification design (set of game elements) to specific user’s char-acteristics (e.g., [29], [8]). In other words, gamified systemshave been personalized by performing static adaptations on thegame elements it features, based on pre-defined characteristics(i.e., behavioral profile), to tailor the gamification designs [21].A recent literature review [21] has found that informa-tion used to drive personalization are, predominantly, users’player/gamer types [22], [23], followed by personality [38].Nevertheless, it has been shown that other user characteristics,such as gaming habits [39] and gender [40], also impact theirpreference, as well as the relationship between user demo-graphics (i.e., age and gender) and player types [22] suggestthe impact of those aspects. In spite of that, these aspectshave been rarely explored in methods for tailoring gamificationdesigns in education [41]. This research addresses this need byintroducing an approach that exploits demographic and gaminghabits as information used to drive the gamified designs’tailoring.Furthermore, the user is not the only factor to be consideredwhen defining gamification designs. A factor that has beenoften discussed as relevant for gamification effectiveness [18],[4], [25], [21], [9], [42], that is rarely considered by tailoring
REPRINT SUBMITTED TO IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES 3 method, is the application context (e.g., geographic location).Specifically in the context of educational systems, an aspect re-searchers have recently argued as relevant, and recommendedto consider when tailoring gamified systems, is the learningactivity [43], [21]. This is related to the recommendation thatgamified designs should match the task [9] and, given thattasks of educational systems are almost ever learning activities,personalizing the gamified designs to these activities should beaccomplished.Despite that, to the best of our knowledge, there are onlytwo approaches for personalizing gamified designs based onlearning activities [33], [20]. In [20], learning activities areconsidered based on their main expected objective, similar tothis article. In [33], the learning activities are activities fromMoodle (e.g., forum and quizzes). Hence, while recommenda-tions from [20] can be extended to any learning activity (linkedto their objective), those from [33] are limited to a specific setof Moodle activities. In addition, both works consider one usercharacteristic, personality trait and player type, respectively.Thus, they provide valuable contributions in terms of exploringlearning activities, as well as presenting recommendations thatconsider the interaction between those and a user characteristic(e.g., player type X, learning activity Y).However, these studies fall into the category of methods thatrely on the most often researched user characteristic, a singleuser characteristic is considered in each one, and the guidelinefrom [33] cannot be generalized to any learning activity.Therefore, the main advances of this article compared to thoseworks are: i) considering multiple user characteristics rarelyexplored simultaneously, ii) taking the context into accountvia learning activities and users’ geographic location, andiii) providing recommendations that consider the interactionbetween all of those aspects that are relevant for users.
C. Learning Tasks
To generally describe a task, one might rely on its desirableoutcomes, behavioral requirements, and/or complexity [44],[9]. Similarly, from the human-computer interaction perspec-tive, a task refers to the activities required to achieve a specificgoal [45]. Consequently, given the context of our study, alearning task refers to a set of activities that aim at someeducational outcome. From this definition, it is possible to notethat numerous tasks might be found in GES, which makesit infeasible to develop a specific personalization approachfor each one. An alternative to that limitation is categorizingthe activities, which can substantially reduce their quantity;consequently, enabling the recommendation of gamificationdesigns to each category.To overcome the numerous learning tasks and categorizethem, we opted to rely on the revision of Bloom’s taxonomyof educational objectives [46], an approach that contributes tothe learning process by matching the educational activities’gamification designs to a cognitive taxonomy [20]. Althoughthere are other options available, the revision of Bloom’staxonomy is a widely cited, well-accepted taxonomy, similarto its original version [47]. It acts as a framework that can beused to classify what is expected from an educational activity (outcome), as well as its complexity [46]. The revised versionis composed of two dimensions: knowledge (concerned withwhat is to be learned; e.g., the subject of matter) and cognitiveprocess (concerned with actions associated with learning; e.g.,how to learn) [46].In the scope of this research, we consider the seconddimension, similar to related work [20]. By categorizinglearning activities based on the cognitive domain of such ataxonomy, we avoid having the gamification focused on theactivity itself (e.g., completing a quiz or answering a forum)and allow it to be aligned with the activity’s expected learningoutcome, addressing the recommendation that gamificationshould match the task [9]. Moreover, as many GES featuretasks of varied subjects, the second dimension choice makesthe approach subject-independent, focusing the gamificationdesigns’ tailoring on the activities’ particular objectives whileallowing it to be used regardless of the system’s educationaltopic.The structure of the cognitive process dimension is split intosix categories: remember, understand, apply, analyze, evaluate,and create. Here, we consider each dimension a differentLAT, wherein their complexity increases following the orderin which they were introduced (i.e., remember is the lesscomplex and create is the most complex). Hereafter, we referto those as LAT1 to LAT6, also following the introducedorder. Furthermore, although an activity might fit in morethan one LAT, our approach considers every activity will havea predominant, main objective to be achieved. Hence, thepersonalization process should be based on that main goal. Itis worth noting that those LAT might be split again, however,we opted to work with the high-level abstraction given that thesimilarities within these sub-categories might be even higher.Thus, this paper contributes a proposal that is based on thesix high-level types of cognitive processes established in [46],that aids in tailoring gamification designs to different LAT,according to their predominant goal.
D. Recommender Systems for Personalized Gamification
An RS can be seen as a technique, or software tool,able to recommend items to users [48]. Such systems areespecially valuable for cases in which several options areavailable, alleviating the burden of human selection by pro-viding recommendations, often based on what other peoplerecommend. Common applications of such systems are e-commerce, movies, and music. Recently, the use of RS hasbeen suggested for personalized gamification [34], which cor-roborates to our research in terms of, for instance, reducing theburden of selecting the most suitable game elements for severalcombinations of user characteristics, geographic location, andLAT. Next, we provide a brief overview of RS for personalizedgamification following the framework by [34].RS have three main elements: inputs, outputs, and process.Inputs concerns all the aspects that are received by the RS to betaken into account before doing the recommendations. Thereare four main types of input: user profile (e.g., demographics,personality, behavioral profile), items (e.g., game elements),transactions (e.g., the relationship between users and items;
REPRINT SUBMITTED TO IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES 4 using or preferring a game element), and context (e.g., ge-ographic location, activities to be done). Outputs are ratingsrelated to the choices that the RS made from the input received.For instance, if items are game elements, the output wouldbe the rating of each one. The process is the core part ofthe RS, concerning the method through which it will performthe recommendations. There also are four main recommen-dation methods. Content-based recommenders are based onknowledge of the application, such as data log, or empiricaland theoretical information. The collaborative filtering methodexclusively depends on data collected implicitly or explicitlyfrom interactions with a system. Context-aware recommendersare those that explore information of the context to make theirchoices. Lastly, hybrid recommenders aim at using two ormore of the previous approaches together.Generally, there is a lack of technological support forgamifying educational environments [49]. Accordingly, theliterature on personalized gamification lacks concrete RS im-plementations, demonstrate by recent literature reviews findingonly four studies that relate to RS or other forms of au-tomating gamification personalization. Among those, one isthe framework proposal itself [34], whereas the remaining aretheoretical/conceptual models with no concrete implementa-tions available for third-parties use [16], [50], [51]. Differently,we present and provide an RS for personalizing gamification,which was built upon findings from the study of this article.Hence, we advance the literature with a free, hybrid RS asit uses both contextual as well as empirical information fromusers’ preferences.
E. Summary
Table I summarizes and demonstrates the points in whichthis study differs from related works based on the discus-sion previously presented. As shown, most studies focus onuser characteristics, few consider the task to be done, andnone but this one take into account geographic locations.Additionally, the few works that consider information fromthe user and the task provide recommendations based ontwo factors (one from each kind). On the other hand, ourapproach was developed considering nine aspects, of whicheight were found to be significant (see Section IV) and,therefore, are considered in the product from our research(see Section V). This final product is another key difference.Whereas previous research only provides conceptual/visualguidelines, this study contributes with technological aid for thedesigning of personalized gamified systems. This also differsfrom research on recommender systems for gamification [16],[50], [51] as those provide no concrete implementations fromtheir proposals. III. S
TUDY
The goal of this research is to understand how to tailorGES to LAT, geographic location, and users’ characteristic.To achieve that goal, we addressed two research questions, asoutlined in Section I. To answer them, we performed a survey-based research asking participants to indicate their preferredgame elements for each LAT. Up to date, this methodology is
TABLE IS
UMMARY COMPARISON OF RELATED WORKS .Recommends game elements based onStudy User Task GL N Factors Product[19] X Conceptual[24] X Conceptual[10] X Conceptual[22] X Conceptual[39] X Conceptual[40] X Conceptual[33] X X 2 Conceptual[20] X X 2 ConceptualThis X X X 8 TechnologyGL = Geographic location. the most used by similar works [32] and has been widely ac-cepted given the number of related research following it [19],[8], [10], [33]. Therefore, we considered it the most adequateapproach to adopt. This study also follows an exploratoryapproach, which aims to understand possible relations betweenthe observable variables, in order to create possible researchguidance [52]. This section presents an overview of this studydevelopment process, as well as further describes the materialand methods followed.
A. Overview
In developing this research, three factors had to be defined:what domain, how to interpret the tasks, and which usercharacteristic to consider.
First , we opted for the educationdomain, which is the one gamification research has focusedthe most [18] and, both positive [4] and negative [5] out-comes have been found, showing the need for further re-search.
Second , given the domain, users will perform learningactivities when using the gamified systems; as one mightcreate numerous of those activities, our approach considersactivities types, based on the revised Bloom’s taxonomy [46],an established, well-accepted taxonomy within the educationalcontext.
Third , we chose to focus on users’ demographiccharacteristics and gaming habits and preferences, deepeninginto aspects that have been discussed as relevant factors[25], [53], [39] but received less attention from the academiccommunity compared to the most used ones [24].Then, to achieve the desired understanding, we developedConditional Decision Trees (CDT), which takes into accountthe interactions between all input variables to determine themost suitable game element for that input set. During datacollection, we operationalized gamification designs as the topthree game elements subjects prefer the most, provided gameelements (N = 21) extracted from the expert-validated taxon-omy by [36], and operationalized LAT as the six cognitiveprocess types defined in [46]. As we considered gamificationdesigns to be composed of participants’ top three game ele-ments, ranked by their preferences.
B. Procedure
The following five steps were performed to develop ourapproach for tailoring gamified designs to LAT and users.1)
Survey development : defining the survey design andsections and the game elements and LAT to consider;
REPRINT SUBMITTED TO IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES 5 Data collection : disclosing the survey online, throughAmazon’s Mechanical Turk (MTurk), to collect partic-ipants opinions.3) Data analysis : running analyses to identify which char-acteristics impact users’ preferences.4)
Users preferences analysis : investigating our findingsto identify how to tailor educational systems’ gamifica-tion designs to users, geographic location, and LAT.5)
RS design : developing a free, ready-to-use resource,based on our findings, to aid those who want to tailortheir educational systems’ gamified designs.
C. Survey
The survey was developed online and can be viewed inthe appendix. Its design was defined in four steps. First,two researchers brainstormed and developed an initial ver-sion. Second, three other researchers revised it and providedfeedback on how to improve it. Following, the survey wasimproved accordingly and, lastly, we ran a pilot study with 50participants.The final version has four sections: consent form, demo-graphics, gaming background, and preferences. In the consentform , all respondents were informed to be participating ina research and agreed all information provided would beused to research ends only. The demographics and gamingbackground captured participants’ gender, age, living country,highest level of education, and MTurk identifier to avoidrepeated completions and for how many years the participantsresearched/worked with gamification (0 for those who didnot), how much time (in hours) they spend with games perweek, and their preferred game genre and playing setting,respectively. Lastly, in the preferences section, participantsranked the top three game elements they prefer the most whenperforming each of the six LAT.The 21 game elements available were: Acknowledg-ment; Chance; Competition; Cooperation; Economy; ImposedChoice; Level; Narrative; Novelty; Objectives; Point; Pro-gression; Puzzles; Rarity; Renovation; Reputation; Sensation;Social Pressure; Stats; Storytelling; Time Pressure. Furtherdescriptions of these elements can be seen in the studyprovided by [36]. The LAT are those introduced in Section II(remember, understand, apply, analyze, evaluate, and create).For further information about each one, see [46]. Thus, the lastsection had seven items, one for each LAT and a repeated itemto assess participant attention/consistency (see next section).Each of those seven items had three sub-items, allowing theparticipant to select the rank-one, -two, and -three game ele-ments, in which the 21 game elements were possible answers.Nevertheless, the same game element could not be selectedtwice within the same item, that is, each participant’s topthree should be composed of three different game elements. Asample question was Indicate the three gamification elementsyou consider will help you the most when performing anactivity you need to REMEMBER something (e.g., rememberwhat the ‘+’ symbol means in arithmetic operations). , whereas Online survey: http://bit.ly/2JWxwqs other items of the same section differed only in the LAT (e.g.,understand instead o remember) and the example at the endof the item. All items had basic mathematical examples dueto the generality of the topic.Additionally, we highlight that this top-three survey designwas adopted due to the number of both game elements (21) andLAT (six), which would lead to a questionnaire with 126 itemsif subjects should, similar to related work [19], [10], providea rating for each gamification element through a Likert-scale.That is, participants would answer to 21 items six times; onetime per LAT. Thus, we opted for one item per LAT, featuringthree options each, to reduce effort, tiredness, and time spent incompleting the survey aiming to improve answers’ reliability.Lastly, note that the survey sections’ order was fixed (the sameas previously introduced) but, within each section, the items’order was randomized.
D. Data Collection and Filtering
We recruited participants through crowdsourcing (MTurk).We made this choice to increase our sample size, similarly torelated research (e.g., [24], [19]), an approach that has beenrecommended in the literature [54], [55] to improve externalreliability [56]. No participant restriction was enforced toavoid selection biases and everyone who completed the surveyreceived a fixed remuneration.Nevertheless, recent studies (e.g., [24], [19]) have employedadditional items to the survey’s long sections to assess whetherparticipants are paying attention and providing consistentanswers. Based on those specific items’ answers, researchersfilter participants according to some assertion threshold (e.g.,discarding those who failed in more than one item [24]). Inthis study, we adopted a similar approach. On the preferences section, we added a repeated question for one LAT, whichallowed us to assess whether the participant was consistentin his or her answer (i.e., did they select the same top-threegame elements in both items?). Participants’ remuneration wasnot conditional to consistently answering, neither participantswere warned about the repeated item, aiming to improve thereliability assessment.Following related work, we adopted a tolerance for in-consistent completions. Hence, discarding all participants thatprovided consistent answers in less than two out of the threegame elements. For instance, one selected Acknowledgment,Chance, and Competition and, then, in the repeated question,selected Acknowledgment, Cooperation, and Economy. Thisparticipant would be discarded by selecting two different gameelements for the same question. In total, 1018 individuals havecompleted the survey, from which 657 answers were discardedbased on our criteria. Thus, the final dataset contains 361consistent answers. The description of these reliable, validanswers is shown in Table II.Overall, our sample is composed of adults (51.5% males,47.4% females, and 1.1% others) with 32 years on average( ±
11) and undergraduate or higher degrees (65.4%). Hence,we might expect our sample to feature responsible peoplewith good educational background. Furthermore, despite thelarge majority never researched gamification (91%), there
REPRINT SUBMITTED TO IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES 6 is an interesting variation in their preferred playing setting(Singleplayer: 59%; Multiplayer: 41%) as well as game genre(20% for the most preferred genre: Role Playing Game), withan overall playing time of 12 hours ( ±
13) per week. Thereby,we might expect participants to be familiar with games andtheir elements.
TABLE IID
ATASET DESCRIPTION .Value N(%) Value N(%)
Gender Preferred game genre
Female 186 (0.515) Role Playing Game 75 (0.208)Male 171 (0.474) Adventure 61 (0.169)Other Gender 4 (0.011) Action 60 (0.166)
Country
Strategy 50 (0.139)United States 259 (0.717) Other Genre 115(0.319)India 22 (0.061)
Preferred playing setting
United Kingdom 20 (0.055) Singleplayer 214(0.593)Canada 18 (0.050) Multiplayer 147(0.407)Brazil 16 (0.044)
Researched gamification
Italy 6 (0.017) No 329 (0.911)Germany 5 (0.014) Yes 32 (0.089)Spain 3 (0.008)
Age
Australia 2 (0.006) Mean 32.615Netherlands 1 (0.003) SD 11.299Albania 1 (0.003) Min. 18.000France 1 (0.003) 25% 24.000Ireland 1 (0.003) 50% 29.000Poland 1 (0.003) 75% 39.000Turkey 1 (0.003) Max. 75.000Austria 1 (0.003)
Weekly playing time (hours)
Nigeria 1 (0.003) Mean 12.874Belize 1 (0.003) SD 13.782Jamaica 1 (0.003) Min. 0.000
Highest education level
25% 4.000Undergraduate 161 (0.446) 50% 10.000High School 81 (0.224) 75% 20.000MsC 63 (0.175) Max. 112.000Technicaleducation 30 (0.083)Other Education 14 (0.039)Ph.D 12 (0.033)
E. Data Analysis
For data analyses, we worked with the party
R package[57]. A decision tree is a classification algorithm that selectsan output based on the interaction between elements from aninput set [58]. Besides handling interactions, which is a keypoint from our objective, the decision tree algorithm providesanother three positive points that led us to choose it. First, itallows visualizing the rules followed to determine the output.Therefore, we can comprehensively discuss and understandhow game elements are selected, given an input set (user dataand LAT). Second, it demonstrates which aspects are more orless important, as the main ones are in the tree’s top, and vice-versa. It also ignores unnecessary inputs, excluding from thethree those that do not contribute. Hence, providing insights onwhich aspects influence users’ preferences, as well as whichare most influencing ones from those we studied. Third, thealgorithm itself determines how each characteristic will be split(e.g., should age be split in 18-28, 29-39 or 18-23, 24-29, 30-39?), removing human bias that are likely to be inserted inthis process. Especially for reducing bias, we chose to use CDT ratherthan the traditional version of the algorithm. Traditional de-cision tree implementations suffer from bias in the selectionprocess when inputs have many splitting points (e.g., how tosplit age, playing time, or country - when there might be morethan 100 countries?), and they are likely to lead to overfittingas the algorithms do not consider whether a splitting/selectionimproves the tree [59]. CDT, on the other hand, address thesegaps by following a statistical approach that takes into accountmeasures’ distributions during splits and variable selection[57].Thus, according to our goal, data captured via our survey(Table II) was entered as input to generate three CDT. Eachtree’s output was users’ preferences for either their first,second, or third selected game elements. Accordingly, eachtree predicts a user’s preferred game element. Note that thedataset described in Table II is in the wide format ; that is,one row per participant, and one column for their preferenceon each LAT (one column for LAT1, one for column LAT2,and so on). Then, to generate trees able to distinct users’preferences from one LAT to another, we converted the datasetto long format ; that is, six rows per participant, a new columnindicating the LAT each row corresponds to, and a singlecolumn indicating the preferred game element from each userfor each LAT. We highlight that, although this increases thesize of the dataset inputted to the CDT, the characteristics’distribution remains the same.IV. R
ESULTS
First, this section presents the overall answers collectedfrom the survey. Then, it explores the CDT generated fromour data, discussing the answers for our research questions inlight of insights gained from them.
A. Overall Survey Responses
Table III demonstrates how much each game element wasselected by the participants, independent of selecting the gameelement as first, second, or third choice. Overall, Acknowledg-ment was the most selected game element (11%), followed byObjectives (10%) and Cooperation (8%). On the other hand,Reputation, Rarity, and Social Pressure are the less chosen,summing up to less than 3% together. Nevertheless, one shouldnote the participants’ selections distribution, with the gameelement selected the most having only 11% of those, and alsoconsidering that the fifth element selected the most - Narrative(7%) - difference to the first was just four percentual points.More specifically, Table IV shows how much each gameelement was chosen as the first, second, and third options. Itdemonstrates Acknowledgment was selected the most as firstand Objectives as both the second and third choices. Again, theproximity in preferences must be noted, with the most selectedgame element being chosen only 20% of the times. Theseresults suggest the variety of participants’ preferences, furthermaking the case for understanding how these preferenceschange according to users’ characteristics as well as the LATthey expect to perform and their geographic location.
REPRINT SUBMITTED TO IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES 7
TABLE IIIO
VERALL PARTICIPANTS ’ PREFERENCES .Game element N(%) Game element N(%)Acknowledgment 740 (0.114) Storytelling 290 (0.045)Objectives 654 (0.101) Stats 281 (0.043)Cooperation 545 (0.084) Point 213 (0.033)Competition 507 (0.078) Time Pressure 180 (0.028)Narrative 420 (0.065) Sensation 126 (0.019)Progression 405 (0.062) Renovation 117 (0.018)Level 381 (0.059) Novelty 106 (0.016)Imposed Choice 363 (0.056) Reputation 76 (0.012)Puzzles 352 (0.054) Rarity 64 (0.010)Economy 327 (0.050) Social pressure 37 (0.006)Chance 314 (0.048)TABLE IVP
ARTICIPANTS ’ PREFERENCES FOR FIRST , SECOND , AND THIRD GAMEELEMENT . D
ATA REPRESENTED AS
N(%).Game element First Second ThirdAcknowledgment
426 (0.197)
130 (0.060) 184 (0.085)Chance 109 (0.050) 105 (0.048) 100 (0.046)Competition 211 (0.097) 158 (0.073) 138 (0.064)Cooperation 176 (0.081) 197 (0.091) 172 (0.079)Economy 81 (0.037) 116 (0.054) 130 (0.060)Imposed Choice 155 (0.072) 116 (0.054) 92 (0.042)Level 104 (0.048) 141 (0.065) 136 (0.063)Narrative 162 (0.075) 168 (0.078) 90 (0.042)Novelty 19 (0.009) 44 (0.020) 43 (0.020)Objectives 191 (0.088)
262 (0.121) 201 (0.093)
Point 37 (0.017) 80 (0.037) 96 (0.044)Progression 57 (0.026) 154 (0.071) 194 (0.090)Puzzles 138 (0.064) 116 (0.054) 98 (0.045)Rarity 13 (0.006) 22 (0.010) 29 (0.013)Renovation 28 (0.013) 41 (0.019) 48 (0.022)Reputation 8 (0.004) 30 (0.014) 38 (0.018)Sensation 23 (0.011) 49 (0.023) 54 (0.025)Social pressure 9 (0.004) 11 (0.005) 17 (0.008)Stats 77 (0.036) 87 (0.040) 117 (0.054)Storytelling 107 (0.049) 84 (0.039) 99 (0.046)Time Pressure 35 (0.016) 55 (0.025) 90 (0.042)
B. Conditional Decision Trees Overview
Concerning participants’ number one choice, that is, thegame element they prefer the most for each LAT, our firstCDT - CDT1 - was constructed. Similarly, our second andthird CDT - CDT2 and CDT3, respectively - concern the gameelement participants selected as the second- and third-preferredones for each LAT. CDT1 is shown in Figure 1, where circlesrepresent decision nodes and rectangles are leaf ones. Decisionnodes function as if/else statements. For instance, the firstnode tests if, for a given input, the preferred game genre isequal to adventure, other genres, role playing game, or strategy(left), or equal to action (right). Based on the answer, it isdecided whether one should follow to the left or right pathof the tree. This procedure is iteratively repeated for eachdecision node until reaching a leaf node. Leaf nodes indicatethe tree’s output, which is the preferred game element in ourcase. Hence, for someone who preferred the game genre isaction and lives in the Netherlands, CDT1 would recommendthe Objectives game element.One should note, however, that the tree in Figure 1 is asimplified version compared to the original tree generatedfrom the R package party . The original version has twomain differences. First, it shows p values from each split, demonstrating they are significant splits. Second, their leafnodes present bar plots, demonstrating to what extent eachgame element was selected for that path’s sample. For in-stance, for those who prefer the game genre action and livein the Netherlands, the tree allows identifying which gameelements this sub-sample selected, as well as the percentageselection of each one. Figure 2 partly shows the originalversion of CDT1, focusing on the case aforementioned. Notethat game elements are numerically represented in the figure’sbarplot for simplicity. Each element’s number is defined al-phabetically, with Acknowledge represented by number oneand Time pressure by number 21 (recall that the 21 gameelements are, in alphabetical order, Acknowledgment, Chance,Competition, Cooperation, Economy, Imposed Choice, Level,Narrative, Novelty, Objectives, Point, Progression, Puzzles,Rarity, Renovation, Reputation, Sensation, Social Pressure,Stats, Storytelling, Time Pressure).From Figure 2, we see the 12 participants who live inNetherlands or Spain and prefer action games chose sixdifferent game elements, with Objectives (number 10 on thex-axis) being the most selected one, followed by Acknowl-edgment (number one on the x-axis). This insight can beused to recommend the most preferred game element (e.g.,Objective) or to provide ratings on the most likely preferredones (e.g., Objective, Acknowledgment, and Point - number 11on the x-axis - are the most preferred elements). Consideringthis context, we highlight Figure 1 only presents the mostpreferred game element for the sake of limited space, as thefull image would be readable within the article template. Forsimilar reasons, CDT2 and CDT3 are not shown in the article.Nevertheless, the full images, with barpots for all leaf nodes,from all CDT we created, are available in the appendix.
C. RQ1: Characteristics that Impact User Preferences
RQ1 concerns finding out which aspects, among usercharacteristics, geographic location, and LAT, impact users’preferences for game elements. Therefore, we analyze whichof those appear in our CDT to identify the ones that influenceparticipants’ preferences.
CDT1 used six of the nine (eight from Table II plus LAT)inputs, namely preferred game genre, LAT, gender, country,experience researching gamification, and education, which ap-peared in the tree in the same order as presented here. Thus, forparticipants number one choice, those are the characteristicsthat impacted their preferences, with preferred game genre andeducation being the most and the less influencing ones.
CDT2 also used six out of the nine inputs; those are country, LAT,preferred game genre, gender, preferred playing setting, andweekly playing time; the order of relevance for the tree is thesame as the one they were presented here. Thus, for partici-pants number two choice, those are the six characteristics thatimpacted their preferences, with country and weekly playingtime being the most and the less influencing ones.
CDT3 used five of the nine inputs, country, preferred game genre,experience researching gamification, LAT, and education, withthe same order of relevance as presented here. Thus, these arethe characteristics that influenced participants’ preferences for
REPRINT SUBMITTED TO IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES 8
PGG Country Gender ERGCountryEducation EducationLATPGGLAT LATCountry CountryCountry LAT LAT Acknow _ledgment Compe _titionAcknow _ledgmentCompe _titionCompe _tition Competition,Cooperaation,Imposed ChoiceObjectivesAcknow _ledgment ObjectivesAcknow _ledgmentCoope _ration Compe _titionAcknow _ledgment Acknow _ledgment Acknow _ledgmentNarrativeAcknow _ledgment YesNoFemaleMale{Canada, USA}{Brazil, Germany, India, Italy, United Kingdom}{Other Education, Ph.D} {High School, MsC, Technical education, Undergraduate}{High School, Undergraduate} {MsC, Technical education}{Albania, Brazil, Canada, Germany, India, Italy, United Kingdom, United States}{Netherlands, Spain}Action{Adventure, Other Genre, Role Playing Game, Strategy}>2<=2{Adventure, Role Playing Game, Strategy} Other Genre<=1 >1 <=4{Australia, France, Italy} {Austria, Belize, Brazil, Canada, Germany, India, Ireland, Jamaica, Nigeria, Poland, Spain, Turkey, United Kingdom, United States}{Austria, Canada, India, Nigeria} {Belize, Brazil, Germany, Ireland, Jamaica, Poland, Spain, Turkey, United Kingdom, United States}{Poland, United Kingdom} {Belize, Brazil, Germany, Ireland, Jamaica, Spain, Turkey, United States}<=3 >3>4 <=5 >5
Fig. 1. Conditional decision tree for participants most preferred game element. Codes refer to preferred game genre (PGG), learning activity type (LAT), andexperience researching gamification (ERG).
PGGp < 0.001 Countryp < 0.001Action{Netherlands, Spain} Albania, Brazil, Canada, Germany, India, Italy, United Kingdom, United States{Adventure, Other Genre, Role Playing Game, Strategy} ... ...
Fig. 2. Partial visualization of a CDT generated from our data (CDT1),illustrating how its leaf nodes demonstrate the distribution from participants’selections. their third choice, in which country was the most relevantone, as opposed to education and LAT that were both the lessrelevant ones.Based on these findings, we might answer
RQ1 withevidence that factors impacting users’ preferences are coun-try, LAT, preferred game genre and playing setting, gender,experience researching gamification, weekly playing time, andeducation. In addition, we also found the order of importanceof these characteristics for each of the three selections. Thisfinding is summarized in Table V, which demonstrates thehighest level of the tree where each characteristic appears(because one might appear multiple times and at differentlevels). Consequently - as the higher the level, the more theimportance - allowing us to identify each one’s importance.
TABLE VL
EVEL IN WHICH EACH CHARACTERISTIC APPEARED IN THE
CDT
OFEACH USERS ’ CHOICES .Choice Cnt LAT PGG PPS G ERG WPT EduFirst 4 2 1 3 4 5Second 1 2 2 5 5 6Third 1 4 2 2 4Cnt = country; LAT = learning activity type; PGG = preferred gamegenre; PPS = preferred playing setting; G = gender; ERS = experienceresearching gamification; WPT = weekly playing time; Edu = education.
D. RQ2: Most useful Game Elements Sets from User’ Prefer-ences
RQ2 concerns identifying which is the most useful gameelements set, given users’ characteristics, the type of the learn-ing activity they will perform, and their geographic location,
REPRINT SUBMITTED TO IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES 9 according to participants’ preferences. From the three CDTwe generated, it is worth mentioning that CDT1, CDT2, andCDT3 have 17, 16, and 15 terminal nodes, respectively. Thismeans that, together, all trees provide recommendations for48 input set combinations. Presenting a complete descriptionof the recommendation for each of these combinations isunfeasible. Nevertheless, we demonstrate recommendationsfor specific cases to illustrate the most useful game elementsset for such cases, according to our findings.First, let us consider the simple case where one wants topersonalize gamification to LAT only, without considering anyuser characteristic. To illustrate that case, we split our datasetin six: each one containing only rows of one LAT. Then, wepredict the output from each of our CDT using each sub-dataset, in which the results are shown in Table VI. Thereare cases (e.g., first, second, and third rows) in which thesame element is recommended as second and third preferred,for instance. Although one participant could not select thesame element for both cases, this corroborates the fact thatthe most select game element as second and third, consideringthe overall sample, was Objectives. Accordingly, our CDTrecommend the same element as the second and third choices.With that in mind, our findings suggest that the most usefulgame elements set, considering LAT and no user characteristic,for LAT1 is Acknowledgment and Objectives, for LAT2 isNarrative and Objectives, and so on.
TABLE VIR
ECOMMENDATIONS FOR PERSONALIZING GAMIFICATION TO
LAT
ONLY , WITHOUT CONSIDERING ANY USER CHARACTERISTIC , BASED ON OURDATASET .LAT First Second Third1 Acknowledgment Objectives Objectives2 Narrative Objectives Objectives3 Acknowledgment Objectives Objectives4 Acknowledgment Objectives Acknowledgment5 Acknowledgment Level Point6 Objectives Objectives Progression
Among the main contributions from our approach, is its abil-ity to handle multiple characteristics simultaneously, as well asthe interaction between these characteristics. Therefore, let usassess the cases in which one wants to personalize gamificationfor a learning activity wherein students need to recall somecontent from long-term memory (remembering, LAT1) andthen perform a second activity in which they need to evaluateothers’ opinions (evaluate, LAT5). Additionally, let us comparethe most useful game elements set for Brazilian and Americansperforming such activities. For the sake of simplicity, let usassume all students are males, never researched gamification,High School degree is their highest education level, playsimilar amounts of time per week (10 hours), and prefer thesame game genre and playing setting: action and singleplayer,respectively . In this context, the recommendations are likelyto vary due to changes in LAT, as well as geographic location(country), as all other relevant characteristics are the same.Table VII demonstrates the recommendations. This fixed combination was selected arbitrarily, aiming to simplify theillustration. Other characteristics were not mentioned as they were found notto influence user preferences (see Table V) TABLE VIIR
ECOMMENDATIONS , DEPENDING ON
LAT
AND COUNTRY , FOR ANARBITRARILY SELECTED SAMPLE : MALES , WHO NEVER RESEARCHEDGAMIFICATION AND HAVE H IGH S CHOOL DEGREE AS THEIR HIGHESTEDUCATION LEVEL , PLAY HOURS PER WEEK , AND PREFER PLAYINGACTION GAMES ALONE .Combination First Second ThirdLAT1 - USA Acknowledgment Competition CompetitionLAT1 - Brazil Competition Competition Time pressureLAT5 - USA Acknowledgment Level PointLAT5 - Brazil Competition Level Point
As shown in Table VII, our findings suggest the most usefulgame element set for LAT1 for Brazilians is Acknowledgmentand Competition, whereas that for Americans is Competi-tion and Time pressure. For LAT5, the recommendation forBrazilians is Acknowledgment, Level, and Point, while that forAmericans differ by suggesting Competition rather than Ac-knowledgment. Hence, highlighting the impact of contextualfactors on users’ preference, in which that differed dependingon the LAT they were expecting to perform, as well as theirgeographic location.In summary, we demonstrated which are the most use-ful game element set for specific combinations of user andcontextual characteristics. We did not show the recommen-dations for all combinations because our CDT have 48 leafnodes, which means they provide recommendations for 48combinations of the characteristics considered in this study,an unfeasible discussion for this article. Despite that, onecan follow CDT1 (Figure 1) to determine the most usefulgame element (number one of three) for any combination ofthe characteristics we studied. Both CDT2 and CDT3 can beanalyzed similarly and accessed in the appendix for findingthe recommendations for users’ second and third preferences.Lastly, we acknowledge that although comprehensible, theprocess for gathering insights from the three CDT can beimproved. Our RS, which contributes to that improvement,is presented in the next section.V. T HE R ECOMMENDER S YSTEM
To cope with the complexity of determining recommen-dations from visual inspection of CDT, we converted ourtree CDT into an RS. This system encapsulates all trees andsimplifies the task of determining which game elements touse given a user, a LAT, and a geographic location. Next,we further describe the characteristics of our RS, then, webriefly present technical concerns on how our CDT wereconverted into a free, easy-to-use system able to automate thepersonalization of GES.We characterize our RS according to the framework forRS for personalized gamification introduced in [34]. Our RSconsiders six user inputs. Those are their preferred gamegenre and playing setting, weekly playing time, gender, highesteducation level, and whether the user researched gamificationbefore. In addition, the user’s living country, as well as theLAT that will be gamified, must also be entered, inputs relatedto the context [60]. Items are the game elements users couldchoose in the survey, the 21 game elements from the taxonomy
REPRINT SUBMITTED TO IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES 10 proposed and validated in [36]. Lastly, the transactions concernusers’ preferred game elements (i.e., user with characteristicsX, from geographic location Y, prefers element Z for LAT W),which is defined according to our findings.The method adopted for output selection characterizes ourRS as a hybrid recommender [34]. The input involves twocontextual characteristics, geographic location and the LAT tobe performed. Accordingly, the method would be character-ized as a context-aware recommender. However, the selectionprocess also relies on empirical information from our findings,which concerns a content-based recommender. Thus, our RSis a hybrid recommender due to exploring the characteristicsof two methods. Lastly, our RS outputs are the ratings forgame elements, defined according to the percentual of eachgame element’s selection for that input. Then, if we considerthe case shown in Figure 2, there would be ratings for sixratings because the other 15 game elements were not chosen,and Objectives game element rating would be roughly 0.30whereas that of Acknowledgment would be around 0.20.Furthermore, as there are tree CDT, the output will show therating of each game element as the first-, second-, and third-preferred game element.Table VIII exemplifies an output of our RS. It demonstratesa full output of the RS with the ratings for all game elementswhen considered as first-, second-, and third-preference. As wediscussed previously, for participants’ number one preference,the game element with the highest rating is Objectives (0.333),followed by Acknowledgment (0.25) and Point (0.167). Forparticipants number two choice, the highest ratings are forCompetition (0.149) and Change (0.144), followed by Coop-eration in third place (0.117). For their third preferred element,Competition holds the highest rating (0.176), followed byAcknowledgment and Cooperation (0.078 for both). Whenusing the RS for other input sets, similar outputs will begiven, likely with different ratings for each game element.Hence, based on outputs as that shown in Table VIII, onecan assess which elements are more likely to be the preferredones for a given situation and define their gamification designaccordingly.Aiming to improve the usability of our RS, we reimple-mented the CDT generated through the R package party [57]in Javascript. Although one could access the R objects, ortry to make some external connection to R code from, forinstance, a web browser, this process could be laborious anddiscouraging. On the other hand, Javascript can be easily run inmost web browsers, as well as be easily plugged-in into a website. Furthermore, as decision trees can be represented througha set of if/else statements, the conversion from R objects toJavascript does not require handling complex programmingtechnical challenges. This is another advantage because theprocedure of transforming our CDT into a Javascript plugincan be replicated to any other programming language.Our RS is freely available (see the appendix), and there aretwo main use cases in which we believe it can be explored. Thefirst one the main case of automating gamification personaliza-tion, in which other systems use it as an external resource/tool.In this case, a gamified system can explore our RS as a plug-in that is consulted to find which game elements should be
TABLE VIIIR
ATINGS OF OUR RS FOR PEOPLE WHO LIVE IN N ETHERLANDS ANDPREFERRED GAME GENRE IS ACTION .Game element First Second ThirdAcknowledgment 0.250 0.080 0.078Chance 0.000 0.144 0.059Competition 0.083 0.149 0.176Cooperation 0.000 0.117 0.078Economy 0.000 0.032 0.049Imposed Choice 0.000 0.080 0.059Level 0.000 0.080 0.059Narrative 0.000 0.043 0.020Novelty 0.000 0.005 0.039Objectives 0.333 0.064 0.069Point 0.167 0.027 0.039Progression 0.083 0.064 0.049Puzzles 0.000 0.037 0.020Rarity 0.000 0.000 0.000Renovation 0.000 0.005 0.010Reputation 0.000 0.005 0.020Sensation 0.000 0.005 0.020Social pressure 0.000 0.011 0.010Stats 0.083 0.011 0.059Storytelling 0.000 0.016 0.059Time Pressure 0.000 0.027 0.029 available for some occasion. To this end, the system wouldcall the plug-in, passing the needed inputs as parameters toreceive the ratings of each game element. Then, the systemcould, for instance, turn on those elements with the highestratings. This procedure could be iteratively repeated, whenthe type of the learning activity to be performed changed, forinstance. Thus, the RS would aid the system in performingdynamic adaptations [21] of its gamification design accordingto the user’s characteristics and geographic locations as wellas the tasks performed. The second case is using our RS as astandalone tool to provide recommendations for one interestedin, for instance, personalizing an unplugged gamified environ-ment [61] or to manually define their system gamification.VI. D
ISCUSSION
Based on participants’ preferences captured though a survey,our findings provided evidence that users’ preferences do differdepending on their characteristics, geographic location, andthe LAT to be performed (RQ1), as well as we were able todevelop an RS that recommends the preferred gamificationdesign for a LAT to be performed by a user with somespecific characteristic in a defined geographic location (RQ2).The main contribution of this research is, therefore, providinga free RS for personalized gamification, built upon a state-of-the-art approach, that aids in automating the tailoring ofgamification designs by suggesting which game elements touse. This RS is based on the three aspects of personalization:domain, user, and task [9], implemented as the educationaldomain, demographics and gaming characteristics, and LATand geographic location, respectively. Additionally, we haveshown which context and user characteristics impact theirpreferences, and which of those are more or less relevant,contributing to expanding and grounding knowledge fromprevious studies (e.g., [19], [39]).Concerning the results on users’ characteristics impactingtheir preferences, our findings are aligned with the literature.
REPRINT SUBMITTED TO IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES 11
Previous studies have shown that, for instance, demographics[40], [22] and attributes related to users’ gaming habits [39]affect user preference. We corroborate those by providing moreempirical evidence that users with different characteristicshave different preferences, as well as presenting which of thoseare more important than others. For instance, we found simpleuser attributes, such as gender and having researched gamifica-tion, are less relevant than gaming-related characteristics (seeTable V), which is in line with previous literature suggestions[62]. Furthermore, it also has been discussed that the task tobe performed influences the perceptions of gamified systems’users [9]. Following that and within the educational context,suggestions to consider learning activities within the tailoringprocess of educational systems have emerged [43], [21]. Ourfindings are aligned with those theories as well, showing thatusers’ preferences differ depending on the LAT they expectto perform (see Figure 1 and Section IV-D). Additionally, wefound geographic location to be another relevant factor, findingalso consistent with recent literature suggestions [32].Concerning the results on users’ preferred gamificationdesign for each LAT given their characteristics, we expandthe literature by i) providing recommendations applicable toany task (by considering its main objective - type) and ii)exploring less studied user characteristics (i.e., demographicsand gaming-related) as well as taking into account theirgeographic location. On one hand, besides not guiding on howto tailor to LAT and geographic location, other personalizationapproaches (e.g., [10], [19]) often rely on user profiles [21].However, as shown by our findings (see Table V), demograph-ics and gaming-related characteristics are relevant as well.On the other hand, despite the recent calls for consideringlearning activities when personalizing [43], [21], availableapproaches considering such aspects are yet limited, mainlydue to considering only two characteristics (one for from userand the learning activity) [33], [20]. Although, if multipleaspects are relevant, they all should be considered, as wellas their interaction [9], [26]. Our research contributes to theseconcerns, guiding how to personalize gamification to users(i.e., demographics and game-related) and contextual (i.e.,LAT and geographic location) aspects simultaneously.Moreover, this article advances the literature by providingan RS for personalized gamification. In [34], a framework forsuch RS has been proposed, however, the literature still lacksconcrete implementations of these systems. On the other hand,recent research has highlighted the need for research to aidin the automation of gamification personalization [32]. Thisarticle contributes to this vein by introducing a free RS forpersonalized gamification that can be both plugged-in gamifiedsystems to automate their personalization process, as well asindependently used as a guide for defining personalized gami-fication designs. As this system is built upon the findings fromthis article, it implements a state-of-the-art personalizationapproach, which addresses a couple of literature challenges,namely the need for considering contextual factors along withuser information, as well as the interaction between all relevantcharacteristics (see Section II).
A. Implications
There are five main implications of our findings. First,demographics and game-related characteristics are moderatorsof user preference that should be prioritized differently. Wehave shown that these characteristics do affect user preferencebut that each one’s importance differs from one to another.Additionally, those exploring gamification effectiveness mightrely on our results to define which data to capture from theirsamples to further assess whether these characteristics alsoplay a role in other aspects (e.g., motivation or learning frominteracting with GES).Second, personalization approaches should be expandedbeyond the user. We have shown that the game elements peopleprefer when expecting to perform a LAT differ from what theyprefer when expecting to perform another; similarly for userswho live in different countries. The implication these findingshave is that rather than just thinking on what users generallyprefer, aspects of the task that will be performed, as well asthe user’s geographic location, should be taken into account,as has been recently advocated by multiple researchers [9],[43], [21], [32].Third, the interaction between relevant characteristics can-not be ignored. Our results demonstrated that the gameelements preferred the most are likely to change when asingle characteristic (e.g., country) changes. For example, wedemonstrated that the recommended game elements for thesame LAT will differ for Brazilian and American users, evenif all other characteristics (e.g., gender, weekly playing time,preferred game genre; see Section VII) are the same. Thus,confirming the need for tailoring gamification designs notonly to the user but also to the context [43], [21] as wellas considering the interaction between different aspects [9],[26]. Hence, the implication is that only one side of the wholeis likely not to work in full potential.Fourth, when designing GES, two people might preferthe same gamification design, but with different priorities.When surveying participants, we asked them to rank thetop three game elements that would help them the most inlearning activities of a specific type. Hence, gathering dataable to inform not only which game elements are the mostpreferred on each occasion, but also the importance order ofthe selected elements. Thus, we imply that when relying onour findings to design GES, one should define the emphasiseach game element will receive based on users’ selection order(see Section IV-D) because despite different individuals mightprefer the same game elements set, they might prefer thosewith different priorities.Lastly, putting together our findings and analyses, one canuse our RS (see the appendix) to automate gamified systems’personalization process as well as be informed on how to tailorgamification designs of educational systems. Practitioners canexploit our RS to define their systems’ gamification designs,as well as researchers can apply its recommendations on theirstudies to assess the effectiveness of users’ preferred designs.To aid those interested in using our RS, we have made itfreely available for use and briefly discussed how it can beeither incorporated into an existing system as well as using
REPRINT SUBMITTED TO IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES 12 it as a guide. Thus, posing a direct implication on the designand development of GES.
B. Limitations
This section discusses limitations that must be consideredwhen interpreting and applying our findings.Concerning the survey: It presented a description for eachLAT and each game element. We adopted this approach toavoid misinterpretations from those completing the survey,seeking to guarantee answers reliability. However, this in-creased the time required to complete the survey and possiblycontributed to tiring the individuals throughout the process. Toaddress this limitation, we added attention questions, whichallowed us to discard inconsistent answers.Concerning the sample: Although we analyzed data from361 consistent subjects, their attributes were highly unbalancedin some characteristics (e.g., country). Consequently, recom-mendations for individuals with a small presence in the dataset(e.g., those reporting a gender other than male and female) areinfluenced due to the sample size.Concerning the RS: Although we selected the revision ofBloom’s taxonomy due to its relevance within the educationcontext, the lack of a systematic selection process also limitsour findings in terms of how our study interprets LAT. Also,as our recommendations are based on averages, it might bethat it will not work for some users. Lastly, although ourRS is a ready-to-use resource, it is a plug-in in its initialversion that can be further enhanced to improve, for instance,its compatibility with other systems, as well as its presentationfor independent use.Concerning recommendations’ effectiveness: This was anexploratory, preference-based research, following a methodol-ogy commonly adopted by related research. Consequently, asthese previous research, we cannot ensure that personalizingto users’ preferences will be effective. Nevertheless, giventhe number of game elements to be considered (21) as wellas LAT (six), thousands of combinations would have to betested in user studies, which is unfeasible. Our survey-basedstudy addresses this limitation by presenting a valuable firststep in suggesting which game elements to use for specificconditions and providing guidance for future studies to test ourpreference-based recommendations. Nevertheless, this limita-tion suggests the need for studies to design ways of testing thelevel of preference between all game elements and all LAT.VII. F
INAL R EMARKS
Personalization emerged as an alternative to improve gami-fication effectiveness. Most studies in this vein exploit userprofiles to tailor the gamified designs. Hence, they ignorethe fact that, besides the user, the tasks and domain alsoplay a significant role in gamification’s success. Additionally,studies often do not consider the interaction between multiplerelevant characteristics, neither offer concrete resources tohelp in automating gamification personalization. To addressthese gaps, this paper introduced a preference-based RS thatsuggests game elements tailored to the user (demographics andgame-related) and the context (LAT - tasks - and geographic location), focused on the educational domain. This RS con-siders the interaction between its inputs and is freely availablefor anyone to use it.Our contributions are twofold. First, we provided practition-ers with a ready-to-use resource able to guide them on howto design GES that are tailored to users’ characteristics, aswell as geographic location, according to the tasks they areup to perform. Second, we expanded the literature on how totailor gamification designs to any learning activity (based onits type) by presenting recommendations that might be empir-ically tested in future research, providing empirical evidenceon which demographics and game-related user characteristicsimpact their preferences, as well as whether one is moreimportant than another, and supporting literature suggestionsby showing that LAT and geographic location do affect userpreference.As future studies, we mainly recommend validating theeffectiveness of our RS recommendations (e.g., ability to im-prove user motivation, flow, academic performance, or learninggains), compared to one-size-fits-all and other personalizationmethods, to identify whether personalizing to users’ prefer-ences will positively impact them as expected. Another lineof future research is improving the RS so that is can be usedas, for instance, a service to mitigate compatibility problemsas well as the need for manually adding the code to the project.Additionally, future studies might tackle the limitation of notassessing the match between all game elements and all LATfrom our methodology, which might be accomplished in steps(e.g., assessing one LAT per experiment) to cope with thecomplexity of testing all at once, as we previously discussed.A
PPENDIX
Appendixes are available at: shorturl.at/aguQT.A
CKNOWLEDGMENTS
The authors would like to thank the funding providedby CNPq, CAPES, and FAPESP (Projects: 2018/07688-1;2018/15917-0; 2016/02765-2; 2018/11180-3).R
EFERENCES[1] K. M. Kapp,
The gamification of learning and instruction . Wiley SanFrancisco, 2012.[2] S. Deterding, D. Dixon, R. Khaled, and L. Nacke, “From game designelements to gamefulness: defining gamification,” in
Proceedings of the15th international academic MindTrek conference: Envisioning futuremedia environments . ACM, 2011, pp. 9–15.[3] C. Dichev and D. Dicheva, “Gamifying education: what is known, whatis believed and what remains uncertain: a critical review,”
Internationaljournal of educational technology in higher education , vol. 14, no. 1,p. 9, 2017.[4] M. Sailer and L. Homner, “The gamification of learning: ameta-analysis,”
Educational Psychology Review , Aug 2019. [Online].Available: https://doi.org/10.1007/s10648-019-09498-w[5] A. M. Toda, P. H. D. Valle, and S. Isotani, “The dark side of gamifi-cation: An overview of negative effects of gamification in education,”in
Higher Education for All. From Challenges to Novel Technology-Enhanced Solutions , A. I. Cristea, I. I. Bittencourt, and F. Lima, Eds.Cham: Springer International Publishing, 2018, pp. 143–156.[6] K. Loughrey and D. Broin, “Are we having fun yet? misapplyingmotivation to gamification,” in . IEEE, 2018, pp. 1–9.
REPRINT SUBMITTED TO IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES 13 [7] B. Morschheuser, L. Hassan, K. Werder, and J. Hamari, “How to designgamification? a method for engineering gamified software,”
Informationand Software Technology , vol. 95, pp. 219–237, 2018.[8] R. Orji, G. F. Tondello, and L. E. Nacke, “Personalizing persuasivestrategies in gameful systems to gamification user types,” in
Proceedingsof the 2018 CHI Conference on Human Factors in Computing Systems .ACM, 2018, p. 435.[9] D. Liu, R. Santhanam, and J. Webster, “Toward meaningful engagement:A framework for design and research of gamified information systems,”
MIS quarterly , vol. 41, no. 4, pp. 1011–1034, 2017.[10] W. Oliveira and I. I. Bittencourt,
Tailored Gamification to EducationalTechnologies . Springer Nature, 2019.[11] G. F. Tondello, “Dynamic personalization of gameful interactive sys-tems,” Ph.D. dissertation, University of Waterloo, 2019.[12] N. Yee, “Gaming motivations align with personality traits,” jan 2016,accessed in January 2020. [Online]. Available: quanticfoundry.com/2016/01/05/personality-correlates/[13] R. Orji, K. Oyibo, and G. F. Tondello, “A comparison ofsystem-controlled and user-controlled personalization approaches,” in
Adjunct Publication of the 25th Conference on User Modeling,Adaptation and Personalization , ser. UMAP ’17. New York,NY, USA: ACM, 2017, pp. 413–418. [Online]. Available: http://doi.acm.org/10.1145/3099023.3099116[14] L. Rodrigues and J. D. Brancher, “Improving players’ profiles clusteringfrom game data through feature extraction,” in
Proceedings of SBGames2018 - Computing Track . IEEE, 2018, pp. 177–186.[15] R. Orji, J. Vassileva, and R. L. Mandryk, “Modeling the efficacy ofpersuasive strategies for different gamer types in serious games forhealth,”
User Modeling and User-Adapted Interaction , vol. 24, no. 5,pp. 453–498, 2014.[16] A. Knutas, R. Van Roy, T. Hynninen, M. Granato, J. Kasurinen,and J. Ikonen, “A process for designing algorithm-based personalizedgamification,”
Multimedia Tools and Applications , vol. 78, no. 10, pp.13 593–13 612, 2019.[17] L. Rodrigues and J. D. Brancher, “Playing an educational game fea-turing procedural content generation: Which attributes impact players’curiosity?”
Revista Novas Tecnologias (RENOTE) , vol. 17, no. 1, pp.254–263, 2019.[18] J. Koivisto and J. Hamari, “The rise of motivational information systems:A review of gamification research,”
International Journal of InformationManagement , vol. 45, pp. 191–210, 2019.[19] G. F. Tondello, A. Mora, and L. E. Nacke, “Elements of gamefuldesign emerging from user preferences,” in
Proceedings of the AnnualSymposium on Computer-Human Interaction in Play . ACM, 2017, pp.129–142.[20] J. Balde´on, I. Rodr´ıguez, and A. Puig, “Lega: A learner-centered gamification design framework,” in
Proceedings of theXVII International Conference on Human Computer Interaction , ser.Interacci&
Transforming Learning with Meaningful Technologies , M. Scheffel,J. Broisin, V. Pammer-Schindler, A. Ioannou, and J. Schneider, Eds.Cham: Springer International Publishing, 2019, pp. 294–307.[22] G. F. Tondello, R. R. Wehbe, L. Diamond, M. Busch, A. Marczewski,and L. E. Nacke, “The gamification user types hexad scale,” in
Proceed-ings of the 2016 annual symposium on computer-human interaction inplay . ACM, 2016, pp. 229–243.[23] L. E. Nacke, C. Bateman, and R. L. Mandryk, “Brainhex: A neurobio-logical gamer typology survey,”
Entertainment computing , vol. 5, no. 1,pp. 55–62, 2014.[24] H. Stuart, A. Serna, J.-C. Marty, G. Lavou´e, and E. Lavou´e,“Factors to Consider for Tailored Gamification,” in
CHI Play .Barcelona, Spain: ACM, Oct. 2019, p. 559–572. [Online]. Available:https://hal.archives-ouvertes.fr/hal-02185647[25] A. Mora, G. F. Tondello, L. Calvet, C. Gonz´alez, J. Arnedo-Moreno,and L. E. Nacke, “The quest for a better tailoring of gameful design:An analysis of player type preferences,” in
Proceedings of the XXInternational Conference on Human Computer Interaction . ACM, 2019,p. 1.[26] P. Buckley and E. Doyle, “Individualising gamification: An investigationof the impact of learning styles and personality traits on the efficacy ofgamification using a prediction market,”
Computers & Education , vol.106, pp. 43–55, 2017.[27] S. Borges, V. Durelli, H. Reis, I. I. Bittencourt, R. Mizoguchi,and S. Isotani, “Selecting effective influence principles for tailoring gamification-based strategies to player roles,” in
Brazilian Symposiumon Computers in Education , vol. 28. Recife, Brazil: SBC, 2017, pp.857–866.[28] S. Nicholson, “A user-centered theoretical framework for meaningfulgamification,” in
Games+Learning+Society 8.0 , 2012, pp. 1–7.[29] W. Oliveira, A. Toda, P. Toledo, L. Shi, J. Vassileva, I. I. Bittencourt,and S. Isotani, “Does tailoring gamified educational systems matter? theimpact on students’ flow experience,” in
Proceedings of the 53rd HawaiiInternational Conference on System Sciences . ScholarSpace, 2020, pp.1226–1235.[30] B. Monterrat, E. Lavou´e, and S. George, “Adaptation of gaming featuresfor motivating learners,”
Simulation & Gaming , vol. 48, no. 5, pp. 625–656, 2017.[31] F. Roosta, F. Taghiyareh, and M. Mosharraf, “Personalization ofgamification-elements in an e-learning environment based on learners’motivation,” in . IEEE, 2016, pp. 637–642.[32] A. C. T. Klock, I. Gasparini, M. S. Pimenta, and J. Hamari, “Tailoredgamification: A review of literature,”
International Journal of Human-Computer Studies , p. 102495, 2020.[33] K. Bovermann and T. J. Bastiaens, “Towards a motivational design?connecting gamification user types and online learning activities,”
Re-search and Practice in Technology Enhanced Learning , vol. 15, no. 1,pp. 1–18, 2020.[34] G. F. Tondello, R. Orji, and L. E. Nacke, “Recommender systemsfor personalized gamification,” in
Adjunct Publication of the 25thConference on User Modeling, Adaptation and Personalization . ACM,2017, pp. 425–430.[35] F. F.-H. Nah, Q. Zeng, V. R. Telaprolu, A. P. Ayyappa, and B. Es-chenbrenner, “Gamification of education: a review of literature,” in
International conference on hci in business . Springer, 2014, pp. 401–409.[36] A. M. Toda, W. Oliveira, A. C. Klock, P. T. Palomino, M. Pimenta,I. Gasparini, L. Shi, I. Bittencourt, S. Isotani, and A. I. Cristea, “Ataxonomy of game elements for gamification in educational contexts:Proposal and evaluation,” in , vol. 2161-377X. IEEE,July 2019, pp. 84–88.[37] A. M. Toda, A. C. Klock, W. Oliveira, P. T. Palomino, L. Rodrigues,L. Shi, I. Bittencourt, I. Gasparini, S. Isotani, and A. I. Cristea,“Analysing gamification elements in educational environments using anexisting gamification taxonomy,”
Smart Learning Environments , vol. 6,no. 1, p. 16, 2019.[38] R. R. McCrae and O. P. John, “An introduction to the five-factor modeland its applications,”
Journal of personality , vol. 60, no. 2, pp. 175–215,1992.[39] M. Denden, A. Tlili, F. Essalmi, and M. Jemni, “An investigation of thefactors affecting the perception of gamification and game elements,” in . Muscat, Sultanate of Oman:IEEE, Dec 2017, pp. 1–6.[40] A. Toda, W. Oliveira, L. Shi, I. Bittencourt, S. Isotani, and A. Cristea,“Planning gamification strategies based on user characteristics anddm: A gender-based case study,” in , 07 2019, pp. 438–443.[41] A. M. Toda, R. M. do Carmo, A. P. da Silva, I. I.Bittencourt, and S. Isotani, “An approach for planning anddeploying gamification concepts with social networks withineducational contexts,”
International Journal of InformationManagement
Human–Computer Interaction , vol. 30, no. 3-4, pp. 294–335,2015.[43] L. Rodrigues, W. Oliveira, A. Toda, P. Palomino, and S. Isotani,“Thinking inside the box: How to tailor gamified educational systemsbased on learning activities types,” in
Proceedings of the BrazilianSymposium of Computers on Education . SBC, 2019, pp. 823–832.[44] R. E. Wood, “Task complexity: Definition of the construct,”
Organiza-tional behavior and human decision processes , vol. 37, no. 1, pp. 60–82,1986.[45] D. Diaper and N. Stanton,
The handbook of task analysis for human-computer interaction . CRC Press, 2003.[46] D. R. Krathwohl, “A revision of bloom’s taxonomy: An overview,”
Theory into practice , vol. 41, no. 4, pp. 212–218, 2002.[47] B. S. Bloom,
Taxonomy of educational objectives: The classification ofeducational goals . New York: Longman, 1956.
REPRINT SUBMITTED TO IEEE TRANSACTIONS ON LEARNING TECHNOLOGIES 14 [48] F. Ricci, L. Rokach, and B. Shapira, “Introduction to recommendersystems handbook,” in
Recommender systems handbook . Springer,2011, pp. 1–35.[49] D. Dicheva, K. Irwin, and C. Dichev, “Motivational factors in educa-tional gamification,” in . IEEE, 2018, pp. 408–410.[50] Y. Xu and Y. Tang, “Based on action-personality data mining, research ofgamification emission reduction mechanism and intelligent personalizedaction recommendation model,” in
International Conference on Cross-Cultural Design . Springer, 2015, pp. 241–252.[51] B. Monterrat, E. Lavou´e, and S. George, “Toward an adaptive gamifica-tion system for learning environments,” in
International Conference onComputer Supported Education . Springer, 2014, pp. 115–129.[52] J. Lazar, J. H. Feng, and H. Hochheiser,
Research methods in human-computer interaction . Morgan Kaufmann, 2017.[53] R. Van Roy and B. Zaman, “Need-supporting gamification in educa-tion: An assessment of motivational effects over time,”
Computers &Education , vol. 127, pp. 283–297, 2018.[54] K. Casler, L. Bickel, and E. Hackett, “Separate but equal? a comparisonof participants and data gathered via amazon’s mturk, social media, andface-to-face behavioral testing,”
Computers in human behavior , vol. 29,no. 6, pp. 2156–2160, 2013.[55] R. N. Landers and T. S. Behrend, “An inconvenient truth: Arbitrary dis-tinctions between organizational, mechanical turk, and other conveniencesamples,”
Industrial and Organizational Psychology , vol. 8, no. 2, pp.142–164, 2015.[56] C. Wohlin, P. Runeson, M. Hst, M. C. Ohlsson, B. Regnell, andA. Wessln,
Experimentation in Software Engineering . Springer Pub-lishing Company, Incorporated, 2012.[57] T. Hothorn, K. Hornik, and A. Zeileis, “Unbiased recursive partitioning:A conditional inference framework,”
Journal of Computational andGraphical statistics , vol. 15, no. 3, pp. 651–674, 2006.[58] S. R. Safavian and D. Landgrebe, “A survey of decision tree classifiermethodology,”
IEEE transactions on systems, man, and cybernetics ,vol. 21, no. 3, pp. 660–674, 1991.[59] J. Mingers, “Expert systems—rule induction with statistical data,”
Jour-nal of the operational research society , vol. 38, no. 1, pp. 39–47, 1987.[60] M. Khemaja and F. Buendia, “Building context-aware gamified apps byusing ontologies as unified representation and reasoning-based models,”in
Serious Games and Edutainment Applications . Springer, 2017, pp.675–702.[61] A. M. Toda, P. T. Palomino, W. Oliveira, L. Rodrigues, A. C. Klock,I. Gasparini, A. I. Cristea, and S. Isotani, “How to gamify learningsystems? an experience report using the design sprint method and a tax-onomy for gamification elements in education,”
Journal of EducationalTechnology & Society , vol. 22, no. 3, pp. 47–60, 2019.[62] R. N. Landers, G. F. Tondello, D. L. Kappen, A. B. Collmus, E. D. Mek-ler, and L. E. Nacke, “Defining gameful experience as a psychologicalstate caused by gameplay: Replacing the term ‘gamefulness’ with threedistinct constructs,”
International Journal of Human-Computer Studies ,vol. 127, pp. 81–94, 2019.
Luiz Rodrigues
Luiz Rodrigues is a Ph.D. candidateat the University of S˜ao Paulo. His research focuseson understanding and improving gamification effec-tiveness within the context of educational systems.His main research interests are gamification, per-sonalization, user modeling, and procedural contentgeneration.
Armando M. Toda
Armando Toda is, currently, aPhD. candidate at University of Sao Paulo workingwith gamification design in educational contexts.Other main areas of research are concerned with:data mining, serious games, evaluation and instru-ment design.
Wilk Oliveira
Wilk Oliveira is a Ph.D. candi-date at the University of S˜ao Paulo and MSc. inComputer Science for the Federal University ofAlagoas with exchange programs at the Universityof Saskatchewan. He was an assistant professor atthe University of S˜ao Paulo and a guest lecturer atthe Tiradentes University Center. His main topicsinclude Artificial Intelligence in Education, Human-computer Interaction, and Gamification.
Paula T. Palomino
Paula Palomino is a PhD. can-didate at University of S˜ao Paulo, researching anddeveloping a gamification content-based frameworkfor educational purposes. Other areas of study in-clude Games, HCI and Cyberculture.
Julita Vassileva
Julita Vassileva is Professor atthe Department of Computer Science, Universityof Saskatchewan. Julita does research in SocialComputing, Persuasive Technologies, Personaliza-tion, User Modeling, AI in Education, Peer-to-Peersystems, Trust and Reputation Mechanisms, andHuman-computer Interaction. She is currently work-ing on Personalized Persuasive Systems, LearningData-Driven Persuasion, and Distributed Ledgers forStoring and Sharing Personal Data.