What is Learning Analytics about? A Survey of Different Methods Used in 2013-2015
ISSN 2414-6099
What is Learning Analytics about? A Survey of Different Methods Used in 2013-2015
Mohammad Khalil
Graz University of Technology,Graz, Austria
Martin Ebner
Graz University of Technology,Graz, Austria
Abstract
The area of Learning Analytics has developed enormously since the first International Conference on Learning Analytics and Knowledge (LAK) in 2011. It is a field that combines different disciplines such as computer science, statistics, psychology and pedagogy to achieve its intended objectives. The main goals illustrate in creating convenient interventions on learning as well as its environment and the final optimization about learning domain’s stakeholders (Khalil & Ebner, 2015b). Because the field matures and is now adapted in diverse educational settings, we believe there is a pressing need to list its own research methods and specify its objectives and dilemmas.
This paper surveys publications from Learning Analytics and Knowledge conference from 2013 to 2015 and lists the significant research areas in this sphere. We consider the method profile and classify them into seven different categories with a brief description on each. Furthermore, we show the most cited method categories using Google scholar. Finally, the authors raise the challenges and constraints that affect its ethical approach through the meta-analysis study. It is believed that this paper will help researchers to identify the common methods used in Learning Analytics, and it will assist by establishing a future forecast towards new research work taking into account the privacy and ethical issues of this strongly emerged field.
Keywords:
Learning Analytics, survey, publications, literacy, techniques
Introduction
Since the first Learning Analytics and Knowledge (LAK) conference in 2011 as well as the Horizon Report in 2013 (Johnson et al., 2013), learning analytics is considered to be an emerging field that would be applied in the different educational settings. This field provides tools and technologies that offer the potentials to do proper interventions and improve education in general. The Society for Learning Analytics and Research (SoLAR) defined it as “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs”. Several studies exchanged views about learning analytics goals. For instance, Khalil and Ebner (2015) introduced learning analytics lifecycle and listed the main surveyed objectives of the past four years of the LAK conferences.
ISSN 2414-6099
They listed interventions, predictions, reflection, awareness, personalization, recommendation and benchmarking as the main goals. These goals conformed to Siemen’s defined techniques back in 2012 (Siemens, 2012). In addition to that, different frameworks have been introduced to define both key objectives and dilemmas of this field. In his paper “The Learning Analytics Cycle: Closing the loop effectively”, Clow argued that a successful learning analytics should be winding up into feeding back the product to learners in order to make effective intervention(s) (Clow, 2012). Whiles Ferguson indexed remarkable challenges of ethics, distinct perspectives from stakeholders’ field of vision and the methods to use in order to make these goals achievable (Ferguson, 2012).
For the time being, there is a large variety of educational environments such as MOOC-platforms, LMS, virtual environments, etc. These educational information systems hold “Big Data” of learners that create huge data repositories. According to learning analytics definition, the data need to be analysed by typical methodologies in order to reflect benefits on learning and teaching. The beginning of lively discussions on the differences between learning analytics and educational data mining were mainly residing to the opposing opinions of using tools and methodologies in both fields (Baker et al., 2012). Nevertheless, educational data mining and learning analytics are enriched by the methods of data mining and analytics in general (Baker & Siemens, 2013). In its first stages, researchers of learning analytics frameworks and structure were discussing methods such as visualizations, data mining techniques (Elias, 2011), social network analysis (Ferguson, 2012), and sentiment analysis (Siemens, 2012), in addition to statistics which was also mentioned as a required tool to build learning prediction models (Campbell, DeBlois & Oblinger, 2007). SoLAR brought to success the annual organization of LAK conferences since 2011. Accordingly, several categories of methods to analyse educational datasets were used. Most of these methods tend to process data quantitatively and qualitatively to discover interesting hidden patterns. Baker and Siemens (2013) mentioned that educational data is what drives new methods to be used in learning analytics. They said: “The specific characteristics of educational data have resulted in different methods playing a prominent role in EDM/LA than in data mining in general, or have resulted in adaptations to existing psychometrics methods”.
In this paper, we survey publications from LAK conference from 2013 to 2015. The purpose is to list the most common methods used in the field of learning analytics in the last three years. We believe this paper provides different benefits because learning analytics becomes an important field by itself and is now completely matured into being adapted in different educational institutions and applications. The main advantages are: 1. It helps learning analytics researchers to identify common methods in use in order to reach intended goals. 2. It determines methods that are highly cited, e.g. by Google scholar (http://scholar.google.com), and establish a future forecast towards new research work. 3. Finally, it assists to compare the beginning view about learning analytics methods and the on-going current version. In addition, the paper aims to guide future researchers into further advances in this field and meets the “Smart Learning Excellence” theme of Innovation Arabia 9 conference which is “The next wave of innovations in Smart-Learning” that mainly considers Big Data and Learning Analytics as a new wave in educational technology.
ISSN 2414-6099
We have organized this paper into the following sections. First, we list the methodology employed to extract methods used in learning analytics publications. We then show statistical data and describe methods in detail with remarks about their types. Finally, we discuss and summarize the main conclusions and list the constraints and the ethical issues of learning analytics.
Research Design
As mentioned before, the conference of learning analytics and knowledge is considered to be the first and the largest repository of learning analytics publications. We mainly focused on it, and surveyed 91 papers from LAK 2013 (Suthers et al., 2013), LAK 2014 (Pistilli, Willis & Koch, 2014), and LAK 2015 (Baron, Lynch & Maziarz, 2015), with an additional supplementary literature from other sources. We excluded papers with topics philosophy, frameworks and conceptual studies of learning analytics for the reason that they address structures and do not accommodate a mechanism for revealing patterns. We also faced papers with unclear methods, and these were excluded too. At the end, 78 publications were set for examination. This study was influenced by the work of Romero and Ventura (Romero & Ventura, 2010), and Dawson et al. (Dawson et al., 2014). The classification of methods was based on reading the abstract, keywords, general terms, methodology section and the conclusion of each paper. In some publications, we paid more analysis into examining literature and the reference list. Furthermore, we collected the total number of citations for each analysed paper from Google scholar and observed the trending topics.
Learning Analytics Methods
Learning analytics is a combination of different disciplines like computer science, statistics, psychology, and education. As a result, we realized different analysis methods that do not only tend to be too technical but rather pedagogical. Before classifying the analysis methods, we have been gravitated towards the beginning topics of the emergence of learning analytics, which briefly described methods and tools for collecting data and analyzing them (Ferguson, 2012; Siemens, 2012). However, the survey reveals more methods being used to examine learners’ data. Our main methods categories, which will be explained in detail in section 3.1, are: (a) data mining techniques; (b) statistics and mathematics; (c) text mining, semantics and linguistic analysis; (d) visualization; (e) social network analysis; (f) qualitative analysis; and (g) gamification. Figure 1 shows grouping of the methods used in learning analytics for LAK publications with the number of papers in each category. It should be noted that some publications might be referenced in a different category. Moreover, a paper could be referenced in multiple methods category.
The bar plot in the figure shows that researchers of 31 publications used data mining techniques and 26 research studies used statistics and mathematics to analyse their data. This makes both of these two methods as the top most employed techniques of analysis. We also see that “Text Mining, Semantics and Linguistic” analyses as well as visualizations are being used in 13 LAK publications equally. However, social network analysis and qualitative analysis as well as gamification were the least used techniques.
ISSN 2414-6099 C ategories This section lists the methods categories in more detail and states relevant publications under each method.
Data Mining Techniques : Data mining tends to make sense out of data. The definition of learning analytics cited the similar idea, namely understanding the data, but in this case, about learners. The survey shows that data mining techniques are the most used method for analyzing and interpreting the learners’ log data. The decision tree algorithm was used to predict the performance drop and the final outcome of students in a Virtual Learning Environment (VLE) (Wolff et al., 2013). Other researchers used several classification techniques such as step regression, Naive Bayes and REP-Trees to study students’ behavior and detect learners who game the system (Pardos et al., 2013). While clustering was used to propose an approach for the purposes of enhancing educational process mining based on the collected data from logs and detecting students at risk (Bogarín et al., 2014). Discovering relations between two factors was observed by using multiple linear regression analysis to forecast the relation between studying time and learning performance (Jo, Kim & Yoon, 2014). Moreover, data mining is used for assessment such as the work at the University of Missouri-Columbia, which proposed an automated tool to enable teachers assess students in online environments (Xing, Wadholm & Goggins, 2014). It was remarked that regression analysis was the common mechanism among data mining techniques.
Statistics and Mathematics : Statistics is the science of measuring, controlling, communicating and understanding the data (Davidian & Louis, 2012). Publications show that researchers have been using descriptive statistics and mathematics, such as the mean, median and standard deviation to signify their results. In addition, inferential statistics was used side by side with data mining in
Figure 1. Number of the examined LAK papers grouped by methods. Some papers share more than single category
ISSN 2414-6099 some of the publications. Markov chain was used to study school students behavior in solving multiplication (Taraghi et al., 2014). Different statistical techniques were operated to build a grading system (Vogelsang & Ruppertz, 2015). Additionally, statistical discourse analysis with Markov chain was employed to study online discussions and summarize demographics (Chiu & Fujita, 2014), as well as examining student problem solving behavior and adapting it into tutoring systems (Eagle et al., 2015).
Text mining, Semantics & Linguistic Analysis : Publications which refer to ontologies, mining texts, discourse analysis, Natural Language Processing, or study of languages are set to be in this category. Some studies refer to text analysis for assessment purposes of short answer questions (Leeman-Munk, Wiebe & Lester, 2014), or to enhance collaborative writing between students (Southavilay et al., 2013). Contextualizing user interactions based on ontologies to illustrate a learning analytics approach (Renzel & Klamma, 2013). Linguistic analysis was clearly used in parsing posts from students for prediction purposes (Joksimović et al., 2015). Finally, online discussion forums were analysed to pioneer an automatic dialogue detection system in order to develop a self-training approach (Ferguson et al., 2013).
Visualization:
When the information is visually presented to the field experts, efficient human capabilities rise to perceive and process the data (Kapler & Wright, 2015). Visual representations take the advantages into expanding human decisions within a large amount of information at once (Romero & Ventura, 2010). There are several studies that cited visualization as a method to analyse the data and deliver information to end users, such as: building a student explorer screen to prepare meetings and identifying at-risk students by the teachers (Aguilar, Lonn & Teasley, 2014). Studying MOOC’s attrition rate and learners’ activities (Santos et al., 2014); Building an awareness tool for teachers and learners (Martinez-Maldonado et al., 2015), and a dashboard for self-reflection goals (Santos et al., 2013). Information can be interpreted into heat maps, scatterplots, diagrams, and flowcharts which were observed in most of the statistical, mathematical and data mining based publications.
Social Network Analysis : Abbreviated as SNA. It focuses on relationships between entities. In learning analytics, SNA can be used to promote collaborative learning and investigate connections between learners, teachers and resources (Ferguson, 2012). Moreover, it can be employed in learning environments to examine relationships of strong or weak ties (Khalil & Ebner, 2015). This category includes network analysis in general and Social Learning Analytics (SLA). The survey observed researchers who: built a collaborative learning environment by visualizing relationships between students about the same topic (Schreurs et al., 2013). A two-mode network was used to study students’ patterns and to classify them into particular groups (Hecking, Ziebarth & Hoppe, 2014). It was also used with a grading system in a PLE to examine the centrality of students and grades (Koulocheri & Xenos, 2013). Again, not so far from this survey study, a network analysis was done to analyse citations of LAK conference papers (Dawson et al., 2014). The authors studied the degree centrality and pointed out the emergence and isolated disciplines in learning analytics. SNA was used to analyse data of connectivist MOOCs by examining interactions of learners from social media websites (Joksimović et al., 2015).
ISSN 2414-6099
Qualitative Analysis:
This category is related to the decisions based on explained descriptions of the analysts. For instance: i) a qualitative evaluation with data mining techniques was made to understand the nature of discussion forums of MOOCs (Ezen-Can et al., 2015); ii) Usage of qualitative interviews, which are answered by words to build a learning analytics module of understanding fractions for school children (Mendiburo, Sulcer & Hasselbring, 2014); iii) Qualitative meta-analysis to investigate teachers’ needs in technology enhanced learning covered by the umbrella of learning analytics (Dyckhoff et al., 2013).
Gamification : It is the use of game mechanics and tools to make learning and instruction attractive and fun (Kapp, 2012). This method is considered as a technique on its own because of its relevant appearance in educational workshops and the requests to make learning entertaining. Some examples are using rewards points and progress bar to enhance the retention rate and building a gamified grading system (Holman, Aguilar & Fishman, 2013), or presenting a competency map with progress bars, pie charts, labels and hints to improve students’ performance (Grann & Bushway, 2014). A significant study on monitoring students in a 3D immersive environment was also advised as another type of gamification techniques (Camilleri et al., 2013).
Prominent Methods and Discussion
In this section, we consider learning analytics methods that have been frequently cited. We used the Google scholar as a foundation to check methods’ popularity. All the data were collected recently and retrieved before the submission date. Figure 2 shows Google scholar citations for the analysed LAK conference papers based on the methods category. The publications with the method type Data Mining and Techniques, were the most cited articles (452 citations). The ultimate number of citations in this survey belongs to the paper of (Kizilcec, Piech & Schneider, 2013) with 236 citations. Although we took into consideration the time span of publications, we see that articles that belong to MOOCs are the most cited papers. Statistics and Mathematics publications were cited 363 times. Qualitative analysis and gamification publications were the least cited articles. In figure 3, we show a density plot of publications’ citations grouped by year. The x-axis records number of citations converted into logarithmic scale to ease the reading. The y-axis records the density of publications per year. Since we did not survey a fair number of publications per year, we intended to use this plot instead of histogram plot, which is highly sensitive to bin size.
Figure 2. Number of Google scholar citations of the examined LAK papers based on methods category. Retrieved on 26 th October, 2015.
ISSN 2414-6099
Figure 3. Density plot of citations grouped by year
ISSN 2414-6099
Some of 2013 publications attracted numerous citations which exceed the expectations such as (Kizilcec, Piech & Schneider, 2013; Pardos et al., 2013). A descriptive analysis of the articles in 2013 leads to: median=8, mean=24.37, max=236; articles in 2014: median=5, mean=5.56, max=17; and articles of 2015: median =1, mean=1.42, max=4. The low number of citations for 2015 is reasonable as the time span between this survey and 2015 LAK publication is around six months.
Challenges & Constraints
The data collection and analysis through learning analytics methods lead to questions related to ownership, privacy and ethical issues (Khalil & Ebner 2015; Khalil & Ebner, 2016). We summarize our experience and previous research studies as the following: A) Privacy, in which the learning analytics specialists need to carefully deliberate the potential privacy issues while collecting, analyzing, and intervene of the students’ data. B) Transparency of disclosing information of learners and the needs to proclaim consent by the students. C) Assuring security and achieving the CIA which is an acronym that refers to Confidentiality, Integrity and Accessibility such as storing of learners records. D) The ownership of the collected and analysed data.
Conclusion
Learning Analytics is a promising area which provides the adequate tools and methods to optimize the learning mechanism in the different environments of educational technology platforms. In this paper, we did a meta-analysis study on publications in the last two years and classified seven different categories of techniques that have been used in that period. We noticed that learning analytics researchers adopt data mining and statistics more often than other techniques. Additionally, 2013 was a stimulating year in showing MOOCs as a desirable article by a distinct number of citations. Moreover, we also see that some publications have had a high impact on education with their peak Google scholar score. In fact, the upcoming learning analytics events might show extinction of methods and an uprising appearance or emergence of new techniques, which can be allocated in our defined categories. Finally, we summarized our experience in this field and listed some of the constraints and dilemmas that negatively affect learning analytics approaches.
References
Aguilar, S., Lonn, S., & Teasley, S. D., 2014. Perceptions and use of an early warning system during a higher education transition program. In
Proceedings of the fourth international conference on learning analytics and knowledge (pp. 113-117). ACM. Baker, R. S. J. D., Duval, E., Stamper, J., Wiley, D., & Buckingham Shum, S., 2012. Panel: educational data mining meets learning analytics. In
Proceedings of 2nd International Conference on Learning Analytics & Knowledge (LAK'12), New York, USA (p. 20). Baker, R. S., & Siemens, G., 2013. Educational Data Mining and Learning Analytics. Baron, J., Lynch, G., Maziarz, N., 2015. Proceedings of the Fifth International Conference on Learning Analytics and Knowledge. ACM, New York, USA. Bogarín, A., Romero, C., Cerezo, R., & Sánchez-Santillán, M., 2014. Clustering for improving educational process mining. In
Proceedings of the Fourth International Conference on Learning Analytics and Knowledge (pp. 11-15). ACM.
ISSN 2414-6099
Camilleri, V., de Freitas, S., Montebello, M., & McDonagh-Smith, P., 2013. A case study inside virtual worlds: use of analytics for immersive spaces. In
Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 230-234). ACM. Campbell, J. P., DeBlois, P. B., & Oblinger, D. G., 2007. Academic analytics: A new tool for a new era.
EDUCAUSE review , (4), 40. Chiu, M. M., & Fujita, N., 2014. Statistical discourse analysis of online discussions: informal cognition, social metacognition and knowledge creation. In Proceedings of the Fourth International Conference on Learning Analytics And Knowledge (pp. 217-225). ACM. Clow, D., 2012. The learning analytics cycle: closing the loop effectively. In
Proceedings of the 2nd international conference on learning analytics and knowledge (pp. 134-138). ACM. Davidian, M., & Louis, T. A., 2012. Why statistics?.
Science , (6077), 12-12. Dawson, S., Gašević, D., Siemens, G., & Joksimovic, S., 2014. Current state and future trends: A citation network analysis of the learning analytics field. In Proceedings of the Fourth International Conference on Learning Analytics and Knowledge (pp. 231-240). ACM. Dyckhoff, A. L., Lukarov, V., Muslim, A., Chatti, M. A., & Schroeder, U., 2013. Supporting action research with learning analytics. In
Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 220-229). ACM. Eagle, M., Hicks, D., Peddycord III, B., & Barnes, T., 2015. Exploring networks of problem-solving interactions. In
Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (pp. 21-30). ACM. Elias, T., 2011. Learning Analytics: Definitions, Processes and Potential. Available from: http://learninganalytics.net/LearningAnalyticsDefinitionsProcessesPotential.pdf. [25 October 2015]. Ezen-Can, A., Boyer, K. E., Kellogg, S., & Booth, S., 2015. Unsupervised modeling for understanding MOOC discussion forums: a learning analytics approach. In
Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (pp. 146-150). ACM. Ferguson, R., 2012. Learning analytics: drivers, developments and challenges. In
International Journal of Technology Enhanced Learning , (5-6), 304-317. Ferguson, R., Wei, Z., He, Y., & Buckingham Shum, S., 2013. An evaluation of learning analytics to identify exploratory dialogue in online discussions. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 85-93). ACM. Grann, J., & Bushway, D., 2014. Competency map: visualizing student learning to promote student success. In
Proceedings of the fourth international conference on learning analytics and knowledge (pp. 168-172). ACM. Hecking, T., Ziebarth, S., & Hoppe, H. U., 2014. Analysis of dynamic resource access patterns in a blended learning course. In
Proceedings of the Fourth International Conference on Learning Analytics and Knowledge . ACM.
ISSN 2414-6099
Holman, C., Aguilar, S., & Fishman, B., 2013. GradeCraft: what can we learn from a game-inspired learning management system?. In
Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 260-264). ACM. Jo, I. H., Kim, D., & Yoon, M., 2014. Analyzing the log patterns of adult learners in LMS using learning analytics. In
Proceedings of the Fourth International Conference on Learning Analytics And Knowledge (pp. 183-187). ACM. Johnson, L., Adams Becker, S., Cummins, M., Estrada, V., Freeman, A., & Ludgate, H., 2013. NMC Horizon Report: 2013 Higher Education Edition. Texas: The New Media. Joksimović, S., Dowell, N., Skrypnyk, O., Kovanović, V., Gašević, D., Dawson, S., & Graesser, A. C., 2015. How do you connect?: analysis of social capital accumulation in connectivist MOOCs. In
Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (pp. 64-68). ACM. Kapler, T., & Wright, W., 2005. GeoTime information visualization.
Information Visualization , (2), 136-146. Kapp, K. M., 2012. The gamification of learning and instruction: game-based methods and strategies for training and education. John Wiley & Sons. Khalil, M., & Ebner, M., 2015. Learning Analytics: Principles and Constraints. In Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications (pp. 1326-1336). Khalil, M., & Ebner, M., 2016. De-Identification in Learning Analytics.
Journal of Learning Analytics . Accepted, in Print. Kizilcec, R. F., Piech, C., & Schneider, E., 2013. Deconstructing disengagement: analyzing learner subpopulations in massive open online courses. In
Proceedings of the third international conference on learning analytics and knowledge (pp. 170-179). ACM. Koulocheri, E., & Xenos, M., 2013. Considering formal assessment in learning analytics within a PLE: the HOU2LEARN case. In
Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 28-32). ACM. Leeman-Munk, S. P., Wiebe, E. N., & Lester, J. C., 2014. Assessing elementary students' science competency with text analytics. In
Proceedings of the Fourth International Conference on Learning Analytics And Knowledge (pp. 143-147). ACM. Martinez-Maldonado, R., Pardo, A., Mirriahi, N., Yacef, K., Kay, J., & Clayphan, A., 2015. The LATUX workflow: designing and deploying awareness tools in technology-enabled learning settings. In
Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (pp. 1-10). ACM. Mendiburo, M., Sulcer, B., & Hasselbring, T., 2014. Interaction design for improved analytics. In
Proceedings of the Fourth International Conference on Learning Analytics And Knowledge (pp. 78-82). ACM. Pardos, Z. A., Baker, R. S., San Pedro, M. O., Gowda, S. M., & Gowda, S. M., 2013. Affective states and state tests: Investigating how affect throughout the school year predicts end of year learning outcomes. In
Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 117-124). ACM.
ISSN 2414-6099
Pistilli, M., Willis, J., Koch, D., 2014.
Proceedings of the Fourth International Conference on Learning Analytics And Knowledge . ACM, New York, USA. Renzel, D., & Klamma, R., 2013. From micro to macro: Analyzing activity in the ROLE sandbox. In
Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 250-254). ACM. Romero, C., & Ventura, S., 2010. Educational data mining: a review of the state of the art.
Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on , (6), 601-618. Santos, J. L., Klerkx, J., Duval, E., Gago, D., & Rodríguez, L., 2014. Success, activity and drop-outs in MOOCs an exploratory study on the UNED COMA courses. In Proceedings of the Fourth International Conference on Learning Analytics and Knowledge (pp. 98-102). ACM. Santos, J. L., Verbert, K., Govaerts, S., & Duval, E., 2013. Addressing learner issues with StepUp!: an Evaluation. In
Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 14-22). ACM. Schreurs, B., Teplovs, C., Ferguson, R., De Laat, M., & Buckingham Shum, S., 2013. Visualizing social learning ties by type and topic: Rationale and concept demonstrator. In
Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 33-37). ACM. Siemens, G., 2012. Learning analytics: envisioning a research discipline and a domain of practice. In
Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (pp. 4-8). ACM. Southavilay, V., Yacef, K., Reimann, P., & Calvo, R. A., 2013. Analysis of collaborative writing processes using revision maps and probabilistic topic models. In
Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp. 38-47). ACM. Suthers, D., Verbert, K., Duval, E., & Ochoa, X., 2013.
Proceedings of the Third International Conference on Learning Analytics and Knowledge . ACM, New York, USA. Taraghi, B., Ebner, M., Saranti, A., & Schön, M., 2014, March. On using markov chain to evidence the learning structures and difficulty levels of one digit multiplication. In
Proceedings of the Fourth International Conference on Learning Analytics And Knowledge (pp. 68-72). ACM. Vogelsang, T., & Ruppertz, L., 2015. On the validity of peer grading and a cloud teaching assistant system. In
Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (pp. 41-50). Wolff, A., Zdrahal, Z., Nikolov, A., & Pantucek, M., 2013. Improving retention: predicting at-risk students by analysing clicking behaviour in a virtual learning environment. In proceedings of the third international conference on learning analytics and knowledge (pp. 145-149). ACM. Xing, W., Wadholm, B., & Goggins, S., 2014. Learning analytics in CSCL with a focus on assessment: an exploratory study of activity theory-informed cluster analysis. In
Proceeding of LAK’14 . ACM.. ACM.