An Application of HodgeRank to Online Peer Assessment
AAn Application of HodgeRank to Online PeerAssessment
Tse-Yu Lin ∗ Yen-Lung Tsai † Abstract
Bias and heterogeneity in peer assessment can lead to the issue ofunfair scoring in the educational field. To deal with this problem, wepropose a reference ranking method for an online peer assessment systemusing HodgeRank. Such a scheme provides instructors with an objectivescoring reference based on mathematics.
In this paper, we construct a reference score for online peer assessments basedon HodgeRank [5]. Peer assessment is a process in which students grade theirpeers assignments [2, 6].A peer assignment system is used to enhance students learning process,especially in higher education. Through such a system, students are given theopportunity to not only learn knowledge from textbooks and instructors, butalso from the process of making judgements on assignments completed by theirpeers. This process helps them understand the weaknesses and strengths in thework of others, and then to review their own.However, there are some practical issues associated with a peer assignmentsystem. For example, students tend to give significantly higher grades thansenior graders or professionals (see [3] for more details). Also, students have atendency to give grades within a range, with the center of such a range oftenbeing based on the first grade they gave. Therefore, bias and heterogeneity canoccur in a peer assignment system.There are various ranking methods on peer assessment problem, such asPeerRank [7] and Borda-like aggregation algorithm [1]. PeerRank, a famousmethod based on a iterative process to solve the fixed-point equation. Peer-Rank has many interesting properties from the view of linear algebra. Borda-like aggregation algorithm, a random matheod based on the theory of randomgraphs and voting theory, whcih provides some probabilistic explanation on peerassessment problem.In this paper, we propose another ranking scheme to deal with peer as-sessment problems that uses HodgeRank, a statistical preference aggregation ∗ Department of Mathematical Sciences, National Chengchi University. † Department of Mathematical Sciences, National Chengchi University. a r X i v : . [ s t a t . M L ] M a r roblem from pairwise comparison data. The purpose of HodgeRank is to finda global ranking system based on pairwise comparison data. HodgeRank cannot only generate a ranking order, but also highlight inconsistencies in the com-parisons (see [5] for more detail). We apply HodgeRank to the problems inonline assessment and display ranking results from HodgeRank and PeerRankin turn.We will briefly introduce HodegRank and its useful properties in next section. HodgeRank, a statistical ranking method based on combinatorial Hodge theoryto find a consistent ranking. Rigorously speaking, HodgeRank is one solutionof a graph Laplacian problem with minimum Euclidean norm.Now, we start from notations borrowed from graph theory.Consider a connected graph G = ( V, E ), where V = { , , · · · , n } is the setof alternatives to be ranked, and E ⊆ V × V , consists of some unordered pairsfrom V .In this paper, V represents the set of students to be ranked by their peers,and E collects the information of pairwise comparisons. i.e., ( i, j ) ∈ E if stu-dents i and j are compared at least once.Denote Λ to be the number of assignments. Then for each assignment α ∈ Λ,pairwise comparison data on a graph G of assignment α , is given by Y α : E → R so that Y α is skew-symmetry. i.e., Y αij = − Y αji for all i, j ∈ V . Y αij > j is higher than student i by Y αij credits. For example, Y αij ∈ [ − , α ∈ Λ, a weight matrix W α = [ w αij ] is associated as follows: w αij > Y αij (cid:54) = 0, and 0 otherwise. Set W = (cid:80) α ∈ Λ W α .Let Y = (cid:80) α ∈ Λ Y α be a n -by- n matrix. The goal of the HodgeRank is find aranking s : V → R so that Y ij = s j − s i for all i, j ∈ V. (1)However, equations (1) can not be admissible in general. Consider the fol-lowing example, Y = − − −
11 1 0 If there exists s : V → R such that (1) hold. Then1 = Y = s − s = ( s − s ) + ( s − s ) = Y + Y = 0which leads to a contradiction. That is, it is impossible to solve (1) for any skew-symmetric matrix Y . Therefore, we should consider the least square solutionof (1) instead. Before we rewrite above problem, we need to introduce somenotations below. 2 efinition 2.1 [5] Denote M G = { X ∈ R n × n | X ij = s i − s j for some s : V → R } , the space of global ranking, and the combinatorial gradient operatorgrad : F ( V, R ) → M G is an operator defined from F ( V, R ), the set of all function from V to R (or thespace of all potential functions), to M G , as follows (cid:0) grad s (cid:1) ( i, j ) = s j − s i . From the example above, it is easy to find that if X = grad ( s ) for some s ∈ F ( V, R ), then X ij + X jk + X ki = 0 for any ( i, j ) , ( j, k ) , ( k, i ) ∈ E . However,the converse might not be true in general. That is, denote A = { X ∈ R n × n | X T = − X } , the set of all skew-symmetric matrices, and let M T = { X ∈ A | X ij + X jk + X ki = 0 } , then M G ⊆ M T .With these notations above, then the above problem becomes the followingoptimization problem:min X ∈M G || X − Y || ,w = min X ∈M G (cid:88) ( i,j ) ∈ E w ij ( X ij − Y ij ) That is, once a graph is given, then the weight on edge E determines anoptimization problem. Conversely, a graph can intuitively arise from the rankingdata.Let { Y α | α ∈ Λ } be a set of n -by- n skew-symmetric matrices, and { W α | α ∈ Λ } is associated as above.Then an undirected graph G = ( V, E ) can be defined by V = { , , · · · , n } and E = { ( i, j ) ∈ V × V | W ij > } . In this case, we can treat X as a edge flow on G in the sense of combinatorialvector calculus.In conclusion, we have the following relation between graph and G = ( V, E ) (cid:40) X T = − XW = (cid:80) α ∈ Λ W α .Hence, the optimization problem of a skew-symmetric least square problemcan be view as an optimization problem of edge flow on a graph.3 efinition 2.2 (Consistency) [5] Let X : V × X → R be a pairwise rankingedge flow on a graph G = ( G, E ). • X is called consistency on { i, j, k } if( i, j ) , ( j, k ) , ( k, i ) ∈ E and X ∈ M T • X is called globally consistency on { i, j, k } if X = grad( s ) for some s ∈F ( V, R )Note that if X is called globally consistency, then X is consistency on any3-clique { i, j, k } , where ( i, j ) , ( j, k ) , ( k, i ) ∈ E .Now, consider the weighted trace induced by W . i.e., < X, Y > = tr (cid:0) X T ( W (cid:12) Y ) (cid:1) = (cid:88) ( i,j ) ∈ E W ij X ij Y ij for X, Y ∈ A , where (cid:12) represents the Hadamard product or elementwise prod-uct.With this weighted inner product, we obtain two orthogonal complement of
A A = M G ⊕ M ⊥ G = M T ⊕ M ⊥ T Since M G ⊆ M T , we have M ⊥ G ⊇ M ⊥ T and we can get further orthogonaldirect sum decomposition of A as follows: A = M G ⊕ M H ⊕ M ⊥ T , where M H = M T ∩ M ⊥ G .This decomposition is called the combinatorial Hodge decomposition. Formore detail about the theory of combinatorial Hodge decomposition, pleaserefer [5] for more detail.We now state one useful theorem in [5]. Theorem 2.1 [5]1. The minimum norm solution s of (1) is the solution of the normal equation:∆ s = − div Y, where ∆ = (cid:80) ( i,j ) w ij if i = j − w ij if j ∈ V with ( i, j ) ∈ E Y )( i ) = (cid:88) js.t. ( i,j ) ∈ E w ij Y ij is the combinatorial curl operator of Y .4. The minimum norm solution s of (1) is s ∗ = − ∆ † div Y, where ∆ † represents the Moore-Penrose pseudo inverse of the matrix ∆ .The Hodge decomposition indicates the solution of (1), while the theorem 2.1shows how to calculate the minimum solution by solving the normal equation.In the next section, we display how to apply HodgeRank to the online peerassessment problem. As previously mentioned, bias and heterogeneity can lead to unfair scoring inonline peer assessments. Students usually grade other students based on the firstscore they gave, which causes bias. However, since scores are usually comparedwith others, we can use this comparison behavior to reconstruct true ranking.The data we used in this section were collected from an undergraduate cal-culus course. In this course, 133 students were asked to upload their GeoGebra[4] assignments. Each student was then asked to review five randomly chosenassignments completed by their peers to receive partial credits in return. Thereare 13 assignments during one semester.Note that ne key point of the HodgeRank is the connectedness of the graphgenerated by pairwise comparison data. From table 1 above, we can easily seethat after half the semester passed, comparison data between students forms aconnected graph. Hence, we can apply HodgeRank to calculate the ranking ofall the students after assignment 7.Table 1: Number of components with respect to the number of assignmentsAssignment ∼ α, β ) = (0 . ,
0) in the setting of PeerRank. For what theseparameters represent in PeerRank, please refer to [7] for more discussion.To compare these results, ranking results were normalized into the interval[0, 1] linearly and sorted in ascending order. In addition, to reveal the tendencyof each ranking method, a steady line was plotted on the graph. There are someinteresting implications that can be observed from this figure.First, the cumulative score offers a ranking higher than the steady line.This reflects the existence of bias and heterogeneity in the cumulative average5igure 1: Final results using different ranking methodsmethod. Second, PeerRank can be viewed as a modification of the averagescoring. Third, sorted ranking result from HodgeRank is a normal distributedcurve. This result can might be an explanation why HodgeRank can be solutionto eliminate bias and heterogeneity by the normality.Note that the reason why HodgeRank and PeerRank show different resultsis their conclusion base are totally different, while former method relies on thepairwise comparison data and latter one is applied on the average score asan initial ranking. Hence, HodgeRank provides instructors with an objectivescoring reference using score difference rather than cumulative or average score.In conclusion, this is the first time HodgeRank has been applied in the field ofeducation. While numerical results were processed using real world data in thisstudy, certain issues, such as how to aggregate the HodgeRank ranking methodinto a peer assessment system, remain unsolved. This task will be attemptedas part of our future work.
References [1] I. Caragiannis et al.,
Aggregating Partial Rankings with Applications to PeerGrading in Massive Online Open Courses , AAMAS ’15 Proceedings of the2015 International Conference on Autonomous Agents and Multiagent Sys-tems, (2015).[2]
N. Falchikov and J. Goldfinch , Student peer assessment in higher ed-ucation: A meta-analysis comparing peer and teacher marks , Review of ed-ucational research, 70 (2000), pp. 287–322.[3]
S. Freeman and J. W. Parks , How accurate is peer grading? , CBE-LifeSciences Education, 9 (2010), pp. 482–488.[4]
M. Hohenwarter , Geogebra–ein softwaresystem f¨ur dynamische geometrieund algebra der ebene [geogebra–a software system for dynamic geometry nd algebra in the plane](unpublished master’s thesis) , Universit¨at Salzburg,Salzburg, Austria, (2002).[5] X. Jiang, L.-H. Lim, Y. Yao, and Y. Ye , Statistical ranking and combi-natorial hodge theory , Mathematical Programming, 127 (2011), pp. 203–244.[6]
K. Topping , Peer assessment between students in colleges and universities ,Review of educational Research, 68 (1998), pp. 249–276.[7]