Leveraging Peer Review in Visualization Education: A Proposal for a New Model
LLeveraging Peer Review in Visualization Education: A Proposal for aNew Model
Alon Friedman and Paul Rosen Abstract — In visualization education, both science and hu-manities, the literature is often divided into two parts: thedesign aspect and the analysis of the visualization. However,we find limited discussion on how to motivate and engagevisualization students in the classroom. In the field of WritingStudies, researchers develop tools and frameworks for studentpeer review of writing. Based on the literature review from thefield of Writing Studies, this paper proposes a new frameworkto implement visualization peer review in the classroom toengage today’s students. This framework can be customizedfor incremental and double-blind review to inspire studentsand reinforce critical thinking about visualization.
Keywords:
Visualization education, peer review, studentmotivation, new model of visual peer review.I. INTRODUCTIONRushmeier et. al [1], defined visual education as a workin progress. Meanwhile, the number of students enrolled invisualization courses, both face-to-face and online, has growndramatically in recent years—a growth that makes it difficultto provide students the feedback they need to improve thequality of their work.Visualization evaluation has been addressed by a numberof researchers [2], [3], [4]. While many techniques andalgorithms can be objectively measured, most recognize thatthere are also significant subjective measurements necessaryto completely evaluate the effectiveness of a visualization [5].Such forms of subjective measurement are not just importantfor the instructor’s assessment of students, but the studentsalso need the ability to critically evaluate a visualization.For practical purposes, visualization education can besplit into two categories. The first is learning the properconstruction of visualization—using the right visualizationprincipals in creating your own visualization. This tends tobe the primary focus of visualization courses. In an era withincreasing proliferation of visualization in science and massmedia, the second category is focused on evaluation of thequality and accuracy of visualizations in general. These skillstend to be taught through less formal methods, such as groupor class discussion. Less formal methods, while critical, leavestudents’ skills underdeveloped.To this end, we propose a framework for engaging stu-dents in peer review as part of their formal visualization *This work was not supported by any organization A. Friedman is with the School of Information, University of SouthFlorida, Tampa, FL 33620, USA alon.friedman at usf.edu P. Rosen is with the Department of Computer Science and Engineer-ing, University of South Florida, Tampa, FL 33620, USA prosen atusf.edu coursework. This framework contains 2 parts. First, weprovide a reconfigurable assessment rubric for students tolearn how to properly peer review visualizations. Second, weprovide a step-by-step approach to educate students in properevaluation techniques by gradually reducing the constraintson their peer reviews over the course of a semester.II. E
DUCATION AND PEER REVIEW
Assessment is the heart of formal higher education. Brans-ford et. al [6] defines assessment as a core componentfor effective learning. Student assessment has many formsin higher education, and one of these forms, used mostcommonly in the liberal and creative arts, is peer review.Peer review was established as an essential component formany professional practices, such as the scholarly publicationprocess. The fundamental principle for peer review is thatexperts in any given domain appraise the professional per-formance, creativity, or quality of scientific work producedby others in their area of competence. Peer review is oftendefined as the evaluation of work by one or more people ofsimilar competence to the producers of the work [7]. Theprinciple component of student peer review is assessment ofstudents by other students, both formative reviews to providefeedback and summative grading.III. M
OXLEY R UBRIC
Moxley [8] reports that the use of peer review technolo-gies is changing the ecology of assessment and studentmotivation. “Rather than happening after the fact, onlineassessment systems are becoming part of the teaching andlearning process, part of the digital experience that studentsare so motivated to be part of.” Moxley utilizes this rubric todevelop online peer review, where the instructor can choosebetween two versions of the common rubric: (1) the numericrubric, which requires students to score rubric criteria (Focus,Evidence, Organization, Style, and Format) based on a five-point scale; and (2) the discussion rubric, which requiresstudents to write textual comments regarding these criteriarather than scores. While the textual feedback provides moredetail, research has shown that instructors have favored thenumeric version of the rubric over the discussion version [8].A reproduction of the rubric is seen in Figure 1.IV. B
ACKGROUND IN V ISUALIZATION E DUCATION
In 2006, the ACM Siggraph Education Committee startedthe discussion on what visualization education should be.Domik [9] recaptured this discussion by posting the ques-tion: “Do we need formal education in visualization?” She a r X i v : . [ c s . H C ] J a n nswered this question by stating that we do at least needinformal visualization education. In the 2000s, humanitiesdepartments started to offer visualization courses to their stu-dents, and as a result, Pat Hanrahan proposed implementationof core topics to fit the humanities, including: data and imagemodels; perception and cognition; interaction; space; color;shape and lines; texture; interior structure with volumetrictechniques; and narrative with animation [10].For this study, we embraced Hanrahan’s core topics in ourclasses, due in part to the different backgrounds our studentsbring to our classes. Understanding the breadth of these topicareas is not necessarily sufficient to be considered skilled invisualization. Gilbert [11] described 5 levels of competencythat give insight to the depth of topical understanding invisualization. His method outlines step-by-step instructionon visualization design that includes: step 1, representationas depiction; step 2, converting early symbolic skills; step 3,the syntactic use of formal representation; step 4, semanticuse of formal representations; and step 5, reflective, rhetoricaluse of presentation. However, we did not find any discussionor references on how to embed these steps into the analysisof visualization or peer review of visualization.V. A PPROACH
In 2013 and 2014, the University of South Florida hiredtwo faculty into the College of Arts and Sciences and Collegeof Engineering, respectively, to build visualization into theircurriculum. This effort was taking place in the context of alarger university-led effort towards improving visualizationskills and access to visualization equipment. Both facultyfocus on teaching visualization as any technique for creatingimages, diagrams, or animations to communicate a messageusing different platforms and programming languages.In the School of Information under the College of Artsand Sciences, a course called “Visualization of Big Data”was offered. The class is part of new area of concentrationin Data Science and was offered to students via an onlinesetting only. In the College of Engineering the Departmentof Computer Science and Engineering offered the classcalled “Scientific Visualization.” The class was offered intraditional classroom format only as an elective. The keydifferences between these courses were format (i.e., onlinevs. classroom) and emphasis (i.e., design for the InformationScience course and algorithmic for the Computer Sciencecourse). In both classes the instructors focused not just ondesign and algorithmic technique, but they also focused onthe analysis of visualizations using models such as the “what,why, and how” model of Liu and Stasko [12] as a frameworkfor discussion.During these 2 courses, we employed several peer reviewtechniques. We collected the students’ peer review commentswith the hope to find patterns. Our premises for the investi-gation was based on the three principals of Monk et al. [13]: • Careless mapping from data to pictures may lead toerroneous interpretation. • A substantial amount of knowledge is necessary togenerate images depicting complex information in a Fig. 1: Reproduction of the Moxley Rubric from [8].way that prevents erroneous interpretation. • Decision making is increasingly based on visual repre-sentations. VI. R
ESULTS
We did not find a single student who directly refereed tothe Monk’s principles in their peer review, as they were cov-ered in class and in their assignments. Many of the studentsselected one aspect of visualization without considering theother aspects that are involved in creating and evaluatingvisualization work. For example, many students simply ig-nored considerations such as the variables embedded, visualpatterns, or time. To illustrate our findings, the following is acommon example from a student peer review. The student inthis case was asked to review their peer’s histogram createdusing R, based on Monk’s principles. The histogram had tobe displayed in multiple colors as part of the assignment.
This is the first time in my life I had to providecomments and criticism on my peers’ work usingopen-source R. I certainly had my struggles withR to create visualization, but in this assignment, Ihave problems understanding to the complexity ofvisualization and visualization creation using R.
Furthermore, many of the comments we examined did nothave any direct connection to the content of the visualizationitself. Thus, we propose a new framework to better teach theevaluation of visualizations.II. P
ROPOSED F RAMEWORK
In the field of Writing Studies, researchers have reportedon the practice of using generic rubrics to make globalclaims about writing development. Beaufort [14], Nowacek[15], and Wardle and Roozen [16] have explored how theability to write well is a skill that can be mastered in onecontext and then simply carried over to another context.Moxley [8] opposed this approach stating that arguments thatmake grand claims about student ability based on a handfulof rubric scores need to be seriously challenged. Students’scores on one rubric are not necessarily predictive of howthey will do when facing alternative genres. To addressthe limited availability of visualization peer review toolsand techniques in the classroom, we propose the followingframework adhering to Moxley’s recommendation.
A. Peer Review Rubric for visualization education
The basic structure of our rubric is to divide topics into5 major assessment categories, with each category havingsub-assessments. The sub-assessments contain 3 scoring cat-egories affixed to a 5-point scale (though no scale is shown).The category on the left is reserved for poor performance onthe assessment, middle for average performance, and rightfor excellent performance. Each sub-assessment contains acomments box for providing details about the scoring.For our rubric, the 5 major assessment categories are:algorithm design, interaction design, visual design, designconsideration, and visualization narrative. The categoriescover the following subjects: The algorithm design categoryis concerned with algorithm selection and implementation.Interaction design is concerned with how the user interactswith the visualization. Visual design is concerned with thetechnical aspects of how data are placed in the visualization.Examples include, visual encoding channels, their expres-siveness, and their effectiveness. Design consideration fo-cuses on composition and aesthetics aspects of the visualiza-tion, such as embellishments. The final category visualizationnarrative is critical in projects where the story surroundingthe visualization is as important as the visualization itself.This category provides a basic framework for assessingwhether the visualization supports the story and vice versa.
B. Customize Rubric
We also consider customization of the rubric, so it can bebuilt based upon the content of an assignment or customizedto fit the course content. Instructors should take our fulltemplate, extract the critical components, and add missingcomponents with respect to their assignment and coursecontent. For example, on a computer science assignment, therubric provides the ability to remove or reduce the numberof components dealing with aesthetics and narrative, whileemphasizing those for algorithm. On the other hand, forprojects that do not implement an algorithm or those forwhich the algorithm is not critical may skip this category.Another aspect in which this rubric can also be customizedis to reduce the constraints of the rubric itself. Assumingthe critical thinking skills of students, particularly in the topic of visualization, are weak, the rubric itself can be atool to help improve those skills. At the beginning of thecourse, the rubric can be provided that includes all scoringcategories to build an analytic framework in the students’minds. As their skills improve, the 5-point scoring scalecan be removed, leaving only the comments section behind.Finally, as students begin to perfect their skills, the sub-assessments can be removed altogether, leaving only themajor categories of evaluation. In this way, students will gofrom highly constrained and guided evaluation to free-formevaluation over the course of a semester.Finally, we foresee the peer review process appearingmuch like conference or journal reviews. First, double-blindreview helps to reduce the risk of collusion or maliciousreview. Second, multiple reviews per visualization help toensure that consensus is built around the evaluation. Inaddition, each student having to produce multiple reviewshelps to reinforce the critical thinking skills. Some of thesefunctionalities are available in learning management systems,such as Canvas.VIII. R
ISKS IN P EER R EVIEW
As we implement this framework in our own classrooms,we are keeping an eye out for a couple of key issues. Forexample, peer review is a compliment to existing educationalapproaches, but the instructor, as the expert, still needs toprovide feedback. Otherwise, some important subtleties maybe missed. Second, the risk for collusion or malice are great,even with double-blind reviews. Students talk, and whenthey talk, they will undoubtedly discover they are evaluatingeach other’s visualizations. Therefore, instructors should takecare to randomize peer reviewers, and grades should onlybe loosely based upon the results of the peer review. Itis also important to remember that there is both art andscience to design with no single optimal design. Therefore, apeer review may not necessarily be the best mechanism forproviding grades. However, peer review naturally supportsiterative design improvements [5].IX. C
ONCLUSION
We have presented a new framework for evaluating studentvisualizations based upon peer review rubric. This rubric hasa couple of important aspects. First, it provided students aframework for critical thinking and evaluation of visualiza-tions. Second, it provides a mechanism to help with issuesof larger class sizes, while still providing students feedbackabout their work.Finally, we do not believe that the rubric as presented willbe the final static version. We anticipate it will be a growingand evolving document as community members provide theirinput and the focus of the visualization community changes.As a result, Appendices A-E represents our proposal for stu-dent peer review rubrics based on specific subjects from algo-rithmic design, interaction design, visual design, design con-sideration, and visualization narrative. For the full proposedvisualization rubric, see Appendix A-E. A L A TEXversion ofhe rubric can be cloned/forked at https://github.com/USFDataVisualization/VisPeerReview .APPENDIX
Appendix A : Algorithmic Design
Selection of Algorithm - Of theavailable algorithm options was the bestselected? What aspects of this algorithmmake it better or worse than theother possible choices? (cid:50)
BelowAverage (cid:50) (cid:50)
Average (cid:50) (cid:50)
AboveAverageComments:Correct Implementation - Does thealgorithm appear to produce the correctresult, given your knowledge of the data? (cid:50) No (cid:50) (cid:50) MinorErrors (cid:50) (cid:50)
AppearsCorrectComments:Efficient Implementation - Is theperformance (speed) of the algorithm whatyou expected? Is is slower? Is it faster? (cid:50)
MuchSlower (cid:50) (cid:50)
AsExpected (cid:50) (cid:50)
MuchFasterComments:Featureful Implementation - Does theimplementation contain the basicrequired features or are additional featuresincluded? (cid:50)
MajorFeaturesMissing (cid:50) (cid:50)
AsExpected (cid:50) (cid:50)
MajorFeaturesAddedComments:Datasets Used - Do the datasets providedgive enough information to evaluate thecorrectness, efficiency, and featurefulnessof the implementation? (cid:50)
NotUseful (cid:50) (cid:50)
AsExpected (cid:50) (cid:50)
BetterThanExpectedComments:
Appendix B : Interaction Design
Interaction Selection - Whatinteraction mechanisms arebeing used? (cid:50)
Linked Views (cid:50)
Filtering (cid:50)
Geometric Zoom (cid:50)
Selection (cid:50)
Aggregation (cid:50)
Pan/Translate (cid:50)
Highlighting (cid:50)
Semantic Zoom (cid:50)
RotateComments:Interaction Effectiveness - Arethe interactions provided highlyeffective? In other words, doesthe visualization react in amanner that makes it easy touse or capable of providing richcontent? (cid:50)
MissingKeyInteractions (cid:50) (cid:50)
Expected (cid:50) (cid:50)
BetterThanExpectedComments:
Appendix C : Visual Design
Visual Encodings - What visual channelswere used to encode data? (cid:50)
Position (cid:50)
Curvature (cid:50)
Area (cid:50)
Color Hue (cid:50)
Depth (cid:50)
Shape (cid:50)
Volume (cid:50)
Texture (cid:50)
Angle (cid:50)
Length (cid:50)
Luminance/Saturation (cid:50)
Motion /AnimationComments:Intended/Unintended Encodings - Do allof the visual encoding appear to beintended, or were some accidentallycreated? (cid:50)
ManyUnintended (cid:50) (cid:50)
FewUnintended (cid:50) (cid:50)
AllIntendedComments:Expressiveness of Encodings - Are thevisual encodings attached to the correcttype of data for that encoding (i.e. arequantitative data attached to quantitativeencodings and categorical data tocategorical encodings)? (cid:50)
ManyErrors (cid:50) (cid:50)
FewErrors (cid:50) (cid:50)
CorrectlyAssignedComments:Effectiveness of Encodings - Have themaximally effective visual encodingsbeen selected in all cases? (cid:50)
ManyIneffective (cid:50) (cid:50)
FewIneffective (cid:50) (cid:50)
MostEffectiveComments:Integral vs. Separable (Conjunction) -When visual encodings are mixed, dothe combined encodings make thevisualization more effective or do theymake interpretation more difficult? (cid:50)
MostlyIneffective (cid:50) (cid:50)
NoneUsed (cid:50) (cid:50)
HighlyEffectiveComments:Effective Use of Color - Is color usedin a same fashion? Do the colors chosenand the application of those colors makethe visualization effective? (cid:50)
MostlyIneffective (cid:50) (cid:50)
NoneUsed (cid:50) (cid:50)
HighlyEffectiveComments:Color Contrast and Harmony - Do theselected colors properly contrast andharmonize? (cid:50) No (cid:50) (cid:50) NoColorUsed (cid:50) (cid:50)
YesComments:Colorblind Safety - Is the visualizationcolorblind safe? Were redundant visualencodings used? (cid:50) No (cid:50) (cid:50) ColorblindSafeColor (cid:50) (cid:50)
RedundantComments:
Appendix D : Design Consideration
Clear, Detailed, and ThoroughLabeling - Is appropriate andcomplete labeling used throughoutor do missing labels requireassumptions about the data? (cid:50)
NoLabels (cid:50) (cid:50)
SomeMissingLabels (cid:50) (cid:50)
CompletelyLabeledComments:Missing Scales - Are scalesprovided for the data? (cid:50)
NoScales (cid:50) (cid:50)
SomeMissingScales (cid:50) (cid:50)
AllScalesPresentComments:Missing Legend - Is a legendprovided for the data? Does thelegend provide useful information? (cid:50)
NoLegend (cid:50) (cid:50)
IncompleteLegend (cid:50) (cid:50)
CompleteLegendComments:Scale Distortion - Is any scaledistortion or deception used inthe visualization? (cid:50)
SevereDistortion (cid:50) (cid:50)
MinorDistortion (cid:50) (cid:50)
No DistortionComments:Lie Factor - Is there any lie factor?How extreme is the lie factor? (cid:50)
MajorLie (cid:50) (cid:50)
MinorLie (cid:50) (cid:50)
NoLieComments:Data/Ink Ratio - Is the data to inkratio reasonable? Could it be moreefficient? (cid:50)
Way TooLittle / (cid:50)
Much Ink (cid:50) (cid:50)
Slightly TooLittle / (cid:50)
Much Ink (cid:50) (cid:50)
PerfectAmountof InkComments:Chart Junk, Embellishments,Aesthetics - Are appropriateembellishments used? Are theembellishments distracting? Dothe embellishments add tothe visualization? (cid:50)
Way Too Few / (cid:50)
ManyEmbellishments (cid:50) (cid:50)
A Bit TooFew / (cid:50)
ManyEmbellishments (cid:50) (cid:50)
PerfectNumber ofEmbellishmentsComments:Data Density - Has too much databeen included in the visualizationmaking interpretation difficult? (cid:50)
TooSparse (cid:50) (cid:50)
Expected (cid:50) (cid:50)
TooDenseComments:Task Selection - Does thevisualization enable appropriatevisual analysis tasks for the datatype and/or dataset? (cid:50)
UnclearTaskSelection (cid:50) (cid:50)
SomeTasksMissing (cid:50) (cid:50)
Wide Rangeof TasksSupportComments:Gestalt Principals - Have Gestaltprincipals been used to improveanalysis? (cid:50)
NoGestaltPrincipals (cid:50) (cid:50)
SomeGestaltPrincipals (cid:50) (cid:50)
ManyGestaltPrincipalsComments:
Appendix E : Visualization Narrative
Description of Visualization - Isa description of the visualizationaccurate and informative? (cid:50)
NoDescription (cid:50) (cid:50)
Incomplete / (cid:50)
Self-Explanatory (cid:50) (cid:50)
CompleteDescriptionComments:Support of Narrative - Does thevisualization support the messageof the narrative? (cid:50)
NoDescription (cid:50) (cid:50)
Incomplete / (cid:50)
Self-Explanatory (cid:50) (cid:50)
CompletelySupportiveComments:Dataset Used - Do the dataset(s)provide enough information anddetail to support the narrative? (cid:50)
Not AtAll (cid:50) (cid:50)
Partially (cid:50) (cid:50)
CompletelyComments: R EFERENCES[1] H. Rushmeier, J. Dykes, J. Dill, and P. Yoon, “Revisiting the needfor formal education in visualization,”
IEEE Computer Graphics andApplications , vol. 27, no. 6, 2007.[2] A. Shamim, V. Balakrishnan, and M. Tahir, “Evaluation of opinionvisualization techniques,”
Information Visualization , vol. 14, no. 4,2014.[3] M. J. Eppler, “A comparison between concept maps, mind maps,conceptual diagrams, and visual metaphors as complementary toolsfor knowledge construction and sharing,”
Information visualization ,vol. 5, no. 3, pp. 202–210, 2006.[4] N. JD and A. Ca˜nas, “The origins of the concept mapping tool andthe continuing evolution of the tool,”
Information Visualization , vol. 5,no. 3, 2006.[5] F. Vi´egas and M. Wattenberg, “Design and redesign in datavisualization,” March 2015. [Online]. Available: https://medium.com/@hint fm/design-and-redesign-4ab77206cf9[6] N. R. Council et al. , How people learn: Brain, mind, experience, andschool: Expanded edition . National Academies Press, 2000.[7] R. Spier, “The history of the peer-review process,”
TRENDS inBiotechnology , vol. 20, no. 8, pp. 357–358, 2002.[8] J. Moxley, “Big data, learning analytics, and social assessment,”
Journal of Writing Assessment , vol. 6, no. 1, 2013.[9] G. Domik, “Do we need formal education in visualization?”
IEEEComputer Graphics and Applications , vol. 20, no. 4, pp. 16–19, 2000.[10] K.-L. Ma, “Teaching visualization,”
ACM Siggraph Computer Graph-ics , vol. 39, no. 1, pp. 4–5, 2005.[11] J. K. Gilbert, “Visualization: An emergent field of practice and enquiryin science education,”
Visualization: Theory and practice in scienceeducation , pp. 3–24, 2008.12] Z. Liu and J. Stasko, “Theories in information visualization: What,why and how,” in
Workshop on the Role of Theory in InformationVisualization , 2010.[13] A. Monk, M. Hassenzahl, M. Blythe, and D. Reed, “Funology: de-signing enjoyment,” in
CHI’02 Extended Abstracts on Human Factorsin Computing Systems . ACM, 2002, pp. 924–925.[14] A. Beaufort,
College writing and beyond: A new framework foruniversity writing instruction . University Press of Colorado, 2008.[15] R. S. Nowacek,
Agents of integration: Understanding transfer as arhetorical act . SIU Press, 2011.[16] E. Wardle and K. Roozen, “Addressing the complexity of writingdevelopment: Toward an ecological model of assessment,”