Enhancing Reading Strategies by Exploring A Theme-based Approach to Literature Surveys
Tanya Howden, Pierre Le Bras, Thomas S. Methven, Stefano Padilla, Mike J. Chantler
EEnhancing Reading Strategies by ExploringA Theme-based Approach to Literature Surveys
T. Howden , P. Le Bras , T. S. Methven , S. Padilla , and M. J. Chantler School of Mathematical and Computer Sciences, Heriot-Watt University, Edinburgh, United Kingdom School of Computing, Edinburgh Napier University, Edinburgh, United Kingdom
Abstract
Searching large digital repositories can be extremely frustrating, as common list-based formats encourage users to adopta convenience-sampling approach that favours chance discovery and random search, over meaningful exploration. We havedesigned a methodology that allows users to visually and thematically explore corpora, while developing personalised holis-tic reading strategies. We describe the results of a three-phase qualitative study, in which experienced researchers used ourinteractive visualisation approach to analyse a set of publications and select relevant themes and papers. Using in-depth semi-structured interviews and stimulated recall, we found that users: (i) selected papers that they otherwise would not have read, (ii)developed a more coherent reading strategy, and (iii) understood the thematic structure and relationships between papers moreeffectively. Finally, we make six design recommendations to enhance current digital repositories that we have shown encourageusers to adopt a more holistic and thematic research approach.
CCS Concepts • Human-centered → Human computer interaction (HCI); Visualization; •
Applied computing → Collaborative learning;
1. Introduction
Large digital repositories of research papers and associated materi-als are ubiquitous and used on almost a day-to-day basis by manyresearchers [Ber96] [ZZYW15]. These repositories combine acces-sibility of information and technology to enable users to instantlyand conveniently search and access resources from diverse collec-tions as described by Cherukodan [CKK13]. As a result, these digi-tal repositories are commonly used by researchers in their standardapproach towards literature discovery and to facilitate their readingstrategies; however, they present challenges and issues.These repositories frequently use a keyword search to high-light resources that may be of relevance to the user; this methodhas been widely observed in current interfaces and broadly re-ported in research [ACM] [Goo] [IBM16] [Spr]. A disadvantage ofsearch methods is their reliance on the users’ expertise and previousknowledge of an area, this causes difficulties when users explorenew domains as described by Kotchoubey et al. [KA11] and Wilsonet al. [WKSS10], for example when they don’t know what to searchfor, or in the case of concept homonymy (e.g. “neural network” inbiology or computer science). Moreover, in these repositories, spe-cific fields of information are quite prominent in the search result(e.g., title and author information) [ACM] [Dir] [Goo] [Mic]; it is,however, unlikely that, for example, a title can adequately representthe whole content of the source. These disadvantages in currentrepositories and search methods increase the chance of users ex- ploring irrelevant sources, advocating for a more time-consumingand frustrating trial and error approach, and being stuck at the startof their literature surveys, a situation commonly experienced by re-searchers.To overcome these issues and challenges, we suggest usinga top-down approach as inspired by Wilson et al. [WKSS10],Padilla et al. [PMCC14] [PMC14] [PMRC17] and in Le Bras etal. [LBGR ∗
20] work, where users begin by browsing an overviewfrom a repository. Furthermore, Blei suggests that rather than find-ing new documents using traditional keyword search approaches, itwould be better for users to take a theme-based approach to exploreand digest collections [Ble12] [BNJ03]. We believe this behavior isa more natural solution to finding resources as it is common for lit-erature sources to be created from a set of themes organized into anarrative.In this paper, we explore user behaviors using thematic analysistools along with data visualization techniques to see if we can visu-alize firstly, theme-based overviews of a paper collection to enableobjective browsing and paper selection, and secondly, if visualiz-ing sequences and quantities of themes within individual papersin a paper set aids the generation of a holistic cross-paper readingstrategy. We conduct our investigation using a three-phase qualita-tive study, a set of tools, and a new six-step thematic methodology,as summarized in Figure 1 inspired by Shneiderman’s Visual In- © 2021 The Author(s) a r X i v : . [ c s . I R ] F e b . Howden, P. Le Bras, T. S. Methven, S. Padilla & M. J. Chantler / Enhancing Reading Strategies formation Seeking Mantra [Shn96] and Wilson’s et al. [WKSS10]exploration of information work.It should be emphasized that our objective is to explore whethertaking a thematic approach to browsing and selecting research pa-pers allows users to adopt a holistic approach to these tasks fol-lowed by developing a reading strategy. We are not exploring issueswith usability and performance of the proposed thematic methodol-ogy compared against commonly-used searching techniques in dig-ital repositories as we want to focus on the user behaviors, gather-ing insights, and suggesting possible add-on enhancements to cur-rent methods and tools.The contributions of this paper, in summary, are:1. We explore visual thematic tools and an associated methodologyfor the selection of a paper set and generation of a cross-paperreading strategy.2. We report insights on the effect of promoting thematic content,contrasted with the recalled experience of commonly-used title-based approaches and tools.3. We propose, from our results, six design recommendations (R1-R6) for enabling effective browsing and selection capabilities toimprove users’ experience and enhance current tools.
2. Background and Related Work
In this section, we look at current approaches for browsing and theselection of content from digital research repositories; we then dis-cuss how visualizations can aid those tasks and motivate our pro-posed methodology.
There are many different definitions of what is considered a digi-tal repository, otherwise known as digital libraries. Chowdhury andChowdhury [CC03] place digital repositories into two major cate-gories based on Borgman’s discussions [Bor99]. These categoriesfirstly look after collecting and organizing literature and secondlyfocus on accessing and retrieving these digital sources. In this pa-per, we concentrate on the latter and consider a digital repository tobe an online platform that allows users to search and retrieve digitalcopies of literature sources.These collections of resources are widely available from the pub-lishers themselves [ACM] [Dir] [IEE] [Spr]. Additionally, compa-nies such as Google and Microsoft provide search engines reachingmultiple repositories [Goo] [Mic]. All of these platforms integratethe same core mechanism for browsing, that is, using keywordsas the basis of the search, with the ability to then filter results us-ing facets such as date published, authors, institutions and publi-cation type [Dir] [XM16]. We believe that Shneiderman’s VisualInformation Seeking Mantra [Shn96] proposes another browsingmechanism: first offering an overview of a research area, then al-lowing the user to focus on particular themes, and finally givingaccess to the sources. A related approach has been partially imple-mented (Research Perspectives [Lab]), its use, however, remainsminor in comparison to the keyword search method. As a result,we believe more research is needed to explore the user behaviors to facilitate the use of such complementary approaches to commonsearch mechanism.Additionally, result listings majorly emphasize title and authorinformation, leaving out the explanation for relevance, and in turnthe order in which results appear. Beel and Gipp found from re-verse engineering techniques that the ranking algorithm by GoogleScholar [Goo] used the number of citations as the highest weightedfactor [BG09a]. They also found that the occurrence of searchterms in the title outweighed their occurrence in the full text, mak-ing no difference to the ranking if the search term appeared onlyonce or multiple times, thus presenting a biased representation ofthe source content [BG09b]. It also emphasizes difficulties in as-sessing the relevance of a source, given the prominence of attractivetitles [Hag04] [RBB16].Modern digital repository platforms have tried to visualize thetheme of the papers using word clouds and similar abstractions[Sco] [IBM16]; however, the main emphasis of their mechanismstill relies upon the search of title keywords to find resources. Toour knowledge, there is a lack of research and tools that offers usersthe ability to see thematic overviews, to explore how much of theirsearch term appears in sources, and that gauges the relevance ofthese to their interests.Finally, there is some work in the manual annotation of themes,for example using crowdsourcing techniques, ConceptScape allowsthe annotation of lecture videos to highlight the content of eachsection, resulting in the facilitation of content discovery [LKW18].Similar results could be achieved with textual content, for example,using analytic hierarchy processes [GWH89] [Kat14], or system-atic literature reviews [Nig09] [XW19]. These methods are; how-ever, time-consuming. Topic modeling [Ble12], and in particularLatent Dirichlet Allocation (LDA) [BNJ03], offers a time-efficientand effective method for uncovering and annotating the thematicstructures within documents. Such methods have successfully beenapplied by Zhao et al. in the context of MOOC video reposito-ries [ZBCS18].
Popular digital repositories, such as Google Scholar [Goo] or ACMDL [ACM], are heavily text-based, with limited amounts of im-agery. Cognitive style research [BK09] [Ric77] [TM10] suggeststhat visual users may not be using these text-based environmentsto their full potential. Therefore, being able to visualize literaturesources, with a focus on themes and thematic structures, couldbetter cater to these users preferred style of information presen-tation. Morris et al. [MFAV18] demonstrated this with dyslexicusers, where the interviewees reported a preference for interfaceuncluttered from substantial textual content. Besides, data visual-izations and pictorial representations allow for better recall [Car99][NRW76]; this highlight why techniques like icons and logos areused rather than text [Nor95].Notable work has been done to visualize search results ratherthan using text-based lists. WebSearchViz incorporates a circularinterface to show result relevance in terms of how close they are tothe center point of the circle [NZ06]. TileBars shows the length ofeach result, highlighting the frequency of the search term [Hea95]. © 2021 The Author(s) . Howden, P. Le Bras, T. S. Methven, S. Padilla & M. J. Chantler / Enhancing Reading Strategies
Figure 1:
Summary of the proposed thematic methodology. We first run an analysis of a large collection of papers (a → b) to generate atheme-based overview. Participants can then explore themes and their relevant papers. Moreover, participants can investigate each of thepapers using visual wheels, highlighting sequences and amounts for different themes. Finally, participants can compare selected resourcesto enable them generate an enhanced cross paper reading strategy (b → c). PubCloud presents a word cloud to summarize each of the listedresults [KHGW07]. Others, like LineUp, explicitly highlight howa result relates to each facet to explain a ranked list [GLG ∗ ∗
14] [DSG ∗
12] [PHRC13] andcompare [DES ∗
15] [AG15] documents in a collections; however,to our knowledge, nobody has focused on using a theme-based ap-proach to give an overview of a large collection of resources, orusing this same approach for analyzing and comparing sources togenerate a reading strategy.We believe that visual representations of collections and indi-vidual sources with a thematic emphasis could allow the users toreflect and recall back to these representations, assisting with theirbrowsing and selection tasks. Additionally, as we will be visualiz-ing sequences of themes to describe the progression of content ina research paper, we have found work has been done on visualiza-tion for sequences. MatrixWave [ZLD ∗
15] visualizes the sequenceof clickstream events on a website. Sankey diagrams are also com-monly used to visualize sequences of objects [RHF05]. We foundthat although these are novel ways of presenting sequences, wewanted a representation that would allow for no training and in-tuitive interaction to allow users to find papers with common quan-tities of their selected themes.
There has been substantial work done on providing insights intoranked search results using data visualization techniques, includinghow similar each resulting item is to one another. To our knowl- edge, however, none of the existing solutions have entirely focusedon using a visual theme-based approach to obtain a interactive vi-sual overview of a large collection of resources, that can be filteredto facilitate comparison and analysis of a paper set, and that assistprimarily with the generation of a holistic reading strategy.
3. Study Design
Our study aims to explore the following research questions:
RQ1:
Does visual thematic analysis as provided by the proposedmethodology and associated tools aid paper selection?
RQ2:
Does visual thematic analysis as provided by the proposedmethodology and associated tools aid generation of cross-paperreading strategies?
RQ3:
What are the advantages and disadvantages of the overallproposed visual thematic approach?To that end, we will ensure our participants are experienced withbrowsing scientific literature and establishing a reading strategy.Given this experience, and to reduce fatigue in the course of thestudy, we will not ask the participants to complete a keyword searchtasks to contrast for performance and usability against theme-basedtasks. However, we will ensure that participants are reminded ofthis approach using pre-study questionnaires, and we gather insightusing stimulated recall semi-structured interviews.We designed two user tasks that we ask our participants to carryout to explore and gather insights. Firstly, A) , browse and selecta set of research papers using an objective, theme-based overviewof a large paper collection. As stated in RQ1 , we are interested inanalyzing whether taking a theme-based approach, using thematicanalysis, aids the selection of papers. This task will also create thebasis for investigating
RQ2 .Secondly, B) , generate a cross-paper reading strategy using a the-matic comparison of a selected paper set. We are interested in fa-cilitating the generation of a reading strategy that considers a set ofpapers rather than individual strategies for each paper ( RQ2 ). © 2021 The Author(s) . Howden, P. Le Bras, T. S. Methven, S. Padilla & M. J. Chantler / Enhancing Reading Strategies Figure 2:
The first 3 stages of our thematic methodology. Tool 1 consist of (a) the selection of a large paper collection of informationresources, (b) thematic analysis of the paper collection resulting in a theme-based overview of the content, and (c) single theme analysishighlighting the relevant papers based on where that theme appears in the source.
Figure 3:
The final 3 stages of our thematic methodology. Tool 2 consists of (a) the focused paper set chosen by the user, (b) cross-paperthematic analysis of the paper set using theme wheels to represent sequences of themes from start to finish in each paper and, (c) generationof a cross-paper reading strategy.
In addition to their responses of these two user tasks, we analyzeperceptions of a theme-based discovery of literature. Throughoutthe user tasks, we are interested in observing behavior from ourparticipants interacting with our theme-based approach to evaluatewhether it allows for high-level insights into research papers, high-lighting its advantages and disadvantages as per
RQ3 . Based on these task requirements, we developed a thematicmethodology consisting of two associated thematic tools for thepresentation of a large paper collection, and the comparison of a pa-per set to facilitate the generation of a cross-paper, holistic readingstrategy. Our methodology can be summarized in six stages (Fig-ures 2 and 3):1. Definition of a large paper collection (Figure 2a);2. Thematic analysis of a large paper collection resulting in a vi-sual thematic map (Figure 2b);3. Upon selection of an individual theme from the thematic map, the top relevant papers are displayed, including the theme loca-tion in their content (Figure 2c);4. Six papers are selected by the user on the basis of their interestsin investigating these papers further (Figure 3a);5. Papers are represented as theme wheels showing the sequencesof themes from start to end, allowing for a cross-paper thematicanalysis (Figure 3b);6. An all-inclusive reading strategy based on all six papers is gen-erated by the user (Figure 3c).
This tool focuses on Task A, i.e. browsing and selecting within alarge paper collection, with the aim to cover stages 1-3 of our the-matic methodology outlined above (Figure 2). An overview of theselected large paper collection is shown using a similarity-basedthematic map (Figure 2b). This thematic map features clusters ofhexagons, each representing a group of similar themes found fromthe paper collection in a concise, structured and efficient setting.Having these themes rendered as clusters of hexagons allows usersto gather insights into the individual themes that are present and © 2021 The Author(s) . Howden, P. Le Bras, T. S. Methven, S. Padilla & M. J. Chantler / Enhancing Reading Strategies
Figure 4:
Thematic map evolving from Tool 1, featuring an overview of the paper collection to Tool 2 that features themes from the selectedpaper set. In the example above, Tool 1 includes 85 topics from 2,782 papers that filters down to only 35 topics from 6 papers in Tool 2. investigate which other areas are closely linked and may be of in-terest.
Clicking on a single theme will display a word-cloud representa-tion of the theme, and a listing of the top ten relevant papers, withan explanation for the ordering of the papers: each paper displaysits relevance percentage to the theme, and its theme wheel (Figure2c). These are donut chart visualizing which parts of the paper wereused to represent the estimation of each theme giving users infor-mation regarding where and by how much a theme occurs in thetext allowing for better insights, for example, establishing whetherthe theme is a minor feature of the background section, or consis-tently used throughout the paper. We chose this method of visual-ization instead of other types (e.g. bar charts) as these are more aes-thetically pleasing and to reinforce relevant percentages [War19],also incorporating images instead of only text can facilitate under-standing as explored in Robb’s et al work [RPKC15b] [RPKC15a][RPM ∗ This tool is implemented using data visualization techniques com-bined with topic modeling algorithms [Ble12] [BNJ03] that use sta-tistical methods to annotate large archives of documents with the-matic information, extracting the common themes among the docu- ments [Ban20] [Ble12]. We used Blei et al. Mallet implementation[McC02] with commonly recommended parameters [BGMN14] tocompile the themes for our study.We split the individual papers from the collection into equaltest chunks. We then use LDA [BNJ03] applying Gibbs Sampling[McC02], to uncover the themes and their distribution in the textchunks. We finally compiled the theme distributions for each pa-per. We visualize the set of uncovered themes in a similarity-basedthematic map, using an agglomerative layout process, as describedby Le Bras et al. [LBRM ∗ This thematic tool focuses on task B, i.e., generating a cross-paperreading strategy using the selected paper set, with the aim to coverstages 4-6 of our thematic methodology outlined above (Figure3). This tool allows for a theme-based analysis of the selectedpaper set, where we produce a truncated thematic map contain-ing only the themes that are relevant to the papers in the selectedset [LBRM ∗
18] (Figure 4).The size of this excerpt map will vary based on the selected paperset. In addition, each paper is represented alongside by its themewheel representing the structure of papers by visualizing the se-quence of themes from start to end (Figure 3b). © 2021 The Author(s) . Howden, P. Le Bras, T. S. Methven, S. Padilla & M. J. Chantler / Enhancing Reading Strategies
Upon interacting with either of these layouts (the thematic map orthe theme wheels), users are presented with a word-cloud to geta detailed description of the themes, emphasizing the relationshipbetween the elements on the screen [YaKSJ07]. This allows usersto analyze and compare a set of research papers, permitting an in-depth exploration of the consistency and changes of the themes thatthe paper authors discuss.The aesthetics of the excerpt map and theme wheels for Tool2 were designed to emphasize the different theme contributions,distinct themselves from task A (Tool 1), and to make it visuallyappealing to users.
Given the selected paper set by participants, the themes coveredby each of the selected papers are noted, and this information isextracted from our thematic map from Tool 1, meaning that thenumber of extracted themes will fluctuate depending on the papers.This creates a smaller thematic map that contains only the relevantthemes for these papers. Each of the themes is then re-evaluatedin terms of how similar they are to each other using agglomerativeclustering algorithms [ABKS14] [LBRM ∗
18] creating our focusedthematic map (Figure 4). Each cluster of themes is assigned a dif-ferent color, allowing for a conceptual link between the clusters andthe theme wheels.
Two pilot studies [Tur05] were completed to evaluate both tools in-dividually. We evaluated Tool 1’s usability with three participants.This evaluation consisted of a set of tasks followed by the comple-tion of SUS [GBKP13]. The set of tasks comprised of using thetool to explore literature about how users interact with data visu-alizations, select up to six papers that were believed to be usefulin gathering this knowledge and explain reasons for this selection.Tool 1 received an average usability score of 76 across participants,indicating good interface usability. It also helped us identify usabil-ity issues which we corrected.We focused on Tool 2’s evaluation on the usability of themewheels. In particular, we looked at how the donut charts were usedto investigate literature sources (lecture notes were used due to ac-cessibility). Five participants were given the task of summarizinga set of lectures, which was repeated twice with the order random-ized – once using a theme wheel of the whole course and onceusing a hard copy of the lecture outline materials. This was fol-lowed by informal semi-structured interviews in order to gain in-sights into how participants felt using the two different resources tocomplete their tasks. We found that the theme-wheels introduceda pictorial representation of the course, allowing for participantsto navigate the lecture materials without opening every documentand skim-reading each one individually. It, therefore, supported ourpremise that theme wheels allow for a broad, intuitive, and objec-tive overview of literature sources.
4. Procedure
In this section, we detail the steps involved in running our study,including how we recruited participants and coded semi-structuredinterviews.
For our study, we followed the thematic methodology that has beenoutlined, making use of our two thematic tools. Our large papercollection is made up of five years’ worth of CHI papers, excludingany extended papers, totaling 2,782 papers.Papers were then each split into 30 equal text chunks (83,460 intotal) and run through LDA [BNJ03] (as noted in the implementa-tion of Tool 1) and generated 85 themes. This number was settledafter exploration sessions and manual adjustments to get detailedthemes whilst keeping this number manageable for participants.Figure 5 shows some examples of uncovered themes.For the second phase, we require the use of Tool 2 which, asdescribed previously, extracts relevant themes based on the selectedpaper set made by a participant. The size of these excerpt thematicmaps varied across participants (n: 10, avg: 28.5, std dev: 7.8, min:10, max: 37).
Figure 5:
We run thematic analysis on papers from 5 years’ worthof CHI (2,782 papers in total) to give an overview of the researchcommunity. A subset of uncovered themes can be seen above.
We recruited 10 experienced participants (P1-P10) in total (5 males;5 females; aged 18-44) using advertisements throughout our orga-nization, which attracted participants across several departments totake part [CC08] [Pat90]. None reported to be color blind and therewas no confusion distinguishing between the colors and shapesused in the tool interfaces. We noticed saturation in the coding ofour results as reported in later sections, validating our sample sizeof participants.Using a pre-study questionnaire, we verified that all participantsare experienced in using digital repositories to browse for literaturesources. These experiences ranged from using digital repositoriesseveral times per week (7 participants), at least once a week (1 par-ticipant) to less than every 1-2 months (2 participants). The stim-ulated recall [AC08] of experience was also used during the semi- © 2021 The Author(s) . Howden, P. Le Bras, T. S. Methven, S. Padilla & M. J. Chantler / Enhancing Reading Strategies structured interviews to allow participants to contrast between theirexperience and our theme-based approach.Our study received ethical approval from our institution, andconsent was collected from the participants. Every participant wascompensated with a $12 voucher for their time. All the results fromthis study were anonymized and unlinked.
We divided our study into three stages with two user tasks, aim-ing to keep participants motivated by breaking down the study intosmaller, manageable tasks [CC08]. These stages follow the taskswe describe above, consisting of: A) browse and select 6 papersusing Tool 1, B) generate a reading strategy using Tool 2, and C)report on the perception of theme-based literature discovery duringa semi-structured interview.This was accompanied by a scenario within which we asked ourparticipants to place themselves in [JBF ∗ ∗ “You are currently planning an experi-ment where you will be looking at how people use different websitesand what they like and dislike about them. You are interested in us-ing focus groups or interviewing techniques to gather additionalinsights from your participants. However, you are not sure whetherthis is the best option for you, so, you want to explore what ap-proaches other similar studies have taken, including how to reporton the data gathered.” This scenario was chosen as it fit into thecommunity of papers that are being displayed and is simple enoughthat participants are not required to have a background in comput-ing to complete the tasks, allowing for us to reach a more diverseaudience [CC08].Stage 1 (paper selection using Tool 1) consisted of the first usertask, (A), that was performed by participants in their own time 1-3 days prior to the rest of the study. This allowed for the task tofeel more relaxing and realistic [JBF ∗
10] and gave the investigatorsenough time to process data before Stage 2. Participants were alsogiven worksheets to complete, in which they communicated theirchoice and reasonings.Stage 2 (reading strategy generation using Tool 2) consisted ofthe second user task, (B), where participants were shown their se-lected 6 papers rendered as theme wheels and were asked to analyzeand interact with the visualization in order to draw out a plan as tohow they would go about investigating the papers further. In par-ticular, we sought to understand their reading strategy in terms ofwhat order they would read the papers and whether they would readonly certain parts within the paper. We then revealed the paper ti-tles to the participants and asked them to describe their impressionof the title, compared to their analysis of the theme wheel. (Figure6 demonstrates the setup).Finally, Stage 3 (semi-structured interview) sought the partici-pants’ opinions and insights about Tool 1 and Tool 2. These inter-views lasted no longer than 30 minutes. In particular, we empha-sized the interviews towards the participants’ usage of the tools,their views on the theme-based approach, their usual procedurewith digital repositories, and the contrasts between the two ap-proaches. The interviews were recorded, with the participants’ agreement,and transcripts were then produced for coding.
Figure 6:
Representation of part 2 of the study. The setup includesthe interactive online tool, hard copies of worksheets, markers,highlighters and an audio recorder.
Coding was done by the investigator using computer-assisted qual-itative data analysis software [Sil13]. An open coding or inductiveapproach was used to develop the codebook [CS14] [Fag10]. Af-ter selecting a random transcript, an initial codebook was drawn,and then verified and adjusted on a second transcript. The rest ofthe transcripts were coded accordingly. A second pass through thedata was made to ensure consistency. We found saturation, val-idating our sample size of experience participants for the study.In addition, we are making the transcribed interviews and an-alyzed data open for future research in this and other areas( strategicfutures.org/publications (CC-BY)).
Figure 7:
The high-level codes from our semi-structured inter-views, measured by analyzing the number of coding referencesmade.
Our codebook comprises of 6 high-level codes (Figure 7): • Application:
This was the largest topic appearing from our in-terviews, where participants were asked to think about what theyliked and disliked about the tools, how they used the thematicmaps and theme wheels, information they thought was missingand how much they interacted with the tools. This brought outany usability issues in terms of features that they did not under-stand or use; © 2021 The Author(s) . Howden, P. Le Bras, T. S. Methven, S. Padilla & M. J. Chantler / Enhancing Reading Strategies • Digital Repositories:
Participants were asked to think aboutprior experiences using digital repositories allowing them togather their thoughts as to what they like and dislike about thesesystems, including what information they thought would be use-ful to help them navigate and select appropriate texts; • Digital Repositories vs Application:
While the previous twocodes reflected on our thematic tools or digital repositories sep-arately, this category focuses on the participants’ contrasts be-tween the two approaches; • Representation of Paper Titles:
Participants were asked to talkthe investigator through each of the paper titles and discuss theirreactions to them whilst comparing the title to the themes shownfrom the theme wheels on Tool 2; • Usual Approach to Browsing:
Discussions around digital li-braries brought up how participants usually browse literature,giving insights into what they consider useful information aboutpapers. The main criteria used to select papers was also dis-cussed; • Reflections on Application & Task:
Participants were asked toreflect upon how they felt completing the tasks, allowing themto consider whether they would change how they approached thetasks given the knowledge that they now have about the tools.
5. Results and Discussion
The research questions that were posed in the introduction will nowbe addressed with design recommendations being made for design-ers to enhance their literature discovery systems like digital repos-itories.
In this section, we discuss RQ1 that focuses on discovering whethera thematic analysis using the proposed theme-based methodologyaids paper selection. Analyzing participants worksheets detailingreasons for their paper set, we found that all participants used fea-tures of Tool 1 to reason their paper selection. P7 focused on thetheme content presented in the thematic map. In addition to thisfeature, P1, P3, P4, and P10 relied on the calculated theme percent-age. P5 combined the theme representation with the theme locationwithin papers. Finally, P2, P6, P8 and P9 made use of all of thesefeatures.During interviews, participants were asked to discuss how theyused the thematic tools to complete the tasks and contrast this ap-proach to how they would have usually completed similar taskswith digital repositories. Upon reflection of selecting papers usinga thematic approach, P8 pointed out that “at first it takes a littlegetting used to because it’s a very different way of considering pa-pers, but it does make you focus on the keywords” . P7 continues onthis point by explaining, “it’s a little more dynamic, your eyes canfirst go to keywords of relevance, so it removes that metric of whereit is in a list of papers” . P2 describes that “it might make narrow-ing down a scope to a few papers from one hundred and, everybodywants to read as few papers as possible” .All ten participants mentioned benefits of this system and it wasnoted that a thematic approach was “better than scrolling through alist of titles” (P3) and helped “pick out the main themes a lot better than you would get with a list of titles” (P5). This prompted partic-ipants to begin to reflect upon their usual approach, and how muchreliance they place on paper titles to help with the selection processas P9 describes “whenever I’m looking at papers, I probably put alot of emphasis into the title than I’ve realized” and P8 recognizesthat “having titles taken away definitely made you think differentlyand focus a bit more on keywords of what you’re going to get outof it” . This highlights problems with titles as participants describedthem as “always trying to be catchy, they’re attention-seeking andthey don’t necessarily say everything” (P8) resulting in sometimesselecting papers and feeling like “this isn’t quite what I expected” (P2).
Design Recommendation (R1):
We found that all participants ap-preciated the theme-based analysis and the thematic mapping of thepaper collection. We found that similarity-based layouts aided fasttheme selection. We would recommend (R1) that designers makeuse of thematic analysis, consider implementing it alongside theirnormal search methods, and use a visual similarity-based map, al-lowing users to easily select themes and explore relevant papers.Also as mentioned, participants began to reflect upon their usualapproach that involved “quickly scroll through and see differenttitles” (P10) or as P3 mentions “check the titles which will usu-ally get me to discard a few” . However, when using a thematicapproach, participants noted being able to gauge the volume andlocation of themes as P8 highlights the usefulness of having pa-pers “ordered using this percentage” and not placing emphasis ona title because “a title can be misleading” . P2 discusses the valueof knowing the locations of themes as the visualization “tells mewhere this keyword is in the paper. . . is it in the introduction, whichmay not be very relevant to me, I might be looking at methods, sothis is very useful!”
P5 also mentions this point as “you can seethe location of different topics, you don’t get that in any digital li-braries that I know of really or certainly graphically, so yeah, I likethat” .When contrasting the thematic approach to participants’ com-mon searching approach, two participants weren’t sure if theywould have selected papers in their paper set based on titles alone– “I’m not really sure whether any of these would leap out at me assomething that I thought that I would need to read for the kind ofresearch I would like to do” (P5). This point is also mentioned byP7: “I have no idea if that would affect the picking of it if I knewthat was the title, it might”.
Design Recommendation (R2):
Users found that the thematic pa-per ranking and particularly the graphical, single-theme represen-tation of both the volume and location of a theme within a paperuseful. This aided assessment of the relevance and use of the themewithin a paper, facilitating the decision to include or not include thispaper within the paper set. We recommend that designers providethese meta-data (paper ordering and theme volume), which are of-ten generated by search engines [BR99] [BG09b] but not normallymade available to users, as they aid selection.As we have seen, a thematic approach facilitates the selectionof research papers, but it also allows for a more objective methodto filtering papers that resulted in participants selecting papers thatthey believe would not have been selected if a traditional approachwas being used. This is due to the functionality allowing papers © 2021 The Author(s) . Howden, P. Le Bras, T. S. Methven, S. Padilla & M. J. Chantler / Enhancing Reading Strategies to be filtered by themes, and the ability to show the volume andlocation of the theme.
In this section, we focus on RQ2, that poses whether the thematicmethodology and associated tools can aid the generation of a cross-paper reading strategy given a selected paper set. In the secondphase of the study, participants were asked to consider the selectedpaper set and develop a reading strategy. From worksheets that par-ticipants described their strategies on, we found that six participantsordered papers for reading based on how much they contained themain themes that the participant was most interested in.P2 describes their answer as “looking at the color coding andlooking at the general themes in the papers” whereas P3 notes theywould “investigate the purple bits because there are a lot them,there are almost three whole purple donuts!”
Others, such as P5,described their approach to investigating the paper set as “scrollingalong here [the theme wheels] and then seeing which ones [themesfrom thematic map] light up and how that relates to the papers thatI picked” to find out “what the predominant color is” .Participants also used the theme-based overviews to eliminatepapers that after having a closer look at, no longer seemed as rel-evant as noted by P5 – “Paper 1, I didn’t end up using becauseI thought it was more specifically for musical learning and it wasquite good I realized that, so it wasn’t used” . P7 summarizes bystating, “I think the visual aspect is helpful because you can almostkind of quickly quantify what a single paper is about whereas withGoogle Scholar it’s kind of just a list of links” .Planning out a reading strategy allows for participants to focuson what they want to get out of each paper to solve a problem.This was highlighted by many participants when they discuss theirusual approach to the discovery of literature as being “very disorga-nized” (P5) or “surfing from paper to paper” (P8). This highlightsthe piecemeal approach that is often adopted using common search-ing techniques, as digital repositories do not allow for a paper setto be considered and evaluated, only individual sources.
Design Recommendation (R3):
We found that when given aside by side comparison of the multi-theme representations of se-quences of themes within a paper set, participants could formulate across-paper reading strategy, ordering paper sections that they planto read based on the quantity and positions of themes within eachindividual paper, promoting a coherent approach to investigatingthe sources. We recommend that designers facilitate the compari-son of a paper set using visualizations of each papers’ sequences ofthemes.It is clear from discussions with participants, that a combinationof the thematic map and the theme wheels were used to develop areading strategy. Due to the clustering in the thematic maps, similarthemes were grouped together. Participants mentioned this func-tionality as it “provides a link to something that might be worth ex-ploring” (P2) but three out of the ten participants also commentedon having difficulties to “find the exact keywords that I noted [inthe previous tasks]” (P6).In order to reduce this problem from occurring, we can imagine a closer integration between Tool 1 and Tool 2. This could be doneusing visual explanations [LBRM ∗
18] to animate the evolution ofthe thematic map from Tool 1 to Tool 2, allowing for users to trace[GB99] interesting themes and see how the tools pull out relevantinformation.
Design Recommendation (R4):
Our result show that participantsfelt they would have benefited from a closer link between the the-matic map of the paper collection provided in Tool 1 and the morefocused thematic map provided for the selected paper set in Tool2 (see Figure 4). We recommend that the thematic maps of the pa-per collection and the user’s paper selection are tightly integrated(e.g. the provision of common highlighting, multiple selections orinteractive transitions).Based on the evidence presented from the in-depth interviews,we have found that not only does following a thematic approach aidthe generation of a reading strategy, but often a strategy that takesinto consideration a set of papers as a whole rather than traditionalapproaches where users adopt a more piecemeal strategy.
In this section, we discuss RQ3 that focuses on the advantages anddisadvantages of the overall thematic approach that has been pro-posed. Exploring RQ1 and RQ2, we have seen that a thematic ap-proach to discovery and analysis of literature gives insights into thestructure, author keywords and sequence of themes as mentionedby participants whilst discussing advantages and disadvantages offollowing a thematic approach.P2 describes being able to “pick out bits of a paper that wereon a particular topic that I might want to focus on, so I could see,oh that’s a bit of waffle, so I can skip through that” while P7 men-tions that “I like how you can see the progress through a paper likethat, being able to see how the topics change or don’t change” .Theme wheels allowed participants to easily identify paper sec-tions (e.g. introduction, background or conclusion), enabling themto map their knowledge and experiences with research papers.With such a focus on themes, six out of ten participants foundthat they interpreted themes differently to the content, which wasbrought to light when the titles were uncovered. For example, P7describes this as “I just saw privacy and thought data privacy andI don’t know if this is actually what this is on or if it’s more actualphysical privacy? But I was thinking more data protection online,so yeah, I was surprised by that” . Le Bras et al. work recommendsinteractivity incorporated into the map for increased user confi-dence and engagement as participants can then interrogate the pro-cess and understand the information at their own pace [LBRM ∗ Design Recommendations (R5):
We found that some participantswould have liked to have been able to obtain a deeper understand-ing of particular themes at both the paper selection and readingstrategy generation stages. We recommend that designers explorehierarchical thematic analysis techniques [GJTB03] to allow userswith different levels of knowledge to investigate themes at multiplelevels of abstraction. © 2021 The Author(s) . Howden, P. Le Bras, T. S. Methven, S. Padilla & M. J. Chantler / Enhancing Reading Strategies
Nine out of our ten participants noted being surprised by at leastone paper title when they saw the titles at the end of the tasks. Thisis emphasized when participants were asked to explain their reac-tions where P1 mentions that “the first two, no, I would never everimagine it was that” and P3 mentions that a paper was “meant tobe for interviewing techniques since the tags were interview, data,survey but the title is nothing like that” . P2 found that the titleswere “totally different but still useful”
Whilst P9 reflects on theirapproach by mentioning “I used the keywords quite a lot, so thetitle was quite different, so it was quite surprising” .Uncovering the titles of the papers right at the end of the studyhighlights our previous point that titles are only one to two lineslong so cannot be expected to reflect the full content. Therefore, byintroducing thematic overviews of the content, participants couldsee the progression of themes from start to finish, giving insightsinto the tools and techniques used but sometimes lacking in givingcontext to the research. For example, eight out of our ten partic-ipants selected a paper titled, Investigating the Suitability of theAsynchronous, Remote, Community-based Method for Pregnantand New Mothers [PGRK ∗ “surprised me a bit.I didn’t see anything in here [the application] that made me thinkof that” or P7 who said, “I definitely had no idea that this was whatthe paper would be about” .Our chosen algorithm aims to uncover the most common themesin a whole corpus of text. It is, therefore, not surprising that preg-nant and new mothers do not come out as a major theme in HCIcommunity. This did not cause issues to participants for their task,as that paper discusses qualitative methods such as focus groupsand interviews, meeting the given scenario and task. If participantshad been given the task of understanding the context of papers, itwould then be likely that they would have struggled to grasp thisinformation from the theme wheels alone.During the interviews, participants were asked whether theythought a thematic approach could be a replacement of current dig-ital repository systems or if it would be more valuable as an add-onfeature. Only two participants thought that our thematic methodol-ogy could replace current systems, with the other eight participantsbelieving that this approach would be best as an add-on feature. P9reasoned this as “getting used to new systems is quite difficult, so itwould be good to have that alongside” or as P8 suggests, “peopleare so stuck in their ways, so I don’t know how open-minded peoplewould be” .Participants began to describe how they would use current sys-tems with a layer of thematic information added. P2 mentions that “I would probably start with this [interface] to get me to a placewhere I think I am ready to look at the text and start looking at theabstracts then and progress from there” while P1 states, “I reallylove this interface, it’s perfect for the first screening but then youneed something else [such as access to digital repositories]” . Design Recommendation (R6):
Participants appreciated the inte-grated thematic approach and its visual representation and inter-face. However, during the study, the participants clearly expectedthe title and abstract fields to be also available and would appreciatea combination of approaches. We recommend that designers incor-porate visual thematic analysis tools with traditional title-abstract search methods to allow users to seamlessly switch between andcombine approaches to get both theme and context information.As we have seen, based on results from our semi-structured in-terviews, there are advantages and disadvantages to the overall pro-posed thematic approach. Advantages included the ability to havea visual representation of a large collection of papers, see the se-quences of themes from start to finish in a paper and visually com-pare a paper set in order to aid the generation of a cross-paper read-ing strategy. The main disadvantages highlighted by participantswere not having an integrated environment with traditional infor-mation such as titles and abstracts available to them, but they ap-preciated that this process did allow for them to reflect upon theircommon approach to the discovery of literature and question theirreliance on commonly used information for their reading strategies.
6. Conclusions
In this paper, we present a study exploring the effects of a newvisual methodology and complementary toolset that helps usersbrowse, select and develop holistic reading strategies. We princi-pally focus on whether our proposed approach enriches paper se-lections, facilitates the development of coherent reading strategies,and allows them to develop high-level holistic reading strategies.To explore these aspects, we carried out a three-phase qualitativestudy using scenario-based, semi-structured interviews that weredesigned to probe insight into to the use of our methodology andtools. We investigated participants’ approaches, user behaviors, andreactions using our thematic methodology and contrasted them totheir experiences with common digital repositories.We believe that our results indicate that adopting a visualthematic methodology encourages a more objective approach tobrowsing and selecting papers. Participants chose papers that theythought they would definitely otherwise would have not selectedand, following selection of paper sets, participants used a com-bination of visual thematic maps and theme wheels to developtheme-based, cross-paper reading strategies. In addition, partici-pants found that the multi-theme paper visualizations gave usefulinsights into the structure, ordering, frequency and commonality ofthemes, allowing participants to quickly gain an overview of con-tent, authors’ writing styles and focus.We make six recommendations aimed at assisting designers thatwish to enhance or develop visual thematic tools and method-ologies that will help users quickly and efficiently explore dig-ital repositories. We certainly believe that such tools should beclosely integrated with existing approaches to provide complemen-tary, rather than replacement functionality, in order to encourage amore holistic and objective approach to developing reading strate-gies.Finally, we hope the insights, visualizations, methodology, toolsand recommendations proposed in this paper will encourage dis-cussion in the community and catalyze the development of new vi-sual thematic-based approaches to developing interfaces to a widevariety of digital repositories, including for example storing video,audio, and multimedia data for educational, entertainment and gov-ernmental applications. © 2021 The Author(s) . Howden, P. Le Bras, T. S. Methven, S. Padilla & M. J. Chantler / Enhancing Reading Strategies
Acknowledgments
The authors would like to thank the participants for their time andinsightful discussions. The data generated for this study can be ac-cessed on request, please email the authors for further details. Fi-nally, visualisations of corpora, open algorithms and data (CC-BY),similar complementary tools [MPCC14] [POC12] [MPC15], andrelated work can be access at strategicfutures.org . References [ABKS14] A
CKERMANN
M. R., B
LÖMER
J., K
UNTZE
D., S
OHLER
C.:Analysis of agglomerative clustering.
Algorithmica 69 , 1 (2014), 184–215. 6[AC08] A
DAMS
A., C OX A. L.: Questionnaires, in-depth interviews andfocus groups.
Cambridge (2008). 6[ACM] ACM:. URL: http://dl.acm.org/ . 1, 2[AG15] A
LEXANDER
E., G
LEICHER
M.: Task-driven comparison oftopic models.
IEEE transactions on visualization and computer graphics22 , 1 (2015), 320–329. 3[Ban20] B
ANSAL
S. B.: Beginners guide to topic mod-eling in python and feature selection, Jan 2020. URL: . 5[Ber96] B
ERRY
J. W.: Digital libraries: new initiatives with worldwideimplications.
Collection Building (1996). 1[BG09a] B
EEL
J., G
IPP
B.: Google scholar’s ranking algorithm: The im-pact of citation counts (an empirical study). In
Beel (05 2009), pp. 439–446. doi:10.1109/RCIS.2009.5089308 . 2[BG09b] B
EEL
J., G
IPP
B.: Google scholar’s ranking algorithm: an in-troductory overview. In
Proceedings of the 12th international confer-ence on scientometrics and informetrics (ISSI’09) (2009), vol. 1, Rio deJaneiro (Brazil), pp. 230–241. 2, 8[BGMN14] B
OYD -G RABER
J., M
IMNO
D., N
EWMAN
D.: Care andfeeding of topic models: Problems, diagnostics, and improvements.
Handbook of mixed membership models and their applications 225255 (2014). 5[BK09] B
LAZHENKOVA
O., K
OZHEVNIKOV
M.: The new object-spatial-verbal cognitive style model: Theory and measurement.
AppliedCognitive Psychology: The Official Journal of the Society for AppliedResearch in Memory and Cognition 23 , 5 (2009), 638–663. 2[Ble12] B
LEI
D. M.: Probabilistic topic models.
Communications of theACM 55 , 4 (2012), 77–84. 1, 2, 5[BNJ03] B
LEI
D. M., N G A. Y., J
ORDAN
M. I.: Latent dirichlet alloca-tion.
Journal of machine Learning research 3 , Jan (2003), 993–1022. 1,2, 5, 6[Bor99] B
ORGMAN
C. L.: What are digital libraries? competing visions.
Information processing & management 35 , 3 (1999), 227–243. 2[BR99] B
ARTER
C., R
ENOLD
E.: The use of vignettes in qualitativeresearch.
Social research update 25 , 9 (1999), 1–6. 7, 8[Car99] C
ARD
M.:
Readings in information visualization: using visionto think . Morgan Kaufmann, 1999. 2[CC03] C
HOWDHURY
G. G., C
HOWDHURY
S.:
Introduction to digitallibraries . Facet publishing, 2003. 2[CC08] C
AIRNS
P. E., C OX A. L.:
Research methods for human-computer interaction.
Cambridge University Press, 2008. 6, 7[CKK13] C
HERUKODAN
S., K
UMAR
G. S., K
ABIR
S. H.: Using opensource software for digital libraries.
The Electronic Library (2013). 1[CS14] C
ORBIN
J., S
TRAUSS
A.:
Basics of qualitative research: Tech-niques and procedures for developing grounded theory . Sage publica-tions, 2014. 7 [DES ∗
15] D
IAKOPOULOS
N., E
LGESEM
D., S
ALWAY
A., Z
HANG
A.,H
OFLAND
K.: Compare clouds: Visualizing text corpora to comparemedia frames. In
Proceedings of IUI Workshop on Visual Text Analytics (2015), pp. 193–202. 3[Dir] D
IRECT
S.: Sciencedirect. URL: . 1, 2[DRRD12] D
ÖRK
M., R
ICHE
N. H., R
AMOS
G., D
UMAIS
S.: Pivot-paths: Strolling through faceted information spaces.
IEEE transactionson visualization and computer graphics 18 , 12 (2012), 2709–2718. 3[DSG ∗
12] D
UNNE
C., S
HNEIDERMAN
B., G
OVE
R., K
LAVANS
J.,D
ORR
B.: Rapid understanding of scientific paper collections: Integrat-ing statistics, text analytics, and visualization.
Journal of the AmericanSociety for Information Science and Technology 63 , 12 (2012), 2351–2369. 3[Fag10] F
AGAN
J. C.: Usability studies of faceted browsing: A literaturereview.
Information Technology and Libraries 29 , 2 (2010), 58–66. 7[GB99] G
REGOR
S., B
ENBASAT
I.: Explanations from intelligent sys-tems: Theoretical foundations and implications for practice.
MIS quar-terly (1999), 497–530. 9[GBKP13] G
RIER
R. A., B
ANGOR
A., K
ORTUM
P., P
ERES
S. C.: Thesystem usability scale: Beyond standard usability testing. In
Proceedingsof the Human Factors and Ergonomics Society Annual Meeting (2013),vol. 57, SAGE Publications Sage CA: Los Angeles, CA, pp. 187–191. 6[GJTB03] G
RIFFITHS
T., J
ORDAN
M., T
ENENBAUM
J., B
LEI
D.: Hi-erarchical topic models and the nested chinese restaurant process.
Ad-vances in neural information processing systems 16 (2003), 17–24. 9[GLG ∗
13] G
RATZL
S., L EX A., G
EHLENBORG
N., P
FISTER
H.,S
TREIT
M.: Lineup: Visual analysis of multi-attribute rankings.
IEEEtransactions on visualization and computer graphics 19 , 12 (2013),2277–2286. 3[Goo] G
OOGLE :. URL: https://scholar.google.com . 1, 2[GWH89] G
OLDEN
B. L., W
ASIL
E. A., H
ARKER
P. T.: The analytichierarchy process.
Applications and Studies, Berlin, Heidelberg (1989).2[Hag04] H
AGGAN
M.: Research paper titles in literature, linguistics andscience: Dimensions of attraction.
Journal of pragmatics 36 , 2 (2004),293–317. 2[Hea95] H
EARST
M. A.: Tilebars: visualization of term distribution in-formation in full text information access. In
Proceedings of the SIGCHIconference on Human factors in computing systems (1995), pp. 59–66. 2[IBM16] IBM: Watson natural classifier, Nov 2016.URL: . 1, 2[IEE] IEEE:. URL: http://ieeexplore.ieee.org/ . 2[JBF ∗
10] J
ENKINS
N., B
LOOR
M., F
ISCHER
J., B
ERNEY
L., N
EALE
J.: Putting it in context: the use of vignettes in qualitative interviewing.
Qualitative research 10 , 2 (2010), 175–198. 7[KA11] K
OTCHOUBEY
B., A
LKHOURY
S.: P9.12 context, and not thedominant meaning, determines word comprehension.
Clinical Neuro-physiology 122 (2011). 1[Kat14] K
ATO
S.: Using analytic hierarchy process in textbook evalua-tion.
TESOL Journal 5 , 4 (2014), 678–697. 2[KHGW07] K UO B. Y., H
ENTRICH
T., G
OOD
B. M., W
ILKINSON
M. D.: Tag clouds for summarizing web search results. In
Proceed-ings of the 16th international conference on World Wide Web (2007),pp. 1203–1204. 3[Lab] L AB S. F.: researchperspectives. URL: . 2[LBGR ∗
20] L E B RAS
P., G
HARAVI
A., R
OBB
D. A., V
IDAL
A. F.,P
ADILLA
S., C
HANTLER
M. J.: Visualising covid-19 research. arXivpreprint arXiv:2005.06380 (2020). 1 © 2021 The Author(s) . Howden, P. Le Bras, T. S. Methven, S. Padilla & M. J. Chantler / Enhancing Reading Strategies [LBRM ∗
18] L E B RAS
P., R
OBB
D. A., M
ETHVEN
T. S., P
ADILLA
S.,C
HANTLER
M. J.: Improving user confidence in concept maps: Explor-ing data driven explanations. In
Proceedings of the 2018 CHI Conferenceon Human Factors in Computing Systems (2018), pp. 1–13. 5, 6, 9[LKW18] L IU C., K IM J., W
ANG
H.-C.: Conceptscape: Collaborativeconcept mapping for video learning. In
Proceedings of the 2018 CHIConference on Human Factors in Computing Systems (2018), pp. 1–12.2[McC02] M C C ALLUM
A. K.: Mallet: A machine learning for languagetoolkit. http://mallet.cs.umass.edu (2002). 5[MFAV18] M
ORRIS
M. R., F
OURNEY
A., A LI A., V
ONESSEN
L.: Un-derstanding the needs of searchers with dyslexia. In
Proceedings of the2018 CHI Conference on Human Factors in Computing Systems (2018),pp. 1–12. 2[Mic] M
ICROSOFT :. URL: http://academic.microsoft.com/ .1, 2[MPC15] M
ETHVEN
T. S., P
ADILLA
S., C
HANTLER
M. J.: I don’t thinkwe’ve met: Encouraging collaboration via topic-based search. In
Pro-ceedings of the 18th ACM Conference Companion on Computer Sup-ported Cooperative Work & Social Computing (2015), pp. 29–32. 11[MPCC14] M
ETHVEN
T. S., P
ADILLA
S., C
ORNE
D. W., C
HANTLER
M. J.: Research strategy generation: avoiding academic’animal farm’. In
Proceedings of the companion publication of the 17th ACM conferenceon Computer supported cooperative work & social computing (2014),pp. 25–28. 11[Nig09] N
IGHTINGALE
A.: A guide to systematic literature reviews.
Surgery (Oxford) 27 , 9 (2009), 381–384. 2[Nor95] N
ORMAN
D. A.: The psychopathology of everyday things. In
Readings in Human–Computer Interaction . Elsevier, 1995, pp. 5–21. 2[NRW76] N
ELSON
D., R
EED
V., W
ALLING
J.: Pictorial superiority ef-fect.
Journal of experimental psychology. Human learning and memory2 (10 1976), 523–8. doi:10.1037/0278-7393.2.5.523 . 2[NZ06] N
GUYEN
T., Z
HANG
J.: A novel visualization model forweb search results.
IEEE Transactions on Visualization and Com-puter Graphics 12 , 5 (Sept. 2006), 981–988. URL: https://doi.org/10.1109/TVCG.2006.111 , doi:10.1109/TVCG.2006.111 . 2[OSR ∗
14] O
ELKE
D., S
TROBELT
H., R
OHRDANTZ
C., G
UREVYCH
I.,D
EUSSEN
O.: Comparative exploration of document collections: a vi-sual analytics approach. In
Computer Graphics Forum (2014), vol. 33,Wiley Online Library, pp. 201–210. 3[Pat90] P
ATTON
M. Q.:
Qualitative evaluation and research methods .SAGE Publications, inc, 1990. 6[PGRK ∗
17] P
RABHAKAR
A. S., G
UERRA -R EYES
L., K
LEINSCHMIDT
V. M., J
ELEN
B., M AC L EOD
H., C
ONNELLY
K., S
IEK
K. A.: Inves-tigating the suitability of the asynchronous, remote, community-basedmethod for pregnant and new mothers. In
Proceedings of the 2017 CHIConference on Human Factors in Computing Systems (2017), pp. 4924–4934. 10[PHRC13] P
ADILLA
S., H
ALLEY
F., R
OBB
D. A., C
HANTLER
M. J.:Intuitive large image database browsing using perceptual similarity en-riched by crowds. In
International Conference on Computer Analysis ofImages and Patterns (2013), Springer, pp. 169–176. 3[PMC14] P
ADILLA
S., M
ETHVEN
T. S., C
HANTLER
M. J.: Is britishhci important? a topic-based comparison with chi. In
Proceedings of the28th International BCS Human Computer Interaction Conference (HCI2014) 28 (2014), pp. 365–370. 1[PMCC14] P
ADILLA
S., M
ETHVEN
T. S., C
ORNE
D. W., C
HANTLER
M. J.: Hot topics in chi: trend maps for visualising research. In
CHI’14extended abstracts on human factors in computing systems . ACM, 2014,pp. 815–824. 1[PMRC17] P
ADILLA
S., M
ETHVEN
T. S., R
OBB
D. A., C
HANTLER
M. J.: Understanding concept maps: A closer look at how people or-ganise ideas. In
CHI (2017), pp. 815–827. 1 [POC12] P
ADILLA
S., O
RZECHOWSKI
P., C
HANTLER
M. J.: Digitaltools for the creative industries.
Digital Futures (2012), 23–25. 11[RBB16] R
USSELL -B ENNETT
R., B
ARON
S.: the importance of thesnappy title.
Journal of Services Marketing (2016). 2[RHF05] R
IEHMANN
P., H
ANFLER
M., F
ROEHLICH
B.: Interactivesankey diagrams. In
IEEE Symposium on Information Visualization,2005. INFOVIS 2005. (2005), IEEE, pp. 233–240. 3[Ric77] R
ICHARDSON
A.: Verbalizer-visualizer: a cognitive style dimen-sion.
Journal of mental imagery (1977). 2[RPKC15a] R
OBB
D. A., P
ADILLA
S., K
ALKREUTER
B., C
HANTLER
M. J.: Crowdsourced feedback with imagery rather than text: Woulddesigners use it? In
Proceedings of the 33rd Annual ACM Conference onHuman Factors in Computing Systems (2015), pp. 1355–1364. 5[RPKC15b] R
OBB
D. A., P
ADILLA
S., K
ALKREUTER
B., C
HANTLER
M. J.: Moodsource: Enabling perceptual and emotional feedback fromcrowds. In
Proceedings of the 18th ACM Conference Companion onComputer Supported Cooperative Work & Social Computing (2015),pp. 21–24. 5[RPM ∗
17] R
OBB
D. A., P
ADILLA
S., M
ETHVEN
T. S., K
ALKREUTER
B., C
HANTLER
M. J.: Image-based emotion feedback: How does thecrowd feel? and why? In
Proceedings of the 2017 Conference on De-signing Interactive Systems (2017), pp. 451–463. 5[Sco] S
COPUS : Scopus. URL: . 2[Shn96] S
HNEIDERMAN
B.: The eyes have it: A task by data type taxon-omy for information visualizations. In
Proceedings 1996 IEEE sympo-sium on visual languages (1996), IEEE, pp. 336–343. 2[Sil13] S
ILVERMAN
D.:
Doing qualitative research: A practical hand-book . SAGE publications limited, 2013. 7[Spr] S
PRINGER :. URL: http://link.springer.com/ . 1, 2[TM10] T
HOMAS
P. R., M C K AY J. B.: Cognitive styles and instructionaldesign in university learning.
Learning and Individual Differences 20 , 3(2010), 197–202. 2[Tur05] T
URNER
J. R.: The role of pilot studies in reducing risk onprojects and programmes, 2005. 6[War19] W
ARE
C.:
Information visualization: perception for design .Morgan Kaufmann, 2019. 5[WKSS10] W
ILSON
M. L., K
ULES
B., S
CHRAEFEL
M., S
HNEIDER - MAN
B.:
From keyword search to exploration: Designing future searchinterfaces for the web , vol. 1. Now Publishers Inc, 2010. 1, 2[XM16] X IE I., M
ATUSIAK
K.:
Discover digital libraries: Theory andpractice . Elsevier, 2016. 2[XW19] X
IAO
Y., W
ATSON
M.: Guidance on conducting a systematicliterature review.
Journal of Planning Education and Research 39 , 1(2019), 93–112. 2[YaKSJ07] Y I J. S., AH K ANG
Y., S
TASKO
J., J
ACKO
J. A.: Toward adeeper understanding of the role of interaction in information visualiza-tion.
IEEE transactions on visualization and computer graphics 13 , 6(2007), 1224–1231. 6[ZBCS18] Z
HAO
J., B
HATT
C., C
OOPER
M., S
HAMMA
D. A.: Flexiblelearning with semantic visual exploration and sequence-based recom-mendation of mooc videos. In
Proceedings of the 2018 CHI Conferenceon Human Factors in Computing Systems (2018), pp. 1–13. 2, 3[ZCCB13] Z
HAO
J., C
OLLINS
C., C
HEVALIER
F., B
ALAKRISHNAN
R.: Interactive exploration of implicit and explicit relations in faceteddatasets.
IEEE Transactions on Visualization and Computer Graphics19 , 12 (2013), 2080–2089. 3[ZLD ∗
15] Z
HAO
J., L IU Z., D
ONTCHEVA
M., H
ERTZMANN
A., W IL - SON
A.: Matrixwave: Visual comparison of event sequence data. In
Proceedings of the 33rd Annual ACM Conference on Human Factors inComputing Systems (2015), pp. 259–268. 3[ZZYW15] Z HA X., Z
HANG
J., Y AN Y., W
ANG
W.: Comparing flowexperience in using digital libraries.
Library Hi Tech (2015). 1(2015). 1