Characterizing the Online Learning Landscape: What and How People Learn Online
1146Characterizing the Online Learning Landscape: What andHow People Learn Online
SEAN KROSS,
University of California San Diego, USA
ESZTER HARGITTAI,
University of Zurich, Switzerland
ELISSA M. REDMILES,
Max Planck Institute for Software Systems, GermanyHundreds of millions of people learn something new online every day. Simultaneously, the study of onlineeducation has blossomed within the human computer interaction community, with new systems, experiments,and observations creating and exploring previously undiscovered online learning environments. In this studywe endeavor to characterize this entire landscape of online learning experiences using a national survey of2260 US adults who are balanced to match the demographics of the U.S. We examine the online learningresources that they consult, and we analyze the subjects that they pursue using those resources. Furthermore,we compare both formal and informal online learning experiences on a larger scale than has ever been donebefore, to our knowledge, to better understand which subjects people are seeking for intensive study. Wefind that there is a core set of online learning experiences that are central to other experiences and these areshared among the majority of people who learn online. We conclude by showing how looking outside of thesecore online learning experiences can reveal opportunities for innovation in online education.CCS Concepts: •
Applied computing → E-learning ; •
Human-centered computing → Human com-puter interaction (HCI) .Additional Key Words and Phrases: Online learning; Adult learning; Survey; MOOCs; YouTube
ACM Reference Format:
Sean Kross, Eszter Hargittai, and Elissa M. Redmiles. 2021. Characterizing the Online Learning Landscape:What and How People Learn Online.
Proc. ACM Hum.-Comput. Interact.
5, CSCW1, Article 146 (April 2021),19 pages. https://doi.org/10.1145/3449220
The subject of how people learn has been fascinating researchers since long before the invention ofthe internet [8, 35, 42, 52]. The innovation of online learning has opened the door for more peopleto learn, and thus, the door for more research into what people are learning about and how theyare doing so.One of the most promising aspects of the internet since its early days has been the opportunitiesit offers for widespread participation thanks to the myriad of resources it makes freely available [6].While much scholarly work has investigated what this means for the development of resourceslike Wikipedia and other wikis (e.g., Shaw & Hill, 2014), researchers have paid much less attentionto how people use diverse online resources for educational purposes [49]. The internet offers aplethora of formal and informal online resources available for free or at significantly lower costcompared to traditional, offline educational opportunities, and has thus widened the availability
Authors’ addresses: Sean Kross, [email protected], University of California San Diego, USA; Eszter Hargittai, Universityof Zurich, Switzerland, [email protected]; Elissa M. Redmiles, Max Planck Institute for Software Systems, Germany,[email protected] to make digital or hard copies of part or all of this work for personal or classroom use is granted without feeprovided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice andthe full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses,contact the owner/author(s).© 2021 Copyright held by the owner/author(s).2573-0142/2021/4-ART146https://doi.org/10.1145/3449220Proc. ACM Hum.-Comput. Interact., Vol. 5, No. CSCW1, Article 146. Publication date: April 2021. a r X i v : . [ c s . H C ] F e b of educational resources. Massive open online courses (MOOCs), video tutorials, how-to articles,online discussion groups, among others, can help people gain new skills with the potential toimprove their job prospects, social mobility, and personal welfare [18].Prior work has considered how a specific subset of people learn (e.g., young adults), how peoplelearn a specific subject, or how people use a particular resource for learning [27, 36, 50, 54]. Yet,little is known about the full ecosystem of what adults learn about online and what educationalresources they use to do so. These can include both formal online courses (e.g., Coursera, thoseoffered by local universities) and less formal resources (e.g., YouTube videos, online discussiongroups). No prior work, to our knowledge, has addressed this full ecosystem by looking at more thanone subject or one resource at a time across all online learners. An important novel contribution forthis paper is that it considers the combination of subjects and resources (e.g., an online universitycourse for learning history, YouTube for learning math) that may generalize to many learners.In this work, we explore three research questions: RQ1)
What do people learn about online?
RQ2)
How do people learn online (i.e., what learning resources do they use)?
RQ3)
What are the core online learning experiences - which subject through what resource -that are common across the majority of online learners?To answer these research questions, we conducted an online survey of 2,260 adults age 18 and overin the U.S. To improve the generalizability of our findings, our survey sample was demographicallybalanced to match U.S. Census statistics on gender, age, and education. We draw on frameworksand findings of past literature on online learning [14, 26, 39] to analyze our findings critically andcharacterize the online learning ecosystem.We find that the vast majority (93%) of those we surveyed had learned something online. At ahigh level, we find that online learners’ interests span a very wide set of subjects (with the mediansubject being learned by 26% of respondents) and resources (with the median resource being usedby 62.5% of respondents). Further, examining online learning experiences (pairs of subjects andresources), we find 12 core online learning experiences shared by the plurality of online learners.The most common of these core online learning experiences, which was reported by over half ofour respondents, was learning how to do something yourself (DIY) using YouTube.The findings from this work provide insight into the ecosystem and experience of online learning;suggest a combination of factors that may drive learners’ choices and affect our progress toward ademocratization of online learning; and identify directions for future work on improving the designand targeting of online learning technologies to draw in new learners and onboard new subjects.
This work is modeled after surveys of participants in sociotechnical systems that are typical through-out the Computer Supported Cooperative Work and Human Computer Interaction communities.Our work is inspired specifically by three analysis frames posited by prior work.First, we draw from the work of Piety et al., who present a framework for the educational datasciences, which delineates studies by (1) the educational stage of the learners studied, and (2) theunit of analysis through which the learning is analyzed [39]. We use this framework to design ourstudy: we choose to study (1) adult learners and (2) our unit of analysis is the individual with afocus on the subjects and resources they consult online for learning.Piety and colleagues additionally explain that a challenge of educational data is that they areoften limited to specific types of online learning resources. In order to address this limitationand maximize the possible set of resources available for us to analyze, we sample a large and
Proc. ACM Hum.-Comput. Interact., Vol. 5, No. CSCW1, Article 146. Publication date: April 2021. haracterizing the Online Learning Landscape: What and How People Learn Online 146:3 demographically diverse set of respondents in our study. By collecting survey responses frompeople whose demographics are proportional to the overall US population on multiple demographicaxes, we aim to capture a set of learners who use a broad set of resources.In addition to the Piety et al. framework for educational data science studies, we draw fromthe analysis frame of Ferguson and Shum. Ferguson and Shum argue for always using networkedanalysis to understand online learning [14]. Their rationale for this methodological approachis that online learners often consult multiple types of online learning resources when trying tounderstand a subject, and therefore the relationship between the different types of resourcesshould be understood better. They point to the proliferation of Open Educational Resources (OERs)that learners are able to consult online as a basis for the need to characterize how the uses ofthese resources intersect. Following their suggestion, we use clustering analysis to understand ourdata and draw from these analyses to identify opportunities for developing new online learningexperiences.Finally, our analysis has been influenced by the work of Kross and Guo, who provide a systematicreview of the literature on how different types of online learning resources connect students toeach other to create new interactions that shape students’ learning trajectories [26]. They highlighthow software systems can be purpose-built to enhance learning experiences for certain subjects.Applying this purpose-driven lens to our results, we will discuss the implications of our empiricalanalysis on the structure of the online learning communities described in their study.
Prior work has investigated various aspects of learners’ online educational experiences. Kizilcecand Schneider developed and deployed the Online Learning Enrollment Intentions scale as aninstrument for understanding the motivations for learners to seek online education [22]. Theyfound that online learners often have social goals when taking a course, which contrasted to thetypical course design built upon assignments meant to be completed alone. They further foundthat the design of online educational resources should incorporate an awareness that learners areusing multiple resources, and that online instructors should direct learners toward other resources.In our study we extend these findings beyond the domain of online courses to quantify the extentto which ten types of online learning resources are used alongside other types of resources.Additionally, Swanson and Walker surveyed young adults between the ages of 18 and 25 togain insight into their usage of several different digital technologies for academic and recreationalpurposes [53]. They found that young adults spend a majority of out-of-classroom academically-focused time using technology. Their results inform how different types of online learning resourcescould be deployed depending on how young adults are using different devices. Our study buildsupon their work by analyzing how different types of online learning resources are used, which maybe influenced by what subjects can be studied effectively given the device that is being used. Bothof these studies aim to understand how technology can be harnessed to meet learners’ expectationsin terms of where they can find online educational resources and how they can be supported tohave fulfilling learning experiences via those resources.
The online education literature includes studies of both more formal online educational experienceslike online courses offered for college credit and massively open online courses that award creden-tials [20, 38]. The literature also includes studies of more casual online educational experiencesthat are related to the field of “free-choice learning,” which includes going to a museum or readinga non-fiction book for pleasure [13]. For our discussion about online learning resources we willdifferentiate between formal and informal resources based on Rosenthal’s work on free-choice
Proc. ACM Hum.-Comput. Interact., Vol. 5, No. CSCW1, Article 146. Publication date: April 2021. learning about YouTube, where both formal and informal resources have learning outcomes butonly formal resources have prescribed learning objectives and informal learning experiences takeplace “outside a formal learning environment” [44]. Additionally, Wenger characterizes informallearning as requiring community participation, versus formal learning that takes place in a class-room or a structured learning environment [55]. In line with prior work, we refer to enrollment inan online course as use of a formal online learning resource, while we refer to all other types ofonline learning resources as informal . We believe that this terminology is justified considering thetime and financial commitment that many online courses require, and the resemblance that onlinecourses have to traditional in-person learning experiences in the classroom.Studies of formal online learning resources greatly outnumber studies of informal learningresources, perhaps due to the high level of access that university researchers have to such learnersand data generated from online university courses [26, 27, 57]. These studies often focus on students’level of engagement with online course materials, how many students complete the course comparedto how many enroll, and how these dynamics differ from in-person educational experiences.Here we highlight a few examples of studies about informal online learning that are especiallysalient to the goals of our study. Many studies of informal online learning are centered around onespecific aspect of the educational experience, usually a specific type of online learning resource, ora study about how a particular subject is pursued online. In an example of the former, Narayan etal. developed and studied the success of an interactive tutorial to onboard new members of theWikipedia community [36]. Although the results from their intervention did not cause participantsto be any more likely to contribute to Wikipedia, it made participants feel more integrated intothe Wikipedia community. Torrey et al. explored teaching and learning crafts online via a series ofinterviews focused on how members of craft communities communicate ideas about space andaesthetics that are not easily communicated with typical online resources [54]. They found thatcuration of specific online resources and persistence in communicating complex ideas about theircraft were key to their success in this informal learning community.Yet other studies on informal online learning focused on how people learn a specific subject.Exemplar studies include that by Shorey et al., who studied how participation in social subgroupswithin an online community for the Scratch educational programming language [41] leads to higherlevels of engagement and enriching interactions that may lead to better learning outcomes [50].These results are further supported by Yang et al. and Gelman et al., who studied how communitiesof informal learning can grow, and how learning trajectories can be mapped in informal learningsettings [16, 56].Our study differs from past work on both formal and informal learning in that we (a) study afar broader set of resources and subjects, aiming to characterize and critically examine the fullecosystem of online learning, and (b) while prior work has called for the comparison of formal andinformal resources [48], our work is the first that we are aware of to compare these mechanisms ofonline learning empirically (specifically, in terms of the subjects that people learn using differingresource types).
We conducted a national survey of American adults to examine the relationship between thesubjects that people learn about online and the types of online learning resources they use to do so.Here, we provide details about the data collection, the statistical procedures we applied to the data,and the limitations of our analysis.
Proc. ACM Hum.-Comput. Interact., Vol. 5, No. CSCW1, Article 146. Publication date: April 2021. haracterizing the Online Learning Landscape: What and How People Learn Online 146:5
We contracted with the survey research firm Cint to administer the study to American adults inJune-July, 2019. People were able to fill out the survey hosted on the Qualtrics platform using acomputer or a mobile device. We included two attention-check verification questions and only thosewho passed both are included in the analyses. We received 2,260 valid responses to our survey.We quota-sampled on gender, age, and education to obtain a diverse sample. Specifically, ourrespondents are 58.6% female and the mean age of our respondents is 41 years (SD=15 years).Additionally, 23.6% of our sample has a high school diploma or less education, 39.8% of our samplehas some college education, while the remaining 36.6% of our sample has a college degree or moreeducation. The survey panel provider compensated respondents for their participation based ontheir preference of cash, gift cards, or donations to charity. All study procedures and materials wereapproved by our Institutional Review Board.
To understand respondents’ online learning experiences, we asked respondents what subjects they had learned about online and what types of resources they used to learn about each of thosesubjects . A respondent who indicated that they used a particular type of resource to learn about aparticular subject had what we define as an online learning experience .To understand what subjects people learned online, we asked “Which of the following topicshave you tried to learn about through online resources?" This was followed by nineteen subjects,the option of “other" and specifying something else, and “none of the above." Subjects ranged fromtraditional academic subjects like history, math, and science to general welfare and lifestyle subjectssuch as: do-it-yourself (DIY) , travel/geography, personal health/health care, makeup/fashion, andonline safety, security, or privacy. The categories of subjects were developed through iterative roundsof 10 cognitive interviews with participants of varied demographics (age, gender, socioeconomicstatus). In these interviews we prompted participants about whether there were any answersthey wanted to provide, but which were not available for them to select. These interviews area commonly-used technique for testing survey questions [40] and are not intended as researchartifacts (see Section 3.3 below for more details).To understand how people learned online – that is, what resources they use to learn – weincluded a matrix question that asked about how participants learned each of the subjects theyindicated in the prior question. Specifically, respondents could select multiple options from a listof ten resources: watched a video (e.g., YouTube), read Wikipedia, took an online course, read aninformational article, read a how-to-guide, used an interactive tutorial, used a practice exam website,used materials from a course, read answers to questions on an online discussion community orforum, and asked questions in an online discussion community or forum.Other than the first three of these response options, the others included at least two examples (seebelow) in parentheses to help respondents understand our categories. Informational articles includearticles on non-Wikipedia websites, such as privately-run wikis like Wikia, online references likedictionaries and thesauri, and research-based resources like museum websites. How-to guidesinclude websites that help people complete projects around their home as well as step-by-stepguides for understanding mathematics like BetterExplained.com. Examples of interactive tutorialson the web we mentioned include platforms like Khan Academy, Codecademy, and Duolingo.Practice exam sites like Kaplan and Magoosh focus on preparation for standardized exams, whilecourse materials refer to open repositories of slides and other artifacts from courses usually taught Respondents had the option to select none of the options, thus indicating that they had not learned online. This was phrased as “how-to (e.g., around the house, cooking/baking, etc.)”Proc. ACM Hum.-Comput. Interact., Vol. 5, No. CSCW1, Article 146. Publication date: April 2021. in person, like the MIT OpenCourseWare project. Finally, discussion communities and forumsinclude social-media forums like Facebook Groups and Reddit, in addition to forums designed forprofessional communities like StackOverflow. Given our interest in comparing informal onlinelearning to formal online learning, which we understood as the online courses, those who indicatedtaking online courses were then presented with an additional question: “What kinds of organizationshave provided the online course(s) that you have enrolled in? (Check all that apply)” The answeroptions were: Online course at a local university; Online course at a university elsewhere; Massiveopen online course (Coursera, FutureLearn, edX, Udacity, Udemy); Other online course (e.g. TheGreat Courses, Lynda.com).Overall, respondents could report up to 190 possible online learning experiences (pairs of subjectsand resources such as learning math on YouTube or learning languages using an interactive tutorial),since we asked about how respondents learned up to 19 subjects using up to 10 online learningresources. During the in-person cognitive interviews, we asked interviewees to think aloud as they werefilling out the survey, and they were encouraged to share their thoughts if they believed that theywere missing answer choices or if they were confused about the wording of a question. Based onthe feedback from these interviews we rewrote and rearranged our survey questions to maximizetheir clarity and to ensure the completeness of the answer choices we offered. Once our surveywas deployed, less than one percent (0.88%) of respondents indicated that their desired answerchoice was not available for any of our questions (they selected “Other" in response to the question),confirming the completeness of the choices we provided.
A respondent is considered to have learned online if they learned about at least one subject usingat least one resource.To investigate what subjects people learn online and what resources they use to learn them (RQ1and RQ2), we use descriptive data analysis including hypothesis tests and 𝜒 proportions tests asappropriate. Additionally, as reviewed in Section 2, prior work has considered formal and informallearning resources independently, but has not examined differences in what these resources areused to learn about. To fill this gap, we make subject-by-subject comparisons between onlinecourses and all other types of online educational resources combined. We used 𝜒 proportionstests to make inferences about whether online courses are used more, less, or equally as oftento learn about certain subjects compared to other types of resources. To reduce the Type I errorrate we applied the Bonferroni-Holm correction to the resulting p-values to account for multiplehypothesis testing.To answer RQ3 we examined commonalities in subjects and resources by using k-modes clus-tering [21] to organize respondents into clusters based on the subjects they reported learningabout online and, separately, the types of resources they reported using. K-modes clustering isan extension of k-means clustering for categorical data. While k-means minimizes the distancebetween the center of each cluster and points belonging to that cluster in euclidean space, k-modesaims to maximize the similarity in categories shared between observations (respondents in ourcase) within a cluster. The number of clusters we selected for clustering respondents accordingto shared types of resources and subjects was chosen using the silhouette method [45], where we The parts of the survey questionnaire that were analyzed in this work can be found at https://doi.org/10.5281/zenodo.4088916.Proc. ACM Hum.-Comput. Interact., Vol. 5, No. CSCW1, Article 146. Publication date: April 2021. haracterizing the Online Learning Landscape: What and How People Learn Online 146:7 calculated the average silhouette width for 𝑘 equal to 3 through 10, and then selected the valueof 𝑘 that maximized the average silhouette width. This method for selecting 𝑘 ensures that theobservations being clustered are the most similar to the cluster they are in and that they are themost dissimilar from clusters from which they are excluded. Respondents were asked to self-report their online learning experiences, therefore this studyexhibits many of the same limitations as other self-report studies. These limitations include over-and under-reporting, which occur when survey results reflect an over- or under-estimate of therate or abundance of an experience. This discrepancy can be caused by respondents misinterpretingsurvey questions, or it can be induced by one of several biases, including desirability bias (whenrespondents give socially desirable, instead of honest answers), and recall bias (when respondentsincorrectly remember an experience). To alleviate the potential for these inconsistencies, we revisedthe survey questions iteratively through a series of interviews, followed by pre-testing the surveyto ensure that respondents thoroughly answered all questions and interpreted them the way wehad intended.Additionally, our survey was conducted online. As our focus is to study online learning, collectingthe data online is appropriate. Cint, the firm through which we collected our data, uses a doubleopt-in procedure to recruit respondents to its panel. Research in the past decade has established thatthere are “few or no significant differences between traditional modes [of survey administration]and opt-in online survey approaches” for research such as that presented here, which is intendedto improve our understanding of human behavior and experiences [3, 4].The results of this survey only reflect the usage of certain types of online learning resources, andthe subjects that respondents were interested in learning about. Therefore we cannot make anyclaims about how often these resources were consulted, or the duration of time that respondentsspent pursuing particular subjects. Our findings are accompanied by a number of theoreticalexplanations that include suggestions for how online educational experiences could be improvedand how they could be studied in the future. This is not an experimental study and we have datafrom one point in time so it is inappropriate to interpret any of the relationships that we present ascausally linked.Finally, it is important to clarify that the conception of “learning” presented in this paper is basedin the computer and cognitive sciences, and therefore represents only one perspective among many.For example, much of the prior work that inspired the design of this study is related or adjacentto studies of “online learning.” This includes both empirical studies of what kinds of activitiesbenefit specific learning outcomes [24], and theoretical work that centers “instructional events”and “learning events” within greater learning frameworks [23]. This is in contrast to many otherapproaches in the learning sciences that focus on the institutions that create learning opportunities,instruction styles, the roles of facilitators in learning environments, and taxonomies of learningoutcomes [34].
Overall, we find that 93% of our 2260 survey respondents reported learning something online. In thissection, we detail the results of our three research questions, examining: (RQ1) what subjects peoplelearn about online; (RQ2) how people learn online; and (RQ3) which online learning experiencesare shared among the majority of online learners.
Proc. ACM Hum.-Comput. Interact., Vol. 5, No. CSCW1, Article 146. Publication date: April 2021.
Fig. 1. (Left) The popularity of different subjects that respondents learned online. (Right) The proportionof books in circulation according to a study by Littman and Connaway [31]. Both the internet and librarycollections are public repositories for learning resources on many different subjects. This comparison showsthat the distribution of subject popularity in our survey mirrors the same power law distribution found in thecirculation of different subjects of library books.
We asked survey respondents to report what subjects they learned about online. They reportedlearning about 19 subjects included in our question . These subjects ranged from non-academictopics such as travel and personal physical fitness to traditional academic subjects such as law andmathematics (see Figure 1 [Left] for a summary).We examine how many of our respondents chose to learn about these subjects. The only subjectthat more than half of our respondents reported learning about was DIY (62.4%), which includesaround-the-house activities such as cooking, baking, and arts-and-crafts. After DIY, the next mostpopular subjects, which were statistically less popular (p < 0.001 with Bonferroni-Holm correctionfor multiple comparisons) , were those about general welfare and lifestyle – physical health andhealthcare (49%), personal physical fitness (37%), and travel and geography (32%) – and history(40%).Our results show that nearly all general-interest subjects were more popular than academic-focused subjects. Academic subjects, aside from history, were significantly less popular (p < 0.005Bonferroni-Holm correction for multiple comparisons) than general interest subjects. Less than30% of respondents reported learning about: fine arts (28%), mathematics (26%), the natural sciences(26%), languages (25%), medicine (23%), politics (22%), social sciences (22%), religion and ethics(19%), computer programming (15%), law (15%), graphic design (14%), and literature (14%).To contextualize our findings, we compare the popularity of subjects learned online with thepopularity of subjects learned offline. We find that while the subjects adults learn online vs. offlinediffer, the popularity of learning various subjects online and offline follows a similar trend: a fewsubjects are very popular while each of the remaining subjects are learned by a sizable minority. As discussed in more detail in the methodology section, we also offered a free-text “Other” option for inputting additionalsubjects. Given that less than 1% of respondents reported a subject not included in the subject list, and there was very littleoverlap in the subjects reported by these respondents. Statistical comparisons for each of the subject categories reported in this section can be found in the appendix.Proc. ACM Hum.-Comput. Interact., Vol. 5, No. CSCW1, Article 146. Publication date: April 2021. haracterizing the Online Learning Landscape: What and How People Learn Online 146:9
This is illustrated by the comparison in Figure 1: on the left, we show the popularity of subjectsfor online learning in our data set while on the right we show the distribution of offline learninginterests as represented by the distribution of library books currently in circulation according tothe Library of Congress Classification [28]. We observe that this distribution follows a power lawrelationship like the Zipf distribution, where only a few subjects are very popular and most subjectsare all approximately equal in their middling popularity.
Fig. 2. Proportion of respondents that reported using each type of educational resource.
Respondents used at least one and at most 10 (mean: 2.93, median: 2) resources to learn about aparticular subject.Of the 2094 respondents to our survey who indicated using at least one resource, the vast majority(88%) relied on YouTube to learn online. Additionally, 77% used Wikipedia or informational articles,respectively, to learn online. Over half of respondents reported learning online using how-to guides(70%) or by using question-asking forums: either asking questions (60%) or reading answers toothers’ questions (65%). The least used online learning resources, but still used by at least halfof respondents, were academic-style resources: interactive tutorials (58%), course materials (e.g.,slides, course notes) (52%), and practice exam websites (50%).Further, 55% of respondents reported enrolling in formal online courses to learn something. Ofthese respondents, 40.5% reported having taken an online course from a local university, while22.7% said they had taken an online course at a university that was not local to them. The differenceof these two proportions suggests that respondents’ awareness of online education is not as globalas the potential online educational opportunities that may be open to them. However, it is alsopossible that the higher percentage of locally-based online course enrollment may be influenced bycollege degree programs that combine in-person and online coursework, rather than by a lack ofawareness about non-local options.Respondents also took online courses through professional services (20.5%) such as Lynda, orthrough massively open online courses such as those offered on Coursera (16.5%). A tenth (10.2%)of respondents reported that they had used more than one type of online course.Figure 2 provides a summary of respondents’ online learning resources. A respondent is considered to have learned online using a resource if they learned about at least one subject using thisresource. Proc. ACM Hum.-Comput. Interact., Vol. 5, No. CSCW1, Article 146. Publication date: April 2021.
Havingidentified the popularity of subjects and of resources, the next step is to look at the relationshipbetween the two. Are certain resources used for more subjects than others? Are certain resourcesmore popular for certain subjects than others?
Fig. 3. The average number of subjects that an individual respondent learned about for each resource.
We find that YouTube is not only the resource used by the most respondents, it is also used byrespondents to learn about the most subjects: a respondent who uses YouTube for learning usesthe platform to learn about an average of 3.56 subjects. YouTube is used to learn about significantlymore subjects (Mann-Whitney U Test, Bonferroni-Holm corrected p < 0.001) than Wikipedia (mean= 2.54 subjects) and other informational articles (mean = 2.25 subjects) or reading (mean = 2.54subjects) and asking questions on Q&A forums (mean = 2.25 subjects). All of these resources wereused by respondents to learn significantly more subjects (Mann-Whitney U Test, Bonferroni-Holmcorrected p < 0.002) than how-to guides, academic-style resources, and online courses, all of whichwere used to learn about less than two subjects, on average. Figure 3 summarizes the averagenumber of subjects that respondents learned about using each resource.While YouTube is the most used resource overall (used by 88% of respondents), it is usedparticularly heavily to learn about certain subjects. Nearly a quarter of respondents who usedYouTube learned about these five subjects: makeup and fashion (used by 36.5% of learners), physicalfitness (30.6%), DIY (28.4%), travel/geography (24.0%) and the arts (23.7%). Across all subjects,YouTube was at least the third most used type of learning resource.The next most popular resource types include informational articles and Wikipedia (used by77% of learners, respectively). Informational articles were particularly popular for learning aboutsubjects related to health, including being the most popular resource for personal health care(27.2%), medicine and nursing (24.6%), and the second most popular resource for physical fitness(15.2%). Wikipedia on the other hand was the most popular resource for learning social scienceand humanities subjects such as history (25.7%), religion and ethics (22.7%), literature and poetry(19.8%), politics (19.7%), and law (18.3%).While how-to guides are, unsurprisingly, used by 20.7% of respondents to learn about DIY, theyare not otherwise used heavily to learn about any other particular subject (an average of 8.2% ofparticipants used how-to guides to learn across all subjects, SD = 4.04%). Similarly, reading andasking questions on Q&A forums made up 11.0% (SD = 2.04%) and 8.9% (SD = 1.64%), respectively,of the resources used by respondents to learn about any given subject.
Proc. ACM Hum.-Comput. Interact., Vol. 5, No. CSCW1, Article 146. Publication date: April 2021. haracterizing the Online Learning Landscape: What and How People Learn Online 146:11
Fig. 4. Proportion of each type of resource that was used to learn about each subject, in order from mostpopular to least popular subject.
Together, interactive online tutorials, course materials, and practice exams made up 40.5% ofthe resources that respondents used to learn about Math. Similarly, these academic-style resourcesmade up 33.3% of the resources used to learn languages, 29.0% of those used to learn programming,and 26.7% and 29.3%, respectively, of the resources used to learn social and natural science. Thespecific type of academic-style resource used most heavily to learn these subjects differed, withpractice exams (14.0% of the resources used) and interactive tutorials (13.6% of the resources used)being particularly popular for math, interactive tutorials being particularly popular for languages(17.8% of the resources used), and course materials being particularly popular for the social andnatural sciences ( 11% of the resources used, respectively) and programming (10.2% of the resourcesused).As mentioned in the introduction of this paper, the vast majority of prior work on online learningfocuses on formal online courses rather than informal resources. This paper is the first, to ourknowledge, to consider both types of resources. As such, we were especially interested in thedifferences between how people use online courses vs. informal online learning resources. Thesedifferences are summarized in Table 1. We find that STEM-related academic subjects as well associal science, literature, languages, and graphic design are more often learned about using onlinecourses than informal resources.On the other hand, informal resources were significantly more likely to be used to learn aboutgeneral interest subjects: DIY, health care, physical fitness, travel and geography, and makeup andfashion. The contrast between these two groups of subjects may be related to the need (or lack ofneed) for a formal environment to become proficient in a subject. Mastering a language requiresan experience approximating immersion into a new social environment, and learning graphicdesign may require hours of orientation with complex user interfaces. In contrast, successfullyplanning a trip or learning how to repair something specific around the house does not requireintense prolonged study. Another difference between these two groups is the extent to which theseactivities are learned through physical interactions. Mathematics and computer programming are
Proc. ACM Hum.-Comput. Interact., Vol. 5, No. CSCW1, Article 146. Publication date: April 2021.
Subject Proportion Formal Proportion Informal Significance Greater ProportionHistory 0.080 0.078Arts 0.056 0.055Medicine/Nursing 0.049 0.045Online Safety 0.039 0.032Religion/Ethics 0.035 0.032Politics 0.034 0.040Law 0.026 0.025Math 0.101 0.063 *** FormalLanguages 0.086 0.045 *** FormalScience 0.076 0.061 * FormalSocial science 0.075 0.048 *** FormalProgramming 0.058 0.036 *** FormalGraphic Design 0.042 0.026 *** FormalLiterature/Poetry 0.037 0.024 ** FormalDIY 0.063 0.132 *** InformalHealth Care 0.047 0.091 *** InformalPhysical Fitness 0.035 0.063 *** InformalTravel/Geography 0.024 0.052 *** InformalMakeup/Fashion 0.022 0.043 *** Informal
Table 1. Pairwise statistical tests comparing the proportion of respondents who used formal versus informalonline learning resources to pursue each subject. Zero * indicates no significant difference, one * indicates p <0.05, two ** indicates p < 0.01, three *** indicates p < 0.001. ‘Greater Proportion’ indicates whether formal orinformal resources were used significantly more often. subjects concerned with manipulating abstract symbols, while training one’s physical fitness orlearning a new sewing or makeup technique have to be physically practiced to be understood.Finally, respondents in our survey were equally as likely to report having used an online courseor an informal resource to learn about: history, arts, religion and ethics, politics, law, medicineand nursing, and online safety. We hypothesize that this is because these subjects straddle the linebetween academic and general interest: for example, an online learner could be studying art historyas part of a degree program or job preparation or an online learner could be passively curious aboutthe art of Michelangelo after a recent trip to Rome.
Finally, going beyond the popularity of what and how people learn online, we consider commonpatterns in people’s practice of online learning. We seek to identify shared experiences in onlinelearners’ interests (what they learn), learning tools (how they learn), and overall online learningexperiences (common pairs of subjects and resources).First, to identify commonalities in online learners’ interests, we used k-modes clustering to grouprespondents who reported learning more than one thing online based on the subjects they reportedlearning. Of those who learned at least two things online, 76.1% of respondents had in commononly that they learned about DIY subjects online. These respondents did not necessarily indicatethat they learned exclusively about DIY subjects, however their choice to learn about DIY subjectsis what they had most in common. An additional 18.8% of respondents had in common that theylearned both about DIY and about at least one of: languages, health care, math, history, natural
Proc. ACM Hum.-Comput. Interact., Vol. 5, No. CSCW1, Article 146. Publication date: April 2021. haracterizing the Online Learning Landscape: What and How People Learn Online 146:13
Fig. 5. Euler diagram representing k-modes clustering of types of online learning resources. Our clusteringanalysis revealed three overlapping core sets of online learning resources. Resources that are contained withina greater number of circles are more central overall to online learning experiences. For example, YouTube iscontained within three circles, compared to Wikipedia which is only contained within two circles. ThereforeYouTube is a more central resource compared to Wikipedia. science, social science, and physical fitness. For the remaining 5.1% of respondents, our clusteringwas unable to identify a common pattern among these respondents’ online learning interests.Second, we consider commonalities in how people learn online. We again used k-modes clusteringto group those respondents who used more than one online learning resource, this time clusteringthe respondents based on the resources they used rather than the subjects they learned.Of respondents who used at least two online learning resources, 17.1% had in common onlythat they used YouTube (the blue cluster in Figure 5), while 12.9% of respondents had in commonthat they used both YouTube and at least one of the following: how-to guides, Wikipedia andinformational articles. An additional 18.8% of respondents had in common that they (1) usedYouTube, (2) used at least one of the aforementioned three resources, and (3) asked questions inQ&A forums. We could not find any pattern in resource use for the remaining 51.2% of respondents.Of the 190 online learning experiences respondents reported, these twelve were experienced mostoften by our respondents: using YouTube to learn about DIY subjects (54.9% of respondents), healthcare (24.0%), makeup/fashion (23.2%), history (21.2%), arts (19.9%), or travel/geography (18.3%);using how-to guides to learn about DIY subjects (40.0%), using informational articles to learn abouthealth care (36.5%), using Wikipedia to learn about history (30.6%), or reading Q&A forums orasking questions on those forums to learn about DIY subjects (25.4% and 20.3%, respectively).
Overall, we find that online learning is a ubiquitous experience: 93% of respondents in our surveyreported learning online. YouTube, in particular, is by far the most popular online learning resource.Not only is YouTube used by the most respondents for learning (88%), it is the most popularresource for learning about 11 out of the 19 subjects examined in our survey. The fact that, onaverage, respondents used YouTube to learn about more than three subjects further underscores theimportance of this resource. Moreover, YouTube is not only popular, it is core to the experience ofonline learning. Our clustering analyses consistently show that YouTube is at the core of other onlinelearning experiences. This result is consistent with previous studies that have found that YouTubespans across formal and informal online learning resources [44]. YouTube recently redesigned an
Proc. ACM Hum.-Comput. Interact., Vol. 5, No. CSCW1, Article 146. Publication date: April 2021. influential page where trending videos are displayed, adding a “Learning” category [9], perhaps inresponse to this trend.In this section, we discuss which mechanisms of online learning are most pervasive and whatthe prevalence of particular mechanisms implies about the online learning ecosystem and howwe design for it. Additionally, we point out directions for future work based on our results. Weconclude with a look toward the future: what implications our findings have for the future of onlinelearning in a world after COVID-19.
As noted earlier, YouTube is the most popular resource for online learning. Perhaps unsurprisingly,informational articles and Wikipedia are the next most popular types of online learning resourcesconsidering that text-based articles and web pages were the first and most fundamental parts of theinternet [7]. Indeed, the four core resources identified by our clustering – YouTube, informationalarticles, Wikipedia, and how-to guides – are built around three legacy internet technologies: video,static text, and wikis. YouTube was founded over 15 years ago [30], static text websites have beenaround since the beginning of the internet, and wikis are a technology that is over 25 years old [12].We hypothesize that the popularity of these legacy resources has four potential causes. First,adult learners may find it easiest – or prefer – to learn through informal video media (e.g., YouTube)and informational articles, and thus leverage these resources the most. Second, due in part to havinga long period of time over which to develop content, these popular learning resources are able tosupply a high volume of content across many topics. For example, YouTube is the leading videoplatform – in terms of both hours of content and revenue [2, 11]. As such, it may be easiest forlearners to find learning content on YouTube, thus leading them to learn using video media. Third,over extended use, Internet users’ may have developed familiarity with learning from video and/orinformational-article style content. This familiarity may lead them to continue to turn to learningmodes from which they are comfortable receiving content instead of exploring newer learningtechnologies such as interactive tutorials, MOOCs and question-and-answer forums, which mayalso have a lower supply of content. Finally, the popularity of these resources may create a cyclicaleffect: because these resources are more popular, and offer more supply, they are easier to findin search results, and thus learners are more likely to turn to them. For example, prior work hassuggested that some of Wikipedia’s popularity could be attributed to how highly it is ranked insearch engine results [33]. These findings raise important questions for future work to investigateregarding whether these resources actually best meet learner needs, or are merely used out ofthe convenience due to supply, familiarity, and/or ease of access. We delve deeper into how ourfindings support or refute each of these potential drivers behind learners’ choice of online learningresources below.
Our work finds that adult online learners have extremely diverse interests, with more than half(54%) learning about 5 or more different subjects. As mentioned above, one of the advantages ofYouTube and Wikipedia (as well as informational articles) is supply: these resources host contentabout a wide array of subjects. However, newer online learning resources may lack this volumeof content, either by choice or due to their relative novelty. This lack of diversity may inherentlyinfluence the mechanism of online learning. Let us take as a case study law and programming.Law and computer programming carry wide-ranging societal influence but were two of the leastpopular subjects among our respondents, with about 15% of respondents indicating they learnedabout either subject. Despite the equal popularity of these subjects, programming was one of themost popular subjects for two types of resources (course materials, interactive tutorials), while law
Proc. ACM Hum.-Comput. Interact., Vol. 5, No. CSCW1, Article 146. Publication date: April 2021. haracterizing the Online Learning Landscape: What and How People Learn Online 146:15 was 12th most popular of the 19 subjects respondents learned using Wikipedia, and ranked lowerin terms of popularity for every other type of online learning resource. While it is possible thatrespondents use a wide variety of resources for learning about law because of the nature of thesubject, this may also be the case because less online course content is available for law vs. forprogramming. Programming has numerous free, online course style resources such as Codecademyand open source scholarly projects like Python Tutor [17, 51] that put interactive programmingtutorials online, in addition to more traditional online courses – either free or payment-based –such as those offered by universities or through MOOCs. Our findings suggest that learners mayturn to newer resources tailored to their learning needs for a particular subject if those resourcesare available. In the absence of such tailored resources, however, online learners may turn to theresources that they know will offer sufficient, affordable, supply: YouTube and Wikipedia.By highlighting this contrast between law and computer science we do not mean to suggestthat low subject diversity among formal online learning resources is necessarily a problem to beaddressed. We believe that this example illustrates with data that the way people learn differentsubjects online is nuanced in ways that are not intuitive. Understanding these kinds of gradationsin the online learning landscape may also call attention to opportunities for new kinds of onlinelearning experiences.
These issues of supply may relate to the ease with which resource creators can develop newcontent. The relatively low popularity of new learning resources (e.g., MOOCS, interactive tutorials)highlights the absence of the diffusion of innovations [43] we might expect given years of interestin new types of educational media and interactive online learning technologies [19]. This absencemay be related to the ease with which these resources can be created. Phones, tablets, and personalcomputers are now often embedded with more than one camera, making the creation of (educational)videos easier than ever before, and the tools for publishing text-based articles are just as widespread.However, relatively less robust and cost-effective support is available for the creation of newerresources. For example, many universities have had to scale up production studios in which tocreate MOOCS [5], purchasing expensive equipment out of reach for many of the learning contentcreators who utilize YouTube. Similarly, many interactive tutorials involve the creation of newsoftware products, requiring significant grant funding and time investment for development [17, 51].Thus, future work on the democratization of online education [27] may wish to consider howto democritize the creation of content, in an effort to improve the learning content available forconsumption.
Our results illustrate the breadth of adult online learning and raise a number of questions regardingwhy adults choose to learn using particular resources. Above, we hypothesized that learners maychoose to learn using certain resources due to the supply of content available from certain resources,learners’ familiarity with the mode of content, the ease with which learners can find that content,and with which educators can create it. However, online learners may also choose to learn particulartopics with particular resources for reasons related to their own internal motivations, rather thanexternalities of the online learning ecosystem.As discussed in prior work, there exists a spectrum of learning methods that may be used by, orwhich appeal to, different learners. This spectrum ranges from free choice learning – e.g., visiting amuseum, watching a documentary for entertainment [13] – to traditional, academic, structuredlearning (e.g., taking a course). One of the contributions of our work is to answer the call [48]
Proc. ACM Hum.-Comput. Interact., Vol. 5, No. CSCW1, Article 146. Publication date: April 2021. to compare online learners’ use of such informal vs. formal online resources. We find significantdifferences in the use of such resources (Table 1).We hypothesize that these differences may relate not only to supply, but to learners’ goals. Forexample, credentials earned online are increasingly popular across many fields, and demand forthese credentials only seems to be growing [10]. We hypothesize that learners who want to earn acredential are using online courses, practice exams, and course materials more often comparedto other online learning resources. In Figure 3 we can see that some of the subjects that mostcommonly use those resources are math, programming, science, and languages, all subjects inwhich it can be valuable to obtain credentials for professional purposes as opposed to the topics thatwe find are learned more frequently using informal resources such DIY and personal health care.Thus, we encourage future work building on the foundation laid by these findings, to investigatehow the professional and credential-related motivations of online learners inform their choice oflearning resources.In addition to professionally-related credentials, our findings also suggest that learners maybe motivated to seek more authoritative information about some topics, even if they are notpursuing a professional qualification. For example, respondents indicated that they used Wikipediamost for learning about history, politics, religion & ethics, and law (see Figure 3), all topics thatmay be presented with significant bias elsewhere. The Wikipedia community actively moderatestheir articles, lending a sense of community authority [46] to their content, which may appealto certain learners, or learners of particular subjects. As such, future work may seek not only toinvestigate learners’ goals, but also the criteria through which they evaluate potential sources oflearning information – much like prior work has studied news consumers’ evaluation of media andmisinformation [29, 47].
During the preparation of this manuscript, COVID-19 emerged causing major disruptions andreorganization to how educational experiences are structured and delivered. Major universities hadto transition all of their courses online quickly to minimize in-person interactions and to complywith new government regulations designed to protect public health [25]. We believe it is reasonableto say that the emergence of COVID-19 is the most significant event in the nascent history of onlineeducation, as the pandemic precipitated a situation where education in any form became nearlysynonymous with online education.Although the long-term effects of COVID-19 on education at large have yet to be realized, severalshort-term effects related to this study are taking form. Since the proliferation of the disease and theresulting lockdowns, more adult learners have been seeking online educational experiences [37].This may be related to the financial recession caused by COVID-19, since it is well understoodthat recessions drive increased interest in adult educational programs [1]. However, given thatmost college campuses are closed, adult learners are more frequently turning to online learningresources [37]. Massively open online course providers in particular have seen a record-breakinggrowth in enrollments [32]. This is a significant development in the trajectory of MOOCs as priorwork has reflected on the role of MOOCs in online education as modeled by their path through theGartner Hype Cycle [15, 26].The data in this study represent a snapshot of the adult online learning landscape just monthsbefore COVID-19. We believe that our results hold valuable insights that can inform the futuredesign of online learning technologies. However, this study also inadvertently and advantageouslyprovides a baseline that can be used in the future to measure how this monumental shift in onlinelearning is changing interests in subjects or the differential use of types of online learning resourcesas both are influenced by the global pandemic.
Proc. ACM Hum.-Comput. Interact., Vol. 5, No. CSCW1, Article 146. Publication date: April 2021. haracterizing the Online Learning Landscape: What and How People Learn Online 146:17
In this work, we presented the results from a survey of 2260 adults about their experiences usingonline learning resources to learn about different topics. Our survey was balanced to include adiverse sample. We found that over 90% of respondents reported learning online about subjects asdiverse as math, politics, and the arts. The most popular subjects that respondents learned aboutonline included DIY, personal health care, history, physical fitness, and travel & geography. Theproportion of respondents interested in the various subjects in our survey followed a power lawdistribution.The most popular types of online learning resources included YouTube, informational articles,and Wikipedia. On average, participants used YouTube to learn about three different subjects, andWikipedia and informational articles to learn about more than two different subjects. Respondentsreported using every resource in our survey to learn about every subject. The most popular subjecton YouTube was makeup & fashion, while the most popular subject on Wikipedia was history, andpersonal health care for informational articles.Further analysis of our results showed that respondents learned about subjects like math, law,and computer programming from more formal online resources. However, for subjects like DIY,physical fitness, and travel & geography, respondents more often learned from informal resources.Some respondents used formal and informal online learning resources equally to learn aboutsubjects like history, the arts, and politics. Finally, a clustering analysis revealed that YouTube isthe resource that has the most overlap among all of our respondents’ online learning experiences.To conclude, we discussed the importance of video and text as technologies that are fundamentalto the web itself, and serve as especially important infrastructure for online education. We alsoexplored how the forces of supply and demand drive the availability of online educational expe-riences. Finally, we set our sights on the future by examining factors that impact resources thatlearners seek out, and how online education will be shaped by the COVID-19 pandemic.
ACKNOWLEDGMENTS
We would like to thank Facebook Research for funding this study as part of their EconomicOpportunity and Digital Platforms Research Award. We would also like to thank Tamara Cleggand Heather Killen for their helpful input.
REFERENCES [1] Donna Airoldi. 2016. Older Americans Went Back To School During The Recession. Did It Pay Off? (December 2016).[2] Julia Alexander. 2020. Creators finally know how much money YouTube makes, and they want more of its.
The Verge (2020).[3] Stephen Ansolabehere and Brian F Schaffner. 2014. Does survey mode still matter? Findings from a 2010 multi-modecomparison.
Political Analysis (2014), 285–303.[4] Reg Baker, Stephen J. Blumberg, J. Michael Brick, Mick P. Couper, Melanie Courtright, J. Michael Dennis, Don Dillman,Martin R. Frankel, Philip Garland, Robert M. Groves, Courtney Kennedy, Jon Krosnick, Paul J. Lavrakas, Sunghee Lee,Michael Link, Linda Piekarski, Kumar Rao, Randall K. Thomas, and Dan Zahs. 2010. Research Synthesis: AAPORReport on Online Panels.
Public Opinion Quarterly
74, 4 (10 2010), 711–781. https://doi.org/10.1093/poq/nfq048arXiv:https://academic.oup.com/poq/article-pdf/74/4/711/5186296/nfq048.pdf[5] Rose M Baker and David L Passmore. 2016. Value and pricing of MOOCs.
Education Sciences
6, 2 (2016), 14.[6] Yochai Benkler. 2006.
The Wealth of Networks: How Social Production Transforms Markets and Freedom . Yale UniversityPress.[7] Tim Berners-Lee and Mark Fischetti. 2001.
Weaving the Web: The Original Design and Ultimate Destiny of the WorldWide Web by Its Inventor . DIANE Publishing Company.[8] Leland P Bradford. 1958. The teaching-learning transaction.
Adult Education
8, 3 (1958), 135–145.[9] Ashley Carman. 2020. YouTube ditches its Trending tab for Explore on mobile.
The Verge (2020).Proc. ACM Hum.-Comput. Interact., Vol. 5, No. CSCW1, Article 146. Publication date: April 2021. [10] Jason Caudill. 2017. The emerging formalization of MOOC coursework: Rise of the MicroMasters. In
EdMedia+ InnovateLearning . Association for the Advancement of Computing in Education (AACE), 1–6.[11] J Clement. 2019. Hours of video uploaded to YouTube every minute as of May 2019.
Statista (2019).[12] Anja Ebersbach, Markus Glaser, Richard Heigl, and Alexander Warta. 2008.
Wiki: web collaboration . Springer Science& Business Media.[13] John H Falk, Martin Storksdieck, and Lynn D Dierking. 2007. Investigating public science interest and understanding:Evidence for the importance of free-choice learning.
Public Understanding of Science
16, 4 (2007), 455–469.[14] Rebecca Ferguson and Simon Buckingham Shum. 2012. Social Learning Analytics: Five Approaches. In
Proceedings ofthe 2nd International Conference on Learning Analytics and Knowledge (Vancouver, British Columbia, Canada) (LAK’12)
Proceedings of the Third (2016)ACM Conference on Learning @ Scale (Edinburgh, Scotland, UK) (L@S ’16) . Association for Computing Machinery,New York, NY, USA, 21–30. https://doi.org/10.1145/2876034.2876052[17] Philip J. Guo. 2013. Online Python Tutor: Embeddable Web-Based Program Visualization for Cs Education. In
Proceedingof the 44th ACM Technical Symposium on Computer Science Education (Denver, Colorado, USA) (SIGCSE ’13) . Associationfor Computing Machinery, New York, NY, USA, 579–584. https://doi.org/10.1145/2445196.2445368[18] Aboozar Hadavand, Ira Gooding, and Jeffrey T Leek. 2018. Can MOOC Programs Improve Student EmploymentProspects?
Available at SSRN 3260695 (2018).[19] Benjamin T Hazen, Yun Wu, Chetan S Sankar, and L Allison Jones-Farmer. 2012. A proposed framework for educationalinnovation dissemination.
Journal of Educational Technology Systems
40, 3 (2012), 301–321.[20] Khe Foon Hew and Wing Sum Cheung. 2014. Students’ and instructors’ use of massive open online courses (MOOCs):Motivations and challenges.
Educational research review
12 (2014), 45–58.[21] Zhexue Huang. 1997. A fast clustering algorithm to cluster very large categorical data sets in data mining.
DMKD
3, 8(1997), 34–39.[22] René F. Kizilcec and Emily Schneider. 2015. Motivation as a Lens to Understand Online Learners: Toward Data-Driven Design with the OLEI Scale.
ACM Trans. Comput.-Hum. Interact.
22, 2, Article 6 (March 2015), 24 pages.https://doi.org/10.1145/2699735[23] Kenneth R. Koedinger, Albert T. Corbett, and Charles Perfetti. 2012. The Knowledge-Learning-Instruction Framework:Bridging the Science-Practice Chasm to Enhance Robust Student Learning.
Cognitive Science
36, 5 (2012), 757–798. https://doi.org/10.1111/j.1551-6709.2012.01245.x arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1551-6709.2012.01245.x[24] Kenneth R. Koedinger, Jihee Kim, Julianna Zhuxin Jia, Elizabeth A. McLaughlin, and Norman L. Bier. 2015. Learning isNot a Spectator Sport: Doing is Better than Watching for Learning from a MOOC. In
Proceedings of the Second (2015)ACM Conference on Learning @ Scale (Vancouver, BC, Canada) (L@S ’15) . Association for Computing Machinery, NewYork, NY, USA, 111–120. https://doi.org/10.1145/2724660.2724681[25] Melissa Korn, Douglas Belkin, and Juliet Chung. 2020. Coronavirus Pushes Colleges to the Breaking Point, Forcing’Hard Choices’ About Education.
The Wall Street Journal (2020).[26] Sean Kross and Philip J. Guo. 2018. Students, Systems, and Interactions: Synthesizing the First Four Years of Learn-ing@scale and Charting the Future. In
Proceedings of the Fifth Annual ACM Conference on Learning at Scale (Lon-don, United Kingdom) (L@S ’18) . Association for Computing Machinery, New York, NY, USA, Article 2, 10 pages.https://doi.org/10.1145/3231644.3231662[27] Sean Kross, Roger D Peng, Brian S Caffo, Ira Gooding, and Jeffrey T Leek. 2020. The democratization of data scienceeducation.
The American Statistician
74, 1 (2020), 1–7.[28] F Wilfrid Lancaster. 1982. Evaluating collections by their use.
Collection Management
4, 1-2 (1982), 15–44.[29] David MJ Lazer, Matthew A Baum, Yochai Benkler, Adam J Berinsky, Kelly M Greenhill, Filippo Menczer, Miriam JMetzger, Brendan Nyhan, Gordon Pennycook, David Rothschild, et al. 2018. The science of fake news.
Science
Business Insider (2020).[31] Justin Littman and Lynn Silipigni Connaway. 2004. A circulation analysis of print books and e-books in on academicresearch library.
Library resources and technical services
48, 4 (2004), 256–262.[32] Steve Lohr. 2020. Remember the MOOCs? After Near-Death, They’re Booming.
The Wall Street Journal (2020).[33] Connor McMahon, Isaac Johnson, and Brent Hecht. 2017. The substantial interdependence of Wikipedia and Google:A case study on the relationship between peer production communities and information technologies. In
EleventhInternational AAAI Conference on Web and Social Media .Proc. ACM Hum.-Comput. Interact., Vol. 5, No. CSCW1, Article 146. Publication date: April 2021. haracterizing the Online Learning Landscape: What and How People Learn Online 146:19 [34] Barbara Means, Marianne Bakia, and Robert Murphy. 2014.
Learning online: What research tells us about whether, whenand how . Routledge.[35] OH Mower. 1960. Learning theory and behavior.
New York (1960).[36] Sneha Narayan, Jake Orlowitz, Jonathan Morgan, Benjamin Mako Hill, and Aaron Shaw. 2017. The WikipediaAdventure: Field Evaluation of an Interactive Tutorial for New Users. In
Proceedings of the 2017 ACM Conference onComputer Supported Cooperative Work and Social Computing (Portland, Oregon, USA) (CSCW ’17) . Association forComputing Machinery, New York, NY, USA, 1785–1799. https://doi.org/10.1145/2998181.2998307[37] Pammy Olson. 2020. Online Classes Boom for Adult Learners, Too.
The New York Times (2020).[38] Ray Pastore and Alison Carr-Chellman. 2009. Motivations for residential students to participate in online courses.
Quarterly review of distance education
10, 3 (2009), 263–277.[39] Philip J. Piety, Daniel T. Hickey, and M. J. Bishop. 2014. Educational Data Sciences: Framing Emergent Practices forAnalytics of Learning, Organizations, and Systems. In
Proceedings of the Fourth International Conference on LearningAnalytics And Knowledge (Indianapolis, Indiana, USA) (LAK ’14) . Association for Computing Machinery, New York,NY, USA, 193–202. https://doi.org/10.1145/2567574.2567582[40] Stanley Presser, Mick P Couper, Judith T Lessler, Elizabeth Martin, Jean Martin, Jennifer M Rothgeb, and EleanorSinger. 2004. Methods for testing and evaluating survey questions.
Public opinion quarterly
68, 1 (2004), 109–130.[41] Mitchel Resnick, John Maloney, Andrés Monroy-Hernández, Natalie Rusk, Evelyn Eastmond, Karen Brennan, AmonMillner, Eric Rosenbaum, Jay Silver, Brian Silverman, et al. 2009. Scratch: programming for all.
Commun. ACM
52, 11(2009), 60–67.[42] Carl R Rogers. 1957. Personal thoughts on teaching and learning.
Merrill-Palmer Quarterly (1954-1958)
3, 4 (1957),241–243.[43] Everett M Rogers. 1985. The diffusion of home computers among households in Silicon Valley.
Marriage & FamilyReview
8, 1-2 (1985), 89–101.[44] Sonny Rosenthal. 2018. Motivations to seek science videos on YouTube: free-choice learning in a connected society.
International Journal of Science Education, Part B
8, 1 (2018), 22–39. https://doi.org/10.1080/21548455.2017.1371357arXiv:https://doi.org/10.1080/21548455.2017.1371357[45] Peter J Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis.
Journal ofcomputational and applied mathematics
20 (1987), 53–65.[46] Gilles Sahut and André Tricot. 2017. Wikipedia: an opportunity to rethink the links between sources’ credibility, trustand authority.
First monday
22, 11 (2017).[47] Dietram A Scheufele and Nicole M Krause. 2019. Science audiences, misinformation, and fake news.
Proceedings of theNational Academy of Sciences
The nextgeneration of distance education . Springer, 139–156.[49] Aaron Shaw and Benjamin M Hill. 2014. Laboratories of oligarchy? How the iron law extends to peer production.
Journal of Communication
64, 2 (2014), 215–238.[50] Samantha Shorey, Benjamin Mako Hill, and Samuel Woolley. 2020. From hanging out to figuring it out: Socializingonline as a pathway to computational thinking.
New Media & Society (2020), 1461444820923674.[51] Zach Sims and C Bubinski. 2011. Codecademy. (2011).[52] Erwin Stengel. 1939. On learning a new language.
International Journal of Psycho-Analysis
20 (1939), 471–479.[53] Joan Ann Swanson and Erica Walker. 2015. Academic versus non-academic emerging adult college student technologyuse.
Technology, Knowledge and Learning
20, 2 (2015), 147–158.[54] Cristen Torrey, Elizabeth F. Churchill, and David W. McDonald. 2009. Learning How: The Search for Craft Knowledgeon the Internet. In
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, MA, USA) (CHI’09) . Association for Computing Machinery, New York, NY, USA, 1371–1380. https://doi.org/10.1145/1518701.1518908[55] Etienne Wenger, Richard Arnold McDermott, and William Snyder. 2002.
Cultivating communities of practice: A guide tomanaging knowledge . Harvard Business Press.[56] Seungwon Yang, Carlotta Domeniconi, Matt Revelle, Mack Sweeney, Ben U. Gelman, Chris Beckley, and Aditya Johri.2015. Uncovering Trajectories of Informal Learning in Large Online Communities of Creators. In
Proceedings of theSecond (2015) ACM Conference on Learning @ Scale (Vancouver, BC, Canada) (L@S ’15) . Association for ComputingMachinery, New York, NY, USA, 131–140. https://doi.org/10.1145/2724660.2724674[57] Saijing Zheng, Mary Beth Rosson, Patrick C. Shih, and John M. Carroll. 2015. Understanding Student Motivation,Behaviors and Perceptions in MOOCs. In
Proceedings of the 18th ACM Conference on Computer Supported CooperativeWork & Social Computing (Vancouver, BC, Canada) (CSCW ’15) . Association for Computing Machinery, New York, NY,USA, 1882–1895. https://doi.org/10.1145/2675133.2675217