Ian Milligan

Lost in the Infinite Archive: The Promise and Pitfalls of Web Archives

Contemporary and future historians need to grapple with and confront the challenges posed by web archives. These large collections of material, accessed either through the Internet Archives Wayback Machine or through other computational methods, represent both a challenge and an opportunity to historians. Through these collections, we have the potential to access the voices of millions of non-elite individuals (recognizing of course the cleavages in both Web access as well as method of access). To put this in perspective, the Old Bailey Online currently describes its monumental holdings of 197,745 trials between 1674 and 1913 as the “largest body of texts detailing the lives of non-elite people ever published.” GeoCities.com, a platform for everyday web publishing in the mid-to-late 1990s and early 2000s, amounted to over thirty-eight million individual webpages. Historians will have access, in some form, to millions of pages: written by everyday people of various classes, genders, ethnicities, and ages....

Content Selection and Curation for Web Archiving: The Gatekeepers vs. the Masses

Any preservation effort must begin with an assessment of what content to preserve, and web archiving is no different. There have historically been two answers to the question “what should we archive?” The Internet Archives broad entire-web crawls have been supplemented by narrower domain-or topic-specific collections gathered by numerous libraries. We can characterize this as content selection and curation by “gatekeepers”. In contrast, we have witnessed the emergence of another approach driven by “the masses” - we can archive pages that are contained in social media streams such as Twitter. The interesting question, of course, is how these approaches differ. We provide an answer to this question in the context of a case study about the 2015 Canadian federal elections. Based on our analysis, we recommend a hybrid approach that combines an effort driven by social media and more traditional curatorial methods.

Desiderata for Exploratory Search Interfaces to Web Archives in Support of Scholarly Activities

Web archiving initiatives around the world capture ephemeral web content to preserve our collective digital memory. In this paper, we describe initial experiences in providing an exploratory search interface to web archives for humanities scholars and social scientists. We describe our initial implementation and discuss our findings in terms of desiderata for such a system. It is clear that the standard organization of a search engine results page (SERP), consisting of an ordered list of hits, is inadequate to support the needs of scholars. Shneidermans mantra for visual information seeking (“overview first, zoom and filter, then details-on-demand”) provides a nice organizing principle for interface design, to which we propose an addendum: “Make everything transparent”. We elaborate on this by highlighting the importance of the temporal dimension of web pages as well as issues surrounding metadata and veracity.

Introduction: Internet histories

For more than four decades, the Internet has grown and spread to an extent where today it is an indispensable element in the communication and media environment of many countries, and indeed of everyday life, culture and society. These precipitous changes have called for the understanding of the innovations, actors, changes and continuities involved in these evolutions, from a technical, but also from a social, scientific, politic or economic point of view. Although the history of the Internet has not been yet very predominant within the academic literature, an increased number of books and journal articles within the last decade attest to the fact that Internet history is an emerging field of study across a number of scholarly disciplines and fields. This is most evident in Internet and new media studies, but also is clear in culture, media, communication and technology research, across the diverse settings and institutional locations where such work may be encountered. A central issue for the advancement of the field is that historical studies of the Internet have mostly been published in journals related to a variety of disciplines, and these journals only rarely publish articles with a clear historical focus. The situation has greatly improved with the various recent issues dedicated to Internet and web histories,

Warcbase: Scalable Analytics Infrastructure for Exploring Web Archives

Web archiving initiatives around the world capture ephemeral Web content to preserve our collective digital memory. However, unlocking the potential of Web archives for humanities scholars and social scientists requires a scalable analytics infrastructure to support exploration of captured content. We present Warcbase, an open-source Web archiving platform that aims to fill this need. Our platform takes advantage of modern open-source “big data” infrastructure, namely Hadoop, HBase, and Spark, that has been widely deployed in industry. Warcbase provides two main capabilities: support for temporal browsing and a domain-specific language that allows scholars to interrogate Web archives in several different ways. This work represents a collaboration between computer scientists and historians, where we have engaged in iterative codesign to build tools for scholars with no formal computer science training. To provide guidance, we propose a process model for scholarly interactions with Web archives that begins with a question and proceeds iteratively through four main steps: filter, analyze, aggregate, and visualize. We call this the FAAV cycle for short and illustrate with three prototypical case studies. This article presents the current state of the project and discusses future directions.

Ten simple rules for collaborative lesson development

The collaborative development methods pioneered by the open source software community offer a way to create lessons that are open, accessible, and sustainable. This paper presents ten simple rules for doing this drawn from our experience with several successful projects.

If these crawls could talk: Studying and documenting web archives provenance

The increasing use and prominence of web archives raises the urgency of establishing mechanisms for transparency in the making of web archives to facilitate the process of evaluating a web archives provenance, scoping, and absences. Some choices and process events are captured automatically, but their interactions are not currently well understood or documented. This study examined the decision space of web archives and its role in shaping what is and what is not captured in the web archiving process. By comparing how three different web archives collections were created and documented, we investigate how curatorial decisions interact with technical and external factors and we compare commonalities and differences. The findings reveal the need to understand both the social and technical context that shapes those decisions and the ways in which these individual decisions interact. Based on the study, we propose a framework for documenting key dimensions of a collection that addresses the situated nature of the organizational context, technical specificities, and unique characteristics of web materials that are the focus of a collection. The framework enables future researchers to undertake empirical work studying the process of creating web archives collections in different contexts.

Web Archiving Democracy

Mary Haberle is a Web Archivist at Archive-It, which is the Internet Archive’s subscription web archiving service. Shes part of a support team that provides training and direct support services to our partners, including the archivists on this panel who are all using Archive-It at their institutions. Dory Bower has been an Archives Specialist at the U.S. Government Publishing Office since 2010, where she has worked on a number of projects to increase access to electronic U.S. government resources. Dory began working with web archiving in 2011 and has played a key role in all aspects of the Federal Depository Library Program Web Archive. Megan Craynon has worked at the Maryland State Archives since 2011, and has spent the majority of that time as a team member on the web archiving project. She currently serves as the Deputy Director of Special Collections. Ben Goldman is the Kalin Librarian for Technological Innovations at Penn State University Libraries, where he has overseen web archiving efforts since 2012. Roger Christman is the Governors’ Records Archivist at the Library of Virginia. In his spare time, he also manages the Library’s web archiving program. Nicholas Worby is the Government Information and Statistics Librarian as well as the Web Archives Program Coordinator at the University of Toronto. Ian Milligan is an associate professor of digital and Canadian history at the University of Waterloo. He’s leading a Mellon-funded project to develop a cloud-based infrastructure for the analysis of web archives.

A Haven for Perverts, Criminals, and Goons: Children and the Battle for and Against Canadian Internet Regulation, 1991-1999

While we today take a largely free and unregulated Internet for granted, our present regulatory environment was established in the 1990s thanks in part to a fight around the role of children on the World Wide Web. Public pressure, coupled with a national debate around cyberporn, led to serious calls for its regulation under the prism of child protection. This article explores the tensions and early fights over whether individuals and families should regulate the Internet, or, as some strenuously argued, the government had a responsibility to impose regulation. Children were the focal point of these debates.