Michael F. Schwartz

The Harvest information discovery and access system

It is increasingly difficult to make effective use of Internet information, given the rapid growth in data volume, user base, and data diversity. In this paper we introduce Harvest, a system that provides a scalable, customizable architecture for gathering, indexing, caching, replicating, and accessing Internet information.

Locating nearby copies of replicated Internet servers

In this paper we consider the problem of choosing among a collection of replicated servers, focusing on the question of how to make choices that segregate client/server traffic according to network topology. We explore the cost and effectiveness of a variety of approaches, ranging from those requiring routing layer support (e.g., anycast) to those that build location databases using application-level probe tools like traceroute. We uncover a number of tradeoffs between effectiveness, network cost, ease of deployment, and portability across different types of networks. We performed our experiments using a simulation parameterized by a topology collected from 7 survey sites across the United States, exploring a global collection of Network Time Protocol servers.

Scalable Internet resource discovery: research problems and approaches

Over the past several years, a number of information discovery and access tools have been introduced in the Internet, including Archie, Gopher, Net nd, and WAIS. These tools have become quite popular, and are helping to rede ne how people think about wide-area network applications. Yet, they are not well suited to supporting the future information infrastructure, which will be characterized by enormous data volume, rapid growth in the user base, and burgeoning data diversity. In this paper we indicate trends in these three dimensions and survey problems these trends will create for current approaches. We then suggest several promising directions of future resource discovery research, along with some initial results from projects carried out by members of the Internet Research Task Force Research Group on Resource Discovery and Directory Service.

A case for caching file objects inside internetworks

This paper presents evidence that several, judiciously placed file caches could reduce the volume of FTP traffic by 42%, and hence the volume of all NSFNET backbone traffic by 21%. In addition, if FTP client and server software automatically compressed data, this savings could increase to 27%. We believe that a hierarchical architecture of whole file caches, modeled after the existing name servers caching architecture, could become a valuable part of any internet.We derived these conclusions by performing trace driven simulations of various file caching architectures, cache sizes, and replacement policies. We collected the traces of file transfer traffic employed in our simulations on a network that connects the NSFNET backbone to a large, regional network. This particular regional network is responsible for about 5 to 7% of NSFNET traffic.While this papers analysis and discussion focus on caching for FTP file transfer, the proposed caching architecture applies to caching objects from other internetwork services.

Internet resource discovery at the University of Colorado

The author discusses aspects of Internets resource discovery problem: how users specify searches, the difference between discovering classes of resources and locating appropriate instances, system-management problems that can be cast as global state discovery searches, issues involved with characterizing resources and with the efficient distribution of characterizing information, and social issues, especially privacy. Results of efforts to address these problems as part of the Networked Resource Discovery project at the University of Colorado are presented. These efforts use a variety of experimental approaches, including prototype systems, network measurement studies, and simulation studies.<<ETX>>

Heterogeneous computing environments: report on the ACM SIGOPS workshop on accommodating heterogeneity

The ACM SIGOPS Workshop on Accommodating Heterogeneity was conducted in December 1985 in Eastbound, Wash., as a forum for an international group of fifty researchers to discuss the technical issues surrounding heterogeneous computing environments.

Customized information extraction as a basis for resource discovery

Indexing file contents is a powerful means of helping users locate documents, software, and other types of data among large repositories. In environments that contain many different types of data, content indexing requires type-specific processing to extract information effectively. We present a model for type-specific, user-customizable information extraction, and a system implementation called Essence. This software structure allows users to associate specialized extraction methods with ordinary files, providing the illusion of an object-oriented file system that encapsulates indexing methods within files. By exploiting the semantics of common file types, Essence generates compact yet representative file summaries that can be used to improve both browsing and indexing in resource discovery systems. Essence can extract information from most of the types of files found in common file systems, including files with nested structure (such as compressed “tar” files). Essence interoperates with a number of different search/index systems (such as WAIS and Glimpse), as part of the Harvest system.

Applying an information gathering architecture to Netfind: a white pages tool for a changing and growing Internet

The Internet is quickly becoming an indispensable means of communication and collaboration, based on applications such as electronic mail, remote information retrieval, and multimedia conferencing. A fundamental problem for such applications is supporting resource discovery in a fashion that keeps pace with the Internets exponential growth in size and diversity. Netfind is a scalable tool that locates current electronic mail addresses and other information about Internet users. Since the time we first deployed Netfind in 1990, it has evolved considerably, making use of more types of information sources. As well as more sophisticated mechanisms to gather and cross-correlate information. In this paper, we describe these techniques, and present a general framework for gathering and harnessing widely distributed information in a diverse and growing Internet environment. At present, Netfind gathers information from 17 different types of sources, providing a particularly thorough demonstration of an information gathering architecture. >

Supporting resource discovery among public Internet archives using a spectrum of information quality

Wide area networks offer access to an increasing number and variety of resources. Yet, it is difficult to locate resources of interest, because of the scale and decentralized nature of the environment. The authors are interested in supporting a global confederation of loosely cooperating systems and users that share far more resources than can be completely organized. Therefore, mechanisms are needed to support incremental organization of the resources, based on the efforts of many geographically decentralized individuals, and a range of different information sources of varying degrees of quality. The authors describe a prototype implementation of a set of mechanisms intended to explore this problem in the specific domain of public Internet archives, accessible via the anonymous file transfer protocol. This is an interesting test case, because it encompasses a very large scale, administratively decentralized collection of resources, with considerably practical value.<<ETX>>

The Changing Global Internet Service Infrastructure

Explains the necessity for measurement of Internet growth so that capacity, commercial potential, etc. may be planned and assessed. Argues that this cannot be measured merely by packet counts and user registrations. Presents detailed analysis of over 13,000 sites, worldwide. Develops a mathematical model which can be used to predict growth, by individual countries and globally. Offers further topics for future research.