Scott Deerwester

Indexing by Latent Semantic Analysis

A new method for automatic indexing and retrieval is described. The approach is to take advantage of implicit higher-order structure in the association of terms with documents (“semantic structure”) in order to improve the detection of relevant documents on the basis of terms found in queries. The particular technique used is singular-value decomposition, in which a large term by document matrix is decomposed into a set of ca. 100 orthogonal factors from which the original matrix can be approximated by linear combination. Documents are represented by ca. 100 item vectors of factor weights. Queries are represented as pseudo-document vectors formed from weighted combinations of terms, and documents with supra-threshold cosine values are returned. initial tests find this completely automatic method for retrieval to be promising.

Information Retrieval using a Singular Value Decomposition Model of Latent Semantic Structure

In a new method for automatic indexing and retrieval, implicit higher-order structure in the association of terms with documents is modeled to improve estimates of term-document association, and therefore the detection of relevant documents on the basis of terms found in queries. Singular-value decomposition is used to decompose a large term by document matrix into 50 to 150 orthogonal factors from which the original matrix can be approximated by linear combination; both documents and terms are represented as vectors in a 50- to 150- dimensional space. Queries are represented as pseudo-documents vectors formed from weighted combinations of terms, and documents are ordered by their similarity to the query. Initial tests find this automatic method very promising.

An Architecture for Full Text Retrieval Systems

A novel architecture for full-text information retrieval systems is described. The architecture’s most distinctive feature is a server that is implemented as an interpreter for a lazily evaluated functional programming language. The consequences of this approach for time and space performance are discussed, concentrating especially on the functionality provided for searching for occurrences of words in textual databases.

Storing text retrieval systems on CD-ROM: compression and encryption considerations

Until a few years ago, large full-text information retrieval systems could only be operated on powerful mainframes. Then the personal microcomputer became more popular and information retrieval. software was adapted to this smaller device, which, however, had only a relatively small memory and therefore could not be used for very large systems. Recently, the CD-ROM (compact disc read only memory) optical disc medium has become widespread, permitting access by a PC to very large amounts of storage at very low cost. It is thus possible to transfer large databases from the mainframes, where they used to be kept, to the cheaper PC’s. This creates new challenges of efficient data handling (see Christodoulakis & Ford [7]). On one hand, retrieval algorithms have to be improved since computation power is usually reduced; on the other hand, even though the storage capacity of a CD is huge (550 to 725 MB),