Samuel DeFazio

Integrating IR and RDBMS using cooperative indexing

The full integration of information retrieval (IR) features into a database management system (DBMS) has long been recognized as both a significant goal and a challenging undertaking. By full integration we mean: i) support for document storage, indexing, retrieval, and update, ii) transaction semantics, thus all database operations on documents have the ACID properties of atomicity, consistency, isolation, and durability, iii) concurrent addition, update, and retrieval of documents, and iv) database query language extensions to provide ranking for document retrieval operations. It is also necessary for the integrated offering to exhibit scaleable performance for document indexing and retrieval processes, To identify the implementation requirements imposed by the desired level of integration, we layered a representative IR application on Oracle Rdb and then conducted a number of database load and document retrieval experiments. The results of these experiments suggest that infrastructural extensions are necessary to obtain both the desired level of IR integration and scaleable performance. With the insight gained from our initial experiments, we developed an approach, called cooperative indexing, that provides a framework to achieve both scalability and full integration of IR and RDBMS technology. Prototype implementations of system-level extensions to support cooperative indexing were evaluated with a modified version of Oracle Rdb. Our experimental findings validate the cooperative indexing scheme and suggest alternatives to further improve performance.

Extensible indexing: a framework for integrating domain-specific indexing schemes into Oracle8i

Extensible indexing is a SQL-based framework that allows users to define domain-specific indexing schemes, and integrate them into the Oracle8i server. Users register a new indexing scheme, the set of related operators, and additional properties through SQL data definition language extensions. The implementation for an indexing scheme is provided as a set of Oracle Data Cartridge Interface (ODCIIndex) routines for index-definition, index-maintenance, and index-scan operations. An index created using the new indexing scheme, referred to as domain index, behaves and performs analogous to those built natively by the database system. The Oracle8i server implicitly invokes user-supplied index implementation code when domain index operations are performed, and executes user-supplied index scan routines for efficient evaluation of domain-specific operators. This paper provides an overview of the framework and describes the steps needed to implement an indexing scheme. The paper also presents a case study of Oracle Cartridges (intermedia text, spatial, and visual information retrieval), and Daylight (Chemical compound searching) Cartridge, which have implemented new indexing schemes using this framework and discusses the benefits and limitations.

Indexing images in Oracle8i

Content-based retrieval of images is the ability to retrieve images that are similar to a query image. Oracle8i Visual Information Retrieval provides this facility based on technology licensed from Virage, Inc. This product is built on top of Oracle8i interMedia which enables storage, retrieval and management of images, audios and videos. Images are matched using attributes such as color, texture and structure and efficient content-based retrieval is provided using indexes of an image index type. The design of the index type is based on a multi-level filtering algorithm. The filters reduce the search space so that the expensive comparison algorithm operates on a small subset of the data. Bitmap indexes are used to evaluate the first filter resulting in a design which performs well and is scalable. The image index type is built using Oracle8i extensible indexing technology, allowing users to create, use, and drop instances of this index type as they would any other standard index. In this paper we present an overview of the product, the design of the image index type, and some performance results of our product.

Database extensions for complex domains

Future versions of the Oracle Server will provide an open and extensible framework for supporting complex data domains including, but not limited to, text, image, spatial, video, and OLAP. This framework encompasses features for defining, storing, updating, indexing, and retrieving complex forms of data with full transaction semantics. The underpinning for these features is an extended Oracle Server that is an object-relational database management system (ORDBMS).

The importance of extensible database systems for e-commerce

Over the last decade, database system products have been extended to provide support for defining, storing, updating, indexing and retrieving complex data with full transaction semantics. Oracle, IBM, Informix and others have used extensibility technology to build database system extensions for text, image, spatial, audio/video, chemical, genetic and other types of complex data. Currently, we find database systems being deployed in support of e-commerce. In many cases, these e-commerce database applications use only simple SQL data types to represent items such as office supplies, computers, books and CDs. There is also a large and important set of e-commerce applications that employ complex data formats such as EDI, SWIFT and HL7. The database extensibility features initially developed to support text, spatial and similar forms of complex data are now being used to build e-commerce applications. Thus, database extensibility technology is evolving into an important mechanism to enable the development of e-commerce systems.

Database extensions for complex forms of data

To adequately support text, image, spatial, message and other complex forms of data, modern database management systems must provide an extensive set of data integration features. The emerging object-relational database systems provide features for defining, storing, updating, indexing, and retrieving complex data types with full transaction semantics. The author describes these features in the context of Oracle8i database technology with examples from existing e-commerce and emerging XML data.