Kazuko Kuriyama

Pooling for a Large-Scale Test Collection: An Analysis of the Search Results from the First NTCIR Workshop

We have conducted a study to: (1) verify the exhaustiveness of pooling for the purpose of constructing a large-scale test collection, and (2) examine whether a difference in the number of pool documents can affect the relative evaluation of IR systems. We carried out the experiments using search topics, their relevance assessments, and the search results that were submitted for both the pre-test and test of the first NTCIR Workshop.Our results verified the efficiency and the effectiveness of the pooling method, the exhaustiveness of the relevance assessments, and the reliability of the evaluation using the test collection based on the pooling method.

NACSIS test collection workshop (NTCIR-1) (poster abstract)

The test collection used in the Workshop consists of more than 330,000 documents and more than half are English-Japanese paired. Although there is a Japanese test collection called BMIR52 consisting of 5,080 newspaper articles[2], enhancement of the Japanese test collection in the both aspects of the variety of text types and the scale is needed. We put emphasis on cross-lingual retrieval since it is critical in the intemet environment and Japanese scientific information retrieval [3].

An evaluation of the Web retrieval task at the third NTCIR workshop

We have investigated the evaluation methods for measuring retrieval effectiveness of Web search engine systems, attempting to make them suitable for real Web environment. With this objective, we conducted ‘Web Retrieval Task’ at the Third NTCIR Workshop (‘NTCIR-3 WEB’) from 2001 to 2002 [1, 2, 3]. Using this NTCIR-3 WEB, we built a re-usable test collection that is suitable for evaluating Web search engine systems, and evaluated the retrieval effectiveness of a certain number of Web search engine systems. TREC Web Tracks [4] are well-known workshops that have an objective to research the retrieval of large-scale Web document data. Past TREC Web Tracks have used data sets extracted from ‘the Internet Archive’or pages gathered from the ‘.gov’ domain as document sets. They assessed the relevance only on information given in English. NTCIR-3 WEB was another workshop that has used 100-gigabyte and/or 10gigabyte document data that were mainly gathered from the ‘.jp’ domain. Relevance judgment was performed on the retrieved documents that are written in Japanese or English, partially considering hyperlinks. By considering the hyperlinks, a ‘hub page’ that gives out-links to multiple ‘authority pages’ [5] may be judged as relevant even if these do not include sufficient relevant information in them. 16 groups enrolled to participate in the NTCIR-3 WEB, and seven of these groups submitted run results.