Sergey Sigelman

Applying the Pyramid Method in DUC 2005

In DUC 2005, the pyramid method for content evaluation was used for the first time in a crosssite evaluation. We discuss the method used in creating pyramid models and performing peer annotation. Analysis of score averages for the peers indicates that the best systems score half as well as humans, and that systems can be grouped into better and worse performers. There were few significant differences among systems. High score correlations between sets from different annotators, and good interannotator agreement, indicate that participants can perform annotation reliably. We found that a modified pyramid score gave good results and would simplify peer annotation in the future.

Columbia's newsblaster: new features and future directions

Columbias Newsblaster tracking and summarization system is a robust system that clusters news into events, categorizes events into broad topics and summarizes multiple articles on each event. Here we outline our most current work on tracking events over days, producing summaries that update a user on new information about an event, outlining the perspectives of news coming from different countries and clustering and summarizing non-English sources.

Applying the Pyramid Method in the 2006 Document Understanding Conference

The pyramid evaluation effort for the 2006 Document Understanding Conference involved twenty-two sites on twenty document sets. Each pyramid content model (one per document set) was constructed from four human summaries. Peer systems were scored using the modified pyramid score introduced in DUC 2005. ANOVAs with score as the independent variable and nine factors yielded three significant factors: document set, peer, and content responsiveness. There were many more significant differences among peer systems in 2006 than for DUC 2005. We speculate this is due to a combination of improved systems and improvements in our evaluation procedures.

Columbia at the Document Understanding Conference 2003

The Columbia Summarizer for DUC 2003, Task 2, isbased on the multi-document summarization system thatwe developed for DUC 2002 (McKeown et al., 2002). Ituses different summarization strategies depending on thetype of documents in the input set. Four different strate-gies are used, one for single events, one for multiple re-lated events, one for biographies and one for discussionof an issue with related events. The summarization strat-egy encoded in M

Personalized Search of the Medical Literature: An Evaluation

We describe a system for personalizing a set of medical journal articles (possibly created as the output of a search engine) by selecting those documents that speci cally match a patient under care. Key element in our approach is the use of targeted parts of the electronic patient record to serve as a readily available user model for the personalization task. We discuss several enhancements to a TF*IDF based approach for measuring the similarity between articles and the patient record. We also present the results of an experiment involving almost 3,000 relevance judgments by medical doctors. Our evaluation establishes that the automated system surpasses in performance alternative methods for personalizing the set of articles, including keyword-based queries manually constructed by medical experts for this purpose.