Elizabeth D. Liddy

The discourse-level structure of empirical abstracts: an exploratory study

Abstract Free-text retrieval is less effective than it might be because of its dependence on notions that evolved with controlled vocabulary representation and searching. The structure and nature of the discourse level features of natural language text types are not incorporated. In an attempt to address this problem, an exploratory study was conducted for the purpose of determining whether information abstracts reporting on empirical work do possess a predictable discourse-level structure and whether there are lexical clues that reveal this structure. A three phase study was conducted, with Phase I making use of four tasks to delineate the structure of empirical abstracts based on the internalized notions of 12 expert abstractors. Phase II consisted of a linguistic analysis of 276 empirical abstracts that suggested a linguistic model of an empirical abstract, which was tested in Phase III with a two stage validation procedure using 68 abstracts and four abstractors. Results indicate that expert abstractors do possess an internalized structure of empirical abstracts, whose components and relations were confirmed repeatedly over the four tasks. Substantively the same structure revealed by the experts was manifested in the sample of abstracts, with a relatively small set of recurring lexical clues revealing the presence and nature of the text components. Abstractors validated the linguistic model at an average level of 86%. Results strongly support the presence of a detectable structure in the text-type of empirical abstracts. Such a structure may be of use in a variety of text-based information processing systems. The techniques developed for analyzing natural language texts for the purpose of providing more useful representations of their semantic content offer potential for application of other types of natural language texts.

The effects of expertise and feedback on search term selection and subsequent learning

Query formation and expansion is an integral part of nearly every effort to search for information. In the work reported here we investigate the effects of domain knowledge and feedback on search term selection and reformation. We explore differences between experts and novices as they generate search terms over 10 successive trials and under two feedback conditions. Search attempts were coded on quantitative dimensions such as the number of unique terms and average time per trial, and as a whole in an attempt to characterize the users conceptual map for the topic under differing conditions of participant-defined domain expertise. Nine distinct strategies were identified. Differences emerged as a function of both expertise and feedback. In addition, strategic behavior varied depending on prior search conditions. The results are considered from both a theoretical and design perspective, and have direct implications for digital library usability and metadata generation, and query expansion systems.

Certainty Identification in Texts: Categorization Model and Manual Tagging Results

This chapter presents a theoretical framework and preliminary results for manual categorization of explicit certainty information in 32 English newspaper articles. Our contribution is in a proposed categorization model and analytical framework for certainty identification. Certainty is presented as a type of subjective information available in texts. Statements with explicit certainty markers were identified and categorized according to four hypothesized dimensions — level, perspective, focus, and time of certainty. The preliminary results reveal an overall promising picture of the presence of certainty information in texts, and establish its susceptibility to manual identification within the proposed four-dimensional certainty categorization analytical framework. Our findings are that the editorial sample group had a significantly higher frequency of markers per sentence than did the sample group of the news stories. For editorials, high level of certainty, writer’s point of view, and future and present time were the most populated categories. For news stories, the most common categories were high and moderate levels, directly involved third party’s point of view, and past time. These patterns have positive practical implications for automation.

Text categorization for multiple users based on semantic features from a machine-readable dictionary

The text categorization module described here provides a front-end filtering function for the larger DR-LINK text retrieval system [Liddy and Myaeing 1993]. The model evaluates a large incoming stream of documents to determine which documents are sufficiently similar to a profile at the broad subject level to warrant more refined representation and matching. To accomplish this task, each substantive word in a text is first categorized using a feature set based on the semantic Subject Field Codes (SFCs) assigned to individual word senses in a machine-readable dictionary. When tested on 50 user profiles and 550 megabytes of documents, results indicate that the feature set that is the basis of the text categorization module and the algorithm that establishes the boundary of categories of potentially relevant documents accomplish their tasks with a high level of performance. This means that the category of potentially relevant documents for most profiles would contain at least 80% of all documents later determined to be relevant to the profile. The number of documents in this set would be uniquely determined by the systems category-boundary predictor, and this set is likely to contain less than 5% of the incoming stream of documents.

Automatic metadata generation & evaluation

The poster reports on a project in which we are investigating methods for breaking the human metadata-generation bottleneck that plagues Digital Libraries. The research question is whether metadata elements and values can be automatically generated from the content of educational resources, and correctly assigned to mathematics and science educational materials. Natural Language Processing and Machine Learning techniques were implemented to automatically assign values of the GEMgenerate metadata element set tofor learning resources provided by the Gateway for Education (GEM), a service that offers web access to a wide range of educational materials. In a user study, education professionals evaluated the metadata assigned to learning resources by either automatic tagging or manual assignment. Results show minimal difference in the eyes of the evaluators between automatically generated metadata and manually assigned metadata.

MetaExtract: an NLP system to automatically assign metadata

We have developed MetaExtract, a system to automatically assign Dublin Core + GEM metadata using extraction techniques from our natural language processing research. MetaExtract is comprised of three distinct processes: eQuery and HTML-based extraction modules and a keyword generator module. We conducted a Web-based survey to have users evaluate each metadata elements quality. Only two of the elements, title and keyword, were shown to be significantly different, with the manual quality slightly higher. The remaining elements for which we had enough data to test were shown not to be significantly different; they are: description, grade, duration, essential resources, pedagogy-teaching method, and pedagogy-group.

Semantic analysis for monitoring insider threats

Malicious insiders’ difficult-to-detect activities pose serious threats to the intelligence community (IC) when these activities go undetected. A novel approach that integrates the results of social network analysis, role-based access monitoring, and semantic analysis of insiders’ communications as evidence for evaluation by a risk assessor is being tested on an IC simulation. A semantic analysis, by our proven Natural Language Processing (NLP) system, of the insider’s text-based communications produces conceptual representations that are clustered and compared on the expected vs. observed scope. The determined risk level produces an input to a risk analysis algorithm that is merged with outputs from the system’s social network analysis and role-based monitoring modules.

Development, implementation and testing of a discourse model for newspaper texts

Texts of a particular type evidence a discernible, predictable schema. These schemata can be delineated, and as such provide models of their respective text-types which are of use in automatically structuring texts. We have developed a Text Structurer module which recognizes text-level structure for use within a larger information retrieval system to delineate the discourse-level organization of each documents contents. This allows those document components which are more likely to contain the type of information suggested by the users query to be selected for higher weighting. We chose newspaper text as the first text type to implement. Several iterations of manually coding a randomly chosen sample of newspaper articles enabled us to develop a newspaper text model. This process suggested that our intellectual decomposing of texts relied on six types of linguistic information, which were incorporated into the Text Structurer module. Evaluation of the results of the module led to a revision of the underlying text model and of the Text Structurer itself.

A Study of Discourse Anaphora in Scientific Abstracts.

Natural language texts are used extensively in a range of information science tasks. Such use requires increased attention to discourse level linguistic phenomena which have the potential for impact on these tasks. One such device, anaphoric reference, was investigated in a frequently used text type, namely, scientific abstracts. Descriptive data on the extent of use of discourse anaphora in abstracts was gathered and rules for distinguishing anaphoric functioning of terms were compiled and tested. Results show a mean use of 3.67 functioning anaphors per abstract in a random sample of 600 abstracts from two databases. Testing of rules indicates high feasibility of future algorithmic recognition of anaphoric uses of terms.

Interpretation of proper nouns for information retrieval

Most of the unknown words in texts which degrade the performance of natural language processing systems are proper nouns. On the other hand, proper nouns are recognized as a crucial source of information for identifying a topic in a text, extracting contents from a text, or detecting relevant documents in information retrieval (Rau, 1991).