Rodney D. Nielsen

Towards comprehensive syntactic and semantic annotations of the clinical narrative

Objective To create annotated clinical narratives with layers of syntactic and semantic labels to facilitate advances in clinical natural language processing (NLP). To develop NLP algorithms and open source components. Methods Manual annotation of a clinical narrative corpus of 127 606 tokens following the Treebank schema for syntactic information, PropBank schema for predicate-argument structures, and the Unified Medical Language System (UMLS) schema for semantic information. NLP components were developed. Results The final corpus consists of 13 091 sentences containing 1772 distinct predicate lemmas. Of the 766 newly created PropBank frames, 74 are verbs. There are 28 539 named entity (NE) annotations spread over 15 UMLS semantic groups, one UMLS semantic type, and the Person semantic category. The most frequent annotations belong to the UMLS semantic groups of Procedures (15.71%), Disorders (14.74%), Concepts and Ideas (15.10%), Anatomy (12.80%), Chemicals and Drugs (7.49%), and the UMLS semantic type of Sign or Symptom (12.46%). Inter-annotator agreement results: Treebank (0.926), PropBank (0.891–0.931), NE (0.697–0.750). The part-of-speech tagger, constituency parser, dependency parser, and semantic role labeler are built from the corpus and released open source. A significant limitation uncovered by this project is the need for the NLP community to develop a widely agreed-upon schema for the annotation of clinical concepts and their relations. Conclusions This project takes a foundational step towards bringing the field of clinical NLP up to par with NLP in the general domain. The corpus creation and NLP components provide a resource for research and application development that would have been previously impossible.

Recognizing entailment in intelligent tutoring systems

This paper describes a new method for recognizing whether a students response to an automated tutors question entails that they understand the concepts being taught. We demonstrate the need for a finer-grained analysis of answers than is supported by current tutoring systems or entailment databases and describe a new representation for reference answers that addresses these issues, breaking them into detailed facets and annotating their entailment relationships to the students answer more precisely. Human annotation at this detailed level still results in substantial interannotator agreement (86.2%), with a kappa statistic of 0.728. We also present our current efforts to automatically assess student answers, which involves training machine learning classifiers on features extracted from dependency parses of the reference answer and students response and features derived from domain-independent lexical statistics. Our systems performance, as high as 75.5% accuracy within domain and 68.8% out of domain, is very encouraging and confirms the approach is feasible. Another significant contribution of this work is that it represents a significant step in the direction of providing domain-independent semantic assessment of answers. No prior work in the area of tutoring or educational assessment has attempted to build such domain-independent systems. They have virtually all required hundreds of examples of learner answers for each new question in order to train aspects of their systems or to hand-craft information extraction templates.

Linguistic Considerations in Automatic Question Generation

This paper describes an automatic question generator that uses semantic pattern recognition to create questions of varying depth and type for self-study or tutoring.

Automatic measurement of physical mobility in Get-Up-and-Go Test using kinect sensor

Get-Up-and-Go Test is commonly used for assessing the physical mobility of the elderly by physicians. This paper presents a method for automatic analysis and classification of human gait in the Get-Up-and-Go Test using a Microsoft Kinect sensor. Two types of features are automatically extracted from the human skeleton data provided by the Kinect sensor. The first type of feature is related to the human gait (e.g., number of steps, step duration, and turning duration); whereas the other one describes the anatomical configuration (e.g., knee angles, leg angle, and distance between elbows). These features characterize the degree of human physical mobility. State-of-the-art machine learning algorithms (i.e. Bag of Words and Support Vector Machines) are used to classify the severity of gaits in 12 subjects with ages ranging between 65 and 90 enrolled in a pilot study. Our experimental results show that these features can discriminate between patients who have a high risk for falling and patients with a lower fall risk.

On multi-task learning for facial action unit detection

In this paper we investigate the use of Multitask Learning (MTL) methods to model the commonalities and variations across a set of facial action units (AUs) and also learn the classifiers for detection of multiple AUs simultaneously by exploiting their inner-relations. We studied three variants of MTL algorithms, the Regularized MTL (RMTL), the Multitask Feature Learning (MTFL) and the Alternating Multi-task Structure Learning (AMTSL). We used two databases to evaluate the performance of the MTL methods; the first one is the extended Cohn-Kanade (CK+) database with posed AUs while the second is the DISFA database consisting of spontaneous AUs. Compared with the canonical Support Vector Machine (SVM) which detects AUs individually without considering their relationships, the MTL-based methods show significant improvements in the F1 reliability measurement. In particular, the RMTL algorithm consistently outperforms the other investigated MTL-based classifiers as well as several state-of-the-art methods on the CK+ database while for spontaneous AUs on the DISFA database the MTFL approach achieves the best performance.

The joint student response analysis and recognizing textual entailment challenge: making sense of student responses in educational applications

We present the results of the joint student response analysis (SRA) and 8th recognizing textual entailment challenge. The goal of this challenge was to bring together researchers from the educational natural language processing and computational semantics communities. The goal of the SRA task is to assess student responses to questions in the science domain, focusing on correctness and completeness of the response content. Nine teams took part in the challenge, submitting a total of 18 runs using methods and features adapted from previous research on automated short answer grading, recognizing textual entailment and semantic textual similarity. We provide an extended analysis of the results focusing on the impact of evaluation metrics, application scenarios and the methods and features used by the participants. We conclude that additional research is required to be able to leverage syntactic dependency features and external semantic resources for this task, possibly due to limited coverage of scientific domains in existing resources. However, each of three approaches to using features and models adjusted to application scenarios achieved better system performance, meriting further investigation by the research community.

Extracting a Representation from Text for Semantic Analysis

We present a novel fine-grained semantic representation of text and an approach to constructing it. This representation is largely extractable by todays technologies and facilitates more detailed semantic analysis. We discuss the requirements driving the representation, suggest how it might be of value in the automated tutoring domain, and provide evidence of its validity.

Leveraging Multiple Views of Text for Automatic Question Generation

Automatic question generation can play a vital role in educational applications such as intelligent tutoring systems. Prior work in question generation relies primarily on one view of the sentence provided by a parser of a given type, such as phrase structure trees or predicate argument structure. In contrast, we explore using multiple views from different parsers to create a tree structure which represents items of interest for question generation. This approach resulted in a 17% reduction in the error rate compared with our prior work, which achieved a 44% reduction in the error rate compared to state-of-the-art question generation systems. Additionally, the work presented in this paper generates with greater question variety than our previous work, and creates 21% more semantically-oriented versus factoid questions.

Pedagogical Evaluation of Automatically Generated Questions

Automatic Question Generation from text is a critical component of educational technology applications such as Intelligent Tutoring Systems. We describe an automatic question generator that uses semantic-based templates. We evaluate the system along with two comparable systems for both linguistic quality and pedagogical value of generated questions and find that our system outperforms prior work.

Predicting changes in systolic blood pressure using longitudinal patient records

OBJECTIVE This paper introduces a model that predicts future changes in systolic blood pressure (SBP) based on structured and unstructured (text-based) information from longitudinal clinical records. METHOD For each patient, the clinical records are sorted in chronological order and SBP measurements are extracted from them. The model predicts future changes in SBP based on the preceding clinical notes. This is accomplished using least median squares regression on salient features found using a feature selection algorithm. RESULTS Using the prediction model, a correlation coefficient of 0.47 is achieved on unseen test data (p<.0001). This is in contrast to a baseline correlation coefficient of 0.39.