Liyuan Zhou

Big Data Small Data, In Domain Out-of Domain, Known Word Unknown Word: The Impact of Word Representations on Sequence Labelling Tasks

Word embeddings -- distributed word representations that can be learned from unlabelled data -- have been shown to have high utility in many natural language processing applications. In this paper, we perform an extrinsic evaluation of five popular word embedding methods in the context of four sequence labelling tasks: POS-tagging, syntactic chunking, NER and MWE identification. A particular focus of the paper is analysing the effects of task-based updating of word representations. We show that when using word embeddings as features, as few as several hundred training instances are sufficient to achieve competitive results, and that word embeddings lead to improvements over OOV words and out of domain. Perhaps more surprisingly, our results indicate there is little difference between the different word embedding methods, and that simple Brown clusters are often competitive with word embeddings across all tasks we consider.

Upgrade of the MIT Linear Electrostatic Ion Accelerator (LEIA) for nuclear diagnostics development for Omega, Z and the NIF

The MIT Linear Electrostatic Ion Accelerator (LEIA) generates DD and D(3)He fusion products for the development of nuclear diagnostics for Omega, Z, and the National Ignition Facility (NIF). Significant improvements to the system in recent years are presented. Fusion reaction rates, as high as 10(7) s(-1) and 10(6) s(-1) for DD and D(3)He, respectively, are now well regulated with a new ion source and electronic gas control system. Charged fusion products are more accurately characterized, which allows for better calibration of existing nuclear diagnostics. In addition, in situ measurements of the on-target beam profile, made with a CCD camera, are used to determine the metrology of the fusion-product source for particle-counting applications. Finally, neutron diagnostics development has been facilitated by detailed Monte Carlo N-Particle Transport (MCNP) modeling of neutrons in the accelerator target chamber, which is used to correct for scattering within the system. These recent improvements have resulted in a versatile platform, which continues to support the existing nuclear diagnostics while simultaneously facilitating the development of new diagnostics in aid of the National Ignition Campaign at the National Ignition Facility.

Benchmarking Clinical Speech Recognition and Information Extraction: New Data, Methods, and Evaluations

Background Over a tenth of preventable adverse events in health care are caused by failures in information flow. These failures are tangible in clinical handover; regardless of good verbal handover, from two-thirds to all of this information is lost after 3-5 shifts if notes are taken by hand, or not at all. Speech recognition and information extraction provide a way to fill out a handover form for clinical proofing and sign-off. Objective The objective of the study was to provide a recorded spoken handover, annotated verbatim transcriptions, and evaluations to support research in spoken and written natural language processing for filling out a clinical handover form. This dataset is based on synthetic patient profiles, thereby avoiding ethical and legal restrictions, while maintaining efficacy for research in speech-to-text conversion and information extraction, based on realistic clinical scenarios. We also introduce a Web app to demonstrate the system design and workflow. Methods We experiment with Dragon Medical 11.0 for speech recognition and CRF++ for information extraction. To compute features for information extraction, we also apply CoreNLP, MetaMap, and Ontoserver. Our evaluation uses cross-validation techniques to measure processing correctness. Results The data provided were a simulation of nursing handover, as recorded using a mobile device, built from simulated patient records and handover scripts, spoken by an Australian registered nurse. Speech recognition recognized 5276 of 7277 words in our 100 test documents correctly. We considered 50 mutually exclusive categories in information extraction and achieved the F1 (ie, the harmonic mean of Precision and Recall) of 0.86 in the category for irrelevant text and the macro-averaged F1 of 0.70 over the remaining 35 nonempty categories of the form in our 101 test documents. Conclusions The significance of this study hinges on opening our data, together with the related performance benchmarks and some processing software, to the research and development community for studying clinical documentation and language-processing. The data are used in the CLEFeHealth 2015 evaluation laboratory for a shared task on speech recognition.

Evaluation Data and Benchmarks for Cascaded Speech Recognition and Entity Extraction

During clinical handover, clinicians exchange information about the patients and the state of clinical management. To improve care safety and quality, both handover and its documentation have been standardized. Speech recognition and entity extraction provide a way to help health service providers to follow these standards by implementing the handover process as a structured form, whose headings guide the handover narrative, and the documentation process as proofing and sign-off of the automatically filled-out form. In this paper, we evaluate such systems. The form considers the sections of Handover nurse, Patient introduction, My shift, Medication, Appointments, and Future care, divided in 49 mutually exclusive headings to fill out with speech recognized and extracted entities. Our system correctly recognizes 10,244 out of 14,095 spoken words and regardless of 6,692 erroneous words, its error percentage is significantly smaller than for systems submitted to the CLEF eHealth Evaluation Lab 2015. In the extraction of 35 entities with training data (i.e., 14 headings were not present in the 101 expert-annotated training documents with 8,487 words in total), the system correctly extracts 2,375 out of 3,793 words in 50 test documents after calibration on 3,937 words in 50 validation documents. This translates to over 90% F1 in extracting information for the patients age, current bed, current room, and given name and over 70% F1 for patients admission reason/diagnosis and last name. F1 for filtering out irrelevant information is 78%. We have made the data publicly available for 201 handover cases together with processing results and code and proposed the extraction task for CLEF eHealth 2016.

Information extraction to improve standard compliance: The case of clinical handover

Clinical handover refers to healthcare workers transferring responsibility and accountability for patient care, e.g., between shifts or wards. Safety and quality health standards call for this process to be systematically structured across the organisation and synchronous with its documentation. This paper evaluates information extraction as a way to help comply with these standards. It implements the handover process of first specifying a structured handover form, whose hierarchy of headings guides the handover narrative, followed by the technology filling it out objectively and almost instantly for proofing and sign-off. We trained a conditional random field with 8 feature types on 101 expert-annotated documents to 36-class classify. This resulted in good generalisation to an independent set of 50 validation and 50 test documents that we now release: 77.9 % F1 in filtering out irrelevant information, up to 98.4 % F1 for the 35 classes for relevant information, and 52.9 % F1 after macro-averaging over these 35 classes, whilst these percentages were 86.2, 100.0, and 70.2 for the leave-one-document-out cross-validation across the first set of 101 documents. Also as a result of this study, the validation and test data were released to support further research.

Information Extraction to Improve Standard Compliance

Clinical handover refers to healthcare workers transferring responsibility and accountability for patient care, e.g., between shifts or wards. Safety and quality health standards call for this process to be systematically structured across the organisation and synchronous with its documentation. This paper evaluates information extraction as a way to help comply with these standards. It implements the handover process of first specifying a structured handover form, whose hierarchy of headings guides the handover narrative, followed by the technology filling it out objectively and almost instantly for proofing and sign-off. We trained a conditional random field with 8 feature types on 101 expert-annotated documents to 36-class classify. This resulted in good generalisation to an independent set of 50 validation and 50 test documents that we now release: 77.9 % F1 in filtering out irrelevant information, up to 98.4 % F1 for the 35 classes for relevant information, and 52.9 % F1 after macro-averaging over these 35 classes, whilst these percentages were 86.2, 100.0, and 70.2 for the leave-one-document-out cross-validation across the first set of 101 documents. Also as a result of this study, the validation and test data were released to support further research.

Information Extraction to Improve Standard Compliance The Case of Clinical Handover

Clinical handover refers to healthcare workers transferring responsibility and accountability for patient care, e.g., between shifts or wards. Safety and quality health standards call for this process to be systematically structured across the organisation and synchronous with its documentation. This paper evaluates information extraction as a way to help comply with these standards. It implements the handover process of first specifying a structured handover form, whose hierarchy of headings guides the handover narrative, followed by the technology filling it out objectively and almost instantly for proofing and sign-off. We trained a conditional random field with 8 feature types on 101 expert-annotated documents to 36-class classify. This resulted in good generalisation to an independent set of 50 validation and 50 test documents that we now release: 77.9 % F1 in filtering out irrelevant information, up to 98.4 % F1 for the 35 classes for relevant information, and 52.9 % F1 after macro-averaging over these 35 classes, whilst these percentages were 86.2, 100.0, and 70.2 for the leave-one-document-out cross-validation across the first set of 101 documents. Also as a result of this study, the validation and test data were released to support further research.