MT-Clinical BERT: Scaling Clinical Information Extraction with Multitask Learning

Abstract

OBJECTIVE\nClinical notes contain an abundance of important, but not-readily accessible, information about patients. Systems that automatically extract this information rely on large amounts of training data of which there exists limited resources to create. Furthermore, they are developed disjointly, meaning that no information can be shared among task-specific systems. This bottleneck unnecessarily complicates practical application, reduces the performance capabilities of each individual solution, and associates the engineering debt of managing multiple information extraction systems.\n\n\nMATERIALS AND METHODS\nWe address these challenges by developing Multitask-Clinical BERT: a single deep learning model that simultaneously performs 8 clinical tasks spanning entity extraction, personal health information identification, language entailment, and similarity by sharing representations among tasks.\n\n\nRESULTS\nWe compare the performance of our multitasking information extraction system to state-of-the-art BERT sequential fine-tuning baselines. We observe a slight but consistent performance degradation in MT-Clinical BERT relative to sequential fine-tuning.\n\n\nDISCUSSION\nThese results intuitively suggest that learning a general clinical text representation capable of supporting multiple tasks has the downside of losing the ability to exploit dataset or clinical note-specific properties when compared to a single, task-specific model.\n\n\nCONCLUSIONS\nWe find our single system performs competitively with all state-the-art task-specific systems while also benefiting from massive computational benefits at inference.