Chuan-Jie Lin

Word segmentation refinement by Wikipedia for textual entailment

Textual entailment in Chinese differs from the way handling English because of the lack of word delimiters and capitalization. Information from word segmentation and Wikipedia often plays an important role in textual entailment recognition. However, the inconsistency of boundaries of word segmentation and matched Wikipedia titles should be resolved first. This paper proposed 4 ways to incorporate Wikipedia title matching and word segmentation, experimented in several feature combinations. The best system redoes word segmentation after matching Wikipedia titles. The best feature combination for BC task uses content words and Wikipedia titles only, which achieves a macro-average F-measure of 67.33% and an accuracy of 68.9%. The best MC RITE system also achieves a macro-average F-measure of 46.11% and an accuracy of 58.34%. They beat all the runs in NTCIR-10 RITE-2 CT tasks.

Rank correlation analysis of NTCIR-10 RITE-2 Chinese datasets and evaluation metrics

Textual Entailment (TE) is the task of recognizing entailment, paraphrase, and contradiction relations between a given text pair. The goal of textual entailment research is to develop a core inference component that can be applied to various domains such as QA. We observed several rank correlations on the data and system results in the NTCIR-10 RITE-2 task, trying to find out correlations between datasets and evaluation metrics. We also constructed RITE4QA datasets in the RITE-2 task under the scenario of QA in order to see the applicability of RITE systems in QA. We find that datasets created from different sources and different ways can hardly predict each other. However, the system ranking on the dataset consisting of expert-made artificial pairs has moderate correlation with the ranking on QA metrics. Both RITE metrics and QA metrics are stable in terms of their own subtasks.

Evaluating and enhancing cross-domain rank predictability of textual entailment datasets

Textual Entailment (TE) is the task of recognizing entailment, paraphrase, and contradiction relations between a given text pair. The goal of textual entailment research is to develop a core inference component that can be applied to various domains, such as IR or NLP. Since the domain that a TE system applies to may be different from its source domain, it is crucial to develop proper datasets for measuring the cross-domain ability of a TE system. We propose using Kendalls tau to measure a datasets cross-domain rank predictability. Our analysis shows that incorporating “artificial pairs” into a dataset helps enhance its rank predictability. We also find that the completeness of guidelines has no obvious effect on the rank predictability of a dataset. To validate these findings, more investigation is needed; however these findings suggest some new directions for the creation of TE datasets in the future.