Roma Robertson

Lessons from a failure: generating tailored smoking cessation letters

STOP is a Natural Language Generation (NLG) system that generates short tailored smoking cessation letters, based on responses to a four-page smoking questionnaire. A clinical trial with 2553 smokers showed that STOP was not effective; that is, recipients of a non-tailored letter were as likely to stop smoking as recipients of a tailored letter. In this paper we describe the STOP system and clinical trial. Although it is rare for AI papers to present negative results, we believe that useful lessons can be learned from STOP. We also believe that the AI community as a whole could benefit from considering the issue of how, when, and why negative results should be reported; certainly a major difference between AI and more established fields such as medicine is that very few AI papers report negative results.

Cost effectiveness of computer tailored and non-tailored smoking cessation letters in general practice: randomised controlled trial

Abstract Objectives: To develop and evaluate, in a primary care setting, a computerised system for generating tailored letters about smoking cessation. Design: Randomised controlled trial. Setting: Six general practices in Aberdeen, Scotland. Participants: 2553 smokers aged 17 to 65. Interventions: All participants received a questionnaire asking about their smoking. Participants subsequently received either a computer tailored or a non-tailored, standard letter on smoking cessation, or no letter. Main outcome measures: Prevalence of validated abstinence at six months; change in intention to stop smoking in the next six months. Results: The validated cessation rate at six months was 3.5% (30/857) (95% confidence interval 2.3% to 4.7%) for the tailored letter group, 4.4% (37/846) (3.0% to 5.8%) for the non-tailored letter group, and 2.6% (22/850) (1.5% to 3.7%) for the control (no letter) group. After adjustment for significant covariates, the cessation rate was 66% greater (−4% to 186%; P=0.07) in the non-tailored letter group than that in the no letter group. Among participants who smoked <20 cigarettes per day, the cessation rate in the non-tailored letter group was 87% greater (0% to 246%; P=0.05) than that in the no letter group. Among heavy smokers who did not quit, a 76% higher rate of positive shift in “stage of change” (intention to quit within a particular period of time) was seen compared with those who received no letter (11% to 180%; P=0.02). The increase in cost for each additional quitter in the non-tailored letter group compared with the no letter group was £89. Conclusions: In a large general practice, a brief non-tailored letter effectively increased cessation rates among smokers. A tailored letter was not effective in increasing cessation rates but promoted shift in movement towards cessation (“stage of change”) in heavy smokers. As a pragmatic tool to encourage cessation of smoking, a mass mailing of non-tailored letters from general practices is more cost effective than computer tailored letters or no letters. What is already known on this topic Brief opportunistic advice on stopping smoking that is given face to face by health professionals increases rates of cessation by 2-3% Intensive, expert-led interventions increase cessation rates by up to 20% or more but are expensive and reach only a small proportion of smokers Written advice tailored to an individuals “stage of change” (intention to stop in a particular period of time) has been claimed to be as effective as intensive interventions, but previous studies of tailored written advice did not biochemically validate cessation What this paper adds A simple standard letter sent to patients of general practices that gave brief advice on stopping smoking increased the biochemically validated rate of cessation by 2% A letter tailored to the individuals “stage of change” was not more effective than the non-tailored standard letter Although the increase in cessation resulting from the non-tailored standard letter was small, this intervention was highly cost effective

Acquiring correct knowledge for natural language generation

Natural language generation (NLG) systems are computer software systems that produce texts in English and other human languages, often from non-linguistic input data. NLG systems, like most ai systems, need substantial amounts of knowledge. However, our experience in two NLG projects suggests that it is difficult to acquire correct knowledge for NLG systems; indeed, every knowledge acquisition (KA) technique we tried had significant problems. In general terms, these problems were due to the complexity, novelty, and poorly understood nature of the tasks our systems attempted, and were worsened by the fact that people write so differently. This meant in particular that corpus-based KA approaches suffered because it was impossible to assemble a sizable corpus of high-quality consistent manually written texts in our domains; and structured expert-oriented KA techniques suffered because experts disagreed and because we could not get enough information about special and unusual cases to build robust systems. We believe that such problems are likely to affect many other NLG systems as well. In the long term, we hope that new KA techniques may emerge to help NLG system builders. In the shorter term, we believe that understanding how individual KA techniques can fail, and using a mixture of different KA techniques with different strengths and weaknesses, can help developers acquire NLG knowledge that is mostly correct.

Knowledge Acquisition for Natural Language Generation

We describe the knowledge acquisition (KA) techniques used to build the STOP system, especially sorting and think-aloud protocols. That is, we describe the ways in which we interacted with domain experts to determine appropriate user categories, schemas, detailed content rules, and so forth for STOP. Informal evaluations of these techniques suggest that they had some benefit, but perhaps were most successful as a source of insight and hypotheses, and should ideally have been supplemented by other techniques when deciding on the specific rules and knowledge incorporated into STOP.

Types of Knowledge Required to Personalise Smoking Cessation Letters

The STOP system generates personalised smoking-cessation letters, using as input responses to a smoking questionnaire. Generating personalised patient-information material is an area of growing interest to the medical community, since for many people changing health-related behaviour is the most effective possible medical intervention. While previous AI systems that generated personalised patient-information material were primarily based on medical knowledge, stop is largely based on knowledge of psychology, empathy, and readability. We believe such knowledge is essential in systems whose goal is to change peoples behaviour or mental state; but there are many open questions about how this knowledge should be acquired, represented, and reasoned with.

Using a Randomised Controlled Clinical Trial to Evaluate an NLG System

The STOP system, which generates personalised smoking-cessation letters, was evaluated by a randomised controlled clinical trial. We believe this is the largest and perhaps most rigorous task effectiveness evaluation ever performed on an NLG system. The detailed results of the clinical trial have been presented elsewhere, in the medical literature. In this paper we discuss the clinical trial itself: its structure and cost, what we did and did not learn from it (especially considering that the trial showed that STOP was not effective), and how it compares to other NLG evaluation techniques.

Using discrete choice experiments to evaluate alternative electronic prescribing systems

Objective — To assess the relative importance to pharmacists and general practitioners (GPs) of different characteristics of electronic prescribing systems.