Paul E. Johnson

The impact of explanation facilities on user acceptance of expert systems advice

Providing explanations for recommended actions is deemed one of the most important capabilities of expert systems (ES). There is little empirical evidence, however, that explanation facilities indeed influence user confidence in, and acceptance of, ES-based decisions and recommendations. This paper investigates the impact of ES explanations on changes in user beliefs toward ES-generated conclusions. Grounded on a theoretical model of argument, three alternative types of ES explanations A trace, justification, and strategy A were provided in a simulated diagnostic expert system performing auditing tasks. Twenty practicing auditors evaluated the outputs of the system in a laboratory setting. The results indicate that explanation facilities can make ES-generated advice more acceptable to users and that justification is the most effective type of explanation to bring about changes in user attitudes toward the system. These findings are expected to be generalizable to application domains that exhibit similar characteristics to those of auditing: domains in which decision making tends to be judgmental and yet highly consequential, and the correctness or validity of such decisions cannot be readily verified.

Impact of Electronic Health Record Clinical Decision Support on Diabetes Care: A Randomized Trial

PURPOSE We wanted to assess the impact of an electronic health record–based diabetes clinical decision support system on control of hemoglobin A1c (glycated hemoglobin), blood pressure, and low-density lipoprotein (LDL) cholesterol levels in adults with diabetes. METHODS We conducted a clinic-randomized trial conducted from October 2006 to May 2007 in Minnesota. Included were 11 clinics with 41 consenting primary care physicians and the physicians’ 2,556 patients with diabetes. Patients were randomized either to receive or not to receive an electronic health record (EHR)–based clinical decision support system designed to improve care for those patients whose hemoglobin A1c, blood pressure, or LDL cholesterol levels were higher than goal at any office visit. Analysis used general and generalized linear mixed models with repeated time measurements to accommodate the nested data structure. RESULTS The intervention group physicians used the EHR-based decision support system at 62.6% of all office visits made by adults with diabetes. The intervention group diabetes patients had significantly better hemoglobin A1c (intervention effect −0.26%; 95% confidence interval, −0.06% to −0.47%; P=.01), and better maintenance of systolic blood pressure control (80.2% vs 75.1%, P=.03) and borderline better maintenance of diastolic blood pressure control (85.6% vs 81.7%, P =.07), but not improved low-density lipoprotein cholesterol levels (P = .62) than patients of physicians randomized to the control arm of the study. Among intervention group physicians, 94% were satisfied or very satisfied with the intervention, and moderate use of the support system persisted for more than 1 year after feedback and incentives to encourage its use were discontinued. CONCLUSIONS EHR-based diabetes clinical decision support significantly improved glucose control and some aspects of blood pressure control in adults with type 2 diabetes.

Effects of Framing on Auditor Decisions

Framing effects occur when an agent (e.g., a manager) constructs a description of some entity (e.g., a company) such that the way information is stated (framed) influences the decisions made by other agents (e.g., auditors, analysts, and investors). Auditors, in particular, are charged by society with evaluating and reporting on the fairness of the descriptions of a company (financial statements and related notes) constructed by its management. The financial statements, notes, and the auditors report provide the financial markets with information on which to base investment and other business decisions. Although auditors have substantial incentives for detecting framing effects that “cover up” misleading financial information, detection is not always achieved. This project was designed as a field (case) study to investigate the cognitive representations used by expert and novice auditors in performing a simulated audit task to evaluate financial data from two actual audit cases in which framing effects were present and were not detected by auditors. The first case contained a deliberately created framing effect (management fraud); the second case contained a naturally occurring framing effect (financial statement error). Thinking-aloud comments given by three expert and three novice auditors were analyzed to determine the representations used in performing the simulated audit task of concurring partner review. In the fraud case, management had manipulated income so that all subjects initially generated a “growth company” representation that was incorrect. Subjects who interpreted cues configurally (i.e., as patterns) were able to construct an alternative representation (a company in decline) and detect the fraud. One novice and one expert who had experience in the industry represented by the fraud case did this. Expert and novice auditors without such experience did not. In the error case in which receivables were grossly overstated, none of the subjects had relevant industry experience. Experts generated a “collections” representation that focused attention on the material risk associated with the problem of collection of accounts receivable. These subjects detected the financial statement error. Novices generated a “collateral” representation that failed to detect the error in accounts receivable.

Detecting deception: adversarial problem solving in a low base-rate world

The work presented here investigates the process by which one group of individuals solves the problem of detecting deceptions created by other agents. A field experiment was conducted in which twenty-four auditors (partners in international public accounting firms) were asked to review four cases describing real companies that, unknown to the auditors, had perpetrated financial frauds. While many of the auditors failed to detect the manipulations in the cases, a small number of auditors were consistently successful. Since the detection of frauds occurs infrequently in the work of a given auditor, we explain success by the application of powerful heuristics gained from experience with deceptions in everyday life. These heuristics implement a variation of Dennett’s intentional stance strategy, which is based on interpreting detected inconsistencies in the light of the Deceiver’s (i.e., management’s) goals and possible actions. We explain failure to detect deception by means of perturbations (bugs) in the domain knowledge of accounting needed to apply these heuristics to the specific context of financial statement fraud. We test our theory by showing that a computational model of fraud detection that employs the proposed heuristics successfully detects frauds in the cases given to the auditors. We then modify the model by introducing perturbations based on the errors made by each of the auditors in the four cases. The resulting models account for 84 of the 96 observations (i.e., 24 auditors x four cases) in our data.

Fraud detection: Intentionality and deception in cognition

Fraud detection is made difficult in part due to the fact that most auditors have relatively little experience with it. We address the issue of what kind of knowledge supports success in financial statement fraud detection by examining the more general information processing problem of detecting a deception. We define deception as a process in which a deceiver (e.g. management) has intentionally manipulated an environment (a financial statement) so as to elicit a misleading representation in a target agent (e.g. an auditor). We develop a theory of the knowledge that the deceiver and the target use for respectively constructing and detecting deceptions. Drawing on the literature in several fields (e.g. cognitive ethology, military strategy, child development) we identify specific strategies and tactics for creating a deception. We then hypothesize that reasoning about a deceivers goals is one of the main strategies for detecting deception. We use the strategies and tactics for creating a deception to propose what the knowledge that would lead to the detection of financial statement fraud must be like based on a proposed hierarchy of the managers (deceivers) goals. We compare the proposed detection knowledge with the knowledge base of a computer (expert system) model of financial statement fraud detection task that was successful in solving several real fraud cases (and was built independently from the proposed theory). We also compare properties of the detection knowledge proposed in our theory with the knowledge employed by several experienced auditors who performed the task of concurring partner review on one of the fraud cases successfully analyzed by the model.

Success and Failure in Expert Reasoning

Discusses causes of error in real-world problem-solving tasks by considering errors as failures of reasoning and by focusing on human experts as a source of insight into the basis for reasoning errors. A theory of expertise for diagnostic tasks based on the hypothetico-deductive method of reasoning and 2 principles of expertise (called coverage and composition) is described. The task of fraud detection in financial statement auditing is used in computer simulations and in comparison with a sample of 24 auditors who were given the financial statements of a subset of the companies. Results indicate that when neither of the 2 principles is violated, the performance of the resulting model is comparable with that of human auditors who are able to detect fraud. (PsycINFO Database Record (c) 2012 APA, all rights reserved)

Understanding variation in chronic disease outcomes.

We propose an explanation for variation in disease outcomes based on patient adaptation to the conditions of chronic disease. We develop a model of patient adaptation using the example of Type 2 diabetes mellitus and assumptions about the process entailed in transforming self-care behaviors of compliance with treatment, compliance with glucose monitoring, and patients knowledge seeking behavior into health outcomes of glycemic control and patient satisfaction. Using data from 609 adults with diagnosed Type 2 diabetes we develop an efficiency (fitness) frontier in order to identify best practice (maximally adapted) patients and forms (archetypes) of patient inefficiency. Outcomes of frontier patients are partitioned by categories of returns to scale. Outcomes for off-frontier patients are associated with disease severity and patient archetype. The model implicates strategies for improved health outcomes based on fitness and self-care behaviors.

Audit judgment research

This paper identifies themes in the audit judgment literature and suggests implications of the research that supports them. Research on outcome behaviors and investigations of the process of audit decision making are stressed. Issues in both methodology and theory are considered. A brief assessment of the current state of knowledge in the field of audit judgment research is provided, and an argument is offered for a direction of future work based on the idea of context and the need to understand the meaning that tasks have for the subjects who perform them. The paper concludes with a brief discussion of data from studies currently in progress at the University of Minnesota.

A Framework for the Development of Measurement Instruments for Evaluating the Introductory Statistics Course

Abstract This paper proposes a framework for the development of instruments to measure content learning and problem-solving skills for the introductory statistics course. This framework is based upon a model of the problem-solving process central to statistical reasoning. The framework defines and interrelates six measurement tasks: (1) subjective reports; (2) reports concerning truth, falsity, or equivalence; (3) supply the appropriate missing information in a message; (4) answer a question based upon a specific message; (S) reproduce a message; and (6) carry out a procedure.

Simulated physician learning intervention to improve safety and quality of diabetes care: a randomized trial.

OBJECTIVE To assess two physician learning interventions designed to improve safety and quality of diabetes care delivered by primary care physicians (PCPs). RESEARCH DESIGN AND METHODS This group randomized clinical trial included 57 consenting PCPs and their 2,020 eligible adult patients with diabetes. Physicians were randomized to no intervention (group A), a simulated case-based physician learning intervention (group B), or the same simulated case-based learning intervention with physician opinion leader feedback (group C). Dependent variables included A1C values, LDL cholesterol values, pharmacotherapy intensification rates in patients not at clinical goals, and risky prescribing events. RESULTS Groups B and C had substantial reductions in risky prescribing of metformin in patients with renal impairment (P = 0.03). Compared with groups A and C, physicians in group B achieved slightly better glycemic control (P = 0.04), but physician intensification of oral glucose-lowering medications was not affected by interventions (P = 0.41). Lipid management improved over time (P < 0.001) but did not differ across study groups (P = 0.67). CONCLUSIONS A simulated, case-based learning intervention for physicians significantly reduced risky prescribing events and marginally improved glycemic control in actual patients. The addition of opinion leader feedback did not improve the learning intervention. Refinement and further development of this approach is warranted.