Marek J. Druzdzel

Building probabilistic networks: "Where do the numbers come from?" guest editors' introduction

Probabilistic networks are now fairly well established as practical representations of knowledge for reasoning under uncertainty, as demonstrated by an increasing number of successful applications in such domains as (medical) diagnosis and prognosis, planning, vision, information retrieval, and natural language processing. A probabilistic network (also referred to as a belief network, Bayesian network, or, somewhat imprecisely, causal network) consists of a graphical structure, encoding a domain’s variables and the qualitative relationships between them, and a quantitative part, encoding probabilities over the variables [29]. Building a probabilistic network for a domain of application involves three tasks. The first of these is to identify the variables that are of importance, along with their possible values. Once the important domain variables have been identified, the second task is to identify the relationships between the variables discerned and to express these in a graphical structure. The tasks of eliciting the variables and values of importance as well as the relationships between them from domain experts is comparable, to at least some extent, to knowledge engineering for other artificial-intelligence representations and, although it may require significant effort, is generally considered doable. The last task in building a probabilistic network is to obtain the probabilities that are required for its quantitative part. This task often appears more daunting: “Where do the numbers come from?” is a commonly asked question. The three tasks in building a probabilistic network are, in principle, performed one after the other. Building a network, however, often requires a careful trade-off between the desire for a large and rich model to obtain accurate results on the one hand, and the costs of construction and maintenance and the complexity of probabilistic inference on the other hand. In practice, therefore, building a probabilistic network is a process that iterates over these tasks until a network results that is deemed requisite. In collaboration with Finn V. Jensen and Max Henrion, we organised in 1995 a workshop devoted to the theme of obtaining the numbers, the most daunting task in building probabilistic networks [14]. The workshop was held in conjunction with the Fourteenth International Joint Conference on Artificial Intelligence (IJCAI’95) and had a programme of presentations of selected contributions and ample slots for flash communications and discussion. Scientists from such disciplines as decision analysis, statistics, and computer science attended the workshop. The interest in the workshop, both during IJCAI’95 and afterwards, prompted us to follow up on the theme. The current issue of IEEE Transactions on Data and Knowledge Engineering is the result.

The emerging science of very early detection of disease outbreaks.

A surge of development of new public health surveillance systems designed to provide more timely detection of outbreaks suggests that public health has a new requirement: extreme timeliness of detection. The authors review previous work relevant to measuring timeliness and to defining timeliness requirements. Using signal detection theory and decision theory, the authors identify strategies to improve timeliness of detection and position ongoing system development within that framework.

AIS-BN: an adaptive importance sampling algorithm for evidential reasoning in large Bayesian networks

Stochastic sampling algorithms, while an attractive alternative to exact algorithms in very large Bayesian network models, have been observed to perform poorly in evidential reasoning with extremely unlikely evidence. To address this problem, we propose an adaptive importance sampling algorithm, AIS-BN, that shows promising convergence rates even under extreme conditions and seems to outperform the existing sampling algorithms consistently. Three sources of this performance improvement are (1) two heuristics for initialization of the importance function that are based on the theoretical properties of importance sampling in finite-dimensional integrals and the structural advantages of Bayesian networks, (2) a smooth learning method for the importance function, and (3) a dynamic weighting function for combining samples from different stages of the algorithm. We tested the performance of the AIS-BN algorithm along with two state of the art general purpose sampling algorithms, likelihood weighting (Fung & Chang, 1989; Shachter & Peot, 1989) and self-importance sampling (Shachter & Peot, 1989). We used in our tests three large real Bayesian network models available to the scientific community: the CPCS network (Pradhan et al., 1994), the PATHFINDER network (Heckerman, Horvitz, & Nathwani, 1990), and the ANDES network (Conati, Gertner, VanLehn, & Druzdzel, 1997), with evidence as unlikely as 10-41. While the AIS-BN algorithm always performed better than the other two algorithms, in the majority of the test cases it achieved orders of magnitude improvement in precision of the results. Improvement in speed given a desired precision is even more dramatic, although we are unable to report numerical results here, as the other algorithms almost never achieved the precision reached even by the first few iterations of the AIS-BN algorithm.

Learning Bayesian network parameters from small data sets: application of Noisy-OR gates

Abstract Existing data sets of cases can significantly reduce the knowledge engineering effort required to parameterize Bayesian networks. Unfortunately, when a data set is small, many conditioning cases are represented by too few or no data records and they do not offer sufficient basis for learning conditional probability distributions. We propose a method that uses Noisy-OR gates to reduce the data requirements in learning conditional probabilities. We test our method on H epar II , a model for diagnosis of liver disorders, whose parameters are extracted from a real, small set of patient records. Diagnostic accuracy of the multiple-disorder model enhanced with the Noisy-OR parameters was 6.7% better than the accuracy of the plain multiple-disorder model and 14.3% better than a single-disorder diagnosis model.

Causality in Bayesian belief networks

We address the problem of causal interpretation of the graphical structure of Bayesian belief networks (BBNs). We review the concept of causality explicated in the domain of structural equations models and show that it is applicable to BBNs. In this view, which we call mechanism-based, causality is defined within models and causal asymmetries arise when mechanisms are placed in the context of a system. We lay the link between structural equations models and BBNs models and formulate the conditions under which the latter can be given causal interpretation.

Importance sampling algorithms for Bayesian networks: Principles and performance

Precision achieved by stochastic sampling algorithms for Bayesian networks typically deteriorates in the face of extremely unlikely evidence. In addressing this problem, importance sampling algorithms seem to be most successful. We discuss the principles underlying the importance sampling algorithms in Bayesian networks. After that, we describe Evidence Pre-propagation Importance Sampling (EPIS-BN), an importance sampling algorithm that computes an importance function using two techniques: loopy belief propagation [K. Murphy, Y. Weiss, M. Jordan, Loopy belief propagation for approximate inference: An empirical study, in: Proceedings of the Fifteenth Annual Conference on Uncertainty in Artificial Intelligence, UAI-99, San Francisco, CA, Morgan Kaufmann Publishers, 1999, pp. 467-475; Y. Weiss, Correctness of local probability propagation in graphical models with loops, Neural Computation 12 (1) (2000) 1-41] and @e-cutoff heuristic [J. Cheng, M.J. Druzdzel, BN-AIS: An adaptive importance sampling algorithm for evidential reasoning in large Bayesian networks, Journal of Artificial Intelligence Research 13 (2000) 155-188]. We tested the performance of EPIS-BN on three large real Bayesian networks and observed that on all three networks it outperforms AIS-BN [J. Cheng, M.J. Druzdzel, BN-AIS: An adaptive importance sampling algorithm for evidential reasoning in large Bayesian networks, Journal of Artificial Intelligence Research 13 (2000) 155-188], the current state-of-the-art algorithm, while avoiding its costly learning stage. We also compared EPIS-BN Gibbs sampling and discuss the role of the @e-cutoff heuristic in importance sampling for Bayesian networks. networks.

Intercausal reasoning with uninstantiated ancestor nodes

Intercausal reasoning is a common inference pattern involving probabilistic dependence of causes of an observed common effect. The sign of this dependence is captured by a qualitative property called product synergy. The current definition of product synergy is insufficient for intercausal reasoning where there are additional uninstantiated causes of the common effect. We propose a new definition of product synergy and prove its adequacy for intercausal reasoning with direct and indirect evidence for the common effect. The new definition is based on a new property matrix half positive semi-definiteness, a weakened form of matrix positive semi-definiteness.

Comparison of Rule-Based and Bayesian Network Approaches in Medical Diagnostic Systems

Almost two decades after the introduction of probabilistic expert systems, their theoretical status, practical use, and experiences are matching those of rule-based expert systems. Since both types of systems are in wide use, it is more than ever important to understand their advantages and drawbacks. We describe a study in which we compare rule-based systems to systems based on Bayesian networks. We present two expert systems for diagnosis of liver disorders that served as the inspiration and vehicle of our study and discuss problems related to knowledge engineering using the two approaches. We finally present the results of a simple experiment comparing the diagnostic performance of each of the systems on a subset of their domain.

The Pittsburgh Cervical Cancer Screening Model: a risk assessment tool.

CONTEXT Evaluation of cervical cancer screening has grown increasingly complex with the introduction of human papillomavirus (HPV) vaccination and newer screening technologies approved by the US Food and Drug Administration. OBJECTIVE To create a unique Pittsburgh Cervical Cancer Screening Model (PCCSM) that quantifies risk for histopathologic cervical precancer (cervical intraepithelial neoplasia [CIN] 2, CIN3, and adenocarcinoma in situ) and cervical cancer in an environment predominantly using newer screening technologies. DESIGN The PCCSM is a dynamic Bayesian network consisting of 19 variables available in the laboratory information system, including patient history data (most recent HPV vaccination data), Papanicolaou test results, high-risk HPV results, procedure data, and histopathologic results. The models graphic structure was based on the published literature. Results from 375 441 patient records from 2005 through 2008 were used to build and train the model. Additional data from 45 930 patients were used to test the model. RESULTS The PCCSM compares risk quantitatively over time for histopathologically verifiable CIN2, CIN3, adenocarcinoma in situ, and cervical cancer in screened patients for each current cytology result category and for each HPV result. For each current cytology result, HPV test results affect risk; however, the degree of cytologic abnormality remains the largest positive predictor of risk. Prior history also alters the CIN2, CIN3, adenocarcinoma in situ, and cervical cancer risk for patients with common current cytology and HPV test results. The PCCSM can also generate negative risk projections, estimating the likelihood of the absence of histopathologic CIN2, CIN3, adenocarcinoma in situ, and cervical cancer in screened patients. CONCLUSIONS The PCCSM is a dynamic Bayesian network that computes quantitative cervical disease risk estimates for patients undergoing cervical screening. Continuously updatable with current system data, the PCCSM provides a new tool to monitor cervical disease risk in the evolving postvaccination era.

Caveats for Causal Reasoning with Equilibrium Models

In this paper we examine the ability to perform causal reasoning with recursive equilibrium models. We identify a critical postulate, which we term the Manipulation Postulate, that is required in order to perform causal inference, and we prove that there exists a general class F of recursive equilibrium models that violate the Manipulation Postulate. We relate this class to the existing phenomenon of reversibility and show that all models in F display reversible behavior, thereby providing an explanation for reversibility and suggesting that it is a special case of a more general and perhaps widespread problem. We also show that all models in F possess a set of variables V′ whose manipulation will cause an instability such that no equilibrium model will exist for the system. We define the Structural Stability Principle which provides a graphical criterion for stability in causal models. Our theorems suggest that drastically incorrect inferences may be obtained when applying the Manipulation Postulate to equilibrium models, a result which has implications for current work on causal modeling, especially causal discovery from data.