Robin Boswell

Validation and verification of knowledge-based systems: report on EUROVAV99

Knowledge-Based (KB) technology is being applied to complex problem solving and safety and business critical tasks in many application domains. Concerns have naturally arisen as to the dependability of Knowledge-Based Systems (KBS). As with any software, attention to quality and safety must be paid throughout development of a KBS, and rigorous Verification and Validation (V&V) techniques must be employed. Research in V&V of KBSs has emerged as a distinct field only in the last decade, and is intended to address issues associated with quality and safety aspects of KBSs, and to provide such applications with the same degree of dependability as conventional applications. In recent years, V&V of KBSs has been the topic of annual workshops associated with the main AI conferences, such as AAAI, IJCAI and ECAI.

Knowledge Refinement for a Design System

The Krust refinement tool has already been successfully applied to a variety of relatively simple classificatory problems, and a generic refinement framework is being developed. This paper describes the application of Krust to a design system Tfs, whose task is tablet formulation for a major pharmaceutical company. It shows how novel components can be included within Krusts underlying knowledge model, and how Krusts refinement mechanisms can be extended as required, by adding new operators to the existing toolsets. New mechanisms have been added whereby proofs of related examples are used to constrain and guide Krusts refinement generation. TFs has provided valuable widening experience for attaining our eventual goal of developing a framework for a generic knowledge refinement toolkit.

Knowledge refinement to debug and maintain a tablet formulation system

Knowledge refinement tools have commonly been applied to diagnostic applications. The paper considers the refinement of a design application. It explores the differences in knowledge content and problem solving steps for design rather than diagnosis systems, and investigates additional refinement operators. Although the problem solving task itself tends to be more complex, this in fact puts more constraints on the results of the problem solving and so the evidence on which the refinement is based can be more rich. Results are provided for a real tablet formulation system, TFS, and experience is reported for two types of refinement tasks: debugging to correct faulty problem solving of an early version of TFS; maintenance of TFS when the formulation task is altered by a change in company policy.

Finding the hidden gems: recommending untagged music

We have developed a novel hybrid representation for Music Information Retrieval. Our representation is built by incorporating audio content into the tag space in a tag-track matrix, and then learning hybrid concepts using latent semantic analysis. We apply this representation to the task of music recommendation, using similarity-based retrieval from a query music track. We also develop a new approach to evaluating music recommender systems, which is based upon the relationship of users liking tracks. We are interested in measuring the recommendation quality, and the rate at which cold-start tracks are recommended. Our hybrid representation is able to outperform a tag-only representation, in terms of both recommendation quality and the rate that cold-start tracks are included as recommendations.

Organising Knowledge Refinement Operators

Knowledge refinement tools seek to correct faulty knowledge based systems (KBSs). The goal of the KrustWorks project is to provide a source of refinement components from which specialised refinement tools tailored to the needs of a range of KBSs can be built. Central to the toolkit is a set of generic refinement operators and a representation language for KBS rules. The language abstracts those properties of rule elements which determine whether they can take part in rule chaining, and how they can be refined. It is used to organise the refinement operators so that each can be applied to similar rule elements found in different KBSs. This organisation facilitates the re-use of refinement operators when new KBSs are encountered, and the development of new ones. The toolkit was evaluated by applying it to a KBS shell significantly different from the ones previously encountered. The rule elements from this shell could be represented within the existing hierarchy, but it was necessary to add some new refinement operators. A KrustTool was then able to fix bugs actually occurring in the application, which confirms the usefulness of the refinement operators, and of our approach to organising and applying them.

Knowledge modelling for a generic refinement framework

Refinement tools assist with debugging the knowledge-based system (KBS), thus easing the well-known knowledge acquisition bottleneck, and the more recently recognised maintenance overhead. The existing refinement tools were developed for specific rule-based KBS environments, and have usually been applied to artificial or academic applications. Hence, there is a need for tools which are applicable to industrial applications. However, it would be wasteful to develop separate refinement tools for individual shells; instead, the KRUSTWorks project is developing reusable components applicable to a variety of KBS environments. This paper develops a knowledge representation that embodies a KBSs rulebase and its reasoning, and permits the implementation of core refinement procedures, which are generally applicable and can ignore KBS-specific details. Such a representation is an essential stage in the construction of a generic automated knowledge refinement framework, such as KRUSTWorks. Experience from applying this approach to CLIPS, POWERMODEL and PFES KBSs indicates its feasibility for a wider variety of industrial KBSs.

Experiences with a Generic Refinement Toolkit

Knowledge refinement tools seek to correct faulty knowledge based systems (KBSs). Most current refinement systems are applicable only to a single KBS shell, and typically they ignore the procedural aspects of KBS reasoning. This paper describes the KrustWorks framework which refines a number of different shells, and can be extended to new ones. Internal knowledge structures represent rules in the target KBS and their interactions, and generic tools manipulate these structures. In this paper KrustWorks is evaluated on two aero-space applications into which various artificial faults have been introduced. KRUSTWorks identifies and fixes these faults, except when the training examples provide insufficient fault evidence. The evaluation demonstrates the effectiveness of KrustWorks as a refinement tool, and confirms that it can represent the knowledge and problem-solving in real expert systems.

Debugging knowledge-based applications with a generic toolkit

Knowledge refinement tools assist in the debugging and maintenance of knowledge based systems (KBSs) by attempting to identify and correct faults in the knowledge that account for incorrect problem-solving. Most refinement systems target a single shell and are able to refine only KBSs implemented in this shell. Our KRUSTWorks toolkit is unusual in that it provides refinement facilities that can be applied to a number of different shells, and is designed to be extensible to new shells. The paper outlines the components of the KRUSTWorks toolkit and how it is applied to faulty KBSs. It describes its application to two real aerospace KBSs implemented in CLIPS and POWER-MODEL to demonstrate its flexibility of application.