2. Planning and Management

The purpose of this chapter is to provide guidance on planning and decision making early in an expert systems project. This concept applies not only to new developments, but to improved thinking and decision making at any stage from development through implementation. This includes planning the verification, validation, and evaluation of an already developed system. The advice given here should aid in developing clear problem definition and thorough system requirements, reflecting realism from both technical and organizational viewpoints. Risk identification information is also provided.

2.1 Introduction

The development, testing and evaluation of an expert system is a demanding process. It is critical in the planning stages that the necessary resources are secured and that the proper development team is assembled. Both the successes and failures in the development of expert systems can usually be traced back to the planning phase of development. The following are important elements of a successful expert system development program:

• Management support in the institution sponsoring the development of the expert system is necessary. This support must include the commitment of both staff and financial resources needed to successfully develop and implement the system.

• The goal of the expert system and the exact uses of the end product must be clearly defined and understood by all involved from executive management to end users. Full knowledge and understanding of the costs and risks involved are also essential.

• Recognized experts in the appropriate technical fields must be available and have sufficient time committed to the expert system development project.

• Influential advocate(s) of the system are essential. Ideally, there should be advocates from both the technical development area and the end user community.

• The end users are pivotal to the development of expert systems and must be involved from the planning through the field evaluation stages. The end users provide definition of the skill level of the user community, information on how problems are addressed in practice versus the prescribed solutions, advice on how the system must function (interact with the user) to be accepted by the user community, and a cadre of supporters to test and promote the system once it is completed.

• Structured planning is recommended for the successful development of a system. This should include the problem/need to be addressed and the system benefits, organizational risk factors, technical risk factors, and user risk factors. Development milestones must be identified and the system demonstrated at each milestone.

• Knowledge elicitation from the experts is vital throughout the duration of the expert system development. It is vital both in terms of building the system and for maintaining interest and continuity throughout the project.

• The verification, validation, and evaluation must be considered in all phases of the system development. Since some aspects of the verification, validation, and evaluation may not be performed by the developers, it is critical that VV&E plans be clearly identified and documented. Maintainability must be considered in all phases of the system development. Since the maintenance will probably not be performed by the developers, the system structure must be clear and straightforward. Logical and understandable names should be used for objects and knowledge structures within the system. Clear and complete system documentation is required for effective maintenance.

• The selection of the development tool for an expert system project should be performed by a qualified knowledge engineer or expert systems developer. This is critical because there are significant differences among the development tools. These differences are not explained in available literature and the application should dictate the selection of the development tool with its specific knowledge handling and operational characteristics.

Figure 2.1 shows the initial project planning process. This process can be applied to either a new development of the VV&E for an existing (but not adequately tested) system, or an existing system.

Figure 2.1: Initial Project Planning

2.2 Identify the Need for an Expert System

Before an expert system can be developed, the need has to be established and the problem to be addressed must be clearly identified and defined. It is strongly recommended that this be done in a structured study to include the following issues (Wentworth 1989):

• The problem/need to be addressed and the system benefits.
• Organizational risk factors.
• Technical risk factors.
• User risk factors.

NOTE: The term risk factors is used in deference to the old adage "if it can go wrong, it will go wrong." The risk factors represent areas where it "will go wrong" if there is any deficiency in planning and common sense.

Once a suitable problem domain has been defined for the expert system, the next task is to narrow the scope of the development effort by clearly defining the set of problems that the system will be expected to solve. The narrower the scope, the better are the chances that the expert system can be successfully built. However, if the scope is too narrow, the application becomes trivial. Judgment must be used in establishing the scope of the system as deterministic methods are not available. In general, it is better to err on the side of too narrow a scope rather than on too broad a scope. If the scope ultimately turns out to be too narrow, it may be relatively easy to broaden the scope by adding more knowledge to the knowledge base. However, if the development tool is too limited, it will be impossible to broaden the scope of the expert system by expanding the knowledge base. This highlights the importance of selecting the proper development tool to fit the particular problem.

Prior to embarking upon an expert system development effort, the expected benefits of such an effort must be clearly defined. There are two categories of benefits that are typically cited as reasons for developing an expert system. One category consists of concrete, quantifiable reasons such as savings of time and money, utility as a training tool, etc. The other category of benefits consists of tangible but not quantifiable reasons. Specifically, the process of developing an expert system will formalize and document the knowledge in a given problem domain, or combine and formalize the expertise from many experts in a given domain. This will result in expanded knowledge and better problem solving techniques in the domain, and provide a mechanism for giving this knowledge wide distribution to the users.

Under the heading of the problem/need to be addressed and system benefits, the following should be accomplished:

• The application or the output and the use of the output must be clearly defined .
• If standardization of results is desirable, the degree to which the expert system will improve standardization must be estimated.
• The use of the expert system to improve conditions by improving quality of results must be estimated.
• The expected utility of the expert system as a training tool must be described.
• End user involvement for the duration of the development process must be assured.
• Time and money savings based on the projected use of the expert system must be estimated.

Under organizational risk factors, suggested requirements and considerations are:

• There must be a dedicated and influential advocate who wants the system to be a success.
• There must be management support for the financial support, staff and time required to build the expert system.
• Management must have realistic expectations regarding the difficulty in developing the expert system.
• Management must have realistic expectations regarding the performance of the developed system.
• The results of the expert system must be applied without excessive management approvals being required.

Once a problem domain has been identified and the initial effort at narrowing the scope of the expert system application completed, the expert(s) whose expertise will be modeled must be selected. The two main criteria that should be used to identify the expert(s) are:

1. The candidate(s) must be an expert in solving problems in the problem domain of interest and must be recognized as such by the potential user community. The need for the candidate to be an expert in the field is essential for the development of the expert system. The need for the expert to be recognized as such by the potential user community is primarily useful in selling the potential users on the viability of the given system as a useful problem solving tool for them.
2. The expert(s) must be dedicated to the successful development, testing, evaluation, and implementation of the system and be available and willing to spend the time (perhaps months) that will be required to accomplish this. The failure to identify such a person or persons and obtain a firm commitment means that the development project should not be undertaken.

Other useful characteristics for the domain expert(s) to have include the ability to communicate effectively, have an orderly mind, patience and the willingness to teach.

In evaluating technical risk factors, the following should be included:

• There must be recognized experts in the field along with general agreement among these experts on the knowledge required to solve the problem the expert system is being developed to address.
• The development team must be identified and arrangements made to assure their dedication to the development and follow-up processes. The availability and personal commitment of all team members must be assured.
• The availability of a manual or automated procedure to be used as a model for the development of the expert system should be considered.
• The required performance of the expert system must be defined (in terms of finding the best solution as compared to senior experts). Unrealistic expectations must be avoided.
• Ambiguity in specifications must be avoided, or if ambiguity does exist, the specifications must be modified to avoid it.
• The scope and range of problems to be addressed by the expert system must be clearly identified.
• Interaction with external programs to run algorithmic routines or for data entry, etc., must be identified.

User risk factors must be considered and resolved in the initial planning phases of the expert system development. If representative end users are not involved in the planning and development stages, the system probably will not be accepted by the user community. Issues include:

The end users must want the system and have a vested interest in its success.

The computer proficiency and other skills and interests of the end users must be accommodated.

The environment or conditions under which the system will be operated must be accounted for.

2.3 The Development Team

There are four categories of participants involved in the expert system building process. These are the advocate who champions the building of the expert system, the end users of the expert system, the domain expert(s) whose problem-solving expertise is to be modeled, and the knowledge engineer who actually builds the system. Although in the process of building a given expert system the same person may at various stages of development take on different roles, it is important to recognize that these roles are distinct. The role of the advocate who champions the development of the expert system is to:

• Identify the need for the system.
• Define the problem domain.
• Identify the intended user community.
• Define the expected benefits that accrue, to the intended audience using the expert system.
• Identify the expert(s) whose expertise will be modeled.
• Choose the knowledge engineer who will develop the system.
• >Maintain (or plan for the maintenance of) the finished product.
• Plan and chaperon the entire development process.

The end user is critical in the development of an expert system and must be involved in the entire development process. The end user provides:

• Definition of the skill level of the user community.
• Information on how problems are addressed in the field versus the prescribed solutions.
• Advice on how the system must function (interact with the user) to be accepted by the intended users.
• A cadre of supporters to test and promote the expert system once it is completed.

The domain expert has a dual role in the expert system development process. First, the expert's problem-solving ability serves as the model for the expert system. Secondly, the expert must assist in quality control on the project and make certain that the expert system faithfully represents a useful portion of the expert's knowledge. In essence, the expert must take some responsibility for ensuring that the expert system faithfully models his expertise. The expert's major task in fulfilling this responsibility is to assist in the design of a comprehensive set of test problems for use in verifying that the expert system actually works.

The knowledge engineer has the task of developing a faithful model of the expert's problem solving ability in the domain of interest. Other tasks which the knowledge engineer must perform are:

• Implement the model of the expert's knowledge.
• Ensure that the implementation is as transparent as possible.
• Document the expert system.
• Test the expert system.

One individual may perform more than one of these functions; however, the end users and their tasks should remain autonomous. If the roles of the domain expert and the knowledge engineer are combined, a second domain expert should review and confirm the technical findings.

2.4 The Test /Evaluation Team

The same four categories of participants involved in expert system verification, validation, and evaluation are involved in the building of the system. However, their roles have changed in some aspects. These are the advocates who champion the building of the expert system, the end users of the expert system, the domain expert(s) whose problem-solving expertise is to be modeled, and the knowledge engineer who actually builds the system. Although in the process of building a given expert system the same person may at various stages of development take on different roles, it is important to recognize that these roles are distinct. The role of the advocate who champions the expert system is to:

• Identify the need for system robustness and usefulness.
• Define the problem domain for testing.
• Identify the intended user community.
• Define the expected benefits that will accrue to the testers of the expert system.
• Identify the sites where testing will be conducted.

The end user is critical and must be involved in the entire process from development through implementation. The end user provides:

• Access to a cadre of supporters to test and promote the expert system.
• Information on how problems are addressed in the field versus the prescribed solutions and knowledge on how to "fix" problems on the fly.
• Advice on how the system must function, i.e. interact with the user, to be accepted by the intended users.

The role of the domain expert and the knowledge engineer are the same as outlined in the previous section.

2.5 Systems Development Milestones

In developing expert systems a series of development milestones should be used to measure progress and to provide a series of "go/no-go" decision points. These milestones should each represent stages of development that would provide an improvement in the state-of-the-practice and points where a formal decision by top management to proceed with the development should be made. It is the responsibility of the development advocate to provide criteria for these decisions and to gain management approval of these formal criteria during the planning of the expert systems development. Figure 2.1.1 illustrates the process of "Go/No Go" decisions and figure 2.1.2, the application of a specfic example. The following example is provided to further illustrate this philosophy.

Situation: A regulatory unit has 2500 paragraphs of regulation to manage. There are about 100 queries per month to these regulations and by mandate responses must be provided in five working days. Files of previous responses are scattered between file cabinets, cardboard boxes and the memory of five remaining experts (all approaching retirement) who know the purpose and the history of the regulations. Solutions: Build an expert system to capture the knowledge of the five remaining experts and to manage the responses to inquiries. Analysis/Recommendations: The solution sounds wonderful and could even be made to work, but competent and thorough planning and management are required. It should never be assumed that an expert system is the logical answer; an expert system is only a tool and should be evaluated along with other possible approaches. The system should be constructed in the following stages: Organize the existing files Develop a system to categorize inquiries Identify typical responses to each category of inquiries Develop a scanning system to automate the reading of inquiries Develop a preliminary response letter based on steps 2,3 and 4 Perform VV&E on the developed system Note that at the end of step 6 a fully developed and tested system will be in place. Also the term "expert system" was not used although in all likelihood an the obvious tool to use steps 2,3,4 and 5. each step represented a clear improvement state-of-the-practice for logical "go/no-go" criteria could be prepared during planning stage of project. >

Figure 2.1.1: System Development Milestones Example

Figure 2.1.2: KB1 Initial Project Planning

[Table of Contents]

[Next]