lOMoARcPSD|35902250

KE Unit 2 notes
 lOMoARcPSD|35902250

KNOWLEDGE ENGINEERING

UNIT - 2 NOTES

METHODOLOGY AND MODELING

 lOMoARcPSD|35902250

Methodologies and Tools for Agent

3 Design and Development

3.1 A CONVENTIONAL DESIGN AND DEVELOPMENT

SCENARIO

3.1.1 Conventional Design and Development Phases

Table 3.1 shows the main development phases for a knowledge-based agent. As in the
case of developing a typical software system, there are feedback loops between all these
phases. We will illustrate these phases with a conventional scenario of developing a
knowledge-based agent. This scenario is an adaptation of the one described in Buchanan
et al. (1983).

3.1.2 Requirements Specification and Domain Understanding

The development process starts with identifying a problem that may be solved by develop-
ing a knowledge-based agent. Table 3.2 shows an example of such a problem.
A knowledge engineer, who is a person specialized in developing knowledge-based
agents, is assigned the job of building an agent that incorporates the expertise of subject
matter experts, who are people familiar with the detection, monitoring, and containment
of the spills of hazardous materials. The goal of the system is to assist its users in
performing such tasks. The knowledge engineer has to identify a subject matter expert
from whom to capture the domain expertise and represent it into the agent’s knowledge
base, as was briefly discussed in Section 1.6.3.1.

Table 3.1 Main Phases in the Development of a Knowledge-based Agent

1. Requirements specification (specifying the types of problems to be solved or hypotheses to be

analyzed, and the agent to be built).
2. Domain understanding.
3. Ontology design and development.
4. Development of problem-solving rules or methods.
5. Refinement of the knowledge base (ontology and rules/methods).
6. Verification, validation, and certification of the agent.

83
 lOMoARcPSD|35902250

84 Chapter 3. Methodologies and Tools

Table 3.2 Problem to Be Solved with a Knowledge-based Agent (from Buchanan et al.,
p. 132).

The director of the Oak Ridge National Lab (ORNL) faces a problem. Environmental Protection
Agency (EPA) regulations forbid the discharge of quantities of oil or hazardous chemicals into or
upon waters of the United States when this discharge violates specified quality standards. ORNL
has approximately two thousand buildings on a two-hundred-square-mile government
reservation, with ninety-three discharge sites entering White Oak Creek. Oil and hazardous
chemicals are stored and used extensively at ORNL. The problem is to detect, monitor,
and contain spills of these materials, and this problem may be solved with a
knowledge-based agent.

Table 3.3 Specification of the Actual Problem to Be Solved (from Buchanan et al., p.133).

When an accidental inland spill of an oil or chemical occurs, an emergency situation may exist,
depending on the properties and the quantity of the substance released, the location of the
substance, and whether or not the substance enters a body of water.
The observer of a spill should:
1. Characterize the spill and the probable hazards.
2. Contain the spill material.
3. Locate the source of the spill and stop any further release.
4. Notify the Department of Environmental Management.

What issues may concern the subject matter expert? First of all, the expert may be
concerned that once his or her expertise is represented into the agent, the organization
may no longer need him or her because the job can be performed by the agent. Replacing
human experts was a bad and generally inaccurate way of promoting expert systems.
Usually, the knowledge-based agents and even the expert systems are used by experts in
order to better and more efficiently solve problems from their areas of expertise. They are
also used by people who need the expertise but do not have access to a human expert, or
the expert would be too expensive.
What are some examples of knowledge-based agents? Think, for instance, of any tax-
preparation software. Is it a knowledge-based agent? What about the software systems that
help us with various legal problems, such as creating a will? They all are based on large
amounts of subject matter expertise that are represented in their knowledge bases.
Once the subject matter expert is identified and agrees to work on this project, the
knowledge engineer and the expert have a series of meetings to better define the actual
problem to be solved, which is shown in Table 3.3.
The knowledge engineer has many meetings with the subject matter expert to elicit his
or her knowledge on how to solve the specified problem. There are several knowledge
elicitation methods that can be employed, as discussed later in Section 6.3. Table 3.4
illustrates the unstructured interview, where the questions of the knowledge engineer and
the responses of the expert are open-ended.
 lOMoARcPSD|35902250

3.1. Conventional Design and Development Scenario 85

Table 3.4 Unstructured Interview to Elicit Subject Matter Expertise

(from Buchanan et al., p. 134)

KE: Suppose you were told that a spill had been detected in White Oak Creek one mile
before it enters White Oak Lake. What would you do to contain the spill?
SME: That depends on a number of factors. I would need to find the source in order to
prevent the possibility of further contamination, probably by checking drains and manholes for
signs of the spill material. And it helps to know what the spilled material is.
KE: How can you tell what it is?
SME: Sometimes you can tell what the substance is by its smell. Sometimes you can
tell by its color, but that's not always reliable since dyes are used a lot nowadays. Oil,
however, floats on the surface and forms a silvery film, while acids dissolve completely in
the water. Once you discover the type of material spilled, you can eliminate any
building that either doesn’t store the material at all or doesn’t store enough of it to account
for the spill.

Table 3.5 Identification of the Basic Concepts and Features Employed by the
Subject Matter Expert

KE: Suppose you were told that a spill had been detected in White Oak Creek one mile
before it enters White Oak Lake. What would you do to contain the spill?
SME: That depends on a number of factors. I would need to find the source in order
to prevent the possibility of further contamination, probably by checking drains
and manholes for signs of the spill material. And it helps to know what the spilled
material is.
KE: How can you tell what it is?
SME: Sometimes you can tell what the substance is by its smell. Sometimes you can
tell by its color, but that's not always reliable since dyes are used a lot nowadays. Oil, however,
floats on the surface and forms a silvery film, while acids dissolve completely in the water. Once
you discover the type of material spilled, you can eliminate any building that either doesn’t store
the material at all or doesn’t store enough of it to account for the spill.

3.1.3 Ontology Design and Development

The main goal of the initial interview sessions is to identify the basic concepts and features
from the application domain. Some of these domain concepts and features are underlined
in Table 3.5.
The identified concepts, instances, and features are used to design and develop the
ontology of the system to be built. A fragment of the developed concept and instance
hierarchy is shown in Figure 3.1.
The nodes in light blue (such as “building 3023”) represent specific objects or
instances. The nodes in dark blue (such as “building”) represent sets of instances and
are called concepts. The instances and concepts may have features that are represented
in green, as for instance, “spill-1 has as type acid” and “spill-1 has as odor vinegar odor.”
 lOMoARcPSD|35902250

86 Chapter 3. Methodologies and Tools

object

subconcept of
building
odor

appearance
instance of

subconcept of
building 3024 source spill

no film
building 3023 instance of
instance of

silver film subconcept of

s6-1 s6-2 spill-1

pungent odor
location
substance
has as type

subconcept of has as odor

subconcept of
vinegar odor
drain manhole water acid
oil

instance of instance of
subconcept of subconcept of

d6-1 d6-2 m6-1 m6-2

diesel oil gasoline sulfuric acid acetic acid

hydraulic oil hydrochloric acid

Figure 3.1. Hierarchy of domain concepts and instances.

feature

has as type has as odor

has as location has as color

has as amount

Figure 3.2. Features used to describe the instances and concepts.

The features are also represented hierarchically, as shown in Figure 3.2. Ontologies are
discussed in detail in Chapter 5.

3.1.4 Development of the Problem-Solving Rules or Methods

Once an initial ontology is developed, the knowledge engineer and the subject matter
expert use the component concepts and features to represent the expert’s methods of
determining the spill material as a set of rules. Consider, for example, the reasoning of
the subject matter expert (SME) from the top of Table 3.6. What IF-THEN rules may
represent this reasoning?
In an iterative process, the knowledge engineer represents the expert’s reasoning as
rules, asks the expert to critique them, and correspondingly updates the rules, as illus-
trated in Table 3.7. As would be expected, this is quite a long and difficult process, for the
reasons discussed in Section 1.6.3.1.
 lOMoARcPSD|35902250

3.1. Conventional Design and Development Scenario 87

Table 3.6 Representing the Expert’s Reasoning as If-Then Rules

SME: Sometimes you can tell what the substance is by its smell. Sometimes you can tell by its color,
but that’s not always reliable since dyes are used a lot nowadays. Oil, however, floats on the
surface and forms a silvery film, while acid dissolves completely in the water.
IF the spill . . .
THEN the substance of the spill is oil
IF the spill . . .
THEN the substance of the spill is acid

Table 3.7 Iterative Process of Rules Development and Refinement (Based on Buchanan et al.,
p.138)

KE: Here are some rules I think capture your explanation about determining the substance of the
spill. What do you think?
IF the spill does not dissolve in water
and the spill forms a silvery film
THEN the substance of the spill is oil
IF the spill dissolves in water
and the spill does not form a film
THEN the substance of the spill is acid

SME: Uh-huh (long pause). Yes, that begins to capture it. Of course, if the substance is silver nitrate,
it will dissolve only partially in water.
KE: I see. Rather than talking about a substance dissolving or not dissolving in water, we should talk
about its solubility, which we may consider as being high, moderate, or low. Let’s add that
information to the knowledge base and see what it looks like.
IF the solubility of the spill is low
and the spill forms a silvery film
THEN the substance of the spill is oil
IF the solubility of the spill is moderate
THEN the substance of the spill is silver-nitrate

SME: If the solubility of the spill is moderate, I would be about 60 percent

sure that the substance of the spill is silver-nitrate.
KE: Okay, we will represent this information in the rule.
IF the solubility of the spill is moderate
THEN the substance of the spill is silver-nitrate with certainty 0.6

3.1.5 Verification, Validation, and Certification

Once the ontology and the rules are defined and represented into the knowledge base of
the agent, the knowledge engineer and the subject matter expert can run the prototype
agent and analyze its solutions, which are generated by the chaining of the rules.
 lOMoARcPSD|35902250

88 Chapter 3. Methodologies and Tools

Testing of the agent involves three types of activity: verification, validation, and certifi-
cation (O’Keefe et al., 1987; Awad, 1996).
In essence, verification attempts to answer the question: Are we building the agent right?
Its goal is to test the consistency and the completeness of the agent with respect to its initial
specification. For example, in the case of a rule-based agent, one would check the rules to
identify various types of errors, such as the existence of rules that are redundant, conflicting,
subsumed, circular, dead-end, missing, unreachable, or with unnecessary IF conditions.
Validation, on the other hand, attempts to answer the question: Are we building the
right agent? In essence, this activity checks whether the agent meets the user’s needs and
requirements.
Finally, certification is a written guarantee that the agent complies with its specified
requirements and is acceptable for operational use.

3.2 DEVELOPMENT TOOLS AND REUSABLE ONTOLOGIES

Various types of tools can be used to develop a knowledge-based agent. We will briefly
discuss three different types: expert system shells, learning agent shells, and learning agent
shells for evidence-based reasoning. We will also discuss the reuse of knowledge in the
development of a knowledge-based agent.

3.2.1 Expert System Shells

These tools exploit the architectural separation in a knowledge-based system between the
knowledge base and the inference engine (see the right-hand side of Figure 1.14, p. 37).
Given this separation, two knowledge-based agents, one for diagnosing ships and the
other for diagnosing airplanes, may potentially use the same inference engine for diagno-
sis. The difference between the two agents will be in the content of their knowledge bases
(e.g., ship parts versus airplane parts).
An expert system shell is a tool that consists of an inference engine for a certain class of
tasks (e.g., planning, design, diagnosis, monitoring, prediction, interpretation, etc.); a repre-
sentation formalism in which the knowledge base can be encoded; and mechanisms for
acquiring, verifying, and revising knowledge expressed in that formalism.
If the inference engine of an expert system shell is adequate for a certain expertise task
(e.g., planning), then the process of building an expert system or agent for that type of
tasks is, in principle, reduced to the building of the knowledge base.
Different expert system shells trade generality (i.e., their domain of applicability)
against power (i.e., the assistance given to the expert and the knowledge engineer in the
development process), covering a large spectrum.
At the generality end of the spectrum are very general shells that can be used to build
almost any type of expert system, but they do not provide too much assistance in building
a specific system. Examples of such shell are:

 OPS (Cooper and Wogrin, 1988), which has a general rule engine
 CLIPS (Giarratano and Riley, 1994), which also has a general rule engine
 CYC (Lenat, 1995; CYC, 2008; 2016), a very large knowledge base with ontologies
covering many domains, and with several rule engines
 EXPECT (Gil and Paris, 1995; EXPECT 2015), a shell that enables the acquisition of
problem-solving knowledge both from knowledge engineers and from end-users
 lOMoARcPSD|35902250

3.2. Development Tools and Reusable Ontologies 89

 JESS (Friedman-Hill, 2003; JESS, 2016), which is a version of CLIPS with a Java-based
rule engine
 CommonKADS (Schreiber et al., 2000), which is a general methodology with support-
ing tools for the development of knowledge-based systems
 Jena, which is a toolkit for developing applications for the Semantic Web (Jena, 2012)
 Pellet, an ontology (OWL2) reasoner that can be used to develop knowledge-based
applications for the Semantic Web (Pellet, 2012)
 Protégé (Musen, 1989; Protégé, 2015), an ontology editor and knowledge base frame-
work, also used to develop knowledge-based applications for the Semantic Web
 TopBraid Composer (Allemang and Hendler, 2011; TopBraid Composer, 2012) Ontology
Development Tool for Semantic Web applications

At the power end of the spectrum are shells that employ much more specific problem-
solving methods, such the propose-and-revise design method used in SALT (Marcus, 1988)
to design elevators. The knowledge for such a system can be elicited by simply filling in
forms, which are then automatically converted into rules. Thus the shell provides signifi-
cant assistance in building the system, but the type of systems for which it can be used is
much more limited.
In between these two types of shells are the shells applicable to a certain type of
problems (such as planning, or diagnosis, or design). A representative example is EMYCIN
(van Melle et al., 1981), a general rule-based shell for medical diagnosis.

3.2.2 Foundational and Utility Ontologies and Their Reuse

The use of an expert system shell facilitates the development of an expert system because
it reduces the process of developing the system to that of developing the knowledge base
for the provided inference engine. However, it is this development of the knowledge base
that is the most challenging part of developing the system.
Despite the fact that building the knowledge base is such a difficult problem, historic-
ally knowledge bases were very rarely reused, primarily for two reasons. First, the know-
ledge in the knowledge base is usually very specific to a particular domain and type of
problems and cannot be applied directly to a different application area. Second, even if the
knowledge base of a system is directly relevant to the new area, its reuse by a system with a
different knowledge representation is likely to be very difficult because of the differences
between the knowledge models of the two systems.
Progress in knowledge engineering has changed this situation. A new architectural
separation has emerged at the level of the knowledge base, which is structured into two
main components: an ontology that defines the concepts of the application domain, and a
set of problem-solving rules or methods expressed with these concepts, as was illustrated
in Section 1.6.3.1. The ontology is the more general part, being characteristic to an entire
application domain, such as military or medicine. In the military domain, for example, the
ontology includes descriptions of military units and of military equipment. These descrip-
tions are most likely needed in almost any specific military application and can therefore
be reused. The rules are the more specific part of the knowledge base, corresponding to a
certain type of application. For example, there may be rules for an agent, such as Disciple-
COA, that assists a commander in critiquing courses of action, or rules for an agent, such
as Disciple-WA, that assists in planning the repair of damaged bridges or roads. These
rules offer much fewer opportunities for reuse, if any.
 lOMoARcPSD|35902250

90 Chapter 3. Methodologies and Tools

As a result of terminological standardization to facilitate automatic processing of infor-

mation, particularly in the context of the Semantic Web, many domain and general-purpose
ontologies have been developed. The general-purpose (domain-independent) ontologies
are also called upper, foundational, or universal ontologies because they provide high-level,
domain-independent concepts and relationships that can be included in the top part of a
domain ontology (Obrst et al., 2012). Examples of such ontologies are Cyc/OpenCyc (2016;
Lenat, 1995), Suggested Upper Merged Ontology (SUMO) (Pease, 2011), WordNet (2012;
Fellbaum 1988), Descriptive Ontology for Linguistic and Cognitive Engineering (DOLCE,
2012), Basic Formal Ontology (BFO, 2012), Object-centered High-level Reference (OCHRE)
(Schneider, 2003), General Formal Ontology (GFO, 2012), Unified Medical Language System
(UMLS) (Humphreys and Lindberg, 1993), and Unified Foundational Ontology (UFO)
(Guizzardi and Wagner, 2005a, 2005b).
There are also utility ontologies (Obrst et al., 2012) that include representations of
commonly used concepts, such as persons, social roles, and organizations (Masolo et al.,
2004; FOAF, 2012), temporal concepts (Hobbs and Pan, 2004; Pan and Hobbs, 2004; 2012),
and geospatial concepts (Ressler et al., 2010; Geonames, 2012).
The Open Knowledge Base Connectivity (OKBC) protocol (OKBC, 2008) has been defined
to facilitate knowledge sharing and reuse (Chaudhri et al., 1998). OKBC is a standard for
accessing knowledge bases stored in different frame-representation systems. It provides a set
of operations for a generic interface to such systems. As a result, OKBC servers for various
frame-based systems, such as Ontolingua (Ontolingua 1997; 2008; Farquhar et al., 1997) and
Loom (Loom, 1999; MacGregor, 1991), have been developed. These servers are repositories of
reusable ontologies and domain theories and can be accessed using the OKBC protocol.
Additionally, there are many tools that can query existing ontologies on the Semantic
Web or simply import them into the knowledge base to be built, including Jena (2012),
OWLIM (2012), Pellet (2012), Protégé (2015), and TopBraid Composer (2012).

3.2.3 Learning Agent Shells

While part of the ontology of the agent to be developed can be imported from previously
developed ontologies, the reasoning rules of the agent are application-specific and cannot
be reused. An alternative approach to rules development is to employ a learning agent that
is able to learn the rules. In addition to containing the knowledge base and the inference
engine, the architecture of a learning agent includes a learning engine consisting of the
programs that create and update the data structures in the knowledge base. The learning
agent may learn from a variety of information sources in the environment. It may learn from
its user or from other agents, either by being directly instructed by them or just by observing
and imitating their behavior. It may learn from a repository of information (such as a
database), or it may learn from its own experience. Building a practical autonomous
learning agent that can acquire and update its knowledge by itself is not yet practical, except
for very simple problems, such as classification. Therefore, a more practical approach is to
develop an interactive learning agent that can interact with an expert. Such an agent can
perform many of the functions of the knowledge engineer. It allows the human expert to
communicate expertise in a way familiar to him or her and is responsible for building,
updating, and reorganizing the knowledge base. We call such a tool a learning agent shell.
A learning agent shell is an advanced tool for building knowledge-based agents. It
contains a general problem-solving engine, a learning engine, and a general knowledge
 lOMoARcPSD|35902250

3.2. Development Tools and Reusable Ontologies 91

base structured into an ontology and a set of rules (see Figure 3.3). Building a knowledge-
based agent for a specific application consists of customizing the shell for that application
and developing the knowledge base. The learning engine facilitates the building of the
knowledge base by subject matter experts and knowledge engineers.
Examples of learning agent shells are Disciple-LAS (Tecuci et al., 1999), Disciple-COA
(Tecuci et al., 2001), and Disciple-COG/RKF (Tecuci et al., 2005b), the last two being
presented in Section 12.4.

3.2.4 Learning Agent Shell for Evidence-based Reasoning

As discussed in Section 1.4.3, tasks in many domains, such as law, intelligence analysis,
cybersecurity, forensics, medicine, physics, chemistry, history, or archaeology, involve
evidence-based reasoning. All these tasks use general evidence-based reasoning concepts
and rules. Examples of general concepts are different types of evidence, such as tangible
evidence and testimonial evidence. An example of a general rule is to assess the credibility of
a human source of information by assessing his or her veracity, objectivity, and observational
sensitivity. Since all this knowledge is domain-independent, it makes sense to develop a
general knowledge base for evidence-based reasoning that can be reused each time we are
developing an agent that also needs to perform this kind of reasoning. To facilitate this
process, the knowledge base of the agent is structured into a hierarchy of knowledge bases,
with the knowledge base for evidence-based reasoning at top of the hierarchy.
A learning agent shell for evidence-based reasoning is a learning agent shell that contains
a hierarchy of knowledge bases the top of which is a knowledge base for evidence-based
reasoning (see Figure 3.4). As will be illustrated in the following sections, building an agent
for a specific evidence-based reasoning application consists in extending the knowledge
base with domain-specific knowledge through learning from a subject matter expert.
An example of a learning agent shell for evidence-based reasoning is Disciple-EBR, the
architecture of which is shown in the center of Figure 3.5. It includes multiple modules for
problem solving, learning, tutoring, evidence-based reasoning, mixed-initiative inter-
action, as well as a hierarchically organized knowledge base with domain-independent
knowledge for evidence-based reasoning at the top of the knowledge hierarchy. The
Disciple-EBR shell can learn complex problem-solving expertise directly from human
experts, and in doing so it evolves into a cognitive assistant that can support experts and
nonexperts in problem solving and can teach expert problem-solving to students.
The outside hexagon in Figure 3.5 summarizes a possible life cycle of a Disciple
cognitive assistant for evidence-based reasoning. The first stage is shell customization,

Problem
Interface

Solving
Ontology
+ Rules
Learning

Figure 3.3. The overall architecture of a learning agent shell.

 lOMoARcPSD|35902250

92 Chapter 3. Methodologies and Tools

Problem EBR KB

Solving

Interface
Domain KB Domain KB Domain KB

Scenario KB Scenario KB
Learning
KB = Ontology + Rules

Figure 3.4. The overall architecture of a learning agent shell for evidence-based reasoning.

1. Shell

Customization
Disciple Developer and Knowledge Engineer

Disciple
Learning Agent Multi-
Disciple Ontology
Shell Strategy
Development
Learning
Modules
Modules

EBR KB

Mixed-
Tutoring
Initiative
Domain KB Domain KB Domain KB
Modules
Interaction

Scenario KB Scenario KB

Knowledge Base
En

Management
d
-U
se
r

Problem- Evidence-

Solving Specific
Specialized
Modules Modules
Agent

End-User

4. Field Use

Figure 3.5. Life cycle of a Disciple agent for evidence-based reasoning.

where, based on the specification of the type of problems to be solved and the agent to be
built, the developer and the knowledge engineer may decide that some extensions of the
Disciple shell may be necessary or useful. It is through such successive extensions during
the development of Disciple agents for various applications that the current version of the
Disciple shell for evidence-based reasoning problems (which includes the EBR knowledge
base) has emerged.
The next stage is agent teaching by the subject matter expert and the knowledge
engineer, supported by the agent itself, which simplifies and speeds up the knowledge
base development process (Tecuci et al., 2001; 2002b; 2005b). Once an operational agent is
developed, it is used for the education and training of the end-users, possibly in a
classroom environment.
The fourth stage is field use, where copies of the developed agent support users in their
operational environments. During this stage, an agent assists its user both in solving
 lOMoARcPSD|35902250

3.3. System Design and Development 93

problems and in collaborating with other users and their cognitive assistants. At the same
time, it continuously learns from this problem-solving experience by employing a form of
nondisruptive learning. In essence, it learns new rules from examples. However, because
there is no learning assistance from the user, the learned rules will not include a formal
applicability condition. It is during the next stage of after action review, when the user and
the agent analyze past problem-solving episodes, that the formal applicability conditions
are learned based on the accumulated examples.
In time, each cognitive assistant extends its knowledge with expertise acquired from its
user. This results in different agents and creates the opportunity to develop a more
competent agent by integrating the knowledge of all these agents. This can be accom-
plished by a knowledge engineer, with assistance from a subject matter expert, during the
next stage of knowledge integration. The result is an improved agent that may be used in a
new iteration of a spiral process of development and use.

3.3 AGENT DESIGN AND DEVELOPMENT USING LEARNING

TECHNOLOGY

In Section 3.1, we briefly illustrated a conventional scenario for building knowledge-based

agents. Now that we have reviewed several learning-based agent building tools, we can
illustrate the development of a knowledge-based agent using learning technology. While the
main design and development phases are, in essence, still those from Table 3.1, the way they
are performed is different. The tasks of the knowledge engineer are reduced because part of
them will be performed by the subject matter expert (e.g., explaining to the agent shell the
sequence of reasoning steps to solve a problem), and part by the agent shell (e.g., learning rules
from example reasoning steps rather than having them defined by the knowledge engineer).

3.3.1 Requirements Specification and Domain Understanding

Let us consider the development of an agent that will assist a PhD student in assessing a
potential PhD advisor, a case study that will be used throughout this book.
The agent should receive a hypothesis, such as “John Doe would be a good PhD advisor
for Bob Sharp,” and should return the probability that this hypothesis is true, such as “very
likely,” together with an easy-to-understand reasoning that has led this assessment.
To build such an agent, one needs first to understand the expertise domain. In this case, a
simple search on the Internet will lead to the discovery of many papers written on this subject
(see Figure 3.6), which may supplement the expertise of a specific subject matter expert.
Reading some of these papers, you will discover that assessing a PhD advisor is a very
complex task. Many questions need to be addressed whose answers need to be aggregated
to evaluate a potential PhD advisor. A few of these questions, identified by the knowledge
engineer and the subject matter expert (in such a case, an experienced professor), are
shown in Table 3.8.

3.3.2 Rapid Prototyping

The next agent development phase is to develop rapidly a prototype that can be validated
by the end-users.
 lOMoARcPSD|35902250

94 Chapter 3. Methodologies and Tools

Figure 3.6. Understanding the expertise domain.

Table 3.8 Some Relevant Questions to Consider When Assessing a Potential PhD Advisor

(1) What is the reputation of the PhD advisor within the professional community at large?
(2) Does the advisor have many publications?
(3) Is his or her work cited?
(4) What is the opinion of the peers of this PhD advisor?
(5) What do the students think about this PhD advisor?
(6) Is the PhD advisor likely to remain on the faculty for the duration of your degree program?
(7) What is the placement record of the students of this PhD advisor? Where do they get jobs?
(8) Is the PhD advisor expert in your areas of interest?
(9) Does the PhD advisor publish with students?
(10) Does the PhD advisor have a research group or merely a string of individual students?
(11) Is the PhD advisor’s research work funded?

An analysis of the questions in Table 3.8 shows that some of them point to necessary
conditions that need to be satisfied by the PhD advisor, while others refer to various
desirable qualities. Which questions from Table 3.8 point to necessary conditions? The
answers to questions (6) and (8) need to be “yes” in order to further consider a potential
PhD advisor.
Now let us consider the desirable qualities of a PhD advisor revealed by the other
questions in Table 3.8. Some of these qualities seem to be more closely related than
others. It would be useful to organize them in classes of quality criteria. Could you identify
a class of related criteria? Questions (2), (3), (4), and (11) all characterize aspects of the
professional reputation of the advisor.
What might be other classes of related criteria suggested by the questions in Table 3.8?
Questions (7) and (9) characterize the results of the students of the PhD advisor, while
questions (5) and (10) characterize their learning experience.
 lOMoARcPSD|35902250

3.3. System Design and Development 95

Based on these observations, we can develop the ontology of criteria from

Figure 3.7.
Each of the criteria from the right side corresponds to one of the questions in Table 3.8.
They are components of the higher-order criteria shown in the middle of the figure, which
are all components of a top-level criterion that characterizes the quality of the PhD
advisor. This is what is called a part-of hierarchy. All these individual criteria are all
instances of the “criterion” concept.
The preceding analysis suggests that, in order to assess a PhD advisor, one needs
to check that the advisor satisfies the necessary conditions and to assess his or her
advisor qualities.
The knowledge engineer and the subject matter expert need to develop a formal,
yet intuitive way of representing the assessment logic. This has to be natural enough,
such that subject matter experts who do not have knowledge engineering experience
are able to express how to solve different problems by themselves, with no or
limited support from knowledge engineers. But the assessment logic also needs to
be formal enough so that an agent can learn general rules from such problem-solving
examples.
A general problem-solving paradigm, called problem reduction and solution synthesis,
which has been illustrated in Section 2.2 and will be discussed in detail in Chapter 4,
satisfies both these requirements. It will be again illustrated in this case.
To clarify the reduction logic, the knowledge engineer and the subject matter expert
consider a particular hypothesis:

John Doe would be a good PhD advisor for Bob Sharp.

criterion

instance of

Q11 research funding criterion

has as criterion

Q2, 3, 4, 11 Q2
research publications criterion
professional

reputation criterion Q3
citations criterion

Q4

peer opinion criterion

has as criterion

has as criterion

PhD advisor
Q7, 9
Q9 publications with advisor criterion
quality criterion
quality of student

results criterion Q7
employers of graduates criterion
has as criterion

Q5, 10
Q10
research group status criterion
student learning

experience criterion
Q5
student opinion criterion

Figure 3.7. Sample criteria ontology for assessing a PhD advisor.

 lOMoARcPSD|35902250

96 Chapter 3. Methodologies and Tools

They express the hypothesis in natural language and select the phrases that may be
different for other similar hypotheses, such as the names of the advisor and student.
The selected phrases will appear in blue, guiding the agent to learn a general hypothesis
pattern:

?O1 would be a good PhD advisor for ?O2.

This top-level hypothesis will be successively reduced to simpler and simpler hypotheses,
guided by questions and answers, as shown in Figure 3.8 and discussed in this section.

John Doe would be a good PhD advisor for Bob Sharp.

Which are the necessary conditions?
Bob Sharp should be interested in an area of expertise of John Doe who should stay on
the faculty of George Mason University for the duration of the PhD dissertation of Bob
Sharp, and should have the qualities of a good PhD advisor.

Therefore, the initial hypothesis can be reduced to three simpler hypotheses:

Bob Sharp is interested in an area of expertise of John Doe.

John Doe will stay on the faculty of George Mason University for the duration of the PhD
dissertation of Bob Sharp.
John Doe would be a good PhD advisor with respect to the PhD advisor quality criterion.

John Doe would be a good PhD advisor for Bob Sharp.

Which are the necessary conditions?

Bob Sharp should be interested in an area of expertise of John Doe, who should

stay on the faculty of George Mason University for the duration of the PhD

dissertation of Bob Sharp and should have the qualities of a good PhD advisor.

Bob Sharp is John Doe will stay on the faculty John Doe would be a

interested in an of George Mason University for good PhD advisor with

area of expertise the duration of the PhD respect to the PhD

of John Doe. dissertation of Bob Sharp. advisor quality criterion.

Is Bob Sharp interested in an Which are the necessary quality criteria for a good PhD advisor?

area of expertise of John Doe?

professional reputation criterion, quality of student results criterion,

Yes, Artificial Intelligence. and student learning experience criterion.

It is certain that Bob Sharp John Doe would be a good John Doe would be a John Doe would be a good

is interested in an area of PhD advisor with respect good PhD advisor with PhD advisor with respect

expertise of John Doe. to the professional respect to the quality of to the student learning

reputation criterion. student results criterion. experience criterion.

Which is a professional reputation criterion?

research funding criterion

John Doe would be a good PhD

advisor with respect to the

research funding criterion.

Figure 3.8. Reduction logic for assessing a specific hypothesis.

 lOMoARcPSD|35902250

3.3. System Design and Development 97

The reductions of the subhypotheses continues in the same way, until solutions are
obtained for them:

Bob Sharp is interested in an area of expertise of John Doe.

Is Bob Sharp interested in an area of expertise of John Doe?
Yes, Artificial Intelligence.

Therefore, one may conclude:

It is certain that Bob Sharp is interested in an area of expertise of John Doe.

Consider now the third subhypothesis of the initial hypothesis:

John Doe would be a good PhD advisor with respect to the PhD advisor quality criterion.

Its reduction is also guided by a question/answer pair:

Which are the necessary quality criteria for a good PhD advisor?
professional reputation criterion, quality of student results criterion, and student learning
experience criterion.

Therefore, the preceding hypothesis can be reduced to three simpler hypotheses:

John Doe would be a good PhD advisor with respect to the professional reputation
criterion.
John Doe would be a good PhD advisor with respect to the quality of student results
criterion.
John Doe would be a good PhD advisor with respect to the student learning experience
criterion.

Each of these subhypotheses can now be reduced to simpler hypotheses, each corres-
ponding to one of the elementary criteria from the right side of Figure 3.7 (e.g.,
research funding criterion). Since each of these reductions reduces a criterion to a
subcriterion, the agent could be asked to learn a general reduction pattern, as shown
in Figure 3.9.
Why is pattern learning useful? One reason is that the pattern can be applied
to reduce a criterion to its subcriteria, as shown in Figure 3.10. Additionally, as will
be illustrated later, the pattern will evolve into a rule that will automatically generate all
the reductions of criteria to their sub-criteria. If, instead of learning a pattern and

John Doe would be a good PhD ?O1 would be a good PhD

advisor with respect to the advisor with respect to the

professional reputation criterion. ?O2.

Pattern

learning

Which is a professional reputation criterion? Which is a ?O2?

research funding criterion ?O3

John Doe would be a good PhD ?O1 would be a good

advisor with respect to the PhD advisor with respect to

research funding criterion. the ?O3.

Figure 3.9. Pattern learning.

 lOMoARcPSD|35902250

98 Chapter 3. Methodologies and Tools

John Doe would be a good PhD advisor with

respect to the PhD advisor quality criterion.

Which are the necessary quality criteria for a good PhD advisor?

professional reputation criterion, quality of student results criterion,

and student learning experience criterion.

John Doe would be a good John Doe would be a good John Doe would be a good

?O1 would be a good PhD advisor with respect to PhD advisor with respect PhD advisor with respect to

PhD advisor with the professional reputation to the quality of student the student learning

respect to the ?O2. criterion. results criterion. experience criterion.

Pattern instantiation

Which is a ?O2? Which is a quality of student results criterion? Which is a quality of student results criterion?

?O3 publications with advisor criterion employers of graduates criterion

?O1 would be a good John Doe would be a good PhD John Doe would be a good PhD

PhD advisor with advisor with respect to the advisor with respect to the

respect to the ?O3. publications with advisor criterion. employers of graduates criterion.

Figure 3.10. Uniform modeling through the instantiation of a learned pattern.

applying it, the user would manually define these reductions, then any syntactic differ-
ences between these reductions would lead to the learning of different rules. These
rules would only be superficially different, leading to an inefficient and difficult to
maintain agent.
After the top-level criterion (i.e., PhD advisor quality criterion) is reduced to a set of
elementary criteria, specific knowledge and evidence about the advisor need to be used
to evaluate John Doe with respect to each such elementary criterion. For example, the
following hypothesis will be evaluated based on favoring and disfavoring evidence from
John Doe’s peers:

John Doe would be a good PhD advisor with respect to the peer opinion criterion.

A learning agent shell for evidence-based reasoning already knows how to assess such
hypotheses based on evidence.
Through this process, the initial hypothesis is reduced to elementary hypotheses for
which assessments are made. Then these assessments are successively combined, from
bottom-up, until the assessment of the initial hypothesis is obtained, as illustrated in
Figure 3.11.
Notice at the bottom-right side of Figure 3.11 the assessments corresponding to the
subcriteria of the quality of student results criterion:

It is likely that John Doe would be a good PhD advisor with respect to the publications
with advisor criterion.
It is very likely that John Doe would be a good PhD advisor with respect to the employers
of graduates criterion.

These assessments are combined by taking their maximum, leading to an evaluation of

“very likely” for the quality of student results criterion:

It is very likely that John Doe would be a good PhD advisor with respect to the quality of
student results criterion.
 lOMoARcPSD|35902250

3.3. System Design and Development 99

John Doe would be a good PhD advisor for Bob Sharp.

very likely

min

Which are the necessary conditions?

Bob Sharp should be interested in an area of expertise of John Doe, who should stay on

the faculty of George Mason University for the duration of the PhD dissertation of Bob

Sharp and should have the qualities of a good PhD advisor.

Bob Sharp is John Doe will stay on the faculty John Doe would be a

interested in an of George Mason University for good PhD advisor with

area of expertise of the duration of the PhD respect to the PhD

John Doe. dissertation of Bob Sharp. advisor quality criterion.

certain almost certain very likely

min

Is Bob Sharp interested in an

Which are the necessary quality criteria for a good PhD advisor?
area of expertise of John Doe?
professional reputation criterion, quality of student results criterion,
Yes, Artificial Intelligence.
and student learning experience criterion.

It is certain that Bob Sharp John Doe would be a good John Doe would be a John Doe would be a good

is interested in an area of PhD advisor with respect to good PhD advisor with PhD advisor with respect to

expertise of John Doe. the professional reputation respect to the quality of the student learning

criterion. student results criterion. experience criterion.

very likely very likely almost certain

max

Which is a quality of student results criterion? Which is a quality of student results criterion?

publications with advisor criterion employers of graduates criterion

John Doe would be a good PhD John Doe would be a good PhD

advisor with respect to the advisor with respect to the

publications with advisor criterion. employers of graduates criterion.

likely very likely

Figure 3.11. Reduction and synthesis tree for assessing a specific hypothesis.

Then this assessment is combined with the assessments corresponding to the other major
criteria (very likely for the professional reputation criterion, and almost certain for the student
learning experience criterion), through a minimum function (because they are necessary
conditions), to obtain the assessment very likely for the PhD advisor quality criterion.
Finally, consider the assessments of the three subhypotheses of the top-level hypothesis:

It is certain that Bob Sharp is interested in an area of expertise of John Doe.

It is almost certain that John Doe will stay on the faculty of George Mason University for
the duration of the PhD dissertation of Bob Sharp.
It is very likely that John Doe would be a good PhD advisor with respect to the PhD
advisor quality criterion.

These assessments are combined by taking their minimum, leading to the following
assessment of the initial hypothesis:

It is very likely that John Doe would be a good PhD advisor for Bob Sharp.

Could you justify the preceding solution synthesis function? We used minimum because
each of the three subhypotheses of the initial hypothesis corresponds to a necessary
condition. If any of them has a low probability, we would like this to be reflected in the
overall evaluation.
 lOMoARcPSD|35902250

100 Chapter 3. Methodologies and Tools

Notice that, at this point, the knowledge engineer and the subject matter expert have
completely modeled the assessment of the specific hypothesis considered. This is the most
creative and the most challenging part of developing the agent. Once such a model for
assessing hypotheses (or solving problems, in general) is clarified, the agent can be rapidly
prototyped by modeling a set of typical hypotheses. The rest of the agent development
process consists of developing its knowledge base so that the agent can automatically
assess other hypotheses. The knowledge base will consist of an ontology of domain
concepts and relationships and of problem/hypothesis reduction and solution synthesis
rules, as was discussed in Section 1.6.3.1 and illustrated in Figure 1.15 (p. 38). As will be
discussed in the following, the way the preceding assessments were modeled will greatly
facilitate this process.

3.3.3 Ontology Design and Development

We want the developed agent to generate by itself reasoning trees like that shown in
Figure 3.11. What knowledge does it need in its ontology to perform this kind of reasoning?
Consider, for example, the reasoning step from the bottom-left of Figure 3.8, shown
again in the left-hand side of Figure 3.12. What knowledge does the agent need in order to
answer the question from this reasoning step? It needs the knowledge from the right-hand
side of Figure 3.12.
But this is just an example. We want the agent to be able to answer similar questions,
corresponding to similar hypotheses. Therefore, the ontological knowledge from the right-
hand side of Figure 3.12 is just a specification for the ontological knowledge needed by
the agent.
What other concepts and instances should we add to the ontology, based on the
specification in Figure 3.12? We would obviously need to consider other areas of expertise,
since we want to develop a general advisor assistant, capable of assessing advisors from
different disciplines. Thus the ontology can be expanded as shown in Figure 3.13.
The knowledge engineer and the subject matter expert will consider all the reasoning
steps from the developed reasoning trees and will correspondingly develop the ontology of
the agent.
What resources could be used to develop the ontology? Obvious resources are the
many foundational and utility ontologies for the Semantic Web that were discussed in
Section 3.2.2.

Bob Sharp is interested in

area of expertise
an area of expertise of

John Doe. Bob Sharp

instance of

is interested in
Is Bob Sharp interested in an

area of expertise of John Doe?

Artificial Intelligence

Yes, Artificial Intelligence.

is expert in

It is certain that Bob Sharp John Doe

is interested in an area of

expertise of John Doe.

Figure 3.12. Modeling-based ontology specification.

 lOMoARcPSD|35902250

3.3. System Design and Development 101

The goal of this phase is to develop an ontology that is as complete as possible. This will
enable the agent to learn reasoning rules based on the concepts and the features from the
ontology, as will be briefly illustrated in the following.

3.3.4 Rule Learning and Ontology Refinement

During this phase, the subject matter expert will interact with the learning agent shell to
explain to it a specific reasoning tree. From each reasoning step, the agent will learn a
general reasoning rule, as shown in Figure 3.14.
Figure 3.15 illustrates the learning of a general hypothesis reduction rule from a specific
hypothesis reduction step. Notice that the specific instances from the example (e.g., Bob
Sharp or John Doe) are generalized to variables (i.e., ?O1, ?O2). In essence, the rule

area of expertise

subconcept of

Mathematics Computer Science Biology

instance of

Bob Sharp

is interested in

Artificial Intelligence Software Engineering

is expert in

John Doe

Figure 3.13. Expanded ontology based on the specification from Figure 3.12.

John Doe would be a good

Learned rule
PhD advisor for Bob Sharp.

Which are the necessary conditions?

Bob Sharp should be interested in an area of expertise of John Doe, who should

stay on the faculty of George Mason University for the duration of the PhD

dissertation of Bob Sharp and should have the qualities of a good PhD advisor.

John Doe will stay on the faculty of John Doe would be a

Bob Sharp is interested
George Mason University for the good PhD advisor with
in an area of expertise
duration of the PhD dissertation of respect to the PhD
of John Doe.
Bob Sharp. advisor quality criterion.
Learned rule

Is Bob Sharp interested in an Which are the necessary quality criteria for a good PhD advisor?

area of expertise of John Doe?

professional reputation criterion, quality of student results criterion,

Yes, Artificial Intelligence. and student learning experience criterion.

It is certain that Bob Sharp is John Doe would be a good John Doe would be a good John Doe would be a good

interested in an area of PhD advisor with respect to PhD advisor with respect PhD advisor with respect to

expertise of John Doe. the professional reputation to the quality of student the student learning

criterion. results criterion. experience criterion.

Which is a professional

reputation criterion?

research funding criterion

John Doe would be a

good PhD advisor with

respect to the research

funding criterion.

Figure 3.14. Rules learning.

 lOMoARcPSD|35902250

102 Chapter 3. Methodologies and Tools

Bob Sharp is

interested an area of

expertise of John Doe.

Is Bob Sharp interested in an

area of expertise of John Doe?

Yes, Artificial Intelligence.

It is certain that Bob Sharp

is interested in an area of

expertise of John Doe.

Figure 3.15. Learning a reduction rule from a reduction step.

indicates the conditions that need to be satisfied by these variables so that the IF
hypothesis can be assessed as indicated in the example. For example, ?O1 should be a
PhD student or possibly a person (the agent does not yet know precisely what concept to
apply because the rule is only partially learned), ?O1 should be interested in ?O3, and ?O3
should be Artificial Intelligence or possibly any area of expertise.
The way the question and its answer from the reduction step are formulated is very
important for learning. What could the agent learn if the answer were simply “yes”? The
agent would only be able to learn the fact that “Bob Sharp is interested in an area of
expertise of John Doe.” By providing an explanation of why this fact is true (“Yes, Artificial
Intelligence” meaning: “Yes, because Bob Sharp is interested in Artificial Intelligence which is
an area of expertise of John Doe”), we help the agent to learn a general rule where it will
check that the student ?O1 is interested in some area ?O3, which is an area of expertise of
the advisor ?O2. This is precisely the condition of the rule that can be easily verified
because this type of knowledge was represented in the ontology, as discussed previously
and shown in Figure 3.12.
What is the difference between the pattern learning illustrated in Figure 3.9 and the rule
learning illustrated in Figure 3.15? The difference is in the formal applicability condition of
the rule, which restricts the possible values of the rule variables and allows the automatic
application of the rule in situations where the condition is satisfied. A learned pattern,
such as that from Figure 3.9, cannot be automatically applied because the agent does not
know how to instantiate its variables correctly. Therefore, its application, during the
modeling phase, is controlled by the user, who selects the instances of the variables.
A remarkable capability of the agent is that it learns a general rule, like the one in
Figure 3.15, from a single example rather than requiring the rule to be manually developed
by the knowledge engineer and the subject matter expert. Rule learning will be discussed
in detail in Chapter 9.
 lOMoARcPSD|35902250

3.3. System Design and Development 103

As indicated in the preceding, the rule in Figure 3.15 is only partially learned, because
instead of an exact applicability condition, it contains an upper and a lower bound for this
condition. The upper bound condition (represented as the larger ellipse from Figure 3.15)
corresponds to the most general generalization of the example (represented as the point
from the center of the two ellipses) in the context of the agent’s ontology, which is used as
a generalization hierarchy for learning. The lower bound condition (represented as the
smaller ellipse) corresponds to the least general generalization of the example.
The next phase is to refine the learned rules and, at the same time, test the agent with
new hypotheses. Therefore, the subject matter expert will formulate new hypotheses,
for example:

Dan Smith would be a good PhD advisor for Bob Sharp.

Using the learned rules, the agent will automatically generate the reasoning tree from
Figure 3.16. Notice that, in this case, the area of common interest/expertise of Dan Smith
and Bob Sharp is Information Security. The expert will have to check each reasoning step.
Those that are correct represent new positive examples that are used to generalize the
lower bound conditions of the corresponding rules. Those that are incorrect are used as
negative examples. The expert will interact with the agent to explain to it why a reasoning
step is incorrect, and the agent will correspondingly specialize the upper bound condition
of the rule, or both conditions. During the rule refinement process, the rule’s conditions
will converge toward one another and toward the exact applicability condition. During this
process, the ontology may also be extended, for instance to include the new concepts used
to explain the agent’s error. Rule refinement will be discussed in detail in Chapter 10.

Dan Smith would be a good PhD advisor for Bob Sharp.

Which are the necessary conditions?

Bob Sharp should be interested in an area of expertise of Dan Smith, who should

stay on the faculty of George Mason University for the duration of the PhD

dissertation of Bob Sharp and should have the qualities of a good PhD advisor.

Bob Sharp is Dan Smith will stay on the faculty Dan Smith would be a

interested in an of George Mason University for good PhD advisor with

area of expertise the duration of the PhD respect to the PhD advisor

of Dan Smith. dissertation of Bob Sharp. quality criterion.

Is Bob Sharp interested in an Which are the necessary quality criteria for a good PhD advisor?

area of expertise of Dan Smith?

professional reputation criterion, quality of student results criterion,

Yes, Information Security. and student learning experience criterion.

It is certain that Bob Sharp Dan Smith would be a good Dan Smith would be a good Dan Smith would be a good

is interested in an area of PhD advisor with respect to PhD advisor with respect to PhD advisor with respect to

expertise of Dan Smith. the professional reputation the quality of student the student learning

criterion. results criterion. experience criterion.

Which is a professional reputation criterion?

research funding criterion

Dan Smith would be a good PhD advisor with

respect to the research funding criterion.

Figure 3.16. Reasoning tree automatically generated by the agent.

 lOMoARcPSD|35902250

104 Chapter 3. Methodologies and Tools

3.3.5 Hierarchical Organization of the Knowledge Repository

Figure 3.17 illustrates the organization of a knowledge repository developed with the
Disciple-EBR learning agent shell, which was trained to assess different types of hypoth-
eses in different domains.
The top of the knowledge repository is the Shared KB, which contains general know-
ledge for evidence-based reasoning applicable in all the domains.
Under the Shared KB are Domain KBs, each corresponding to a different application
domain. One of these domain KBs will correspond to PhD advisor assessment. This PhD
advisor domain KB will contain general knowledge for assessing a PhD advisor, such as the
criteria for assessing a PhD advisor shown in Figure 3.7, and the learned reasoning rules. This
knowledge is, in principle, applicable for assessing any PhD advisor from any university.
Let us assume that we are using this agent to assess PhD advisors at three different
universities: George Mason University, Old Dominion University, and the University of
Virginia. How many scenario knowledge bases do we need and what knowledge will be
stored in each of them?
For each university there is at least one Scenario KB containing knowledge specific to
that university, such as individual professors and their characteristics.
When we need to assess PhD advisors at a specific university, the agent will use only the
corresponding Scenario KB, the Domain KB for PhD advisor assessment, and the Shared
KB for evidence-based reasoning.
The left-hand side of Figure 3.17 shows a reasoning step in the assessment of John Doe.
In which knowledge base will the following elements be represented: John Doe, Bob Sharp,
Artificial Intelligence, and certain?
John Doe and Bob Sharp are individuals at a specific university and are therefore
represented as specific instances in the Scenario KB corresponding to that university.
Artificial Intelligence is an instance to be used in reasoning in the context of any
university and is therefore represented as a generic instance in the Domain KB corres-
ponding to the assessment of PhD advisors.

Bob Sharp is Shared KB Domain-independent

interested in an Evidence-based concepts, generic

area of expertise reasoning knowledge

kn instances, and rules

of John Doe.

Is Bob Sharp interested

in an area of expertise of
Domain-
John Doe?
dependent
generic i
Domain KB Domain KB
Yes, Artificial Intelligence.
concepts,
instance Domain knowledge … Domain knowledge
generic
adv
(PhD advisor) (Insider threat)
It is certain that Bob instances,

Sharp is interested in and rules

the area of expertise

of John Doe.

specific
Domain-
instance i
Scenario KB Scenario KB
dependent
Scenario knowledge … Scenario knowledge
specific
(George Mason Univ.) (Univ. of Virginia)
instances

Figure 3.17. Organization of the knowledge repository of a Disciple-EBR agent.

 lOMoARcPSD|35902250

3.3. System Design and Development 105

Finally, certain is a constant to be used in evidence-based reasoning in any application

domain and is therefore represented in the Shared KB.

3.3.6 Learning-based Design and Development Phases

Figure 3.18 summarizes the main phases of agent development when using learning
technology. The first phase is agent specification, during which a knowledge engineer and
a subject matter expert define the types of hypotheses to assess (or problems to be solved)
by the agent. For example, the hypotheses might be to assess potential PhD advisors.
The second phase is rapid prototyping, where the knowledge engineer supports the
subject matter expert to develop reasoning trees for specific but representative hypotheses
(or problems). An example of a reasoning tree is the one from Figure 3.8. In this reasoning
tree, a complex hypothesis is assessed as illustrated in Figure 3.11 by:

Agent PhD advisor

specification assessment

P1 S1
P1 S1
QuestionP1 S1
Question
Answer
Question
Answer
Question
Answer
Question
Answer
Question
Answer
Rapid Answer IF the problem to solve is P1g
P1 S1
1 1 P 1 S1
n n
IF the problem to solve is P1g
IF the problem to solve is P1g
IF the problem to solve is P1g
P11 S11 Pn1 S1n THEN solve its sub-problems
THEN solve its sub-problems

prototyping
P11 S11 Pn1 S1n 1 … P
P1g 1
THEN
Png
1 … P solve
1
THEN
its sub-problems
solve its sub-problems
Question 1g Png
1 … P 1
Question
Answer Examples: E1, … , Ek 1g Png
1 … P1
ng
Question
Answer Examples: E1, … , Ek 1g
Question
Answer Examples: E1, … , Ek
Question Examples: E1, … , Ek
Answer
Question
Answer
P21 S2 Answer
2 2
1 Pm Sm
P21 S21 2
Pm 2
Sm
P21 S21 2
Pm 2
Sm

actor

Ontology
person

organization

development university employee student

graduate educational
faculty member staff member company
student organization

instructor professor PhD advisor PhD student college university

actor

IF the problem to solve is P 1g

person
IF the problem to solve is P organization
1g
IF the problem to solve is P
1g
Condition IF the problem to solve is P
1g
Condition IF the problem to solve is P
1g
Condition

Rule learning and

IF the problem to solve is
P
Except-When Condition 1g
Condition
… Except-When Condition university employee student
Condition
… Except-When Condition
Condition
… Except-When Condition
Except-When Condition
… Except-When Condition
Except-When Condition educational

ontology refinement
… Except-When Condition company
Except-When Condition organization
… faculty member staff member
Except-When Condition
THEN solve its sub-problems graduate
Except-When Condition
THEN solve its sub-problems
P
1 1
… THEN
P Except-When Condition student
1g 1ng 1
solve its sub-problems
P … THEN
P solve its sub-problems
1ng 1
1g
P … THEN
P solve its sub-problems
1g 1ng 1
P … THEN
P solve its sub-problems instructor professor PhD advisor
1g 1ng 1
P … P
1g 1 ng 1
P … P college university
1g ng

assistant associate full

MS student PhD student
professor professor professor

P1 S1
P1 S1
QuestionP1 S1
Question
Answer IF the problem to solve is P1g
Question
Answer IF the problem to solve is P1g
Question
Answer
Question IF the problem to solve is P1g
Answer THEN solve its sub-problemsIF the problem to solve is P1g
Question
Agent use and
Answer THEN
1 … P
P1g
solve
1 its sub-problems
Answer THEN solve its sub-problems
P1 S1
1 P
1
1 S1
n n
Png
1 … P
1g
1
THEN
Png
1 … P solve
1 its sub-problems
Examples: E1, … , Ek 1g Png
1 … P1
P11 S11 Pn1 S1n Examples: E1, … , Ek 1g
ng

personalization P1 S1 Pn1 S1n Examples: E1, … , Ek

Question 1 1 Examples: E1, … , Ek
Question
Answer
Question
Answer
Question
Answer
Question
Answer
Question
Answer
P2 S2 Answer
2
1 P 1 S2 m m
P21 S21 2
Pm 2
Sm
P21 S21 2
Pm 2
Sm
actor

person
IF the problem to solve is P1g organization

IF the problem to solve is P1g

IF the problem to solve is P1g
Condition IF the problem to solve is P1g
Condition IF the problem to solve is P1g
Condition
Except-When Condition IF the problem to solve is P1g university employee student

Condition IF the problem to solve is P1g

… Except-When
… Except-When
Condition Condition
Condition IF the problem to solve is P1g
Agent
Condition educational
… Except-When
Except-When Condition
… Except-When
Condition Condition
Condition organization
company

Except-When Condition Condition faculty member staff member

… Except-When
Except-When Condition Condition graduate undergraduate

THEN solve its sub-problemsExcept-When … Except-When

Condition Condition student student

… Except-When Condition
optimization
THEN Except-When Condition
1 … P1
P1g
solve
THEN
its sub-problems
solve its sub-problemsExcept-When …
Condition
Png its sub-problemsExcept-When Condition
1 … P 1
THEN solve instructor professor PhD advisor

Png its sub-problemsExcept-When Condition

1g 1 … P 1
THEN solve
1g Png
1 … P 1
THEN solve its sub-problems BS student college university
1g Png
1 … P 1
THEN solve its sub-problems
1g Png
1 … P 1
THEN solve its sub-problems
1g Png
1 … P
1g
1
ng
1 1 assistant associate full
P1g … Png professor professor professor
MS student PhD student

Figure 3.18. Main phases of agent development when using learning technology.
 lOMoARcPSD|35902250

106 Chapter 3. Methodologies and Tools

 Successively reducing it, from the top down, to simpler and simpler hypotheses
 Assessing the simplest hypotheses
 Successively combining, from the bottom up, the assessments of the subhypotheses,
until the assessment of the top-level hypothesis is obtained

During this phase, the agent also learns reduction patterns.

The next phase is that of ontology development. The guiding question is: What are
the domain concepts, relationships, and instances that would enable the agent to
automatically generate the reasoning trees developed during rapid prototyping, as well
as similar ones? From each reasoning step, the knowledge engineer and the subject
matter expert identify the ontology elements mentioned in it, as illustrated in Figure 3.12.
Such semantic network fragments represent a specification of the needed ontology.
In particular, this fragment suggests the need for the ontology fragment shown in
Figure 3.13. Based on such specifications, and using the ontology development tools
of the Disciple-EBR shell, the knowledge engineer develops an ontology that is suffi-
ciently complete. As part of ontology development, a knowledge engineer may reuse
concepts and relationships from previously developed ontologies, including those on
the Semantic Web.
The next phase in agent development is that of rule learning and ontology refinement.
From each hypothesis reduction step of a reasoning tree developed during rapid proto-
typing, the agent will learn a general hypothesis reduction rule by using the ontology as a
generalization hierarchy. This was illustrated in Figures 3.14 and 3.15. During this phase,
the expert teaches the agent to assess other similar hypotheses. The expert instantiates a
learned hypothesis pattern, such as “Dan Smith would be a good PhD advisor for Bob
Sharp,” and the agent automatically generates a reasoning tree, by applying the learned
rules, as was illustrated in Figure 3.16. The expert then critiques the agent’s reasoning,
guiding it in refining the rules.
Now the assistant is ready for the typical end-user, as part of the next phase, agent use
and personalization. Typically, the user will specify the hypothesis to assess by simply
instantiating a corresponding pattern. Then the agent will automatically generate a reduc-
tion tree like the one in Figure 3.16 by automatically applying the learned reduction rules.
This tree reduces the top-level hypothesis to elementary hypotheses to be directly assessed
based on knowledge and evidence. The assessments of these elementary hypotheses are
then automatically combined, from the bottom up, until the probability of the initial
hypothesis is obtained.
The end-user may also assess an entirely new hypothesis, but in this case he or she also
needs to indicate its decomposition, either to elementary hypotheses or to known hypotheses.
So when does the additional learning take place? It may be the case that the agent does
not know how to decompose a specific hypothesis. In such a case, the user can indicate
the desired decomposition, which will be used in the current analysis. As a result, the
agent will automatically learn a decomposition pattern, such as the one from the right-
hand side of Figure 3.9.
As mentioned previously, an important difference between a learned rule and a learned
pattern concerns its use in problem solving. A learned rule is always automatically applied,
while a pattern will be proposed to the user who may decide to select it and instantiate it
appropriately. If this is done, the corresponding instances will be added to the pattern.
Thus this type of learning is much simpler for the end-user, who only needs to specify, in
natural language, the decomposition of a hypothesis into simpler hypotheses.
 lOMoARcPSD|35902250

3.4. Hands On 107

Periodically, the agent can undergo an optimization phase, which is the last phase in
Figure 3.18. During this phase, the knowledge engineer and the subject matter expert will
review the patterns learned from the end-user, will learn corresponding rules from them,
and will correspondingly refine the ontology. The current version of the Disciple-EBR shell
reapplies its rule and ontology learning methods to do this. However, improved methods
can be used when the pattern has more than one set of instances, because each represents
a different example of the rule to be learned. This is part of future research.
Figure 3.19 compares the conventional knowledge engineering process of developing a
knowledge-based agent (which was discussed in Section 3.1) with the process discussed in
this section, which is based on the learning agent technology.
The top part of Figure 3.19 shows the complex knowledge engineering activities that are
required to build the knowledge base. The knowledge engineer and the subject matter
expert have to develop a model of the application domain that will make explicit the way
the subject matter expert assesses hypotheses. Then the knowledge engineer has to
develop the ontology. He or she also needs to define general hypotheses decomposition
rules and to debug them, with the help of the subject matter expert.
As shown at the bottom of Figure 3.19, each such activity is replaced with an equivalent
activity that is performed by the subject matter expert and the agent, with limited assistance
from the knowledge engineer. The knowledge engineer still needs to help the subject matter
expert to define a formal model of how to assess hypotheses and to develop the ontology.
After that, the subject matter expert will teach the agent how to assess hypotheses, through
examples and explanations, and the agent will learn and refine the rules by itself.
The next chapters discuss each phase of this process in much more detail.

3.4 HANDS ON: LOADING, SAVING, AND CLOSING

KNOWLEDGE BASES

The knowledge bases developed with Disciple-EBR are located in the repository folder,
which is inside the installation folder. As shown in Figure 3.17, the knowledge bases are

Subject
ct Dialogue
Conventional
Matter
er Knowledge
Knowledge Engineer
Engineering
Expert
rtt
Programming
Model Define Verify and
Understand Develop
problem reasoning update rules
domain ontology
solving rules and ontology

Develop
D Define concepts Provide and Analyze
Explain
reasoning with hierarchical explain agent’s
errors
model organization examples solutions

Knowledge
Assist in Extend ontology Learn Learn
Engineer Refine
(support role)) solution based on modeled reasoning ontology
rules
development solutions rules elements

Learning-based Subject Mixed-Initiative

Knowledge Matter Dialogue
COGNITIVE ASSISTANT
Engineering Expert

Figure 3.19. Conventional knowledge engineering versus learning-based knowledge engineering.

 lOMoARcPSD|35902250

108 Chapter 3. Methodologies and Tools

organized hierarchically, with the knowledge base for evidence-based reasoning at the top
of the hierarchy. The user cannot change this knowledge base, whose knowledge elements
are inherited in the domain and scenario knowledge bases. From the user’s point of view,
each knowledge base consists of a top-level domain part (which contains knowledge
common to several applications or scenarios in a domain) and one scenario part (con-
taining knowledge specific to a particular application or scenario). As illustrated in
Figure 3.17, there can be more than one scenario under a domain. In such a case, the
domain and each of the scenarios correspond to a different knowledge base. Loading,
saving, or closing a scenario will automatically load, save, or close both the scenario part
and the corresponding domain part of the knowledge base.
Loading and selecting a knowledge base are described in Operation 3.1 and illustrated
in Figure 3.20.

Operation 3.1. Load and select a knowledge base

 In the System Workspace Manager, click on the knowledge base icon containing the
plus sign (+).
 The Knowledge Manager window opens, showing all the knowledge bases from the
repository.
 Click on the plus sign (+) of the domain knowledge base to be loaded, to display all its
scenario knowledge bases.
 Click on the scenario knowledge base to be loaded.
 Click on the Select button. This both loads the scenario and domain KBs and selects
them as the current ones to work with. Their names will be shown in bold in the
Knowledge Manager window.

1. Click on this icon to load a KB

2. System displays all KBs

3. Click on + to expand the KB to be loaded

5. Click on Select

4. Click on Scen

Figure 3.20. Loading and selecting a knowledge base.

 lOMoARcPSD|35902250

3.4. Hands On 109

Once a knowledge base is selected, you can invoke different modules of Disciple-EBR to
use it. Each module is accessible in a specific workspace. As illustrated in Figure 3.21,
there are three workspaces:

Evidence workspace: “Evidence: repository\KB-name\Scen”

Scenario workspace: “repository\KB-name\Scen”
Domain workspace: “repository\KB-name”

The user can switch between the workspaces to use the corresponding modules. For
example, you must switch to the Evidence workspace to work with evidence items. Then,
to save the knowledge base, you must switch to the Scenario workspace.
The user should work with only one set of the three workspaces at a time (correspond-
ing to the same KB). Therefore, you should close the workspaces corresponding to a
knowledge base (by clicking on the icon with the minus sign [–]) before opening the
workspaces corresponding to another knowledge base.
The steps to save all the knowledge bases loaded in memory are described in Operation
3.2 and illustrated in Figure 3.22.

Operation 3.2. Save all the knowledge bases

 Select the Scenario workspace (“repository\KB-name\Scen”).
 Select the System menu.
 Select Save All.

You may click on this icon to close a workspace

Scenario workspace

Domain workspace
Evidence workspace

Figure 3.21. Workspaces in Disciple-EBR.

. Select Scenario workspace

2. Scenario workspace

3. Select System
Save All

Figure 3.22. Saving all the knowledge bases.

 lOMoARcPSD|35902250

110 Chapter 3. Methodologies and Tools

It is highly recommended to have only one knowledge base loaded in memory.

Therefore, before loading a new knowledge base, you should close all the opened
ones by following the instructions described in Operation 3.3 and illustrated in
Figure 3.23.

Operation 3.3. Close a knowledge base

 Click on the knowledge base icon containing the plus sign (+) situated at the right of
the workspace selector.
 The Knowledge Manager window opens, showing all the knowledge bases.
 Click on the scenario knowledge base to be closed.
 Click on the Close button in the right side of the window.
 Click on the X button to close the Knowledge Manager window.

The predefined knowledge base “00-Reference-KB” contains general knowledge for

evidence-based reasoning, an empty domain, and an empty scenario. Users can create
their knowledge bases as renamed copies of “00-Reference-KB,” as indicated in Operation
3.4. They should never work with “00-Reference-KB,” which should be kept as a reference
knowledge base.

Operation 3.4. Create a user knowledge base

 Open the “repository” folder from the “installation” directory in Windows Explorer.
 Make a copy of the entire “00-Reference-KB” folder and give it a new name.
 Use the Disciple-EBR modules to develop the newly created knowledge base.

. Click on this icon

2. System displays all KBs

3. Click on the Scen KB to be closed

5. Click on the x button to close the window

4. With Scen selected, click on Close

Figure 3.23. Closing a knowledge base.

 lOMoARcPSD|35902250

3.6. Project Assignment 2 111

3.5 KNOWLEDGE BASE GUIDELINES

The following are knowledge engineering guidelines for knowledge base development.

Guideline 3.1. Work with only one knowledge base loaded in memory
To maintain the performance of the Disciple-EBR modules, work with only one knowledge
base loaded in memory. Therefore, close all the knowledge bases before loading a new one.
If several knowledge bases are loaded, work only with the set of three workspaces
corresponding to the knowledge base you are currently using. Close all the other
workspaces.

Guideline 3.2. Create a knowledge base and save successive versions

A user can introduce errors into the knowledge base while developing it. When Disciple-
EBR attempts to load a knowledge base, it makes a rigorous check of its correctness and
will not load the knowledge base if it contains errors. It is therefore important to save
successive versions of the knowledge base being developed in order to fall back to a
previous version in case the latest one contains errors.
Create your knowledge base by making a copy of “00-Reference-KB” and giving it a new
name, for instance “WA.” From now on, you will work only with “WA,” but as you develop it,
you will have to save successive copies of it with different names, as explained in the following.
Suppose that you have developed “WA” to contain part of the modeling for a hypothesis.
Save “WA” in Disciple-EBR, then using Windows Explorer make a copy of it, and rename the
copy as “WA-1m.” Continue working with “WA” and expand the modeling of the hypothesis.
Save and make a copy of “WA.” Then rename the copy as “WA-2m,” and so on.
Through such a process, you will save a sequence of knowledge bases: “WA-1m,” “WA-
2m,” “WA-3o,” “WA-4o,” and so on, each corresponding to a given stage in your develop-
ment of “WA.” In this way, if your “WA” knowledge base is damaged for any reason, you can
always resume from the most recently saved version, as illustrated in the following scenario:

 “WA” has errors, and the most recently saved version is “WA-4o.”
 Delete “WA” in Windows Explorer, copy “WA-4o,” and rename this copy as “WA.”
 Continue with the development of “WA.”

3.6 PROJECT ASSIGNMENT 2

Finalize the project team, specify the type of hypotheses to be analyzed by the agent to
be developed, and study the application domain. Prepare a short presentation of the
following:

 The application domain of your agent and why your agent is important.
 A bibliography containing the expertise domain and your current familiarity with the
domain, keeping in mind that you should choose a domain where you already are or
could become an expert without investing a significant amount of time.
 Three examples of hypotheses and a probability for each.
 lOMoARcPSD|35902250

112 Chapter 3. Methodologies and Tools

3.7 REVIEW QUESTIONS

3.1. Which are the main phases in the development of a knowledge-based agent?

3.2. Which are the required qualifications of a knowledge engineer?

3.3. Which are the required qualifications of a subject matter expert?

3.4. Briefly define “verification,” “validation,” and “certification.”

3.5. Consider the fact that a knowledge-based agent may need to have hundreds or
thousands of rules. What can be said about the difficulty of defining and refining
these rules through the conventional process discussed in Section 3.1.4?

3.6. Use the scenario from Section 3.1 to illustrate the different difficulties of building a
knowledge-based agent discussed in Section 1.6.3.1.

3.7. What is an expert system shell?

3.8. Which are different types of expert system shells?

3.9. What is a learning agent shell?

3.10. What is a learning agent shell for evidence-based reasoning?

3.11. What is the organization of the knowledge repository of a learning agent shell for
evidence-based reasoning?

3.12. What is the difference between a specific instance and a generic instance? Provide
an example of each.

3.13. Are there any mistakes in the reasoning step from Figure 3.24 with respect to the
goal of teaching the agent? If the answer is yes, explain and indicate corrections.

3.14. Which are the main stages of developing a knowledge-based agent using learning
agent technology?

3.15. Compare the manual knowledge engineering process of developing a knowledge-

based agent, as described in Section 3.1, with the process using learning agent
technology, described in Section 3.3.

John Doe will stay on the faculty of

George Mason University for the duration

of the PhD dissertation of Bob Sharp.

Is John Doe likely to stay on the faculty of George Mason

University for the duration of the PhD dissertation of Bob Sharp?

Yes

It is almost certain that John Doe is likely to stay

on the faculty of George Mason University for the

duration of the PhD dissertation of Bob Sharp.

Figure 3.24. Reduction step.

 lOMoARcPSD|35902250

Modeling the Problem-Solving

4 Process

4.1 PROBLEM SOLVING THROUGH ANALYSIS AND SYNTHESIS

Analysis and synthesis, introduced in Section 1.6.2, form the basis of a general divide-and-
conquer problem-solving strategy that can be applied to a wide variety of problems. The
general idea, illustrated in Figure 4.1, is to decompose or reduce a complex problem P1 to
n simpler problems P11, P12, . . . , P1n, which represent its components. If we can then find
the solutions S11, S12, . . . , S1n of these subproblems, then these solutions can be combined
into the solution S1 of the problem P1.
If any of the subproblems is still too complex, it can be approached in a similar way, by
successively decomposing or reducing it to simpler problems, until one obtains problems
whose solutions are known, as illustrated in Figure 4.2.
Figures 4.3 and 4.4 illustrate the application of this divide-and-conquer approach to
solve a symbolic integration problem. Notice that each reduction and synthesis operation
is justified by a specific symbolic integration operator.

4.2 INQUIRY-DRIVEN ANALYSIS AND SYNTHESIS

Cognitive assistants require the use of problem-solving paradigms that are both natural
enough for their human users and formal enough to be automatically executed by the
agents. Inquiry-driven analysis and synthesis comprise such a problem-solving paradigm
where the reduction and synthesis operations are guided by corresponding questions and
answers. The typical questions are those from Rudyard Kipling’s well-known poem “I Keep
Six Honest . . .”

P1 S 1

P11 S11 P12 S12 … P1n S1n

Figure 4.1. Problem reduction (decomposition) and solutions synthesis (composition).

113
 lOMoARcPSD|35902250

114 Chapter 4. Modeling the Problem-Solving Process

I keep six honest serving-men

(They taught me all I knew);
Their names are What and Why and When
And How and Where and Who.

A complex problem P is P1 S1
1

solved by:

• Successively reducing

it to simpler and

simpler problems;
P11 S11 P12 S12 … P1n S1n
• Finding the solutions

of the simplest

problems;

• Successively

combining these P1n1 S1n1 … P1nm S1nm

solutions, from

bottom up, until the

solution of the initial

problem is obtained

(synthesized). P1n11 S1n11 … P1n1q S1n1q

Figure 4.2. Problem solving through analysis and synthesis.

Figure 4.3. Reduction of a symbolic integration problem.

 lOMoARcPSD|35902250

4.2. Inquiry-Driven Analysis and Synthesis 115

Figure 4.4. Symbolic integration through analysis and synthesis.

Problem 1

Question Q
Question Q
Answer B
Answer A

Problem 2 Problem 3 Problem 4 Problem 5

Figure 4.5. Inquiry-driven problem reduction.

We have already illustrated this paradigm in Sections 2.2 and 3.3.2. To better understand
it, let us consider the simple abstract example from Figure 4.5. To solve Problem 1, one
asks Question Q related to some aspect of Problem 1. Let us assume that there are two
answers to Q: Answer A and Answer B. For example, the question, “Which is a sub-
criterion of the quality of student results criterion?” has two answers, “publications with
advisor criterion” and “employers of graduates criterion.”
Let us further assume that Answer A leads to the reduction of Problem 1 to two simpler
problems, Problem 2 and Problem 3. Similarly, Answer B leads to the reduction of
Problem 1 to the other simpler problems, Problem 4 and Problem 5.
Let us now assume that we have obtained the solutions of these four subproblems. How
do we combine them to obtain the solution of Problem 1? As shown in Figure 4.6, first the
 lOMoARcPSD|35902250

116 Chapter 4. Modeling the Problem-Solving Process

solutions to Problem 2 and Problem 3 are combined to obtain a solution to Problem

1 corresponding to Answer A (called Solution A). Also the solutions to Problem 4 and
Problem 5 are combined to obtain a solution to Problem 1 corresponding to Answer
B (called Solution B). Each of these two synthesis (or composition) operations is called a
reduction-level synthesis because it corresponds to a specific reduction of the top-level
problem. Second, we need to combine Solution A with Solution B into the final solution of
Problem 1. This synthesis operation is called problem-level synthesis because it corres-
ponds to all reductions of Problem 1.
Figures 4.7 and 4.8 illustrate inquiry-driven analysis and synthesis in the context of the
military center of gravity (COG) determination problem (Tecuci et al., 2005b; 2008b),
which will be discussed in Section 12.4. Figure 4.7 shows the reduction of a problem to
three simpler problems, guided by a question/answer pair.
Figure 4.8 illustrates the composition of the solutions of the three subproblems
into the solution of the problem. One aspect to notice is that the reduction-level
synthesis operation is guided by a question/answer pair, while the problem-level syn-
thesis operation is not. Thus the use of questions and answers is actually optional.

Problem 1

Solution

of Problem 1

Problem-

level

synthesis
Question Q
Question Q
Answer B
Answer A

Solution B
Solution A
of Problem 1
of Problem 1

Reduction-

level

synthesis
Problem 2 Problem 3 Problem 4 Problem 5

Solution Solution Solution Solution

of Problem 2 of Problem 3 of Problem 4 of Problem 5

Figure 4.6. A more detailed view of the analysis and synthesis process.

Test whether President Roosevelt has the critical capability to maintain support.

Which are the critical requirements for President Roosevelt to maintain support?

President Roosevelt needs means to secure support from the government, means to

secure support from the military, and means to secure support from the people.

Test whether President Roosevelt Test whether President Roosevelt Test whether President Roosevelt

has means to secure support from has means to secure support from has means to secure support from

the government. the military. the people.

Figure 4.7. Illustration of a reduction operation in the COG domain.

 lOMoARcPSD|35902250

4.2. Inquiry-Driven Analysis and Synthesis 117

Test whether President Roosevelt has the critical capability to maintain support.

President Roosevelt has the critical capability to maintain support because President

Roosevelt has means to secure support from the government, has means to secure

support from the military, and has means to secure support from the people.

critical capability is

an emerging property

President Roosevelt has the critical capability to

Which are the critical requirements for President
maintain support because President Roosevelt
Roosevelt to maintain support? President Roosevelt

needs means to secure support from the government, has means to secure support from the government,

means to secure support from the military, and means has means to secure support from the military, and

to secure support from the people.

has means to secure support from the people.

Does President Roosevelt have the critical capability to maintain support?

Yes, because President Roosevelt has all the needed critical requirements.

Test whether President Roosevelt has Test whether President Roosevelt Test whether President Roosevelt

means to secure support from the has means to secure support from has means to secure support from

government. the military. the people.

President Roosevelt has means to President Roosevelt has means to President Roosevelt has means to

secure support from the government. secure support from the military. secure support from the people.

Figure 4.8. Illustration of reduction and synthesis operations in the COG domain.

Moreover, it is assumed that the questions guiding the synthesis operations may have
only one answer, which typically indicates how to combine the solutions. Allowing more
questions and more answers in the synthesis tree would lead to a combinatorial explo-
sion of solutions.
Another interesting aspect is that the three leaf solutions in Figure 4.8 are about the
means of President Roosevelt, while their composition is about a capability. Thus this
illustrates how synthesis operations may lead to emerging properties.
A third aspect to notice is how the reduction-level composition is actually performed. In
the example from Figure 4.8, the solutions to combine are:

President Roosevelt has means to secure support from the government.

President Roosevelt has means to secure support from the military.
President Roosevelt has means to secure support from the people.

The synthesized solution is obtained by concatenating substrings from these solutions

because, as indicated in corresponding question/answer pair, President Roosevelt has all
the needed critical requirements:

President Roosevelt has the critical capability to maintain support because President
Roosevelt has means to secure support from the government, has means to secure
support from the military, and has means to secure support from the people.

The preceding synthesis operation in the interface of Disciple-COG is shown in

Figure 12.28 (p. 373) from Section 12.4.2.
 lOMoARcPSD|35902250

118 Chapter 4. Modeling the Problem-Solving Process

Figure 4.9 shows another example of reduction and synthesis in the COG domain. In
this case, the solutions to combine are:

PM Mussolini has means to secure support from the government.

PM Mussolini has means to secure support from the military.
PM Mussolini does not have means to secure support from the people.

In this case, the synthesized solution is no longer obtained by concatenating substrings

from these solutions because, as indicated in corresponding question/answer pair, PM
Mussolini does not have all the needed critical requirements:

PM Mussolini does not have the critical capability to maintain support because PM
Mussolini does not have means to secure support from the people.

Additional examples of solution synthesis from the COG domain are presented in
Figures 12.27 (p. 372), 12.28 (p. 373), 12.29 (p. 373), and 12.30 (p. 374) from Section 12.4.2.
As suggested by the preceding examples, there are many ways in which solutions may
be combined.
One last important aspect related to problem solving through analysis and synthesis is
that the solutions of the elementary problems may be obtained by applying any other type of
reasoning strategy. This enables the solving of problems through a multistrategy approach.
Chapter 12 presents Disciple cognitive assistants for different types of tasks, illustrating
the use of the inquiry-driven analysis and synthesis in different domains. Section 12.2
discusses this problem-solving paradigm in the context of military engineering planning.
Section 12.3 discusses it in the context of course of action critiquing. Section 12.4 discusses
it in the context of center of gravity analysis, and Section 12.5 discusses it in the context of
collaborative emergency response planning.

Test whether PM Mussolini has the critical capability to maintain support.

PM Mussolini does not have the critical capability to maintain support because

PM Mussolini does not have means to secure support from the people.

Which are the critical requirements for PM Mussolini to PM Mussolini does not have the critical

maintain support? PM Mussolini needs means to secure capability to maintain support because PM

support from the government, means to secure support from Mussolini does not have means to secure

the military, and means to secure support from the people. support from the people.

Does PM Mussolini have the critical capability to maintain support?

No, because PM Mussolini does not have all the needed critical requirements.

Test whether PM Mussolini has Test whether PM Mussolini has Test whether PM Mussolini has

means to secure support from the means to secure support from means to secure support from the

government. the military. people.

PM Mussolini has means to secure PM Mussolini has means to PM Mussolini does not have means

support from the government. secure support from the military. to secure support from the people.

Figure 4.9. Another illustration of reduction and synthesis operations in the COG domain.
 lOMoARcPSD|35902250

4.3. Evidence-based Reasoning 119

4.3 INQUIRY-DRIVEN ANALYSIS AND SYNTHESIS FOR

EVIDENCE-BASED REASONING

4.3.1 Hypothesis Reduction and Assessment Synthesis

In this section, we discuss the specialization of the inquiry-driven analysis and synthesis
paradigm for evidence-based reasoning where one assesses the probability of hypotheses
based on evidence, as was illustrated in Section 2.2. In this case, a complex hypothesis is
assessed by:

 Successively reducing it, from the top down, to simpler and simpler hypotheses
(guided by introspective questions and answers).
 Assessing the simplest hypotheses based on evidence.
 Successively combining, from the bottom up, the assessments of the simpler hypoth-
eses, until the assessment of the top-level hypothesis is obtained.

Figure 4.10 shows a possible analysis of the hypothesis that Country X has nuclear
weapons.

Country X has nuclear weapons.

likely

max

What are the necessary and sufficient conditions?

Reasons, desires, and capabilities.

likely

min

Country X has reasons to Country X desires Country X has the capability

have nuclear weapons. nuclear weapons. to obtain nuclear weapons.

almost certain very likely likely

max

Which is a strategy to obtain Which is a strategy to obtain

nuclear weapons? Build nuclear weapons? Buy

likely no support

max max

Country X can build Country X can buy

nuclear weapons. nuclear weapons.

likely no support

max

Which is an indicator?

Capability to produce enriched uranium.

likely

very likely indicator

Country X can produce

enriched uranium.

likely

Figure 4.10. An example of different types of reductions and corresponding synthesis functions.
 lOMoARcPSD|35902250

120 Chapter 4. Modeling the Problem-Solving Process

As pointed out by Alvin Toffler (1984, p. xi):

One of the most highly developed skills in contemporary Western civilization

is dissection; the split-up of problems into their smallest possible compon-
ents. We are good at it. So good, we often forget to put the pieces back
together again.

Evidence-based reasoning offers an opportunity to simplify the more complex synthesis

process. Indeed, in this case, the solution of a hypothesis may no longer need to be a complex
phrase or expression, but just its probability of being true, given the available evidence. The
actual solution synthesis function depends on the type of reduction strategy used.
In the following section, we will review different types of reduction strategies and the
corresponding synthesis functions.

4.3.2 Necessary and Sufficient Conditions

Ideally, a hypothesis would be reduced to several subhypotheses that would represent
necessary and sufficient conditions. That is, the hypothesis is true if and only if all the
subhypotheses are true. An example is the top-level reduction from Figure 4.10, where the
top-level hypothesis is reduced to three subhypotheses. Let us assume that we have
obtained the following assessments of these subhypotheses: almost certain, very likely,
and likely, respectively. Then the assessment of the top hypothesis, corresponding to this
necessary and sufficient condition, is the minimum of the three assessments (i.e., likely),
because each of the three subhypotheses would need to be true to ensure that the top-
level hypothesis is true. This value and the minimum (min) function that produced it are
associated with the question/answer pair.
In general, as will be illustrated later in this chapter, there may be more than one
strategy to reduce a hypothesis to simpler hypotheses, each resulting in a possibly different
assessment. In such a case, the assessment of the hypothesis should be taken as the
maximum of all these possible assessments. In this particular example, since we have only
one strategy, the assessment of the top-level hypothesis is max(likely) = likely.

4.3.3 Sufficient Conditions and Scenarios

Many times it is not easy or even possible to identify necessary and sufficient conditions to
reduce a hypothesis. In such a case, a second-best reduction would be a sufficient
condition. This means that if the subhypotheses are true, then the hypothesis is true.
But, as we have discussed previously, there may be more than one sufficient condition for
a hypothesis. For example, the middle part of Figure 4.10 shows two possible strategies for
Country X to obtain nuclear weapons: It can build them, or it can buy them. Each strategy
has to be assessed and the maximum assessment represents the assessment of the
hypothesis that Country X has the capability to obtain nuclear weapons.
A special type of a sufficient condition for a hypothesis is a scenario in which the
hypothesis would be true, such as the one illustrated in Figure 2.9 (p. 63) from Section
2.2.4. But, as we have discussed in Section 2.2.4, there may be multiple alternative
scenarios. Figure 4.11 shows an abstract example where there are two alternative scenarios
for Hypothesis 1 to be true. Scenario 1 consists of action 2 and action 3. For this scenario to
have happened, both these actions should have happened. Therefore, we combine their
 lOMoARcPSD|35902250

4.3. Evidence-based Reasoning 121

Hypothesis 1

Assessment of

Hypothesis 1

max

Which is a possible scenario? Which is a possible scenario?

Scenario 1 Scenario 2

Assessment 1 of Hypothesis 1 Assessment 2 of Hypothesis 1

min min

Hypothesis (action) 2 Hypothesis (action) 3 Hypothesis (action) 4 Hypothesis (action) 5

Assessment of Hypothesis 2 Assessment of Hypothesis 3 Assessment of Hypothesis 4 Assessment of Hypothesis 5

Figure 4.11. Reductions and syntheses corresponding to two sufficient conditions (scenarios).

corresponding assessments with a minimum function, as shown at the bottom-left of

Figure 4.11. Hypothesis 1, however, would be true if either of the two scenarios would
happen. Therefore, we combine the assessments corresponding to the two scenarios
through a maximum function.

4.3.4 Indicators
Many times when we are assessing a hypothesis, we have only indicators. For example, as
shown at the bottom part of Figure 4.10, having the capability to produce enriched uranium
is an indicator that a country can build nuclear weapons. An indicator is, however, weaker
than a sufficient condition. If we determine that a sufficient condition is satisfied (e.g., a
scenario has actually happened), we may conclude that the hypothesis is true. But we
cannot draw such a conclusion just because we have discovered an indicator. However, we
may be more or less inclined to conclude that the hypothesis is true, based on the relevance
(strength) of the indicator. Therefore, given the symbolic probabilities from Table 2.5, we
distinguish between three types of indicators of different relevance (strength): “likely indica-
tor,” “very likely indicator,” and “almost certain indicator.”
A “likely indicator” is one that, if discovered to be true, would lead to the conclusion
that the considered hypothesis is likely. Similarly, a “very likely indicator” would lead to the
conclusion that the hypothesis is very likely, and an “almost certain indicator” would lead to
the conclusion that the hypothesis is almost certain.
In the example from the bottom part of Figure 4.10 it is likely that Country X can
produce enriched uranium, and this is a very likely indicator that Country X can build
nuclear weapons. Therefore, we can conclude that the probability of the hypothesis
that Country X can build nuclear weapons is likely, the minimum between likely (the
probability of the indicator) and very likely (the strength of the indicator).
In general, the probability of a hypothesis H based on an indicator I is the minimum
between the probability of the indicator and the relevance (strength) of the indicator (which
could be likely, very likely, or almost certain).
 lOMoARcPSD|35902250

122 Chapter 4. Modeling the Problem-Solving Process

It makes no sense to consider the type “certain indicator,” because this would be a
sufficient condition. Similarly, it makes no sense to consider the type “no support indica-
tor,” because this would not be an indicator.
As an abstract example, Figure 4.12 shows a hypothesis that has two likely indicators,
A and B, if only one of them is observed. However, if both of them are observed, they
synergize to become an almost certain indicator.
As a concrete example, consider Person , who has been under surveillance in
connection with terrorist activities. We suspect that will attempt to leave the country
in a short while. Three days ago, we received information that sold his car. Today, we
received information that he closed his account at his bank. Each of these is only a likely
indicator of the hypothesis that plans to leave the country. He could be planning to buy
a new car, or he could be dissatisfied with his bank. But, taken together, these two
indicators suggest that it is almost certain that is planning to leave the country.
Coming back to the abstract example in Figure 4.12, let us assume that indicator A is
almost certain and indicator B is very likely. In such a case, the assessment of Hypothesis
1, based only on indicator A, is minimum(almost certain, likely) = likely. Similarly, the
assessment of Hypothesis 1, based only on indicator B, is minimum(very likely, likely) =
likely. But the assessment of Hypothesis 1, based on both indicators A and B, is
minimum(minimum(almost certain, very likely), almost certain) = very likely. Also, the
assessment of Hypothesis 1 based on all the indicators is the maximum of all the
individual assessments (i.e., very likely), because these are three alternative solutions
for Hypothesis 1.

4.4 EVIDENCE-BASED ASSESSMENT

Now we discuss the assessment of the leaf hypotheses of the argumentation structure,
based on the identified relevant evidence. Let us consider an abstract example where the
leaf hypothesis to be directly assessed based on evidence is Q (see Figure 4.13).
We begin by discussing how to assess the probability of hypothesis Q based only on one
item of favoring evidence Ek* (see the bottom of Figure 4.13). First notice that we call
this likeliness of Q, and not likelihood, because in classic probability theory, likelihood is

Hypothesis 1

very likely

max

Which is an indicator? Which is an indicator? Which is an indicator?

Indicator A Indicator B Indicators A and B

likely likely very likely

likely indicator likely indicator almost certain indicator

Indicator A Indicator B Indicator A Indicator B

almost certain very likely almost certain very likely

Figure 4.12. Hypothesis assessment based on indicators.

 lOMoARcPSD|35902250

4.4. Evidence-based Assessment 123

Q very likely

on balance

Inferential force of evidence on Q

c
Q based only on almost Q based only on
likely
favoring evidence certain disfavoring evidence

max

Inferential force of favoring evidence on Q

Q based almost Q based

likely
only on Ei* certain only on Ek*

min

Inferential force of Ek* on Q

How likely is Q based only on Ek*?

Relevance of Ek to Q likely Believability of Ek* very likely

How likely is Q, based only on Ek* How likely is it that Ek* is true?

and assuming that Ek* is true?

Figure 4.13. The relevance, believability, and inferential force of evidence.

P(Ek*|Q), while here we are interested in P(Q|Ek*), the posterior probability of Q given Ek*.
To assess Q based only on Ek*, there are three judgments to be made by answering
three questions:

The relevance question is: How likely is Q, based only on Ek* and assuming that Ek* is
true? If Ek* tends to favor Q, then our answer should be one of the values from likely
to certain. If Ek* is not relevant to Q, then our answer should be no support, because
Ek* provides no support for the truthfulness of Q. Finally, if Ek* tends to disfavor Q,
then it tends to favor the complement of Q, that is, Qc. Therefore, it should be used
as favoring evidence for Qc, as discussed later in this section.
The believability question is: How likely is it that Ek* is true? Here the answer should be
one of the values from no support to certain. The maximal value, certain, means
that we are sure that the event Ek reported in Ek* did indeed happen. The minimal
value, no support, means that Ek* provides us no reason to believe that the event
Ek reported in Ek* did happen. For example, we believe that the source of Ek* has
lied to us.
The inferential force or weight question is: How likely is Q based only on Ek*? The agent
automatically computes this answer as the minimum of the relevance and believ-
ability answers. What is the justification for this? Because to believe that Q is true
based only on Ek*, Ek* should be both relevant to Q and believable.

When we assess a hypothesis Q, we may have several items of evidence, some favoring Q
and some disfavoring Q. The agent uses the favoring evidence to assess the probability of
Q and the disfavoring evidence to assess the probability of Qc. As mentioned previously,
because the disfavoring evidence for Q is favoring evidence for Qc, the assessment process
for Qc is similar to the assessment for Q.
When we have several items of favoring evidence, we evaluate Q based on each of them
(as was explained previously), and then we compose the obtained results. This is illus-
trated in Figure 4.13, where the assessment of Q based only on Ei* (almost certain) is
 lOMoARcPSD|35902250

124 Chapter 4. Modeling the Problem-Solving Process

composed with the assessment of Q based only on Ek* (likely), through the maximum
function, to obtain the assessment of Q based only on favoring evidence (almost certain). In
this case, the use of the maximum function is justified because it is enough to have one
item of evidence that is both very relevant and very believable to persuade us that the
hypothesis Q is true.
Let us assume that Qc based only on disfavoring evidence is likely. How should we
combine this with the assessment of Q based only on favoring evidence? As illustrated at
the top of Figure 4.13, the agent uses an on-balance judgment: Because Q is almost certain
and Qc is likely, it concludes that, based on all available evidence, Q is very likely.
In general, as indicated in the right and upper side of Table 4.1, if the assessment of Qc
(based on disfavoring evidence for Q) is higher than or equal to the assessment of Q
(based on favoring evidence), then we conclude that, based on all the available evidence,
there is no support for Q. If, on the other hand, the assessment of Q is strictly greater than
the assessment of Qc, then the assessment of Q is decreased, depending on the actual
assessment of Qc (see the left and lower side of Table 4.1).
One important aspect to notice is that the direct assessment of hypotheses based on
favoring and disfavoring evidence is done automatically by the agent, once the user
assesses the relevance and the believability of evidence.
Another important aspect to notice is that the evaluation of upper-level hypotheses (such
as those from Figure 4.10) requires the user to indicate what function to use when
composing the assessments of their direct subhypotheses. This was discussed in Section 4.3.

4.5 HANDS ON: WAS THE CESIUM STOLEN?

To illustrate further the divide-and-conquer approach to hypothesis analysis, let us

continue with the cesium example introduced in Section 2.2, where we have already

Table 4.1 An “On-Balance” Synthesis Function

c
Q based only on disfavoring evidence
Q based

on all no very almost

likely certain
support likely certain
evidence
Q based only on favoring evidence

no no no no no no

support support support support support support

no no no no
likely likely
support support support support

very very no no no
likely
likely likely support support support

almost almost very no no

likely
certain certain likely support support

almost very no
certain certain likely
certain likely support
 lOMoARcPSD|35902250

4.5. Hands On: Was the Cesium Stolen? 125

established that the cesium-137 canister is missing (see Figure 2.8 on p. 62). The next step
is to consider the competing hypotheses:

H2: The cesium-137 canister was stolen.

H2’: The cesium-137 canister was misplaced.
H2”: The cesium-137 canister is used in a project without being checked out from the
XYZ warehouse.

We have to put each of these hypotheses to work, to guide the collection of relevant
evidence. In Section 2.2.4, we have already discussed, at a conceptual level, the collec-
tion of evidence for hypothesis H2. Table 4.2 shows the result of our information
collection efforts.
The collected information from Table 4.2 suggests that the cesium-137 canister was
stolen with the panel truck having Maryland license MDC-578. This has led to the
development of the analysis tree in Figure 2.9 (p. 63). In this case study, you are going
to actually perform this analysis. You have to identify the “dots” in the information from

Table 4.2 Additional Information on the Missing Cesium-137 Canister

INFO-003-Clyde: We talked to a professional locksmith named Clyde, who said that

the lock had been forced, but it was a clumsy job.
INFO-004-SecurityCamera: The security camera of the XYZ warehouse contains a
video segment showing a person loading a container into a U-Haul panel truck.
INFO-005-Guard: There is a security perimeter around the XYZ warehouse and
employee parking area having just one gate that is controlled by a guard. On the day before the
missing canister was observed, the security guard, Sam, recorded that a panel truck having
Maryland license plate MDC-578 was granted entry at 4:45 pm just before the XYZ closing hour
at 5:00 pm. The driver of this vehicle showed the guard a manifest containing items being
delivered to the XYZ warehouse. This manifest contained a list of packing materials allegedly
ordered by the XYZ Company. The vehicle was allowed to enter the parking area. At 8:30 pm,
this same vehicle was allowed to exit the parking area. A different guard was on duty in the
evenings and noticed that his records showed that this vehicle had been permitted entry, and so
he allowed the vehicle to exit the parking area.
INFO-006-TRUXINC: Maryland DOT’s record indicates that the panel truck carrying
the license plate number MDC-578 is registered in the name of a truck-rental company called
TRUXINC, located in Silver Spring, MD. The manager of this agency showed records indicating
that this truck was rented to a person who gave his name as Omer Riley, having as his
listed address 6176 Williams Ave. in Silver Spring. The truck was rented on the day before
Willard’s discovery about the missing cesium-137, and it was returned the day after he made
the discovery.
INFO-007-SilverSpring: Silver Spring city record according to which there is no
residence at 6176 Williams Ave. in Silver Spring, MD.
INFO-008-InvestigativeRecord: An examination of the panel truck rented by Omer
Riley, using a Geiger counter, revealed minute traces of cesium-137.
INFO-009-Grace: Grace, the Vice President for Operations at XYZ, tells us that no one
at the XYZ Company had checked out the canister for work on any project the XYZ Company was
working on at the time. She says that the XYZ Company had other projects involving hazardous
materials, but none that involved the use of cesium-137.
 lOMoARcPSD|35902250

126 Chapter 4. Modeling the Problem-Solving Process

Table 4.2, which are fragments representing relevant items of evidence for the leaf
hypotheses in Figure 2.9. These dots are presented in Table 4.3.
This case study has several objectives:

 Learning to associate with a hypothesis in an argument the evidence that is relevant

to it
 Learning to evaluate the relevance and the believability of evidence
 Learning to select synthesis functions
 Better understanding the process of evaluating the probability or likeliness of a hypoth-
esis based on the available evidence

When you associate an item of evidence with a hypothesis, the agent automatically
generates a decomposition tree like the one in Figure 4.14. The bottom part of Figure 4.14
shows the abstraction of the tree that is automatically generated by the agent when you
indicate that the item of evidence E005-Ralph favors the leaf hypothesis “The XYZ hazardous
material locker was forced.”
The agent also automatically generates the reduction from the top of Figure 4.14, where
the leaf hypothesis, “The XYZ hazardous material locker was forced,” is reduced to the
elementary hypothesis with the name, “The XYZ hazardous material locker was forced,” to be
directly assessed based on evidence. Although these two hypotheses are composed of the
same words, internally they are different, the latter being an instance introduced in the
agent’s ontology. This elementary hypothesis corresponds to the hypothesis Q in
Figure 4.13. The agent decomposes this hypothesis as shown in the bottom part of
Figure 4.14, which corresponds to the tree in Figure 4.13 except that there is only one

Table 4.3 Dots from Table 4.2

E006-Clyde: Locksmith Clyde’s report that the lock was forced.

E007-SecurityCamera: Video segment of the security camera of the XYZ warehouse,
showing a person loading a container into a U-Haul panel truck.
E008-GuardReport: The record, made by Sam, security guard at the XYZ Company,
that a panel truck bearing Maryland license plate number MDC-578 was in the XYZ parking area on
the day before Willard’s discovery about the missing cesium-137 canister.
E009-MDDOTRecord: Maryland DOT’s record that the truck bearing license plate
number MDC-578 is registered in the name of the TRUXINC Company in Silver Spring, MD.
E010-TRUXINCRecord1: TRUXINC’s record that the truck bearing MD license plate
number MDC-578 was rented to a man who gave his name as Omer Riley on the day before
Willard’s discovery of the missing cesium-137 canister.
E011-TRUXINCRecord2: TRUXINC’s record that Omer Riley gave his address as
6176 Williams Ave.
E012-SilverSpringRecord: Silver Spring city record according to which there is no
residence at 6176 Williams Ave. in Silver Spring, MD.
E013-InvestigativeRecord: Investigative record that traces of cesium-137 were found
in the truck bearing license plate number MDC-578.
E014-Grace: Grace, the Vice President for Operations at XYZ, tells us that no one at
the XYZ Company had checked out the canister for work on any project.
 lOMoARcPSD|35902250

4.5. Hands On: Was the Cesium Stolen? 127

Figure 4.14. Evidence-based assessment of an elementary hypothesis.

item of favoring evidence, namely E005-Ralph. After that, you have to assess the relevance
of this item of evidence to the considered hypothesis (e.g., likely), as well as its believability
(e.g., very likely), and the agent automatically composes them, from the bottom up, to
obtain the assessment of the leaf hypothesis. When you add additional items of evidence
as either favoring or disfavoring evidence, the agent extends the reasoning tree from
Figure 4.14 as indicated in Figure 4.13.
Figure 4.15 illustrates the selection of a synthesis function indicating how to evaluate
the probability of a node based on the probability of its children. You have to right-click on
the node (but not on any word in blue), select New Solution with. . ., and then select the
function from the displayed list.
Now you can perform the case study. Start Disciple-EBR, select the case study know-
ledge base “02-Evidence-based-Analysis/Scen,” and proceed as indicated in the instruc-
tions from the bottom of the opened window.
This case study illustrates several basic hypothesis analysis operations described in the
following.

Operation 4.1. Associate evidence to hypotheses

 In the Evidence workspace, click on the Evidence menu at the top of the window.
 Notice the four modes of operations from the top part of the left panel. Because the
selected one is [AVAILABLE EVIDENCE], the left panel shows the current evidence from
the knowledge base.
 In the left panel, click on the item of evidence you would like to associate with a leaf
hypothesis from the current argumentation. As a result, the upper part of the right
panel shows the main characteristics of this item of evidence, followed by all the leaf
hypotheses in the analysis tree (see Figure 4.16). You will have to decide whether the
selected item of evidence favors or disfavors any of the hypotheses under the Irrelevant
to label, and indicate this by clicking on [FAVORS] or [DISFAVORS] following that
hypothesis.
 Clicking on [FAVORS] or [DISFAVORS] automatically creates an elementary hypoth-
esis to be assessed based on evidence, and moves it under the Favors (or Disfavors)
 lOMoARcPSD|35902250

128 Chapter 4. Modeling the Problem-Solving Process

Figure 4.15. Selecting the synthesis function for a node.

Figure 4.16. Associating an evidence item to a hypothesis.

label. Clicking on [REMOVE] will restore the leaf hypothesis under the Irrelevant
to label.
 To associate another evidence item to a hypothesis, click on it in the left panel and
repeat the preceding operations.
 To return to the Reasoner module, click on [REASONING] following the hypothesis.
 lOMoARcPSD|35902250

4.5. Hands On: Was the Cesium Stolen? 129

Operation 4.2. Update the name of an elementary hypothesis

 In the right panel of the Reasoner module, right-click on a hypothesis that was
reduced to an elementary hypothesis to be assessed based on evidence, and select
Improve Phrasing.
 In the opened editor, update the phrasing of the hypothesis and then click outside
the box.
 Notice that both this hypothesis and the corresponding elementary hypothesis have
been updated accordingly.

Operation 4.3. Assess evidence

 In the Reasoner module, in the left panel, click on the name of the item of evidence to
assess. You may need to right-click on the top hypothesis and select Expand, to make
the evidence item visible. As a result, the right panel shows the decomposition of
evidence assessment into a relevance assessment (the left leaf) and a believability
assessment (the right leaf), as illustrated in Figure 4.17.
 If the right panel does not show the solutions of the hypotheses, then click on [SHOW
SOLUTIONS] at the top of the panel.
 In the right panel, right-click on the left (relevance) leaf and select New Assumption.
As a result, the agent proposes the default solution (e.g., certain).
 If necessary, click on the default solution (the underlined text) and, from the displayed
list, select (double-click) the appropriate value.
 In the right panel, right-click on the right (believability) leaf and select New Assump-
tion (as illustrated in Figure 4.17). As a result, the agent proposes the default solution.
 If necessary, click on the default solution and, from the displayed list, select the
appropriate value.
 The agent automatically determines the inferential force of the item of evidence.

Operation 4.4. Select the synthesis function

 In the Reasoner module, in the right panel, right-click on the node for which you have
to select the synthesis function, select New Solution with. . ., and then select the
function from the displayed list (see Figure 4.18).
 To select a function for a node, all its children must have solutions.

Figure 4.17. Assessing an item of evidence.

 lOMoARcPSD|35902250

130 Chapter 4. Modeling the Problem-Solving Process

Figure 4.18. Selecting a synthesis function.

Figure 4.19. Evidence collection guidance for a selected hypothesis.

4.6 HANDS ON: HYPOTHESIS ANALYSIS AND EVIDENCE

SEARCH AND REPRESENTATION

The objective of this case study is to learn how to use Disciple-EBR to analyze hypotheses based
on evidence retrieved from the Internet, by associating search criteria with elementary hypoth-
eses, invoking various search engines (such as Google, Yahoo!, or Bing), identifying relevant
information, extracting evidence from it, and using the evidence to evaluate the hypotheses.
This case study concerns the hypothesis that the United States will be a global leader in
wind power within the next decade.
To search for evidence that is relevant to a leaf hypothesis, the agent guides you to
associate search criteria with it and to invoke various search engines on the Internet.
Figure 4.19 shows the corresponding interface of the Evidence module. Because the
[COLLECTION GUIDANCE] mode is selected in the left panel, it shows all the leaf hypotheses
and their current evidential support. If you click on one of these hypotheses, such as
“United States imports huge quantities of oil,” it displays this hypothesis in the right panel,
enabling you to define search criteria for it. You just need to click on the [NEW] button
following the Search criterion label, and the agent will open an editor in which you can
enter the search criterion.
Figure 4.20 shows two defined search criteria: “oil import by United States” and “top oil
importing countries.” You can now invoke Bing, Google, or Yahoo! with any one of these
criteria to search for relevant evidence on the Internet. This will open a new window with
the results of the search, as shown in Figure 4.21.
 lOMoARcPSD|35902250

4.6. Hands On: Hypothesis Analysis 131

Figure 4.20. Defined search criteria for a selected hypothesis.

Figure 4.21. Searching relevant evidence on the Internet.

You have to browse the retrieved documents shown in Figure 4.21 and determine
whether any of them contains information that is relevant to the hypothesis that the
United States imports huge quantities of oil. Such a document is the second one, whose
content is shown in Figure 4.22.
You can now define one or several items of evidence with information copied from the
retrieved document, as illustrated in Figure 4.23. In the left panel of the Evidence module,
you switch the selection mode to [AVAILABLE EVIDENCE] and then click on [NEW]. As a
result, the right panel displays a partial name for the evidence E001- to be completed by
you. You then have to click on the [EDIT] button, which opens an editor where you can
copy the description of this item of evidence from the retrieved document. The result is
shown in the right panel of Figure 4.23.
You can define additional characteristics of this item of evidence, such as its type (as
will be discussed in Section 4.7), and you should indicate whether this item of evidence
favors or disfavors the hypothesis that the United States imports huge quantities of oil, as
explained previously.
 lOMoARcPSD|35902250

132 Chapter 4. Modeling the Problem-Solving Process

Figure 4.22. Selected document providing relevant information.

Figure 4.23. Defining an item of evidence.

In this case study, you will first select the hypothesis, ”United States will be a global
leader in wind power within the next decade,” and then you will browse its analysis tree to
see how it is reduced to simpler hypotheses that you have to assess by searching evidence
on the Internet. You will associate specific search criteria with the leaf hypotheses, invoke
specific search engines with those criteria, identify relevant Web information, define
evidence from this information, associate evidence with the corresponding hypotheses,
and evaluate its relevance and believability, with the goal of assessing the probability of the
top-level hypothesis.
Start Disciple-EBR, select the case study knowledge base “03-Evidence-Search/Scen,”
and proceed as indicated in the instructions from the bottom of the opened window.
This case study illustrates the following hypothesis analysis operation:

Operation 4.5. Associate search criteria with hypotheses

 In the Evidence workspace, click on the Evidence menu and then click on [COLLECTION
GUIDANCE]. The left panel shows the leaf hypotheses and their evidential support.
 lOMoARcPSD|35902250

4.7. Believability Assessment 133

 In the left panel, select a hypothesis.

 In the right panel, after Search criterion, click on [NEW] to define a new criterion.
 Type the search criterion and click on [SAVE].
 You may define additional criteria by repeating the preceding two steps.
 Select one of the search criteria by clicking on it.
 After Search with, click on one of the available search engines (i.e., [BING], [GOOGLE],
[YAHOO]) to search the Internet with the selected criterion.
 Browse the documents returned by the search engine, select the relevant ones,
and define items of evidence based on them, as indicated in Operation 4.6, later in
this chapter.

4.7 BELIEVABILITY ASSESSMENT

In the previous sections, we have discussed and illustrated how you may directly assess
the believability of an item of evidence. However, the Disciple-EBR agent has a significant
amount of knowledge about the various types of evidence and its believability credentials,
enabling you to perform a much deeper believability analysis, as will be discussed in this
section. You may wish to perform such a detailed believability analysis for those items of
evidence that are critical to the final result of the analysis. We will start with presenting a
classification or ontology of evidence.
Attempts to categorize evidence in terms of its substance or content would be a fruitless
task, the essential reason being that the substance or content of evidence is virtually
unlimited. What we have termed a substance-blind classification of evidence refers to a
classification of recurrent forms and combinations of evidence, based not on substance or
content, but on the inferential properties of evidence (Schum, 1994 [2001a], pp. 114–130;
Schum, 2011). In what follows, we identify specific attributes of the believability of various
recurrent types of evidence without regard to their substance or content.
Here is an important question you are asked to answer regarding the individual
kinds of evidence you have: How do you stand in relation to this item of evidence? Can
you examine it for yourself to see what events it might reveal? If you can, we say that the
evidence is tangible in nature. But suppose instead you must rely upon other persons
to tell you about events of interest. Their reports to you about these events are
examples of testimonial evidence. Figure 4.24 shows a substance-blind classification
of evidence based on its believability credentials. This classification is discussed in
the following sections.

4.7.1 Tangible Evidence

There is an assortment of tangible items you might encounter. Both imagery intelligence
(IMINT) and signals intelligence (SIGINT) provide various kinds of sensor records
and images that can be examined. Measurement and signature intelligence (MASINT)
and technical intelligence (TECHINT) provide various objects, such as soil samples and
weapons, that can be examined. Communications intelligence (COMINT) can provide
audio recordings of communications that can be overheard and translated if the commu-
nication has occurred in a foreign language. Documents, tabled measurements, charts,
maps, and diagrams or plans of various kinds are also tangible evidence.
 lOMoARcPSD|35902250

134 Chapter 4. Modeling the Problem-Solving Process

evidence

tangible testimonial missing authoritative

evidence evidence evidence record

real demonstrative unequivocal equivocal

tangible tangible testimonial testimonial

evidence evidence evidence evidence

unequivocal unequivocal testimonial completely probabilistically

testimonial evidence testimonial evidence equivocal equivocal

based upon direct evidence obtained based on testimonial testimonial

observation at second hand opinion evidence evidence

Figure 4.24. Substance-blind classification of evidence.

There are two different kinds of tangible evidence: real tangible evidence and demon-
strative tangible evidence (Lempert et al., 2000, pp. 1146–1148). Real tangible evidence is
an actual thing and has only one major believability attribute: authenticity. Is this object
what it is represented as being or is claimed to be? There are as many ways of generating
deceptive and inauthentic evidence as there are persons wishing to generate it. Docu-
ments or written communications may be faked, captured weapons may have been
tampered with, and photographs may have been altered in various ways. One problem
is that it usually requires considerable expertise to detect inauthentic evidence.
Demonstrative tangible evidence does not concern things themselves but only repre-
sentations or illustrations of these things. Examples include diagrams, maps, scale models,
statistical or other tabled measurements, and sensor images or records of various sorts
such as IMINT, SIGINT, and COMINT. Demonstrative tangible evidence has three believ-
ability attributes. The first concerns its authenticity. For example, suppose we obtain a
hand-drawn map from a captured insurgent showing the locations of various groups in his
insurgency organization. Has this map been deliberately contrived to mislead our military
forces, or is it a genuine representation of the location of these insurgency groups?
The second believability attribute is accuracy of the representation provided by the
demonstrative tangible item. The accuracy question concerns the extent to which the
device that produced the representation of the real tangible item had a degree of sensitiv-
ity (resolving power or accuracy) that allows us to tell what events were observed. We
would be as concerned about the accuracy of the hand-drawn map allegedly showing
insurgent groups locations as we would about the accuracy of a sensor in detecting traces
of some physical occurrence. Different sensors have different resolving power that also
depends on various settings of their physical parameters (e.g., the settings of a camera).
The third major attribute, reliability, is especially relevant to various forms of sensors
that provide us with many forms of demonstrative tangible evidence. A system, sensor,
or test of any kind is reliable to the extent that the results it provides are repeatable or
 lOMoARcPSD|35902250

4.7. Believability Assessment 135

consistent. You say that a sensing device is reliable if it provides the same image or report
on successive occasions on which this device is used.
The left side of Figure 4.25 shows how the agent assesses the believability of an item of
demonstrative tangible evidence Ei* as the minimum of its authenticity, accuracy, and
reliability.
Here are additional examples involving evidence that is tangible and that you can
examine personally to see what events it reveals.
Have a look at evidence item E009-MDDOTRecord in Table 4.3 (p. 126). The Maryland
DOT record, in the form of a tangible document, could be given to the analyst to verify
that the vehicle carrying MD license plate number MDC-578 is registered in the name of
the TRUXINC Company in Silver Spring, Maryland.
Now consider evidence item E008-GuardReport in Table 4.3. Here we have a document
in the form of a log showing that the truck bearing license plate number MDC-578 exited
the XYZ parking lot at 8:30 pm on the day in question. This tangible item could also be
made available to analysts investigating this matter.

4.7.2 Testimonial Evidence

For testimonial evidence, we have two basic sources of uncertainty: competence and
credibility. This is one reason why it is more appropriate to talk about the believability of
testimonial evidence, which is a broader concept that includes both competence and
credibility considerations. The first question to ask related to competence is whether this
source actually made the observation the source claims to have made or had access to the
information the source reports. The second competence question concerns whether this
source understood what was being observed well enough to provide us with an intelligible
account of what was observed. Thus competence involves access and understandability.
Assessments of human source credibility require consideration of entirely different
attributes: veracity (or truthfulness), objectivity, and observational sensitivity under the
conditions of observation (Schum, 1989). Here is an account of why these are the major
attributes of testimonial credibility. First, is this source telling us about an event this source
believes to have occurred? This source would be untruthful if he or she did not believe the
reported event actually occurred. So, this question involves the source's veracity. The
second question involves the source's objectivity. The question is: Did this source base a
belief on sensory evidence received during an observation, or did this source believe
the reported event occurred because this source either expected or wished it to occur?

believability of Ei* believability of Ek*

min min
i

Source’s Source’s

competence credibility

authenticity of Ei* reliability of Ei* min

min

Source’s veracity Source’s objectivity

accuracy of Ei* Source’s access Source’s understandability

Source’s observational sensitivity

Figure 4.25. Assessing the believability of evidence with Disciple-EBR.

 lOMoARcPSD|35902250

136 Chapter 4. Modeling the Problem-Solving Process

An objective observer is one who bases a belief on the sensory evidence instead of desires
or expectations. Finally, if the source did base a belief on sensory evidence, how good was
this evidence? This involves information about the source's relevant sensory capabilities
and the conditions under which a relevant observation was made.
As indicated in Figure 4.24, there are several types of testimonial evidence. If the source
does not hedge or equivocate about what the source observed (i.e., the source reports that
he or she is certain that the event did occur), then we have unequivocal testimonial
evidence. If, however, the source hedges or equivocate in any way (e.g., "I'm fairly sure
that E occurred"), then we have equivocal testimonial evidence. The first question we
would ask a source of unequivocal testimonial evidence is: How did you obtain information
about what you have just reported? It seems that this source has three possible answers to
this question. The first answer is, "I made a direct observation myself.” In this case, we have
unequivocal testimonial evidence based upon direct observation. The second possible
answer is, "I did not observe this event myself but heard about its occurrence (or
nonoccurrence) from another person." Here we have a case of second hand or hearsay
evidence, called unequivocal testimonial evidence obtained at second hand. A third answer
is possible: "I did not observe event E myself nor did I hear about it from another source.
But I did observe events C and D and inferred from them that event E definitely occurred."
This is called testimonial evidence based on opinion, and it requires some very difficult
questions. The first concerns the source's credibility as far as his or her observation of
events C and D; the second involves our examination of whether we ourselves would infer
E based on events C and D. This matter involves our assessment of the source's reasoning
ability. It might well be the case that we do not question this source's credibility in
observing events C and D, but we question the conclusion that the source has drawn
from his or her observations that event E occurred. We would also question the certainty
with which the source has reported the opinion that E occurred. Despite the source’s
conclusion that “event E definitely occurred," and because of many sources of uncertainty,
we should consider that testimonial evidence based on opinion is a type of equivocal
testimonial evidence.
There are two other types of equivocal testimonial evidence. The first we call completely
equivocal testimonial evidence. Asked whether event E occurred or did not, our source
says, "I don't know," or, "I can't remember."
But there is another way a source of HUMINT can equivocate: The source can provide
probabilistically equivocal testimonial evidence in various ways: "I'm 60 percent sure that
event E happened”; or "I'm fairly sure that E occurred”; or, "It is very likely that
E occurred." We could look upon this particular probabilistic equivocation as an assess-
ment by the source of the source’s own observational sensitivity.
The right side of Figure 4.25 shows how a Disciple-EBR agent assesses the believability
of an item of testimonial evidence based upon direct observation Ek* by a source, as the
minimum of the source’s competence and credibility. The source’s competence is
assessed as the minimum of the source’s access and understandability, while the source’s
credibility is assessed as the minimum of the source’s veracity, objectivity, and observa-
tional sensitivity.
Here are some examples involving testimonial evidence from human sources that is not
hedged or qualified in any away.
Evidence item E014-Grace in Table 4.3 (p. 126) is Grace’s testimony that no one at the
XYZ Company had checked out the canister for work on any project. Grace states this
unequivocally. You should also note that she has given negative evidence saying the
 lOMoARcPSD|35902250

4.7. Believability Assessment 137

cesium-137 was not being used by the XYZ Company. This negative evidence is very
important, because it strengthens our inference that the cesium-137 canister was stolen.
E006-Clyde in Table 4.3 is unequivocal testimonial evidence. It represents positive evidence.
Here are some examples involving testimonial evidence given by human sources who
equivocate or hedge in what they tell us.
Consider the evidence item E005-Ralph in Table 2.4 (p. 60). Here Ralph hedges a bit by
saying that the lock on the hazardous materials storage area appears to have been forced.
He cannot say for sure that the lock had been forced, so he hedges in what he tells us.
In new evidence regarding the dirty bomb example, suppose we have a source code-
named “Yasmin.” She tells us that she knew a man in Saudi Arabia named Omar al-
Massari. Yasmin says she is “quite sure” that Omar spent two years “somewhere” in
Afghanistan “sometime” in the years 1998 to 2000.

4.7.3 Missing Evidence

To say that evidence is missing entails that we must have had some basis for expecting we
could obtain it. There are some important sources of uncertainty as far as missing
evidence is concerned. In certain situations, missing evidence can itself be evidence.
Consider some form of tangible evidence, such as a document, that we have been unable
to obtain. There are several reasons for our inability to find it, some of which are more
important than others. First, it is possible that this tangible item never existed in the first
place; our expectation that it existed was wrong. Second, the tangible item exists, but we
have simply been looking in the wrong places for it. Third, the tangible item existed at one
time but has been destroyed or misplaced. Fourth, the tangible item exists, but someone is
keeping it from us. This fourth consideration has some very important inferential implica-
tions, including denial and possibly deception. An adverse inference can be drawn from
someone's failure to produce evidence.
We should not confuse negative evidence with missing evidence. To adopt a common
phrase, “evidence of absence (negative evidence) is not the same as absence of evidence
(missing evidence).” Entirely different conclusions can be drawn from evidence that an
event did not occur than can be drawn from our failure to find evidence. We are obliged to
ask different questions in these two situations.
Consider our discussion on the cesium-137 canister. Upon further investigation, we
identify the person who rented the truck as Omar al-Massari, alias Omer Riley. We tell him
that we wish to see his laptop computer. We are, of course, interested in what it might
reveal about the terrorists with whom he may be associating. He refuses to tell us where it
is. This we referred to as the nonproduction of evidence.

4.7.4 Authoritative Record

This final category of evidence would never oblige an analyst to assess its believability.
Tabled information of various sorts such as tide tables, celestial tables, tables of physical or
mathematical results such as probabilities associated with statistical calculations, and
many other tables of information we would accept as being believable provided that we
used these tables correctly. For example, we would not be obliged to prove that tempera-
tures in Iraq can be around 120 degrees Fahrenheit during summer months or that the
population of Baghdad is greater than that of Basra.
 lOMoARcPSD|35902250

138 Chapter 4. Modeling the Problem-Solving Process

4.7.5 Mixed Evidence and Chains of Custody

We have just described a categorization of individual items of evidence. But there are
situations in which individual items can reveal various mixtures of the types of evidence
shown in Figure 4.24. One example is testimonial evidence about tangible evidence where
a source describes a weapon observed at a scene of a crime. Another example is a tangible
document containing a testimonial assertion based on other alleged tangible evidence.
Figure 4.26, for example, shows how one would need to assess the believability of tangible
evidence about testimonial evidence.
Here is an example of a mixture of two or more items of tangible evidence; it is called a
passport. A passport is a tangible document alleging the existence of other tangible
documents recording the place of birth and country of origin of the holder of the passport.
In other words, a passport sets up a paper trail certifying the identity of the holder of the
passport. In addition to needing to check the authenticity of the passport itself, we are
also interested in the authenticity of all the other tangible documents on which this
passport is based.
Here is another mixture of forms of evidence, this time recording a mixture of tangible
and testimonial evidence. We return to our asset “Yasmin,” who has given us further
evidence about Omar al-Massari in our cesium-137 example. Suppose we have a tangible
document recording Yasmin’s account of her past experience with Omar al-Massari. This
document records Yasmin’s testimony about having seen a document, detailing plans for
constructing weapons of various sorts, that was in Omar al-Massari’s possession. As far as
believability issues are concerned, we first have the authenticity of the transcription of her
testimony to consider. Yasmin speaks only in Arabic, so we wonder how adequate the
translation of her testimony has been. Also, we have concerns about Yasmin’s competence
and credibility to consider in her recorded testimony. Finally, we have further interest in
the authenticity of the document she allegedly saw in Omar al-Massari’s possession.
But the believability analysis of an item of evidence can be even more complicated. For
example, very rarely, if ever, has an analyst access to the original evidence. Most often,
what is being analyzed is an item of evidence that has undergone a series of transform-
ations through a chain of custody (Schum et al., 2009). Here we have borrowed an

believability of Ei*

min

Source’s

believability
authenticity of Ei* accuracy of Ei*
min

Source’s Source’s

competence credibility

min
min

Source’s Source’s

Source’s Source’s veracity objectivity

access understandability

Source’s

observational sensitivity

Figure 4.26. Believability analysis of tangible evidence about testimonial evidence.

 lOMoARcPSD|35902250

4.7. Believability Assessment 139

important concept from the field of law, where a chain of custody refers to the persons or
devices having access to the original evidence, the time at which they had such access, and
what they did to the original evidence when they had access to it. These chains of custody
add three major sources of uncertainty for intelligence analysts to consider, all of which
are associated with the persons in the chains of custody, whose competence and credibil-
ity need to be considered. The first and most important question involves authenticity:
Is the evidence received by the analyst exactly what the initial evidence said, and is it
complete? The other questions involve assessing the reliability and accuracy of the
processes used to produce the evidence if it is tangible in nature or also used to take
various actions on the evidence in a chain of custody, whether the evidence is tangible or
testimonial. As an illustration, consider the situation from Figure 4.27. We have an item of
testimonial HUMINT coming from a foreign national whose code name is “Wallflower,”
who does not speak English. Wallflower gives his report to the case officer Bob. This
report is recorded by Bob and then translated by Husam. Then Wallflower’s translated
report is transmitted to the report’s officer Marsha, who edits it and transmits it to the
analyst Clyde, who evaluates it.
Figure 4.28 shows how a Disciple-EBR agent may determine the believability of the
evidence received by the analyst. A more detailed discussion is provided in Schum
et al. (2009).
The case officer might have intentionally overlooked details in his recording of Wall-
flower’s report. Thus, as shown at the bottom of Figure 4.28, the believability of the
recorded testimony of Wallflower is the minimum between the believability of Wallflower
and the believability of the recording. Then Husam, the translator, may have intentionally
altered or deleted parts of this report. Thus, the believability of the translated recording is
the minimum between the believability of the recorded testimony and the believability of
the translation by Husam. Then Marsha, the report’s officer, might have altered or deleted
parts of the translated report of Wallflower’s testimony in her editing of it, and so on.

Wallflower’s

received Wallflower’s

reported Wallflower’s
testimony about
Wallflower’s
testimony about translated
Emir Z. in English.
recorded
Emir Z. in English. testimony about
testimony about Wallflower’s
Emir Z. in English.
Emir Z. in Farsi. testimony about

E005-Emir- Emir Z. in Farsi.

E004-Marsha-
Iran
E003-Husam-
report
E002-Bob-
translation
E001-Wallflower-
recording
testimony

recording
translation
Clyde editing
transmission

Wallflower

• Competence
• Credibility
Bob
Husam A.
Marsha • Competence ce
e
Marsha
• Competence • Competence
• Competence • Veracity Sony Recorder
ce
e
• Credibility • Credibility
• Veracity SN 247
• Fidelity
• Fidelity • Reliability
• Reliability
• Security

Figure 4.27. The chain of custody of the Wallflower’s testimony.

 lOMoARcPSD|35902250

140 Chapter 4. Modeling the Problem-Solving Process

Believability of transmitted, edited,

translated, recorded testimony of Wallflower

min

Believability of Believability of

edited translation transmission by Marsha

min
Believability of Believability of

translated recording editing by Marsha

min
Believability of Believability of

recorded testimony translation by Husam

min
Believability of Believability of

Wallflower recording by Bob

Figure 4.28. Assessing the believability of Wallflower’s testimony.

The result of these actions is that the analyst receiving this evidence almost certainly
did not receive an authentic and complete account of it, nor did he receive a good account
of its reliability and accuracy. What Clyde received was the transmitted, edited, translated,
and recorded testimony of Wallflower. Although the information to make such an analysis
may not be available, the analyst should adjust the confidence in his conclusion in
recognition of these uncertainties.

4.8 HANDS ON: BELIEVABILITY ANALYSIS

This case study, which continues the analysis from Section 4.5 with the analysis of the
hypothesis, “The cesium-137 canister is used in a project without being checked-out from
the XYZ warehouse,” has two main objectives:

 Learning to define a more detail representation of an item of evidence

 Better understanding the process of believability analysis

In Section 4.6, we have presented how you can define an item of evidence, and Figure 4.23
(p. 132) shows the definition of E001-US-top-oil-importer with type evidence. You can
specify the type by clicking on the [CHANGE] button. Figure 4.29, for instance, shows the
definition of E014-Grace. After you click on the [CHANGE] button, the agent displays the
various evidence types from the right panel. You just need to click on the [SELECT] button
following the correct type, which in this case is unequivocal testimonial evidence based upon
direct observation.
Once you have selected the type of E014-Grace, the agent displays it after the label Type
and asks for its source, which is Grace (see Figure 4.30).
As shown in Figure 4.30, we have also indicated that this item of evidence disfavors the
hypothesis “The missing cesium-137 canister is used in a project at the XYZ company.” As a
result, the agent introduced it into the analysis tree and generated a more detailed analysis
of its believability, which is shown in Figure 4.31.
You can now perform a more detailed believability analysis, as illustrated in Figure 4.32,
where we have assessed the competence, veracity, objectivity, and observational sensitiv-
ity of Grace, and the agent has automatically determined her believability.
In this case study, you will practice the preceding operations. You will first select the
hypothesis, “The cesium-137 canister is used in a project without being checked out from
 lOMoARcPSD|35902250

4.8. Hands On: Believability Analysis 141

Figure 4.29. Selecting the type of evidence.

Figure 4.30. Definition of an item of evidence.

 lOMoARcPSD|35902250

142 Chapter 4. Modeling the Problem-Solving Process

Figure 4.31. Decomposition of the believability assessment for an item of testimonial evidence.

Figure 4.32. More detailed believability analysis.

the XYZ warehouse.” Then you will browse its analysis to see how it is reduced to simpler
hypotheses that need to be assessed based on the evidence. After that, you will represent a
new item of evidence, will associate it with the hypothesis to which it is relevant, assess its
relevance, evaluate its believability by assessing its credentials, and browse the resulting
analysis tree.
 lOMoARcPSD|35902250

4.9. Drill-Down Analysis 143

Start Disciple-EBR, select the case study knowledge base “04-Believability-Analysis/

Scen,” and proceed as indicated in the instructions from the bottom of the opened window.
This case study illustrates the general operation of defining an item of evidence,
summarized as follows.

Operation 4.6. Define an item of evidence

 In the Evidence workspace, click on the Evidence menu at the top of the window.
 Notice the four modes of operations from the top part of the left panel. Because the
selected one is [AVAILABLE EVIDENCE], the left panel shows the current evidence (if any)
from the knowledge base.
 In the left panel, click on [NEW]. The right panel now shows a partially defined item of
evidence, such as E002-. You will complete the definition of this item of evidence.
 Complete the name E. . .- at the top of the right panel and click on [SAVE].
 Click on [EDIT] for Description, click inside the pane and type the description of the item
of evidence.
 Click on [SAVE].
 You may now provide additional information about the item of evidence (as indicated
in the following steps) or define additional items of evidence (by repeating the
preceding steps).
 After “Type: evidence,” click on [CHANGE] to specify the type of this item of evidence.
 Inspect the different evidence types and click on [SELECT] following the type corres-
ponding to the current item of evidence.
 Provide the additional, type-related information, requested by the system (e.g., the
source in the case of a testimonial item of evidence).

4.9 DRILL-DOWN ANALYSIS, ASSUMPTION-BASED

REASONING, AND WHAT-IF SCENARIOS

An important feature of the Disciple-EBR agent is that it allows you to perform analyses at
different levels of detail. What this means is that a hypothesis may be reduced to many levels
of subhypotheses or just a few levels that are then assessed based on relevant evidence. The
same applies to assessing the believability of evidence. You may directly assess it, as was
illustrated in Figure 4.14 (p. 127), where the believability of E005-Ralph was assessed as very
likely. But if an item of evidence has an important influence on the analysis, then you may
wish to perform a deeper believability analysis, as was illustrated in Figure 4.32, where the
user assessed lower-level believability credentials. The user could have drilled even deeper
to assess the source’s access and understandability instead of his or her competence.
It may also happen that you do not have the time or the evidence to assess a
subhypothesis, in which case you may make various assumptions with respect to its
probability. Consider, for example, the analysis from the case study in Section 4.5, partially
shown in Figure 4.15 (p. 128) and the four subhypotheses of the top-level hypothesis. The
first three of these subhypotheses have been analyzed as discussed in the previous
sections. However, for the last subhypothesis, you have made the following assumption:

It is certain that the MDC-578 truck left with the cesium-137 canister.

Assumptions are distinguished from system-computed assessments by the fact that the
assumed probabilities have a yellow background.
 lOMoARcPSD|35902250

144 Chapter 4. Modeling the Problem-Solving Process

You may provide justifications for the assumptions made. You may also experiment
with various what-if scenarios, where you make different assumptions to determine their
influence on the final result of the analysis.
Thus the agent gives you the flexibility of performing the analysis that makes the best
use of your time constraints and available evidence.
The Disciple-EBR shell includes a customized modeling assistant to model the hypoth-
esis analysis process. The following two case studies demonstrate its use.

4.10 HANDS ON: MODELING, FORMALIZATION,

AND PATTERN LEARNING

The objective of this case study is to learn how to use Disciple-EBR to model the analysis of
a hypothesis. More specifically, you will learn how to:

 Specify a new hypothesis

 Specify a question/answer pair that suggests how the hypothesis can be reduced to
simpler hypotheses
 Specify the subhypotheses suggested by the question/answer pair
 Select ontology names to be used in hypotheses, questions, and answers
 Convert a hypothesis to an elementary solution (assessment)
 Formalize a reasoning tree or a part of it to learn reduction patterns
 Convert formalized nodes back to modeling to further update them

This case study will guide you through the process of defining and analyzing a hypothesis
by using, as an example, the following hypothesis: “CS580 is a potential course for Mike
Rice.” You will first define the reduction tree shown in Figure 4.33. Then you will formalize
it and specify the synthesis functions.
Start Disciple-EBR, select the case study knowledge base “05-Modeling-Learning/Scen,”
and proceed as indicated in the instructions from the bottom of the opened window.

Figure 4.33. Hypotheses reduction tree.