Rethinking Failure

Managing the “Reliability Cycle”: An Alternative

Approach to Thinking About Intelligence Failure
Scott J. Hatch

Soon after becoming the director of and organizational literature on so-

the Intelligence Success and Failure called high reliability organizations
Course of CIA University’s Kent (HRO) and normal accident theory.
School for Intelligence Analysis, I In this article I have adapted the
realized that much of the literature on material contained in the literature to
intelligence success and failure made the domain of intelligence analysis. I

“
The IC should consider
applying the lessons
High Reliability
no mention of insights from profes-
sions outside of our intelligence
domain. Many of these professions
also face severe consequences for
failure. Increasingly, as I taught the
believe a shift in our thinking about
this subject would allow the Intelli-
gence Community to think more pro-
actively and holistically about the
ways to increase the reliability of our
Organizations have class, I came to draw from business intelligence analysis.
learned in thinking about
vvv
failures of intelligence
analysis. During the past few decades, busi- those faced by HROs, I believe the
ness researchers have produced a IC should apply the lessons HROs

” substantial body of literature on

organizations that achieve high reli-
ability under conditions of dynamic
uncertainty, inherent complexity,
high risk, and potentially cata-
have learned in thinking about fail-
ures of intelligence analysis. Doing
so may yield not only additional les-
sons for the community but could
help managers of intelligence analy-
strophic costs should they fail. The sis think more effectively about their
researchers have focused on are in own environments in order to avert
fields like aviation safety, nuclear or mitigate risk of failure and
power plant operations, chemical or improve prospects for success.
oil processing, medicine, and wild-
fire control. In this essay, I will translate HRO
and accident-management insights
Given that Intelligence Commu- to the domain of intelligence analy-
nity (IC) organizations face chal- sis and sketch out an HRO frame-
lenges of ensuring reliability under work for intelligence analysis. 1
conditions in some ways similar to More work, of course, would need

1 Among the leading researchers in studying HROs are Karl E. Weick, Kathleen M. Sutcliffe, Karlene

Roberts and David van Stralen. Weick and Sutcliffe together authored, Managing the Unexpected:
Resilient Performance in an Age of Uncertainty (Jossey-Bass, 2007), probably the most frequently ref-
erenced book in the field.

All statements of fact, opinion, or analysis expressed in this article are those of the
author. Nothing in the article should be construed as asserting or implying US govern-
ment endorsement of its factual statements and interpretations.

Studies in Intelligence Vol. 57, No. 2 (June 2013) 29

Rethinking Intelligence Failure

Impact, informing decisionmaking, is what we aim for as intelli- policymakers. This is clear-cut: if
gence analysts. we do not do these things, then we
have failed. At the same time, meet-
ing these two conditions is neces-
to be done to apply this framework lowing way. Our goal in analysis sary—but not sufficient—for
to daily work practices in the IC’s should be to success. Success goes beyond “get-
analytic components. ting it right”: It concerns impact and
• have a positive impact in inform- achieving consistency over time.
ing our consumer’s decisionmak-
What We Are Aiming For ing… The goal of analysts and managers
Any effort to improve organiza- is to have policymakers and policy
• by delivering to the consumer the implementers keep coming back to
tional performance must begin with
right insights … analysts over time. While luck may
a clear sense of aims so that bench-
marks can be set up to gauge prog- be—and often is—a component of
• in a timely and useful manner… any given success or failure, we can-
ress.
• consistently over time. not rely on it. Thus, by always being
In my experience, analysts in rigorous in tradecraft, persuasively
CIA’s intelligence successes and Impact—informing decisionmak- presenting assessments, and manag-
failures program have found it chal- ing—is what we aim for as intelli- ing relations with our consumers, we
lenging to define failure. Much of gence analysts. Even granting that demonstrate the marks of reliability.
the literature on intelligence failure impact may be difficult to measure, To use a manufacturing analogy, it is
has focused on “making the right considering impact forces us to look not enough to minimize production-
call” and identifying the cognitive at real measures of effectiveness line defects. To achieve success we
elements that might have gone into rather than just at the numbers of must actively manage our corporate
“failed” analysis. products produced, briefings deliv- brand, and that means striving for
ered, and other similar quantitative reliability.
But intelligence analysts know measures (number of graphics, for
they must aim for more than just the example). These latter attributes Lastly with the above four-part
“right call.” They rightly observe only have indirect effects on impact. definition of our mission we have a
that factors having to do with orga- Our goal as analysts is not to simply way to identify partial success or
nizational and policy environments write papers, throw them over some partial failure so that we can think
are always involved as well. More- official’s transom, and hope they get about how to do things better and
over, because most studies of intelli- read. Stressing impact enables us to avoid the overgeneralization inher-
gence success and failure tend to be start thinking about how to measure ent in today’s use of the terms “suc-
case-specific, it is natural to fixate effectiveness, not just performance. cess” and “failure.” Moreover, it
on specific events, rather than on provides more clarity for accurate
success and failure as part of a pro- Second, the “right insights” can be benchmarking.
cess that transcends particular defined as those insights that accu-
moments or events. rately describe a situation, add value Developing Attitudes to Facilitate
for a consumer, are rigorously
In reality, I believe we should not Reliability
arrived at, and are soundly rea-
be interested in a “win-loss” bal- soned. Accuracy and value-added Having a sharp definition of goals
ance sheet but in how our successes are essential to having the right is only a first step toward greater reli-
and failures factor into our ongoing insights, and without them one is left ability. A second is adoption of the
efforts to be consistently reliable in with either something that is wrong appropriate attitude toward failure.
supporting our consumers in the or merely obvious. Failure would Organizations typically either
many ways they demand of us. With come from the absence of the right acknowledge failure or they deny it.
this in mind, I believe we should insights or the delivery of insights A denial mentality is often character-
think of our analytical mission—and (even if the right ones) in ways that ized by the phrase (attributed to Gene
hence the ways in which we mea- were neither timely nor useful to Kranz, the NASA flight director dur-
sure success and failure—in the fol-

30 Studies in Intelligence Vol. 57, No. 2 (June 2013)

 Rethinking Intelligence Failure

ing the Apollo 13 mission), “Failure ous and forward-looking process for what indicators, if any, suggest
is not an option.” In contrast, an dealing with the possibility of fail- things are going wrong.
“acknowledgement mentality” is cap- ure. The former looks for lessons
tured in the sentence, “We are always after a failure. The latter identifies In intelligence work, this means
one step away from failure.” Both the risks and potential causes of fail- analysts and managers need to diag-
attitudes set up formal and informal ure and works to avoid them. nose the situations they are in, iden-
incentives throughout an organiza- tify potential vulnerabilities, and
tion. The latter attitude facilitates The following is my view of the monitor signals for evidence of
learning; the former does not. roles and responsibilities of analysts weakness. Their goal is not to antici-
and managers in this cycle. pate every possible failure—that
The saying “failure is not an would be impossible—but to address
option” may be well intended. At the Phase 1: Anticipation the most evident and biggest poten-
same time, however, some manag- The “preoccupation [of HROs] tial problems. At the same time, they
ers and employees may draw from with failure,” as Weick and Sutcliffe gain familiarity with their systems
the expression the sense that failure put it, might seem paralyzing, but so they will be able to anticipate and
should not even be considered. that focus leads to constant self react to unexpected developments
When failure does occur, such atti- awareness. HROs consistently ask more quickly or establish means for
tudes could create incentives for how things are supposed to work, prevention, mitigation, and recov-
individuals to look for ways of deny- how they are working, what could ery. The less done at this stage, the
ing it has occurred or to try to go wrong, what the consequences more that will need to be done if
deflect responsibility. Such behavior would be if things did go wrong, and failure does occur.
is unproductive and costs organiza-
tions energy, time, and focus that
could be better spent recovering
from failure. (U) The Reliability Cycle

ANTICIPATION PHASE
The opposite mentality, the one
Thinking about how things work,
that adopts the attitude of “we are how they can go wrong, and what
always one step away from failure” the consequences are of failure

is the mark of the high reliability

organization. It is an attitude that, Anticipation
proponents of the HRO concept
argue, produces different organiza- BUILDING RELIABILITY MITIGATION PHASE
tional incentives.
Cultivating mindfulness as a Mitigation Identifying actions to take
Building
matter of routine, improving and/or now to prevent, mitigate, or
Reliability
Prevention
organizational capabilities prepare for the possibility
HROs and the Reliability Cycle and learning of failure

According to Weick and Sutcliffe, Recovery RECOVERY PHASE

HROs constantly try to anticipate (from failure)
Dealing with the consequences
failure, and they recover quickly and of failure by balancing the
need for reassessment with
effectively when failures occur. In near-term demands
the field of intelligence analysis, I
suggest these qualities can be fur- Reevaluate
(near failures
ther refined into the five elements and successes)

shown in the graphic below. I

believe management of this cycle REEVALUATION

holds the key to increasing our ana- Mitigation efforts may avert
lytic reliability. Its application would failures but they need to be
examined to assess what success
move us from dealing purely retro- lessons are repeatable
spectively with failure to a continu- DI Design Center/MPG 464612ID 8-12
UNCLASSIFIED

Studies in Intelligence Vol. 57, No. 2 (June 2013) 31

Rethinking Intelligence Failure

Unlike cognitive or policy-environmental failures, organization- Organizational Failure: 9/11

al failures seldom offer single causes to be remedied. Organizational (or systems) fail-
ure may be the most difficult kind of
problem we can face. Counterterror-
Anticipation requires understand- gory for the intelligence world ism analysts knew before the 9/11
ing the factors that contribute to fail- would be failures caused by factors attacks that al-Qa‘ida was planning a
ure in intelligence analysis, a topic in the “[security] policy environ- major attack in the United States, but
that has been explored in the writ- ment” or the failure of analysis to they did not know where, when,
ings of Richard Betts, Richards engage with those in that environ- how, or what kind of targets.
Heuer, Robert Jervis, and others. ment. 2
Analysts typically offer as reasons The 9/11 Commission Report and
for failure incompetence, insuffi- the declassified CIA Inspector Gen-
cient data, or the fact that the prob- Introducing COPE eral’s Report on Accountability With
lems they tackle are intrinsically My shorthand for a taxonomy of Respect to the 9/11 Attacks detailed
hard. failure that adopts these three cate- organizational issues that contrib-
gories is COPE, which I illustrate uted to the US government’s failure
But it is not enough, in my view, to using three examples below. The to act before the attacks. 3 These
list specific issues in specific cases elements of each category are included problems with watchlist-
after the fact. Just as Heuer has detailed in the table on the facing ing, poor communication within and
given names to mindsets, biases, and page. between agencies, unclear lines of
logical fallacies, so too a structured authority, murky legal authorities,
taxonomy of reasons for failure Cognitive Failure: Iraq WMD and so forth.
would allow us more readily to diag- The Iraq WMD case was first and
nose situations more precisely and foremost a cognitive failure: the IC Unlike cognitive or policy-environ-
act more quickly to prevent or miti- judged that Iraq had ongoing WMD mental failures, organizational fail-
gate the effects of failure. programs and stockpiles of WMD, ures seldom offer single causes to be
even though Saddam Hussein’s remedied: rather, they usually involve
For the business world, Max regime had destroyed what it had multiple breakdowns that, in the
Bazerman, professor of business and was only trying to preserve a aggregate, cause the failure. In intelli-
administration at the Harvard Busi- capability to reconstitute aspects of gence work, tackling this kind of fail-
ness School, and Michael Watkins, a the program when sanctions ended. ure requires examination of analytical
consultant in leadership strategy, While organizational and policy- and work processes and their individ-
have done work along these lines environmental factors contributed to ual vulnerabilities. Often this requires
that offers a model for a taxonomy. the failure, it was nevertheless a cog- analysis of processes across bureau-
They have named three categories of nitive failure driven by mind-set cratic boundaries.
failure: cognitive, organizational issues. Had the cognitive factors
(process or systems), and political, been recognized early on—probably
Policy-Environmental Failure:
which, with the exception of the years earlier—the IC, using struc-
CIA and Vietnam Analysis
third, easily parallel failures in intel- CIA’s pessimistic assessments of
tured analytic techniques or other
ligence analysis. The last category the situation in Vietnam for much of
methods, might have reexamined its
refers to failures of businesses to the 1960s were largely accurate, and
assumptions and considered alterna-
address the political system within cognitive challenges (though they
tive judgments about Saddam and
which they must operate (e.g, lobby- existed) had little or no bearing on
his programs.
ing for legislation or regulatory analysis. The challenges lay in the
changes). A more appropriate cate- problems senior CIA officers faced

2 Max H. Bazerman and Michael D. Watkins, Predictable Surprises: The Disasters You Should Have Seen Coming, and How to Prevent Them (Harvard Busi-
ness Review Press, 2008).
3 The 9/11 Commission Report: Final Report of the National Commission on Terrorist Attacks Upon the United States (US Government Printing Office, 2004)

and CIA Inspector General, Report on Accountability With Respect to the 9/11 Attacks (Central Intelligence Agency: 2007), declassified/redacted available at
www.cia.gov.

32 Studies in Intelligence Vol. 57, No. 2 (June 2013)

 Rethinking Intelligence Failure

The second way in which the

(U) Table 1: A Taxonomy of Failure
framework can help is in providing
Cognitive Challenges Organizational Policy Environmental approaches to diagnosing a very
in Analysis Challenges Challenges
complex process. Intelligence analy-
MIND-SET ISSUES GROUP MIND-SET ISSUES CONSUMER MIND-SETS
» Paradox of expertise (incl. » Groupthink or denial » Overconfidence in their sis has been evaluated from a num-
overconfidence), especially
on estimates;
(unwillingness to see a problem
is real or serious);
own ideological beliefs
or capabilities;
ber of angles, each more advanced
» Mind-sets and biases » Intangibility (reluctance to » Overreliance on their own than the simple five-part loop that is
(e.g. confirmation bias, invest in a future that is distant judgments or experience,
mirror-imaging, rational actor, or vague); perception of having better known as the traditional intelligence
etc.);
» Denial and deception.
» Abstractness (hard to focus on
a problem not experienced or
information or insights;
» Experience breeds the need
cycle. Rob Johnston’s taxonomy of
imagined vividly). for certainty, not hedging intelligence analysis variables in his
NATURE OF CHANGE or ambiguity;
» Complexity of change; RESOURCING ISSUES » Tendency to focus on persons Analytic Culture in the U.S. Intelli-
» Recognizing paradigm shifts; » Inadequate personnel and/or
» Identifying the salient resources to cover the issue;
rather than systemic issues;
» Perception that analysts
gence Community, for example,
underlying drivers. » Individuals acting in their own lack experience, judgment, illustrates the complexity in the four
narrow self-interest deplete perspective or loyalty.
DIFFICULTY THINKING IN TIME common resources; types of variables he lists in a taxon-
» Status quo bias; » Competing or gapped coverage.
» Limited ability to
CONSUMER LIMITATIONS AND
GAUGING IMPACT omy of factors that influence analy-
imagine discontinuities;
» Rapid or incremental change;
PROCESS ISSUES
» Overly rigid or, conversely,
» Distractions, as they are sis: systemic (those factors that
beset by immediate and
» Limited or lack of ambiguous lines of authority; pressing problems; affect the intelligence organization
alternative scenarios. » Subordinates tend to stress » Have to respond to
good news or what their boss constituencies, limiting options, and the analytical environment); sys-
LITTLE CLEAR REPORTING
» Collection not tasked because
wants to hear;
» Data collection based on
flexibility, or ability to develop a tematic (factors, especially external
long-term horizon;
gaps not clearly identified
or understood;
consumer demands but » Beholden to or advocates of influences, that affect the analytical
unaligned with actual needs;
» Collection tasked, but » Information compartmentation
a specific approach, program,
or option; environment); idiosyncratic (mat-
reporting insufficient to answer
specific problems.
between components or units;
» Legal, customary, or
» Inadequate capabilities to ters that affect individuals and their
resolve or exert leverage on
POOR QUALITY REPORTING
internal policy constraints the problem. analytic performance); and commu-
or prohibitions. » Factoring in US actions.
» Overreliance on single or nicative (those that affect communi-
limited sources;
» Reporting very fragmentary
RISK MANAGEMENT
CALCULATIONS
EFFECTIVELY MANAGING
RELATIONS WITH CONSUMERS
cation between groups involved in
or indirect. » Easier to ignore the harm » Meeting consumer needs; the analytic process). 4
from inaction or pay more » Providing solid argumentation;
SIGNIFICANT NOISE OVER CLEAR
SIGNALS
attention to harm from action » Soliciting feedback.
» Heavy volume of reporting;
than to take steps with small,
known costs; PROVIDING EFFECTIVE WARNING
For the purposes of this discussion
» Significant processing needed
to be useful.
» Willingness to incur a large but » Warning not given at all or I prefer to think in terms of five crit-
low-probability risk rather than not given clearly enough for
accept a smaller, sure loss now. the decisionmaker; ical areas of vulnerability, each of
» Cry-wolf syndrome (repeated
warnings become ignored
which has elements that can be mon-
over time). itored during an analytic process or
UNCLASSIFIED DI Design Center/MPG 464610ID 8-12
examined in the event of a failure.
These points are:
in engaging presidential administra- mate failure) was on the policy-envi-
tions that declined to accept CIA ronmental side of the equation. • Assessment—the cognitive ele-
analysis. let alone act on it, which ments of the analytical problem.
represents failure to have an impact. • Collection—the continuous effort
Indeed, intelligence histories tout as COPE’s Utility
to expand knowledge about a situ-
successful CIA’s analytic perfor- The COPE framework can clarify ation.
mance during the period, but that causes of failure in three ways. First,
analysis cost CIA one director, John the mere act of determining which of • Support—provision to consumers
McCone, who resigned in frustra- the three types of failure a situation of products, warning memos,
tion, and kept CIA out of Oval falls into will help triage it to make efforts to brief them, etc.
Office deliberations on the issue for further diagnosis easier.
nearly two years after he left. In this • Response from consumers—feed-
case, the real challenge (and ulti- back, further tasking, etc.

4 Dr. Rob Johnston, Analytic Culture in the U.S. Intelligence Community (Central Intelligence Agency, Center for the Study of Intelligence, 2005), 33–44.

Studies in Intelligence Vol. 57, No. 2 (June 2013) 33

Rethinking Intelligence Failure

Identification of responsibilities and degrees of influence over By using the COPE framework,
given situations will set components up to address prevention analysts can also improve their abil-
and mitigation. ity to identify areas in which they
can (and should) take the lead in
addressing shortcomings. It can also
• Organizational—resourcing and always have complete knowledge of help them recognize areas in which
process issues, group mind-sets, or the interactions of their people higher management is needed or
poor risk management calcula- within the system. When multiple cases in which a component—or
tions. systems are involved and are organization as a whole—must react.
involved at higher levels, the chal- The more analysts can anticipate
The third way in which the COPE lenge grows substantially. management concerns, the more
framework can help is in identify- they and management can be proac-
ing what individual or component tive in risk-reduction and post-fail-
would be best able to remedy prob- Phase 2: Prevention, Risk ure activities.
lems that caused a failure (after the Reduction, and Mitigation
fact) or appear to be contributing to Accepting that we cannot always
an increasing risk of failure (before Phase 3: Recovery and/or
prevent failure, we should always
one occurs). Identification of Reevaluation
think about the things that could be
responsibilities and degrees of influ- done to reduce the risk of failure. At This is the phase of the process
ence over given situations will set the same time, we should position usually given the least thought in
components up to address preven- ourselves to deal with failures and organizations that work from the
tion and mitigation. mitigate their consequences. assumption that “failure is not an
option.” When failure does happen,
In the case of cognitive issues or As with anticipation, mitigation two simultaneous, interrelated tasks
cognitive failures, analysts are involves shared responsibilities. must follow:
likely to bear the most responsibil- While managers will make the deci-
ity. They will also have the greatest sions on resources and processes, the • A retrospective of what went
ability to address problems. In the analysts closest to the substance of a wrong must be completed.
policy-environmental arena, ana- problem can speak most authorita-
lysts should be aware of dynamics, tively on the consequences in a • The organization must rebuild
but managers will most likely have region or subject area, should the credibility with both higher man-
to take the lead in addressing issues. unforeseen or unlikely actually take agement and consumers.
Neither analyst nor manager is place. Their understanding of how
likely to have much influence over things could unfold, including These tasks will have to be com-
the consumer environment, as each dynamics previously unforeseen that pleted in an environment of
is most likely to be in a reactive could affect the actors in the region increased workloads as consumers
mode as they see it unfold, espe- and US interests will provide the will demand intelligence support to
cially in a relatively new situation. basis for decisions about resources manage the new situation.
and processes to follow. In this stressful time, the natural
Analysts and managers most likely
will have to share responsibility for Analysts may be weaker in their temptation is to postpone a reassess-
resolving organizational issues. understanding of the consequences ment until things quiet down—
Managers will have decisionmaking of a failure for their component and which often leads to never doing one
responsibility and depend on ana- the larger organization than manag- at all. But a rapid assessment is vital.
lysts to contribute substantive and ers. Experience, training, and man- The more quickly mental models are
working-levels insights on pro- agement engagement can help adjusted, the sooner a component
cesses to inform decisionmaking. sensitize them to these dimensions can begin to reestablish its credibil-
This is likely to be more difficult and help them contribute more effec- ity. Being proactive and taking
than it might seem on the surface. tively to the decisions management responsibility for failure will buy
Even first line managers don’t will have to make. goodwill, improve the confidence of

34 Studies in Intelligence Vol. 57, No. 2 (June 2013)

 Rethinking Intelligence Failure

higher ups, and possibly earn more In short, components should look at both success and failure to
latitude to tackle the situation. sharpen lessons learned.

Management must play a bigger

role than analysts in the recovery coupling”—for example, by adding intelligence success or averted an
phase. First-line managers in particu- redundant backup systems—or adds outright failure? It is important to
lar need to lead the reassessment, but complexity to its processes, such as assess these situations as well, even
they must do so without alienating by adding new procedures. While when a failure has not occurred.
individuals—analysts, other manag- Perrow is focused on technological
ers, or policymakers. The first-line systems, these insights can be In this case, one would want to
managers also need to preserve team applied to organizational behavior engage in a reevaluation of the suc-
cohesion and deal with resource chal- more generally. cess or avoided failure. The natural
lenges created by new circum- organizational tendency is simply to
stances. In such an environment, the These insights also dovetail with accept a success and to allow it to
potential goes up for missteps and conclusions Richard Betts and others become a new “template,” without an
counterproductive reactions. It is have made about the inevitability of examination of what might have made
vital, therefore, to understand what intelligence failure.6 In particular, the seemingly successful procedures
kind of reactions are the most and reflexive organizational responses to work and what actual limitations
least helpful. On this score, some use- intelligence failure have increased remain. Determining why success was
ful insights can be gleaned from nor- redundancy, multiplied organiza- achieved and what characteristics
mal accident theory. tional components, and added more were unique or repeatable helps to
procedures while making work pro- make a clear headed determination of
Normal Accident Theory cesses more complex, burdening ana- what should be emulated in the future.
Normal accident theory was intro- lysts and managers alike with more In short, components should look at
duced by Charles Perrow in his 1984 tasks. These changes, it could be both success and failure to sharpen
book in which he observed that com- argued, have made the IC system lessons learned.
plex technological systems are more more vulnerable to failure by increas-
likely to fail when “tight coupling” ing the incentives people have to
and “interactive complexity” occur. 5 ignore even good practices to “get the Phase 4: Building Reliability
“Tight coupling” describes a situa- job done.” Unlike the other three phases of
tion in which incidents in one part of the Reliability Cycle, this phase is
a system will have prompt and major A better response, per Perrow, less tied to a specific situation,
effects on other parts of the same would have been to find ways to although it can flow out of one. In
system. In a sense, “tight coupling” “loosen the coupling” and/or this phase, the focus is on how orga-
defines rigidities in systems. “Inter- “reduce complexity.” In the environ- nizations can improve their ability to
active complexity” describes a situa- ment of intelligence analysis, this learn and more consistently culti-
tion in which two or more individual could involve substituting greater vate the practices necessary for high
events or failures in a system inter- ownership, accountability, and learn- reliability.
act and create unexpected effects on ing in place of adding redundancy. It
the system as a whole. In short, the could also mean streamlining proce- Harvard business professor David
more complex the system, the more dures and components rather than Garvin and others have described at
likely “normal accidents” are to multiplying them. least four characteristics of learning
occur. organizations: (1) a supportive learn-
While recovery assumes that fail- ing environment; (2) concrete learn-
For Perrow, an ineffective response ure has happened, what about those ing processes; (3) a leadership that
to failure would be one in which an situations in which anticipation and reinforces learning; and (4) the
organization either “tightens the mitigation efforts have led to an transfer of knowledge throughout

5 Charles Perrow, Normal Accidents; Living with High-Risk Technologies (Princeton University Press, 1984 updated 1999).
6 Richard K. Betts, “Analysis, War, and Decision: Why Intelligence Failures Are Inevitable.” World Politics 31, no. 1 (Oct. 1978): 61–69.

Studies in Intelligence Vol. 57, No. 2 (June 2013) 35

Rethinking Intelligence Failure

Inculcating practices of mindfulness and reliability will be done • They can establish performance
mostly at work, and that is mainly the responsibility of manag- benchmarks for individuals and
ers—especially first-line managers. their components.

• They can encourage and provide

the organization. 7 It should be added Training. Training is necessary, the means their analysts can use to
that building reliability requires but it is only part of the process. pursue outreach to bring new or
looking at the organizational pro- Training can facilitate skills devel- different ideas to their teams.
cesses that may be inhibiting effec- opment and spread values within
tiveness and thinking through how Accomplishing all this, however,
organizations, but inculcating prac-
to realign them to be more condu- places a premium on deliberate plan-
tices of mindfulness and reliability
cive to intelligence success. The key ning on the part of the manager.
will be done mostly at work, and
here is not simply compiling lessons that is mainly the responsibility of Knowledge Capture. Also
learned but finding ways of integrat- managers—especially first-line man- needed are improvements in the cap-
ing and habituating them into daily agers. ture and transfer of knowledge
work processes. within components. This is not about
Management Focus. Managers
Climate and Culture. To be better information sharing, which
set the tone in their units and should
effective, efforts to build reliability usually is about getting access to
foster environments in which ana-
need to operate on two organiza- more data from outside of compo-
lysts are free to present minority
tional levels, climate and culture. 8 nents. Knowledge capture and trans-
viewpoints and alternative views and
Climate is the perception within an fer is about preserving insights
to question key assumptions. Man-
organization that senior leaders are gained within a component, enabling
agers have levers for doing this:
committed to achieving greater reli- their recovery and regular reexami-
ability and are actively facilitating • They have the power of example. nation, and passing them along to
the effort. Culture alludes to how the Experience shows that the tone set new and future members of the com-
values have been adopted in the by first-line managers and senior ponent. Despite improvements in our
rank-and-file and have become part analysts, whether positive or nega- IT systems over the years, we are
of daily processes. tive, will be embraced by more arguably doing worse in knowledge
junior analysts. capture than we have in the past. In
A true HRO will operate on both my experience components shared
levels simultaneously. If senior man- • Managers have the power to common sets of “read” files for all
agement is trying to promote a reli- reward behavior that contributes to team members. Today analysts tend
ability climate but is not thinking constructive questioning environ- to maintain their own personal files
about how these values and prac- ments or curb behavior that under- which few others can see or use.
tices are inculcated at the working cuts them.
level, then efforts to become more Willingness to Countenance
• They can mandate papers that Failure. On a day-to-day basis,
reliable are likely to falter. Con-
question existing points of view or managers must demonstrate willing-
versely, a good culture of reliability
institute regular “stand downs” to ness to discuss near failures and see
and tradecraft can be eroded and
review analytic lines or explore them as such. Organizationally, there
undermined if working-level person-
vulnerabilities that could lead to can be a strong disincentive to do
nel perceive that senior managers are
failure. this, as higher management and out-
only mouthing slogans. Harmoniz-
siders could perceive such discussion
ing these two levels is a significant
as an indication of poor perfor-
challenge.

7 David A. Garvin, Amy C. Edmondson, and Francesca Gino, “Is Yours a Learning Organization?” Harvard Business Review, March 2008. See also, David A.

Garvin, “Building a Learning Organization” Harvard Business Review, July–August 1993.

8 Anthony Ciavarelli and Jeffrey Crowson, “Organizational Factors in Accident Risk Assessment,” unpublished paper presented to the Safety Across High-

Consequences Industries Conference, 9–10 March 2004, 1. Ciavarelli and Crowson are from the Naval Postgraduate School and are focused on aviation
safety. I have taken their distinction between climate and culture and adapted it to the issue of intelligence reliability.

36 Studies in Intelligence Vol. 57, No. 2 (June 2013)

 Rethinking Intelligence Failure

By seeing the analytical process in a more integrated and ho-

mance. In reality, however, discus- listic way, we can develop a better sense of where discrete ac-
sion of smaller or near failures tions fit into the process and how they may affect other parts of
actually can help components get bet-
the process and its outcomes.
ter reads on situations they face and
better position themselves to help
prevent or mitigate big failures. Conclusion: The Value of This sense of where discrete actions fit
Paradigm into the process and how they may
Consumer Relationships. affect other aspects of the process
Focus must be kept on developing The value of the conceptual frame- and its outcomes. Such recognition
relationships with consumers. This work sketched out here is in its com- would enable us to go beyond either
can improve reliability by helping to prehensiveness and forward-leaning reorganizations or ad hoc solutions
better focus support for them, but it orientation. At a minimum, it can and short-term fixes and allow us to
can also help mitigate one effect of give us a more consistent vocabu- develop better organizational and
failure. While consumers are never lary as we continue to explore intel- systemic approaches for improving
happy about failure, a component ligence success and failure. No our reliability.
with a strong track record as a reli- doubt elements of this framework
able partner that does due diligence touch on existing practices, although All of this, however, hinges on
will be more likely to be given lee- they are probably carried out in ad attitudes toward failure. Component
way after a failure, especially if the hoc and inconsistent ways. No doubt leaders who deny that failure is a
component demonstrates positively as well that much more could be said possibility almost certainly have set
that it is taking responsibility for its about specific parts of the COPE themselves up for failure. If they
mistakes and is learning from and framework and practices that would acknowledge they are almost always
correcting them. flesh it out in even more practical one step away from failure and
terms. apply the reliability cycle to manage
the risks they face, they will have
Nevertheless, by seeing the analyt- taken their organizations to the place
ical process in a more integrated and they should aim—and need—to be.
holistic way we can develop a better

vvv

Studies in Intelligence Vol. 57, No. 2 (June 2013) 37