1.

1 Performance Management for Research and

Development
An R&D environment poses particular challenges.

First, in an R&D environment, performance is difficult to measure and the outcome of

R&D activities often cannot be quantified in advance. Some benefits may be
monetizable (i.e., measured in units of currency such as dollars), while others likely are
not.

Second, the timeliness of the data is a concern because of the often long time span (e.g.,
several decades) between starting an R&D effort and realization of those benefits.

Third, in R&D there are many unknowns, which cannot be measured, as well as
generally higher risks of failure.

R&D aims to grow capacity (i.e., mental abilities) and knowledge. From the macroeconomic
point of view, when money is moved into certain areas of research, there is historically a
secondary movement in the percentage of experts (e.g., PhD candidates) in those areas. Despite
the longstanding interest in increasing accountability of R&D programs, there are relatively few
models that program managers can follow to evaluate the effectiveness of R&D.

2 Practices of R&D Organizations

MITRE investigated performance management practices of R&D organizations. Section 3.1
portrays commercial industry practices, and Section 3.2 describes government practices. Eleven
case study examples are provided, including five from commercial industry and six from
government. Additional information is available in Appendix A.
What follows are several examples from commercial industry - Lucent Technologies, Xerox,
Hewlett-Packard, and IBM - detailing new ways to measure and analyze performance in
companies or business units whose sole focus is R&D.
3.1.1 Lucent Technologies
Lucent Technologies developed a decision support model called the Value Creation Model
(VCM) in an attempt to better measure the operations of its Advanced Technologies (AT) Group's
R&D Operations. At the time it was rolled out in 1997, Lucent had applied the model to a
portfolio of over 500 research-driven innovation projects that required up to five years to
transition from invention to commercialization. The projects were investment intensive, spanned
numerous market settings and were composed of a wide array of differing, and often emerging,
technologies.1
The major component of the VCM is the Portfolio Value Metric (PVM) which is defined as the
ratio of the NPV of expected future cash flows resulting from the commercialization activities
attributed to the NPV of the R&D expense. In addition to the PVM, a risk calculation can also
be generated for individual projects or portfolios. The parameters for a triangular distribution2
1
2 A triangular distribution is typically used as a subjective
description of a population, often in cases when only limited data
is available. It is a continuous probability distribution with a
1

are created from three (pessimistic, realistic and optimistic) cash flow estimates for each project.
The resulting calculation of risk is then displayed as a histogram projected by the VCM.3
Although the PVM could be categorized as a financial metric the VCM analysis includes six
other qualitative, non-financial, attributes from the following four categories:

Strategic Initiatives
e.g., an internal strategic initiative that dealt with broadband
technology

Market Categories
e.g., the market category of the investment itself, the life cycle
stage of the market

Intellectual Property
e.g., category of intellectual property, the life cycle stage

Business Units
e.g., business units from Lucent's internal organizational
structure that are supported by the particular R&D project

These additional qualitative metrics serve to create a more well-rounded measurement of the
investment or portfolio performance. Analysis of the qualitative data, due diligence on multiple
estimate scenarios (i.e. pessimistic, realistic and optimistic), and linkages of projects to other key
initiatives and business units within the company all serve to provide Lucent management with
much more data to base future decisions. It also allows the managers within Lucent's AT Group
to better defend their past work and to make a stronger argument for why funding should
continue in the future.
3.1.2 Xerox
Like many large and mature technology-based firms, Xerox employs a budgeting process for
R&D. Its internal process uses a key planning metric for the coming year, which it calls "R&D
intensity." This metric is defined as the planned R&D investment divided by the anticipated
revenue. The R&D intensity metric is periodically compared to competing firms, and it is kept
relatively constant year over year.
Similar to other large technology firms, Xerox organizes its R&D budget into two main parts:
product development and research laboratories. Approximately 80% of the total R&D budget is
allocated to product development and managed by the business divisions. The remaining 20% is
distributed to its research laboratories and is controlled at the corporate level.
Xerox makes tactical cost adjustments
The annual process for
determining the next
and reconsiders any strategic
year's R&D budget coincides
with the corporate-wide
implications at the beginning of the next
budget activities and is tightly
aligned with projected
planning cycle.
revenues and profits. The
specific R&D budget is
created by categorizing those requests that fall under the scope of product development versus
research laboratories, being sure to identify the number of years anticipated for the realization
of the investments. Based on the current and anticipated economic environment, updated

lower limit (minimum), upper limit (maximum), and modal

(most likely) value.
3
2

financial targets are established for the following year and costs across the corporation are
adjusted to meet the new values. Unlike the model at Lucent which ties potential R&D
investments to strategic initiatives at the outset, these cost adjustments at Xerox are tactical; any
strategic implications are reconsidered at the beginning of the next planning cycle. Any
decision to increase R&D spending is usually tied to next year's anticipated revenue, with
revisions possible depending on short-term affordability. This approach implicitly assumes that
the R&D budget followed revenue and profit growth, rather than driving it.4
3.1.3 Hewlett-Packard
Hewlett-Packard is the
At Hewlett-Packard, the continual investment in
future product lines is a key activity that serves to
new product revenue
establish a constant stream of revenue for the
product innovation
company even as older product lines are phased out
growth. The study
by
which a company converts internal resources

focus of a study on
and the link between
activities and revenue
defines the process

(e.g., labor, materials) into products, the products are consumed by its customer base, the
company earns revenue, and the revenue can then be reinvested into current operations as well as
future R&D for innovative and new product lines. The continual investment in future product
lines is a key activity that serves to establish a constant stream of revenue for the company even
as older product lines are
phased out of production.
Three factors are identified that A large company's revenue contributions drive revenue growth:
the fraction of revenue invested of a particular new-product year (or
in product innovation,
vintage) fall into a regular pattern over
new product revenue gain, and time, which enables a company to
the behavior of revenue
over time for a particular
determine mathematical relationships for business. Using a graph
called a product vintage chart, revenue growth as a function of R&D
a large company's
investment and new product revenue
revenue contributions of a
particular new-product
growth.
year (or vintage) fall into a
regular pattern over
time, which enables a company to determine mathematical relationships for revenue growth as
a function of R&D investment and new product revenue growth. In this way, senior managers
can gain clearer understanding of the interplay between product innovation, R&D investment,
revenue growth, and profitability over time.5
3.1.4 IBM
Many companies believe there is a strong correlation between future revenue growth and internal
investments made in R&D.
Some argue that R&D
John Armstrong, former Vice President of
should increase spending,
regardless of the specific
Research and Technology at IBM, claims
investment. However, there "you can spend too much on R&D."
is some level of R&D
spending that will not yield
additional revenue

4 Hartmann, George, "Planning Your Firm's R&D Investment,"

Research Technology Management, March 2006.
5 Patterson, Marvin L., "From Experience: Linking Product
Innovation to Business Growth," Journal of Product Innovation
Management, 1998.
3

return. John Armstrong, former Vice President of Research and Technology at IBM, claims "you
can spend too much on R&D."6
In an attempt to quantify the value of its e-business initiatives, IBM established the Risk and
Opportunity Assessment process to assist in selecting and prioritizing e-business initiatives.
Within this process, IBM uses the Value Chain Modeling Tool to analyze and model the value
chain of its enterprise. This internal IBM approach has been successfully used to improve the
financial and operating performance of several of its business units.7
The Risk and Opportunity Assessment process includes the following stages:
1. Collection of data about the R&D initiative: Includes any
background information on the project as well as documented
assumptions. This results in a data collection plan, definitions of
data requirements, and the actual collection of data.
2. Modeling and Analysis: A baseline model is built, defining key
financial and operational drivers for the initiative as well as
highlighting various scenarios which could positively or
negatively impact the success of the project.
3. Development: A cost-benefit analysis is performed, potential
solutions are prioritized, and a final choice is made.

3 R&D Metrics
Section 4 provides example R&D metrics used by commercial industry and government
organizations and programs.

3.1 Commercial Industry Practices

As mentioned in Section 3.1, a common industry practice is to categorize projects as either
product development or research innovation, and funding is allocated at approximately 80% and
20%, respectively. However, the case examples show that many companies think about R&D
budget allocation and spending trends in different ways. Metrics commonly used by commercial
industry include:

Net Present Value (NPV)

ROI

Many organizations rely on metrics to

measure R&D performance instead of
relying on the traditional use of
shortterm financial metrics alone.

Total Cost of Ownership

(TCO)
Discounted Cash Flow
Budget variance (delivering on or below allocated budget)
Quality measurements (meeting specifications/requirements)
Risk
6 Hartmann, George, "Planning Your Firm's R&D Investment,"
Research Technology Management, March 2006.
7 Nassar, Ayman, "A System-Based Approach for Defining IT
Operations Value Proposition," Management Science and
Engineering, October 2006.

 Alignment with corporate strategies

Cost
Schedule
Organizational flexibility
Intellectual Property factors
Market Lifecycle factors
Fit with existing product portfolio
Market Share
Since R&D projects are unique compared to other corporate spending projects, many firms have
expanded the list of metrics used to measure R&D performance instead of relying on the
traditional use of short-term financial metrics alone.

3.1.1 Example Efficiency Measures

For illustrative purposes, example efficiency measures developed by commercial organizations Alcoa, Dow Chemical, IBM, and Procter & Gamble - are shown in the table below.8
Table 3. Commercial Industry Example Efficiency Measures
Agency or
Organization
Alcoa

Alcoa
Alcoa
Alcoa
Alcoa
Alcoa
Alcoa
Alcoa
Alcoa
Alcoa
Alcoa

Efficiency Measure
Return-on-investment calculation:
(FY 2005) Improve existing ARIS by converting its mainframe system into Web-based system
designed by OAR and IC representatives in consultation with contractor
Variable cost improvement
Margin impact from organic growth
Capital avoidance
Cost avoidance
Annual impact of these four metrics over 5-year period becomes numerator; denominator is total
R&D budget
Metric is used most often to evaluate overall value of R&D program and current budget focus
(Atkins 2007)
Time (Atkins 2007)
Cost (Atkins 2007)
Customer demand (Atkins 2007)
Risk (Atkins 2007)
Impact on business (Atkins 2007)
Impact on customers (Atkins 2007)
Location (Atkins 2007)
Intellectual property (Atkins 2007)
Aggregate R&D expenditures by laboratory group or by identifiable programs and publish value
capture or success rate for each on annual basis (Atkins 2007)
ROI on R&D spending; success rate of launched products (Atkins 2007)

8 "Evaluating Research Efficiency in the U.S. Environmental

Protection Agency," National Research Council, 2008.
5

Dow
Chemical
IBM
IBM
IBM
IBM
IBM
IBM
Procter &
Gamble
Procter &
Gamble
Procter &
Gamble

Publications; participation and leadership in scientific community (collaborative research efforts;

trade associations; ILSI-HESI; external workshops; adjunct faculty positions, journal or book
editors, professional societies) (Bus 2007)
ROI on Summer Internship Program and Graduate Fellowship Program: what percentage return
as regular IBM research employees?
Bureaucracy Busters Initiative to reduce bureaucracy in laboratory support,
informationtechnology support, HR processes, and business processes (Kenney 2007)
Tracking of patent-evaluation process (Kenney 2007)
Customer-satisfaction surveys for support functions to evaluate effect of service reductions
(Kenney 2007)
Measurement of response time and turnaround for external contracts (Kenney 2007)
Measurement of span of responsibility for secretarial support (Kenney 2007)
Time saved in product development (Daston 2007)
Increased confidence about safety (Daston 2007)
External relations benefits (although not quantifiable) (Daston 2007)