WORLD CLASS

DESIGN TO MANUFACTURE

The R&D Effectiveness Index:

A Metric for Product Development Performance
Michael E. McGrath and Michael N. Romeri
Pittiglio Robin Todd & McGrath

“How effective are my company’s product their strategies and opportunities are similar and
development efforts? Is our product development their product development processes are equally
process improving, or possibly deteriorating? We effective.
can measure specific projects but have no way of Studies have generally shown little correlation
measuring how we are doing overall.” This between the level of R&D spending and success.
CEO’s comments are typical of the frustration in Some[2,6] have measured limited correlation
measuring overall product development success. between the percentage of R&D spending and
Many companies are now realizing that both revenue growth. Although these studies show
the number and the success of new products rely some relationship, it is not enough to suggest that
on the performance of their product development spending more on R&D is always better.
process. Until recently, improving this process Spending 50% of revenue on R&D, for example,
was such a clear opportunity that companies did will likely increase revenue, but it is unlikely that
not need to measure performance. They just the return on investment would justify this for
started to improve their processes. However, now very long.
that some companies have made major Product development cycle times are a key
improvements to their product development performance measure of the product development
process, they need an appropriate way to measure process[3]. Short cycle times generally increase
its performance. product success and improve development
However, product development is much more productivity. However, this does not measure
difficult to measure than other business processes success across products or guide the amount of
and because of this, there are no broadly accepted spending on R&D.
performance metrics as there are for other Companies are now beginning to focus on the
business processes. For example, in effectiveness of their R&D investment instead of
manufacturing, inventory turnover and gross how much they invest. They want to get more
margin percentage can be used as metrics of the from the same level of investment instead of
manufacturing process. increasing their investment. Even Business Week’s
Some metrics are applied. The investment annual R&D Scoreboard[1], which ranks
community and the press, for example, have companies by R&D spending, points out that
resorted to using the percentage of R&D spending R&D efficiency is becoming a strategic necessity.
as a comparative measure – with more spending Pittiglio Rabin Todd & McGrath (PRTM)
as better. This is a little like saying that more recently conducted a comprehensive study of
spending on plant and equipment is always better, development metrics for electronic systems
even though the appropriate level of capital products. The study included extensive
investment considers expected return and a benchmarking of time-to-market, sources and
company’s long-term strategy. In the same way,
business executives sometimes compare their Reprinted by permission of the publisher from “The
R&D expenditure (as a percent of sales) to that of R&D Effectiveness Index: A Metric for Product
their competition. The comparison is only valid if Development Performance” by Michael E. McGrath
and Michael N. Romeri, Journal of Product Innovation
Management, Vol. 11 No. 3, pp. 213-220.  1994 by
World Class Design to Manufacture, Vol. 1 No. 4, 1994, pp. 24-31
MCB University Press, 1352-3074 Elsevier Science Inc.

24
 VOLUME 1 NUMBER 4
1994

uses of R&D, performance to targets, and best projects and is therefore much more difficult to
practices. Forty-five companies participated in the measure. Unsuccessful projects and wasted efforts
study. Although they varied by size and industry, need to be offset against successful ones. Projects
all developed similar products – electronic with varying degrees of success need to be
systems where there is both electronic and weighted together. Finally, R&D investment that
software content. Table I shows a profile of the spans several periods needs to be matched against
participating companies. a payback that spans future periods.
Study participants included American, Although these complications limit the
European, and Asian companies. Each provided precision of any aggregate performance metric, a
detailed data on its product development metric – even an approximate one – is needed
activities, enabling the analysis of the data by to measure overall performance. Without such a
product complexity, type of product, and level of
metric, management of the product development
effort involved. They study’s benchmarks are
process is purely subjective.
used by participants to evaluate their product
development process and provide cycle-time The R&D effectiveness index (EI) compares
guidelines for scheduling new development. The the profit from new products to the investment in
study showed that the participants did not new product development, using the following
consistently use any single overall metric to formula (all% are stated as a percentage of
measure their product development process, but revenue):
they indicated that one was badly needed. EI =
As part of this study, PRTM introduced a new % New Product Revenue × (Net Profit % + R&D %)
metric for measuring overall product development R&D %
performance – the R&D effectiveness index.
As a simple interpretation, the index computes the
ratio of increased profits from new products
divided by the investment in product
R&D Effectiveness Index
development. When the index is above 1.0, the
The R&D effectiveness index is an aggregate return from new products is running at a rate
measure of the overall success of a company’s
greater than the investment.
product development efforts. It fills the void for
For example, assume that a company has a 9%
an overall metric at a level above individual
net profit. Also assume that it invests 6% of
products and development projects, and provides
management with a tool to measure the long-term revenue in R&D and that 40% of its revenue is
effectiveness of its product strategy and product derived from new products. Its R&D effectiveness
development process. index would be 1.0 [computation: 40% × (9% +
Overall or aggregate performance involves 6%) / 6%]. On the other hand, if this same
many new products and product development company derived only 20% of its revenue from
new products then it would have an index of 0.5
(20% × 15% / 6%). It would not be getting as
much from its investment in R&D.
Industries %
The R&D effectiveness index was computed
Communication systems 35.6 for companies participating in the study. The
Industrial/Medical electronics 22.2 results were then analyzed and correlated to other
Consumer electronics 15.6 performance factors. Figure 1 shows the
Computers 13.3
Peripherals 13.3
100
21%
Size (revenue) <0.5
39% 0.5 - 1.0
$250 million + 28.2
$100-$250 million 25.6 1.0 - 1.25
>1.25
$50-$100 million 20.5 18%
$25-$50 million 10.3
<$25 million 15.4
100 22%

Table I. Figure 1.
Profile of Participating Companies R&D Effectiveness Index Distribution from the Study

25
 WORLD CLASS
DESIGN TO MANUFACTURE

distribution of R&D effectiveness index values begins to flatten at that point, consistent with the
derived from the study. 2.5-year new product definition. In other
Of the participants, 39% (21% + 18%) had an industries, the curve would be different.
R&D effectiveness index greater than 1.0, Arguments could be made for different
indicating that new products generated more definitions of new products. For example, the
profit than the investment made in R&D. If any entire product life-cycle could be used in the
time-value of money is considered, then only 21% definition, because the entire life-cycle is used to
(companies with an EI greater than 1.25) evaluate a new product opportunity. This
generated a return on their investment in R&D. definition would include all revenue and could
Interpreting the index properly requires an not distinguish between new and old products.
understanding of the numerator and denominator: The index would simply be the ratio of profit to
new product profit and R&D investment. These R&D investment, losing the importance of new
are subject to differences in interpretation. products.
The definition could be longer or shorter, but
the half-line concept is reasonable. Consistency is
New Product Profit the most important consideration in determining
Computing new product profit starts with the the definition of a new product. Comparisons are
definitions of new products. In the R&D possible when the definition is consistent, and the
effectiveness index, new products are defined as real objective of the metric is relative
those that are still in the first half of their product comparisons, not absolute performance.
life-cycle. For electronic systems, the average New product profit is computed by multiplying
product life-cycle is approximately five years, so the percentage of revenue from products
new product revenue covers products introduced introduced in the last three years by the rate of net
in the last three years. (This averages to be 2.5 profit combined with the percentage of R&D
years because products introduced in the last half spending. Net profit is used instead of gross
of a year are less than 0.5-years-old, whereas margin because selling and administrative costs
others are more than 0.5-years-old.) must be covered by new product revenue. The
In this study, product life-cycle was supported percentage of R&D investment is added back into
by analyzing the new product curve for profitability because the index computes how well
participating companies. The curve plots the profit covers this investment. An EI of 1.1
percentage of revenue from products introduced a represents a return of the original investment, plus
year ago, two years ago, etc. Figure 2 illustrates 10%.
the new product curve for the electronic systems Either actual profit from new products or a
companies in the study. representative average profit can be used. Actual
For 1992, approximately 12% of revenue came profit is more precise, but may not be accurate.
from products introduced in the last year, whereas The allocation of administrative and other indirect
approximately 50% came from products costs makes it difficult for many companies to
introduced in the last three years. The curve accurately identify the actual profit of individual
products. Both alternative methods were
compared in the study, and there was no
Average %
of 1992 revenue significant difference in results. The profit rate
70 –
-
- may also need to be adjusted for cases where
-
-
60 –
-
there are significant nonrecurring expenses.
-
-
-
50 –
-
-
-
-
40 –
-
-
R&D Investment
-
-
30 –
-
The investment in R&D is based on spending as a
-
-
- percentage of current revenue. Although this does
20 –
-
-
-
not match the time period of the profit derived
-
10 –
-
-
from this investment, it does provide a reasonable
-
0–
- approximation. Companies tend to invest a
92 91 90 89 88
Years in which products were introduced
consistent percentage of revenue in product
development from year to year. The EI is
Figure 2. therefore a running average.
New Product Revenue as a Percentage of Total The definition of R&D investment also varies
Revenue by company. Some define R&D as spending by

26
 VOLUME 1 NUMBER 4
1994

their engineering and research departments, driven by time-to-market (TTM), whereas the
regardless of whether it is spent on new product second is driven by the characteristics of the
development or used to support existing products. product as well as time-to-market.
Companies that divert a significant portion of The profitability of new products is a result of
their R&D investment to product support would how successful the products are in meeting the
have a lower index than if they invested more in customer needs compared to competitive
new products. Some companies include the cost products. New product profit is the product of the
of other departments involved in product sales volume and the rate of profitability. It may
development, whereas some do not. Sensitivity be reduced by a high degree of problems, such as
analysis of these variations did not show any engineering change orders (ECOs), when new
significant difference in results; however, an products are released.
individual company with accounting practices at R&D spending comprises the number of
the extreme of the variation could have development projects and product development
noncomparable results. productivity. Time-to-market is the most
significant driver of productivity. If a product is
developed in less time with fewer delays, then the
run rate (spending per month) costs are incurred
for fewer months. The cost of wasted
Drivers of the R&D Effectiveness Index development is another productivity factor. This
The R&D effectiveness index provides insights is the amount spent on the development of
into the dynamics of the product development products that never come to market.
process and how it affects profitability. In fact, the Understanding each of these dynamics and how
R&D effectiveness index itself is the result of key they influence the R&D effectiveness index is
drivers of the product development process. This essential to applying it to measure the success of
is illustrated in Figure 3. the product development process. Decreasing
The percentage of total revenue derived from time-to-market, improving product profitability,
new products is a result of how long it takes to get increasing product development productivity, and
these new products to market and their early sales reducing wasted development all increase the
success once they are released. The first factor is R&D effectiveness index.

From new products


Profit
released in year Y (start date + TTM ) 
Profit From new products  Cost of product
released in year Y-1(start date + TTM +1)  launch Inefficiencies
From new products 
Profit released in year Y-2(start date + TTM +2)

New product
profit in
year Y
R&D
= effectiveness
index year Y
R&D spending in
year Y


Project 1 run rate X TTM  Portion in

Project 2 run rate X TTM year Y 
. 
+
. 
.
 Spending on 
Project n run rate X TTM  cancelled 
projects

Figure 3.
Drivers of the R&D Effectiveness Index

27
 WORLD CLASS
DESIGN TO MANUFACTURE

Application of the Index Benchmarking provides a powerful tool to

The R&D effectiveness index is a useful understand performance relative to other
quantitative metric of product development companies, particularly “best-in-class”
performance that companies can use to manage companies. Benchmarking studies, like the
their product development process. However, it one referenced here, can provide participants
should be interpreted with caution, as should all with this comparison. Any such study,
metrics. Proper application requires an however, needs to be thorough enough to
understanding of its basis and limitations. The enable comparability of the benchmarks.
following are some suggestions for application:
(1) The index is best used as a metric of relative Case Study: The R&D Effectiveness Index
performance. Consistent improvement in the
product development process should be the
as a Management Tool
goal of all companies, and the R&D The following case study provides an illustration
effectiveness index can be used to measure of how the R&D effectiveness index can be used
this continuous improvement. It is a long- as a management tool guide and measure
term metric. Improvements to the product improvement to the product development process.
development process will be seen in future The company has six divisions, each developing
years, when the benefits are achieved. different products. Its CEO was not satisfied with
the results of its new product development efforts.
(2) The absolute value of the index can be used to He believed intuitively that while the company
measure ROI. An index of less than 1.0 was investing heavily in research and
implies that a company may not be getting a development, it was not getting enough in return.
sufficient return on its investment in R&D. “Our growth has been less than 10% per year,”
There are some limitations to this he said, “while the industry has been growing at
interpretation, including the definition of new 25%. Our profit margins are shrinking every year,
product profit. It is not a conclusive metric; it and we don’t seem to have as many new products
is an indicative metric. Companies with an as our competitors.”
index well below 1.0 would be wise to He did not know what to do. “The board of
investigate the reasons behind this, including directors thinks that I should cut R&D spending.
their portfolio of R&D projects. They also believe that we shouldn’t be investing
(3) The index can measure performance across the same percentage in R&D for each division,
multiple divisions. Large companies can use but they don’t have any specific suggestions on
the index as a comparative measure of how to shift it. Marketing and engineering, of
performance across all divisions. Because course, say that we are not spending enough.
market opportunities for each division can be Lately, I’ve begun to wonder if we have
very different, the absolute index value may fundamental problems in our product
not compare the success of their product development processes.”
development process. Differences in the index The CEO did not have a way to measure the
across divisions may identify which divisions performance of his product development process,
have better opportunities. Relative changes and as a result he did not know how to fix it. The
will measure their improvement. R&D effectiveness index provided him with the
performance measure he needed.
(4) The index measures the differences in Overall the company had an R&D effectiveness
opportunities across industries. There was an index of 0.5. Approximately 20% of its revenue
indication in the study that the R&D came from new products, and the average profit
effectiveness index varies by industry. Some on these products was 12%. It invested 8% of
industries have a much higher payback on revenue in R&D. The index of 0.5, [20% × (12%
their R&D investments than others. They + 8%)] / 8%, however, varied widely by division,
have a higher potential, whereas others have as can be seen in Table II.
significantly less. In some cases, a low R&D Even though it invested heavily in new product
effectiveness index may be more an development, Division A had the highest index
indication of an industry’s potential than the due to a higher percentage of new products and a
performance of a particular company in that higher rate of profitability. Divisions C and D also
industry. invested heavily in R&D, but their index was
(5) The index can be used as a benchmark lower because new products were less profitable.
comparison to other companies. Division B, E and F has a low percentage of new

28
 VOLUME 1 NUMBER 4
1994

New 50 –
Effectiveness index
– 1.40

product Profit (%) Investment Effective- 45 –

– 1.20
revenue of new in R&D ness 40 –
(%) products (%) index

Percentage of revenue
35 – New product – 1.00

Effectiveness index
revenue
Division A 30 25 9 1.13 30 –
– 0.80
Division B 15 18 7 0.54 25 –
Division C 20 10 8 0.45 20 –
– 0.60
Profit %
Division D 30 10 9 0.63 15 – – 0.40
Division E 10 10 6 0.27 10 – R&D %
Division F 5 12 7 0.14 – 0.20
5–

0– – 0.00
Table II. 0 1 2 3 4 5
Division Performance Year

Figure 4.
products, and as a result had a low R&D Performance Goals
effectiveness index.
No one, including the CEO, knew why each goals for the company as a whole; each division
division invested what it did in R&D. A budget of has its own individual goals.
6% to 9% seemed to be acceptable within the
The company’s plan included a number of
company, and the R&D budget just continued
changes to its product development process and
from year to year, increasing at about the same
the way it did business:
rate that the division grew.
Differences in the R&D effectiveness index ● A new product development process[4] would
across divisions were not easily explained. be implemented throughout the company. This
Perhaps some had better market opportunities was expected to reduce time-to-market and
than others. Division A, for example, was in a increase R&D productivity. As a result, more
“hot” market, whereas Division F was in a very new products were expected starting in year 2.
mature market. Perhaps some divisions had a ● More rigorous financial screening of new
better product development process than others. product proposals was implemented as part of
There was no consistency in the way each a new phase review process. This would stress
developed products. higher profitability as a requirement for
The CEO set an ambitious goal for funding new projects.
improvement. Using the R&D effectiveness index
● A strategic evaluation was also initiated,
as the overall metric, he set a target to improve
effectiveness from 0.5 to 1.4 over the next five focusing on the growth opportunities in
years. The underlying goals were to increase new Divisions E and F with the expectation that
product content of revenue from 20% to 40%, R&D spending would be reduced in these
improve the new product profitability from 12% divisions if they did not have acceptable new
to 15%, and reduce R&D investment from 8% to product opportunities.
6% of revenue. This would result in an index of Although the work was just beginning, the CEO
1.4 [40% × (15% + 6%)] / 6%. Each of these of the company now felt like he was in control,
underlying goals, as well as the R&D having set a clear direction. “We certainly have
effectiveness index, were set by year. These are our work cut out for us, but we know what we
charted in Figure 4. need to do and how we are going to get there. The
These individual goals were the expected result R&D effectiveness index is our overall guide.
of an aggressive improvement plan. The product There is no magic to the goal of 1.4. Perhaps we
development process would be upgraded to could do even better. I do know, however, that an
world-class status. The percentage of R&D improvement of that magnitude is both achievable
spending would be reduced starting in year 2. and badly needed.”
Improvements in the number of new products and This case study illustrates the R&D
product profitability were not expected until year effectiveness index in action as a management
2, with most of the improvement not coming until tool. It provided the metric that the company
year 3. New product profit was expected to needed to guide its efforts to become more
increase starting in year 2. These were composite competitive. The case also illustrates how a

29
 WORLD CLASS
DESIGN TO MANUFACTURE

company can sort through the complexity of growth also shows a strong relationship between
managing new product development without growth and the R&D effectiveness index.
oversimplifying this complex problem. Companies with a high index grew twice as fast
as the average from 1988 to 1992.
Other characteristics of companies with a high
Characteristics of Companies with a High R&D effectiveness index are also worth noting:
R&D Effectiveness Index ● They invested less on R&D and got more out
The benchmarking study also provided an of it. As a percentage of revenue, the
interesting analysis of the characteristics of companies with a high R&D effectiveness
companies with a high R&D effectiveness index. index invested approximately a fifth less than
These characteristics help not only to understand average in R&D. At the same time, they also
the index a little better, but also provide insights had approximately a third more revenue from
into what it takes to perform well. new products. This reflects the productivity
Companies with a high R&D effectiveness aspect measured by the index.
index (> 1.25) outperformed the average in
● They wasted less on products that did not
growth and profitability. These comparisons are
come to market. Companies with a high R&D
shown in Figure 5. Although higher profitability
effectiveness index cancelled marginal
would be expected (as it is part of the
computation), the magnitude of this difference is projects much earlier in the development
surprising. These companies were twenty-two cycle. As a result, they lost only 14% of their
times more profitable! The difference in revenue R&D budget on work that never came to
market, compared to 21% for the others.
Companies with the lowest R&D effectiveness
index (>0.5) lost 28%.
Profitability
25 – (% revenue) ● They invested proportionally more of their
-
-
-
22.3 R&D budget on totally new products and
-
20 – platforms. They invested 30% of their R&D
-
-
-
budget on totally new platforms compared to
-
15 –
-
less than 25% for the other companies. This
-
- supports the importance of managing the
-
10 –
-
amount invested in new product
-
- platforms[5,7].
-
5–
-
-
● They met their original project goals more
0.98
-
- often. Companies with a high R&D
0–
High R&D Average effectiveness index met their revenue goals
effectiveness
index 50% more often, their profitability goals 20%
more often, and their return on investment
goals twice as often. This suggests that their
Revenue growth product development process was more in
1988-1992
350 –
-
(% change in revenue) control.
- 318.6
-
-
300 –
-
-
-
-
250 –
-
-
Conclusion
-
-
200 –
- This article introduces the R&D effectiveness
-
-
-
150 –
154.9 index as an overall measure of product
-
-
-
-
development success. An aggregate metric like
100 –
-
- this is clearly needed by companies that want to
-
-
50 –
-
measure their improvement, as well as by
-
0–
-
- companies that are not satisfied by their results.
High R&D Average The R&D effectiveness index reflects the
effectiveness
index dynamics of the product development process.
Decreased time-to-market, improved product
Figure 5. profitability, increased development productivity,
R&D Effectiveness Index Correlated to Revenue Growth and reduced development waste all increase the
and Profitability index.

30
 VOLUME 1 NUMBER 4
1994

A study of product development in 45 7. Wheelwright, S. and Clark, K. Revolutionizing

electronic systems companies provided data to Product Development, New York: The Free
compute and analyze a profile of index values. Press, 1992.
This analysis showed a strong relationship
between the index and both growth and
profitability. The study also provided data to
analyze the characteristics of companies with a
high R&D effectiveness index. These companies Michael E. McGrath is a founding Director of
show some outstanding attributes in their product Pittiglio Rabin Todd & McGrath (PRTM), an
development process. internationally recognized management
The R&D effectiveness index has several consulting firm to technology-based companies.
applications as a management tool. It is strongest He is based in the firm’s Weston, MA office.
at measuring relative performance, but can also Seven years ago, Michael initiated the firm’s
measure absolute performance similar to ROI. It product-development consulting practice referred
can identify performance differences across to as PACE (Product and Cycle-time Excellence).
divisions and opportunity differences among Michael has written numerous articles on product
industries. It can also be used as a benchmark development, operations management,
comparison. Some of these are illustrated in a manufacturing strategy, and international
case study. manufacturing and is co-author of the book
Product Development: Success Through Product
and Cycle-time Excellence and author of the book
n Product Strategy for High-Technology Companies
to be published by Irwin Professional Publishing.
Prior to founding PRTM, Michael worked for
References Texas Instruments, Computervision, Price
1. Coy, P, In the Labs, the Fight to Spend Less, Waterhouse & Co., and McCormack & Dodge.
Get More, Business Week (New York: He graduated magna cum laude from Boston
McGraw-Hill), pp. 102-104 (July 5, 1993). College with a BSc in Computer Science and
2. Gilman, J. Inventivity, New York: Van Management Science and received an MBA from
Nostrand Reinhold, 1992, pp. 7-27. Harvard Business School. He is also a CPA and
CMA.
3. Griffin, A. Metrics for measuring product
Michael N. Romeri is a Director in PRTM’s
development cycle time, Journal of Product
Innovation Management 10:112-125 (March Weston, MA office. He works extensively in the
1993). firm’s product development consulting practice,
directing large international projects. Mike has
4. McGrath, M., Anthony, M. and Shapiro, A., worked extensively within the electronics
Product Development: Success Through industry since 1979. Prior to joining PRTM, he
Product and Cycle-time Excellence,
worked with AMD and Flextronics. Mike’s
Stoneham, MA: Butterworth-Heinemann,
academic experience includes a BA from Harvard
1992.
College and an MBA from Harvard Business
5. McGrath, M. Product Strategy in High- School.
Technology Companies, Burr Ridge, IL: Irwin For further information, contact Michael
Professional Publishing, 1994 (in press). McGrath, Managing Director, PRTM, 9 Riverside
6. Morbey, G. R&D: Its relationship to company Rd, Weston, MA 02193, USA. Tel: (617) 647
performance, Journal of Product Innovation 2800; Fax: (617) 647 2804.
Management 5: 191-200 (March 1988).

31