Project Portfolio Management Tools: Which Approach is Best?

Part 6: Evaluating Tools

Part 6: Evaluating Tools

Part 1 identified available tools for project portfolio management, Part 2 described key differences, Part 3 summarized costs and risks, Part 4 identified the critical weakness of most tools, and Part 5 described tool components. This part provides a framework for evaluating tools. How can we determine whether a decision model/tool is adequate? Six criteria are relevant: (1) accuracy, (2) logical soundness, (3) completeness, (4) practicality, (5) effectiveness, and (6) acceptability [1]. The subsections below clarify these considerations.

The Tool Must Be Accurate

The value of a PPM tool depends, most obviously, on its ability to produce accurate recommendations. Important questions include: Can the tool be counted on to produce reliable estimates? Is it biased toward or against certain projects, interests, or considerations? Are results highly sensitive to untested or untestable assumptions? Does the tool produce outputs with an acceptable level of confidence and precision? Does the tool indicate the confidence level or precision associated with outputs? Flaws and omissions in decision models can produce large errors in recommendations. As evidence, see the examples in the boxes below.

Intuitive and Reasonable-Sounding Tools Can Be Inaccurate

The Los Alamos National Laboratory (LANL) was seeking a tool for prioritizing equipment purchases, seismic upgrades, and other highcost investments. The lab reviewed a half dozen prioritizing models used in industry and by government agencies, noting that each involved a very different approach. A team consisting of the author plus experts from LANL, two other national laboratories and a university, was formed to develop an improved prioritization model, called the Laboratory Integration and Priority System (LIPS) [2]. Unlike the other models, LIPS was based on multiattribute utility analysis, a formal theory for valuing projects [3]. A test was conducted to compare project rankings produced by LIPS with those produced by other models. The results surprised many. Even though the other models seemed intuitive and reasonable, projects that ranked the highest using LIPS were often ranked low by other models.

Being Well-Established Is No Guarantee that a Tool Is Accurate

As another example, in the late 1980's Congress ordered the Environmental Protection Agency to evaluate whether its Hazard Ranking System (HRS), used to place contaminated sites on the National Priorities List for accelerated clean up, was the best-available site-ranking model. The analysis consisted of comparing site rankings produced by the HRS and other models (used by various states and other government agencies) with a ranking produced by an expert panel. The analysis also included developing a model based on multiattribute analysis (MUA). Once again, the results were surprising. Although the expert and MUA model rankings were very highly correlated, none of the siteranking models produced rankings that correlated with each other or with the ranking provided by the expert panel. Several models, in fact, produced rankings that were negatively correlated with the expert ranking [4].

www.prioritysystem.com

27

c 2003-2009 Lee Merkhofer Consulting

Project Portfolio Management Tools: Which Approach is Best? Part 6: Evaluating Tools

Unfortunately, inaccuracies in recommendations often only surface after many applications of a tool. It is difficult to collect empirical data for validating a tool's recommendations (few organizations are willing to fund recommended and not-recommended projects for the purpose of testing a tool!). Tool providers may have some evidence of their tools accuracy, but they will naturally emphasize the positive and minimize (or ignore) the negative. The data they have may not be representative of the particular tool configuration or types of projects relevant to your organization. Conducting a pilot test before fully committing to a tool is essential. Choose a variety of projects with different characteristics and see what recommendations the tool makes. Be skeptical of any odd patterns, such as projects with certain characteristics consistently being ranked either high or low. Resist the natural temptation to rationalize the results (garbage in, gospel out). Drill down to fully understand why the results come out the way they do. Since extensive tool testing is usually impossible, assessments of accuracy must, at least in part, be based on a detailed evaluation of how the tools decision model works. In this regard, two other considerations are useful. As described below, the tool must be logically sound and it must be complete.

The Tool Must Be Logically Sound

From a practical standpoint, the logical defensibility of a tool is primarily a function of the degree to which it can be justified by accepted and proven theory. If a model can be shown to faithfully implement accepted theory, more confidence can be attributed to the predictions made by that model. Conversely, if a model is not grounded in theory, it will almost certainly not consistently produce reliable predictions. What exactly is a theory? A theory is a logic predicting what outcome will result from specified actions and why. The "why" is important. A theory must explain cause and effect. To do this, a theory contains an explanation of how things work. A theory is proven by showing that its predictions consistently match observations. To illustrate the importance of theory, consider initial attempts at manned flight. Observing birds, early researchers thought that they could fly if they strapped feathered wings onto their arms and jumped off cliffs. Many animals fly by flapping feathered wings, but this is not the fundamental reason that they can fly. Manned flight only became possible after Daniel Bernoulli developed
The Concepts Need to Be Right
Success may be in the details, but the first step is to get the concepts right. As an example, an organization was using a scoring approach wherein candidate projects were awarded points based on judged performance along various dimensions (financial reward, strategic fit, leverage, probability of success, etc.). The points were added and the result used to rank projects. Although no one objected to the logic, there was a suspicion that something was wrong. Large projects were nearly always high on the list and small projects near the bottom. One major flaw of the tool is that it fails to account for the resource constraint. The relevant logic is ranking projects by the ratio of benefit-to-cost, not benefit alone. Another flaw is that adding the points awarded in the various dimensions does not measure project benefit. The aggregation equation must correspond to the way that the various performance dimensions actually influence value (as described by an applicable theory, such as MUA). For example, points indicating probability of success should not be added. Probability of success should be a multiplierIf a project has zero probability of success the project will generate no benefit whatever.

www.prioritysystem.com

28

c 2003-2009 Lee Merkhofer Consulting

Project Portfolio Management Tools: Which Approach is Best? Part 6: Evaluating Tools

theory explaining how airflow around objects can produce lift. Many models are like the feathered wings of early would-be aviators. Instead of being derived from theory, they are heuristicsrule-of-thumb relationships based on observations that certain characteristics tend to be associated with certain outcomes. For example, a popular heuristic for constructing project portfolios is balance. Project portfolios of successful companies often contain a balance of low-payoff sure things and high-payoff, long-shot gambles. However, balance is not the fundamental characteristic that makes a project portfolio successful. Thus, choosing balanced project portfolios will not ensure success.

Ranking Projects Based on Strategic Alignment is Not Logically Sound

As described in the paper on this website on Mathematical Theory, ranking projects based on the ratio of benefit-to-cost (bang for the buck) can in some circumstances be a reasonable prioritizing technique (provided that there are no project interdependencies, e.g., infrastructure projects and projects that use that infrastructure). Choosing projects that implement organizational strategy is important. However, strategic alignment is not a surrogate for "bang-for-the-buck." Therefore, the approach will not lead to value-maximizing project portfolios.

As noted previously, relevant theories for selecting and prioritizing projects include decision analysis [5], multi-attribute utility analysis (MUA) [6], modern portfolio theory [7], portfolio optimization theory [8], and real options [9]. These theories are well-established within the technical and academic communities and have been proven in many real-world applications.
Modern Portfolio Theory
Modern portfolio theory is a theory of decision making that seeks to construct a portfolio of investments offering maximum expected returns for a given level of risk. The theory quantifies the benefits of diversification as a means of reducing risk.

Real Options
Real options analysis is a valuation theory that views projects as creating options for dealing with an uncertain future. For example, a new factory created by a project carries options to shut down, abandon, or expand, depending on market conditions. Project value depends on the options created. The theory includes methods for computing project value based on the market prices of related assets.

Decision Analysis
Decision analysis is theory and collection of methods for making decisions under uncertainty. The approach involves constructing and analyzing a model of the decision problem to identify the choice, or sequence of choices, leading to outcomes most consistent with the preferences of the decision maker.

The above theories are sometimes termed axiomatic theories. Each begins with a set of axioms (assumptions, or hypotheses) about how things work, for example, about how people or organizations ought to value and choose projects. These axioms can be accepted or rejected based on observations or other evidence. The theory then derives (through mathematical "proofs") conclusions that follow from its axioms. If you can demonstrate that the axioms are acceptable (and the math is correct), the rest of the theory follows. See the side box for an example.

Theories Derive from Axioms

As an example, decision analysis is derived from axioms intended to define rational choice [10]. One such axiom, known as transitivity, states: If there are 3 possibilities, labeled A, B, and C, then, if you prefer A to B and you prefer B to C, you will prefer A to C. The theory shows that if you accept the axioms, then there is a way to mathematically measure your preference for alternatives based on the characteristics of those alternatives and your answers to some basic questions about what you like.

www.prioritysystem.com

29

c 2003-2009 Lee Merkhofer Consulting

Project Portfolio Management Tools: Which Approach is Best? Part 6: Evaluating Tools

By the way, the fact that there are multiple theories does not mean that you will get different answers depending on the theory that you choose to apply. The well-established theories for selecting projects typically give the same answers, provided that each theory is able to address all the relevant factors, the theories are properly applied, and the assumptions for the analyses are the same. The situation is analogous to that for theories in other fields, such as physics. For example, the mathematics for Newtons laws of motion and Einsteins theory of relativity look very different, but both theories predict the same trajectories for everyday objects under everyday conditions. The predictions only differ in situations (e.g., speeds close to the speed of light or extremely small objects) that invoke considerations that arent addressed by Newtons much simpler laws of motion. Likewise, decision analysis and real-options, for example, look very different, but they give exactly the same answers to any problems that both can fully address provided that each theory is correctly applied [11]. Nevertheless, the choice of the theory is critical. Choosing a solution approach based on ill-suited theory will make it extremely difficult, or impossible, to obtain useful and meaningful answers. Rather than cite the theories on which their tools are based, most providers merely reference the analytic techniques that their tools employ, such as balanced scorecards, strategic alignment, decision trees, linear programming and Monte Carlo simulation. Such techniques, by themselves, provide no explanation for why recommended portfolios should be preferred. Instead, they merely describe or refer to mathematical calculations that are performed. For example, scorecards typically use an equation that weights and adds scores assigned to projects. Projects are ranked based on total weighted score. There is no reason why this should lead to the identification of preferred projects. Weighted scoring techniques can sometimes be used to effectively apply appropriate theories, but only if the scoring scales and weights are structured to match the requirements of the theory. For example, MUA theory for valuing projects can sometimes be implemented using scorecards and a weight-and-add equation. However, for a weight-andadd equation to work, the metrics that are scored must meet a condition known as additive independence, scaling functions must be assigned so that the computed performance measures are proportional to value, and the weights must quantify the value of specified improvements against objectives (the weights are often assigned using an assessment technique known as the "swing weight method [13]). Unless these conditions are met, the aggregated score will not measure value and will not serve as an indicator of decision-maker preference.
www.prioritysystem.com

Tools Not Based on Sound Theories Are Exposed When Subjected to Technical Review
Congress required the Department of Energy (DOE) to rank potential sites for disposing of radioactive waste from nuclear power plants. To select the best site, the DOE initially used a scorecard approach. Each site was rated against each of the objectives established for a good site, the objectives were weighted, and the rates and weights were combined to rank the sites. Hanford, a site in Washington State, ranked highest, and the DOE published the results in a draft Environmental Impact Statement. The choice was criticized, especially by officials from Washington State, and the DOE asked a board of the National Academy of Sciences (NAS) to review the ranking methodology. The board, which included experts in decision theory, responded that DOE's method was unsatisfactory, inadequate, undocumented, and biased [12]. DOE was told to redo its analysis (a site ranking model based on MUA was developed). The new analysis ranked Yucca Mountain, Nevada, highest. DOE was forced to change its decision, causing the agency considerable embarrassment.

30

c 2003-2009 Lee Merkhofer Consulting

Project Portfolio Management Tools: Which Approach is Best? Part 6: Evaluating Tools

Thus, the use of a weighted scorecard does not in any way ensure that the requirements of any accepted theory are being met. The defensibility of the recommendations made by any decision model depends on whether the techniques used to value and prioritize projects are consistent with some defensible theory and on how faithfully the model implements the requirements of that theory. As an example of the dangers of using tools not based on sound theories, see the side box example above describing the Department of Energys initial attempt to rank potential sites for a nuclear waste repository. Logical defensibility is particularly important when using a tool to help make controversial decisions. Although most project decisions aren't as controversial as nuclear waste, some (for example, an electric utility's decision to acquire right-of-way to construct a transmission line) can be. Using a logically sound approach avoids errors associated with unsound methods and reduces the risk of successful challenges to the credibility of decisions. Although logical soundness does not guarantee accuracy (see below), it is far safer and wiser to use a tool based on sound theory than one that merely "seems" reasonable.

The Tool Must Be Complete

Being complete means accounting for all significant and relevant considerations. A logically sound tool that is incomplete gives, at best, the right answer to the wrong question. A tool might leave out important considerations because those considerations are difficult to accommodate (e.g., it may be too hard or too costly to obtain the necessary input data) or because they are impossible to include given the nature of the selected decision model (e.g., dynamic considerations within a static model). As noted earlier, project decisions tend to produce broad, enterprise-level impacts. This makes creating a complete model challenging. As illustrated by the example in the side box, a complete model is often large, requiring many inputs and sophisticated mathematical algorithms. One way to assess the completeness of a model is through sensitivity analysis. Try varying the description of a complicated project in subtle ways that should logically affect the attractiveness of that project. Do the inputs permitted by the model allow you to reflect such considerations? Do the priorities established by the model behave as they should? No model can capture everything, but a complete model should properly address those considerations most important to project-selection decisions.
Completeness Requires Addressing "Hard" and "Soft" Considerations
A water utility desired a system for prioritizing capital improvements. To fully value such investments, it was necessary to include "soft" project benefits, such as enhanced public health and safety, reduced flood risk, increased community recreational opportunities, and expanded capability and knowledge. Providing inputs for quantifying project value required information from databases associated with many different applications, including asset utilization systems, customer information systems, GIS, work management systems, and investment modeling applications. Furthermore, it was clear that field personnel had local knowledge of needs and the effectiveness of proposed projects that was not captured by historical data. Thus, the resulting priority system included both an extensive scoring logic (to capture judgments) plus analytic techniques for logically combining empirical and judgmental data.

www.prioritysystem.com

31

c 2003-2009 Lee Merkhofer Consulting

Project Portfolio Management Tools: Which Approach is Best? Part 6: Evaluating Tools

The Tool Must Be Practical

Accuracy, logical soundness, and completeness are the reasons that prioritization tools must often include sophisticated decision models. But, a tool must also be practical. Building a quality tool that is still practical is a significant challenge facing expert tool designers. To be practical, users must have sufficient expertise to understand and apply the tool. The required inputs must be available. Computational resources must be adequate, and applications must be completed within available time. All this must be accomplished without unacceptably degrading the accuracy, logical soundness, or completeness of the model.

Experience Creates Capability and Demand for More Sophisticated Tools

In the early 1990's, a client operating an oil and gas pipeline sought help in designing a system for prioritizing investments to reduce risk. The new system was much more detailed and formal than existing practices. It required training, implementing new processes, constructing additional databases, and creating computationally sophisticated solution algorithms. The system manual was over 200 pages long! Many outside the development team argued that the system was "far too complicated." However, backing of senior management was secured when it was demonstrated that the system uncovered ways to significantly reduce risks without increasing total costs. The client has continued to use the system independently for many years, and each year the system has been expanded. Experience, better documentation and internal training led to increased confidence and expertise, promoting the desire for more capability and accuracy.

How can such challenges be addressed? Obviously, skill is required and experts should guide the development of the decision model used by the tool. At the same time, the organization can take steps to make it practical for it to use more sophisticated tools. These steps can include investing in internal training, reassigning responsibilities, developing new sources of data, and adjusting budgeting schedules. Although the above recommendations may seem daunting, it is worth noting that it is common for an organization to initially view a quality tool as "too complex," and then later, after gaining experience and comfort with its use, to want to expand the tool and make it even more sophisticated (see the side box above for an example). Thus, organizations should avoid simplistic tools and tools whose decision model cannot be improved as experience and understanding grows.

The Tool Must Be Effective

A PPM tool cannot be effective unless it fits the way that the organization actually makes decisions. The considerations to which the tool may need to be sensitive include: Timing. When are decisions made? Annually, quarterly, monthly, or just in time? Once a decision is made, is the commitment over multiple years, or can it be revisited at any time? Decision process. How are choices made? Through consensus? A dictatorship? Who makes decisions? The Board? Multi-functional steering committee? A project portfolio manager? Project management. How are needs identified, project solutions conceptualized, and a preferred solution identified? How are projects planned, scheduled, and staffed? If a project stage gate process is used, how are decisions made about whether projects proceed through each gate?
www.prioritysystem.com c 2003-2009 Lee Merkhofer Consulting

32

Project Portfolio Management Tools: Which Approach is Best? Part 6: Evaluating Tools

Performance monitoring. What information is available to help the organization track its performance? What types of performance data are collected? Such considerations must be understood and the tool and its application process must be designed to ensure that there is a good fit to the organization. See the side box for an example. Being effective also means achieving the specific goals that motivate using a tool. For example, a tool intended for back-room use would be designed differently than one whose purpose is to demonstrate to regulators and local citizens that the organization's project decisions are in the best interests of the community. Not only would the latter have different user characteristics and features, the definitions of project benefits would be quite different.
A Centralized Project Ranking Tool May Not Fit a Decentralized Decision-Making Environment
An electric utility desired a system for allocating its $250 million annual O&M budget. The organization had a decentralized decisionmaking structure. The managers of business units (e.g., line clearance, meter reading, the call center) had historically been given authority to decide how best to spend within their respective areas. A single, centralized, project ranking system was unacceptable. It would dictate to each manager how the projects within his or her area should be prioritized. On the other hand, allowing each business unit to have its own, separate priority system would not provide a consistent approach for the enterprise as a whole. The selected system [14] involved a tiered approach to PPM. It allowed each department to apply a tool to determine the value generated under different funding levels, based on the projects that would be conducted under those funding levels. This design allowed area managers to retain authority to prioritize and select projects within their respective areas while rewarding managers who propose projects that create value consistent with corporate objectives.

Being Effective May Require a New Approach

A recent effort was conducted for an electric utility to provide a system for prioritizing bids for long-term (greater than 10-year) energy contracts. Standard contract valuation methods were overly sensitive to discount rate assumptions. Therefore, an alternative model not dependent on discount rates was developed based on real options theory. Real options [9] adopts the view that projects and assets create options (e.g., a contract creates the option to buy energy according to a predetermined price rule). The real options tool did not replace the utility's conventional contract valuation tool. Rather, it is used to make broad choices about contract durations while the utility's more conventional tool is still used to support specific decisions.

Before building or purchasing a tool, think carefully about what the tool needs to do in order to be effective. See the second side box for another example of an application requiring a specialized approach. Although a tool may appear to offer lots of flexibility, it can only be adjusted within the limitations dictated by its underlying decision model. Few tools allow users to change the mathematical logic by which projects are valued or optimal portfolios are identified.

The Tool Must Be Acceptable to Stakeholders

A tool that is practical and effective is not always acceptable to decision makers and other stakeholders. An acceptable tool must be compatible with existing organizational processes and culture. It must be understandable and understood. A tool that impacts funding decisions will be perceived as a threat to some interests. All key stakeholders must have confidence that the tool will help them, as well as the organization, to succeed.

www.prioritysystem.com

33

c 2003-2009 Lee Merkhofer Consulting

Project Portfolio Management Tools: Which Approach is Best? Part 6: Evaluating Tools

Gaining adequate acceptance is critical to the success of the tool. For a dramatic example of a tool that was widely regarded as technically defensible, complete, accurate, and practical and, yet, was rejected, read "The Rise and Fall of a Risk-Based Priority System" [15]. The most effective way of generating acceptance is to involve in the design effort those who will use and be impacted by the tool. A collaborative process helps ensure that the tool will have exactly those characteristics necessary to best suit the organization and its needs. Equally important, involving stakeholders in the design of the decision model creates buyin by allowing skeptics to express their concerns and see first hand how those concerns are addressed. See the side box for an example.

Developing the Tool as a Collaborative Effort Promotes Buy-In

A client was investing heavily in information technology. A system was desired that would prioritize proposals submitted by different departments. Department managers were naturally concerned about how the new system would affect their ability to obtain funding. One department head, responsible for core systems, was particularly pessimistic about the ability of a tool to fairly evaluate his projects. "How can you quantify the value of adding capacity? Our projects don't increase revenue or reduce costs, and they are invisible to our customers. The only time we ever get more funding is when things stop working." The manager only agreed to participate after attempts to get his projects exempted from the system failed. To address the concerns, the selected design included a feature not common to priority systems--If an infrastructure project enabled or facilitated another project, the enabling project obtained a share of the benefit derived from the enabled project. The approach worked, and the core systems department obtained approval for several projects that had previously been denied funding.

Notes
1. These criteria are discussed at greater length in M. W. Merkhofer, Decision Science and Social Risk Management, Reidel, 1987, and V. Covello and M. W. Merkhofer, Risk Assessment Methods, Plenum, 1993. 2. R. G. Anderson, A. Bendure, S. Strait, and A. Kann. "Supporting Documentation: Laboratory Integration and Prioritization System," Los Alamos National Laboratory, Los Alamos, New Mexico, 1994. 3. See, for example, Ward Edwards and J. Robert Newman, Multiattribute Evaluation, Sage University Papers Series on Quantitative Applications in Social Sciences, Beverly Hills CA 1982. 4. The analysis and results are documented in H. Call and M. W. Merkhofer, "A MultiAttribute Utility Analysis Model for Ranking Superfund Sites", published in "Superfund '99, The Proceedings of the 9th National Conference," Washington, D. C., November 2830, 1988. 5. Robert T. Clemen, Making Hard Decisions: An Introduction to Decision Analysis, PWSKent Publishing Company, Boston, 1997. 6. Ralph L. Keeney and Howard Raiffa, Decisions with Multiple Objectives, Wiley, New York, 1976. 7. For example, E. J. Elton and M. J. Gruber, Modern Portfolio Theory and Investment Management, 4th ed., Wiley, New York, 1991.

www.prioritysystem.com

34

c 2003-2009 Lee Merkhofer Consulting

Project Portfolio Management Tools: Which Approach is Best? Part 6: Evaluating Tools

8. For example, R. K. Sundaram, A First Course in Optimization Theory, Cambridge, 1996. Also see Mathematical Theory for Prioritizing Projects and Optimally Allocating Capital, located on this website. 9. For example, T. Copeland, V. Antikarov, and T. E. Copeland, Real Options: A Practitioner's Guide, Texere, 2001. 10. J von Neumann and O. Morgenstern, Theories of Games and Economic Behavior, Princeton University Press, 1947. 11. James E. Smith and Robert F. Nau, Valuing Risky Projects: Options Pricing Theory and Decision Analysis, Management Science, 41 (4) 1995. 12. For a complete description, see M. W. Merkhofer and R. L. Keeney, "A Multiattribute Utility Analysis of Alternative Sites for the Disposal of Nuclear Waste," Risk Analysis, Vol. 7, No. 2, 1987, 173-194. 13. D. Von Winterfeldt and W. Edwards, Decision Analysis and Behavioral Research, New York: Cambridge University Press, 1986. 14. E. Martin and M. W. Merkhofer, "Lessons Learned - Resource Allocation based on MultiObjective Decision Analysis", Proceedings of the First Annual Power Delivery Asset Management Workshop, New York, June 3-5, 2003. This reference also appears as a paper on this website 15. K. E. Jenni, M. W. Merkhofer, and C. Williams, "The Rise and Fall of a Risk-Based Priority System: Lessons from DOE's Environmental Restoration Priority System," Risk Analysis, Vol. 15, No. 3, 1995, 397-409.

www.prioritysystem.com

35

c 2003-2009 Lee Merkhofer Consulting