Portfolio decision quality

University of Massachusets Boston From the SelectedWorks of Jefrey Keisler 2011 Portfolio decision quality Jefrey Keisler, University of Massachusets Boston Available at: htp://works.bepress.com/jefrey_keisler/38/ Portfolio Decision Quality Jeffrey Keisler1 Abstract. The decision quality framework has been useful for integrating decision analytic techniques into decision processes in a way that adds value. This framework extends to the specific context of portfolio decisions, where decision quality is determined at both the project level and the portfolio level, as well as in the interaction between these two levels. A common heuristic says that the perfect amount of decision quality is the level at the additional cost of improving an aspect of the decision is equal to the additional value of that improvement. A review of several models that simulate portfolio decision making approaches to illustrates how this value added depends on characteristics of the portfolio decisions, as does the cost of the approaches. 2.1 Introduction The nature of portfolio decisions suggests particular useful interpretations for some of the elements of decision quality (DQ). Because of the complexity of these problems, portfolio decision makers stand to benefit greatly from applying DQ. This chapter aims to facilitate such an application by: characterizing the role of DA in portfolios; describing elements of portfolio decision quality; defining levels of achievement on different dimensions of DQ; relating such achievement to value added (using a value-of-information analogy); and considering the drivers of cost in conducting portfolio decision analysis (PDA). It is helpful to start by characterizing the role of DA in portfolio problems, particularly in contrast to the role of optimization algorithms. Portfolio optimization algorithms range from simple to quite complex, and may require extensive data inputs. DA methods largely serve to elicit and structure these often subjective inputs, thereby improving the corresponding aspects of DQ: A portfolio decision process should frame the decision problem by defining what is in the portfolio under consideration and what can be considered separately, who is to decide, and what resources are available for allocation. At their simplest, Jeffrey Keisler Management Science and Information Systems Department, College of Management, University of Massachusetts Boston, Boston, MA, USA, e-mail: jeff.keisler@umb.edu 1 portfolio alternatives are choices about which proposed activities are to be supported, e.g., through funding. But there may also be richer alternatives at the project level, or different possible funding levels, as well as portfolio level choices that affect multiple entities. For each project, information may be needed about the likelihood of outcomes and costs associated with investments – and for portfolio decisions especially, the resulting estimates should be consistent and transparent across projects. The appropriate measure of the value of the portfolio may be as simple as the sum of the project expected monetary values, or as complex as a multi-attribute utility function of individual project and aggregate portfolio performance; this involves balancing tradeoffs among different attributes as well as between risk and return, short term and long term, small and large, etc. Decisions that logically synthesize information, alternatives and values across the portfolio can be especially complex, as they must comprehend many possible interactions between projects. Implementation of portfolio decisions requires careful scheduling and balancing to assure that sufficient resources and results will be available when projects need them. In practice, it has been common to calculate the value added by PDA as the difference between the value of the “momentum” portfolio that would have been funded without the analysis and the value of the (optimal) portfolio ultimately funded. This idea can be refined to understand the value added by specific analysis that focuses on any of the DQ elements. This is similar to an older idea of calculating the value of analysis by calculating the value of information “revealed” to the decision maker by the analysis, and it suggests specific, conceptually illuminating, structural models for the value of different improvements to portfolio decision quality (PDQ). It is productive to think in terms four steps in the different dimensions of DQ: No information, information about the characteristics of the portfolio in general, partial specific information about some aspects of the portfolio in a particular situation, or complete information about those aspects. These steps often correspond (naturally, as it were) to discrete choices about formal steps to be used in the portfolio decision process. Thus, it’s possible to discuss clearly rather detailed questions about the value added by PDA efforts. This chapter synthesizes some of my past and current research (some mathematical, some simulation, some empirical) where this common theme has emerged. Without going to the level of detail of the primary research reports, this paper will describe how various PDQ elements can be structured for viewing through this lens. A survey of results shows that no one step is clearly most valuable. Rather, for each step, there are conditions (quantitative characteristics of the portfolio) that make quality relatively more or less valuable (perhaps justifying a low-cost pre-analysis or an organizational diagnosis step assessing those conditions). Several related queries help to clarify how PDQ could apply in practical settings. We consider (briefly) how choices about analysis affect the cost of analysis, e.g., number of variables, number of assessments, number of meetings, etc. Be- cause these choices go hand-in-hand with DQ levels that can be valued, this provides a basis for planning to achieve DQ. We review (briefly) case data from several organizations to understand where things stand in current practice and which elements of DQ might need reinforcement. Finally, in order to illustrate how the PDQ framework facilitates discussion of decision processes, we review several applications that describe approaches taken. The chapter concludes with a PDQ driven agenda to motivate future research. A richer set of decision process elements could be modeled in order to deepen understanding of what drives PDQ. Efforts to find more effective analytic techniques can be focused on aspects of portfolio decisions that have the highest potential value from increased quality. Alternatively, in areas where lower quality may suffice, research can focus on finding techniques that are simpler to apply. The relative value of PDQ depends on the situation, and it may be that simple and efficient approaches ought to be refined for one area in one application setting, while more comprehensive approaches could be developed for another setting. 2.2 Decision portfolios A physical portfolio is essentially a binder or folder in which some related documents are carried together, a meaning which arises from the Latin roots port (carry) and folio (leaf or sheet). Likewise, an investment portfolio is a set of individual investments which a person or a firm considers as a group, while a project portfolio is a set of projects considered as a group. Portfolio decision analysis (PDA) applies decision analysis (DA) to make decisions about portfolios of projects, assets, opportunities, or other objects. In this context, we can speak more generally of (a) portfolio (of) decisions. We define a portfolio decision as a set of decisions that we choose to consider together as a group. I emphasize the word choose because the portfolio is an artificial construct – an element is a member of the portfolio only because the person considering the portfolio deems it so. I emphasize the word decisions because portfolio decision analysis methods are applied to decisions not to projects or to anything else. A portfolio of project decisions starts with a portfolio of projects and (often) maps each project simply to the decision of whether or not to fund it. Likewise, other portfolios of concern may be mapped to portfolios of decisions. Considering these decisions in concert is harder than considering them separately. There are simply more facts to integrate and there are coordination costs. Therefore, decisions are (or ought to be) considered as a portfolio only when there is some benefit to doing so. That benefit is typically that the best choice in one decision depends on the status of the other decisions. This chapter will discuss portfolio decision quality (PDQ). This concept builds on the decision quality (DQ) framework, but considers how characteristics specific to portfolio decisions may change how DQ is applied so that portfolio managers can better understand how to design their decision processes. 2.3 Decision quality Decision quality (Matheson and Matheson, 1998, Howard, 1988, McNamee and Celona, 1990) is a framework developed by practitioners associated with Stanford University and Strategic Decisions Group. Their basic story is as follows: At the time a decision is made, its quality cannot be judged by its outcome. Rather, the quality of a decision is judged by the process used to make it. There are six dimensions of decision quality: The six elements are: Framing, Alternatives, Information, Values, Logic, and Implementation. The quality of the decision is only as high as its quality in its weakest dimension. In each dimension, a quality level of 100% is defined to be the point at which marginal effort would not be justified by the benefit it would produce. We now aim to extend this idea to portfolio decisions. In this context, we must consider how DQ manifests at the project (or individual decision) level, at the portfolio level, and in the interchange between these two levels. That is, in order for the portfolio decision to be of high quality, the preparation and consideration of the individual decisions in the portfolio must be of high quality in their own right. They must also be considered well as a portfolio, and they must be considered at the individual level in a way that facilitates high quality consideration at the portfolio level and vice versa. Portfolio level Frame Interchange Alternatives Project level Information Logic Values Commitments Fig. 2.1 Decision quality is created the project and the portfolio level A 100% decision quality level is surely a worthwhile target. But the benefit of additional analysis is usually hard to quantify. There have been efforts to quantify the value of additional analysis as if it were additional information (Watson and Brown, 1978). This approach is hard to apply in general. With portfolio decisions, there are reasons why it’s not so hopeless. The ultimate value resulting from the decisions about a portfolio decision is a function of the analytic strategy used (steps taken to improve decision quality in different dimensions) and certain portfolio characteristics. I have used an approach that has been dubbed optimization simulation (Nissinen, 2008) to model the value added by the process. This approach works if there is an appropriate way to specify the steps taken and to quantify the portfolio characteristics. I am not sure if it is applicable as a precise tool to plan efforts for a specific portfolio decision, but it is able to generate results for simulated portfolios and these give insight that may guide practice. 2.4 Portfolio decision quality We now discuss how the decision quality framework applies in the context of portfolio decisions. Note, this is a perspective piece, and so the classifications that follow are somewhat arbitrary (as may be the classifications of the original decision quality framework), but are intended as a starting point for focusing discussion on the portfolio decision process itself. 2.4.1 Framing In DA practice, “framing” (drawing rather loosely on Tversky and Kahneman’s (1981) work on framing effects) typically refers to understanding what is to be decided and why. In many decisions (single or portfolio), before even discussing facts, it is necessary to have in mind the right set of stakeholders, as it is their views that drive the rest of the process. There are various methods for doing so (e.g., McDaniels et al, 1999) and because portfolios involve multiple decisions, it is typical for there to be multiple stakeholders associated with them. In portfolio settings, a first step in framing is to determine what the portfolio of decisions is. Sometimes the decisions are simple fund/don’t fund decisions for a set of candidate projects. In other situations, it is not as clear. For example, there might be a set of physical assets the disposition of each of which has several dimensions that can be influenced by a range of levers. Then the portfolio might be viewed in terms of the levers (what mix of labor investment, capital improvements, new construction, outsourcing, and closure should be applied), the dimensions of disposition (how much should efforts be directed toward achieving efficiency, effectiveness, quality, profitability, growth, and societal benefit), the assets themselves (how much effort should be applied to each site), or richer combinations of these. Once the presenting portfolio problem has been mapped to a family of decisions, a key issue in framing is to determine what is in and out of the portfolio of decisions. The portfolio is an artificial construct, and decisions are considered together in a portfolio when the decision maker deems it so. There is extra cost to considering decisions as a portfolio rather than as a series of decentralized and one-off decisions. Thus, the decisions that should be joined in a portfolio are the ones where there is enough benefit from considering them together to offset the additional cost. (Note, Cooper et al, 2001, discuss the related idea of strategic buckets, elements of which are considered together). Such benefit arises when the optimal choice on one decision depends on the status of other decisions or their outcomes. A common tool for framing decisions is to use the decision hierarchy (Howard, 2007), to characterize decisions as already decided policy, downstream tactics (which can be anticipated without being decided now), out of scope (and not linked to each other in an important way at this level), or as the strategic decisions in the current context. Portfolios themselves may also form a hierarchy, e.g., a company has a portfolio of business units each of which has a portfolio of projects (e.g., Manganelli and Hagen, 2003). The decision frame, like a window frame, determines what is in view and what is not – what alternatives could reasonably be considered, what information is relevant, what values are fundamental for the context, etc. In the standard project selection problem, the most basic interaction between decisions arises from their drawing on the same set of constrained resources. Thus, with a set of decision variables x1, …, xn (e.g., the amount of funding for projects 1 to n, or simply whether indicator variable of whether or not to fund the projects) the decision problem is Max X V(X) s.t. C(X) <= B. The first part of the framing problem is to determine the elements of X, and the related question of identifying the constraints B. The set of all project proposals can be divided into clusters in terms of timing (e.g., should all proposals received within a given year be considered in concert, or should proposals be considered quarterly?), department or division (should manufacturing investments compete with funds alongside marketing investments?) geography, level, or other dimensions. Montiebeller et al (2009) give some attention to this grouping problem. Even if there are no interactions between projects other than competition for resources, the larger the set of projects considered within a portfolio, the less likely it is that productive projects in one group (often a business unit or department) will go without resources while unproductive projects elsewhere (often in another business unit or department) obtain them. Related to bounding of decisions in the portfolio problem is defining the constraints. Of course, a portfolio consisting of all proposals received in a 3-month period ought to involve allocating a smaller budget than a larger portfolio of proposals received over a longer time. But sometimes the budget has to be an explicit choice. For example, Sharpe and Keelin, 1998, describe their success with the portfolio at Smithkline Beecham, and in particular how they persuaded management to increase the budget for the portfolio when the analysis showed that there were worthwhile projects that could not be funded. In this case, the higher budget required the company to align its R&D strategy with its financial strategy for engaging in capital markets, essentially allowing the portfolio of possible investments to spread over a greater time span. In other cases, allowing the portfolio budget to vary could imply that R&D projects are competing for funds with investments in other areas, so that the relevant portfolio contains a wider range of functions. We see here that portfolio decisions involve distinctions that are not salient in general. Mapping from a portfolio of issues to a portfolio of decisions (what is to be decided) frames at the level of the individual projects, although this also necessarily drives how analysis proceeds at the portfolio level. Scoping frames explicitly at the portfolio level, but it automatically affects the primary question at the project level of whether a proposal is even under consideration. Bounding the solution space in terms of budget/resource constraints frames at the portfolio level, and has little bearing on efforts at the project level. 2.4.2 Alternatives In the DQ framework, the quality of alternatives influences the quality of the decision because if the best alternatives are not under consideration, they will simply not be selected. High quality alternatives are said to be well-specified (so that they can be evaluated correctly), feasible (so their analysis is not a waste of time), and creative (so that surprising potential sources of value will not be overlooked). In a portfolio, there are alternatives defined at the project level (essentially, these are mutually exclusive choices with respect to one object in the portfolio), and at the portfolio level (e.g., the power set of projects). At the project level, the simplest alternatives are simply “select” or “don’t select”. A richer set of alternatives may contain different funding levels and a specification for what the project would be at each funding level; although these variations take time to prepare, richer variation at the project level allow for a better mix at the portfolio level. A related issue is that the level of detail with which projects are defined determines what may be recombined at the portfolio level. For example, if a company is developing two closely related products, it may or may not be possible to consider portfolio alternatives including one but not the other product, depending on whether they are defined as related projects (with detail assembled for each) or as a single project. At the portfolio level, one may consider as the set of alternatives the set of all feasible combinations of project-level decisions. Here we get into issues of what is computationally tractable, as well as practical for incorporating needed human judgments. How alternatives are defined (and compared) at the portfolio level af- fects what must be characterized at the project level. Where human input is needed, high quality portfolio level alternatives may be organized with respect to events (following Poland, 1999), objectives (following value-focused thinking, Keeney, 1993), or resources (e.g., strategy tables, as in Spradlin and Kutolski, 1999), or constraints, or via interactive decision support tools (e.g., visualization methods such as heatmaps, as in Kiesling et al, in this volume). 2.4.3 Information The quality of information about the state of the world (especially as it pertains to the value of alternatives) is driven by its completeness, its precision, and its accuracy. High quality information about what is likely to happen enables estimates that closely predict actual value of alternatives. In portfolio decisions, the quality of information at the project level largely has the same drivers. But rather than feeding a go/no-go decision about the project in isolation, , this information feeds choices about what is to be funded within a portfolio, and this can mean that less detailed estimates of project value are needed. On the other hand, because projects may be in competition for resources used, project level information about costs may be more significant in feeding the portfolio decision. Furthermore, as has been noted, consistency across projects is important – a consistent bias may have less impact on the quality of the portfolio decision (and its implementation) than a smaller bias that is less consistent. At the level of the portfolio, interactions between projects are important – synergies and dissynergies, dynamic dependencies /sequencing, and correlations may all make the value of the portfolio differ from the value of its components considered in isolation. When these characteristics are present, the search for an optimal portfolio is not so simple as ranking projects in order of productivity index to generate the efficient frontier. Therefore, it is important to not only collect this portfolio information appropriately (and perhaps iteratively), but to structure it so as to facilitate later operations. 2.4.4 Values In decision analysis and other prescriptive approaches to decision making, decision makers seek to select the most preferred option. Much effort may go into identifying preferences and values in order to facilitate that selection. In valuing a portfolio of projects, since the options are different possible portfolios, it helps to construct a value function that comprehends preferences and then represents them in a form amenable to making the necessary comparisons, i.e., the options that have higher values ought to be the ones the decision maker prefers. If project value is additive, then high quality on values requires mainly that the value function include the right attributes and the right weights for each project. If the value function is additive across attributes and across projects, then the best portfolio decision arises from setting the right value function to be incorporated across the set of projects. A more complex value function, e.g., a non-linear multi-attribute utility function over the sum of project level contributions, requires that the projects be characterized and measured in the right terms, but the hard judgments must be made at the portfolio level. Finally, with portfolio decisions there are often more stakeholders affected by the set of projects, and who therefore have values to be integrated. This is an area within PDA where there are numerous common approaches. Of particular note, at the portfolio level there is often a range of interacting objectives (or constraints). Rather than undertaking the sometimes prohibitive task of formally structuring a utility function for them all, the portfolio manager may strive for “balance” (see Cooper et al, 2001, the “SDG grid” in Owen, 1983 and elsewhere, or Farquhar and Rao, 1976), which is commonly done through 2x2 matrices showing how much of one characteristic and how much of another each project has. This could include balance: between risk and return, across risks (i.e., diversification), benefits and costs, short term and long term, internal and external measure, one set of stakeholders and another (i.e., fairness), resources of different types used, balance over time, or other characteristics. Balance grids are easier to work with conceptually than are convex multi-attribute utility functions with interactive terms. With such grids, and with projects mapped to them, the decision maker can then envision the effect of putting in or pulling out individual projects, thus directly relating decisions about individual projects to portfolio level value. Other interactive approaches utilizing feedback or questions based on an existing portfolio model may also help to identify values and interactions between them, e.g., Argyris et al, in this volume. 2.4.5 Logical synthesis In a standard DA setting, the logic element of DQ means the assurance that information, values and alternatives will be properly combined to yield identify the course of action most consistent with the decision maker’s preferences. Standard DA utilizes devices such as decision trees, probability distributions and utility functions to ensure consistency that the decision maker’s actions and beliefs conform to normative axioms. Certainly, much of this still applies at the project level within portfolios. Detailed inputs at the project level ought to be logically synthesized to obtain value scores that are then incorporated in the portfolio level decision. For example, in the classic SDG style PDA, decision trees at the project level identify the ENPV and cost of each project. If there is minimal interaction between projects, the portfolio level synthesis is simply to rank projects by productivity index and fund until the budget is ex- hausted. However, interactions – as mentioned previously – can include synergies and dissynergies, logical constraints where under some circumstances some combinations of activities may be impossible, while in other cases certain activities may only be possible if other activities are also undertaken. In simpler cases, it may be possible to still capture this primarily with simple spreadsheet-level calculations based on the DA derived inputs, but often optimization and math programming techniques are required and in their absence, it is unlikely that an unaided decision maker would be able to approach the best portfolio. When such techniques are to be used, it is especially important to have coherence between the algorithms to be deployed and the project level inputs. Furthermore, because optimization models often require simplifying assumptions (as do all models to some extent), this element of quality may require a feedback loop in which managers review the results of the model and refine it where necessary. Such feedback is commonly prescribed in decision modeling (see Fasolo et al, in this volume). 2.4.6 Commitment and implementation. Producing the desired results once portfolio decisions are made (which SDG calls Value Delivery) requires effort at the border of project and portfolio management. At the portfolio level, resources must be obtained and distributed to projects as planned. The portfolio plan itself, with more precise timing, targets, and resource requirements must be translated back into detailed project plans, as the initial specifications for all individual approved projects must be organized into a consistent and coherent set of activities for the organization. Project managers monitor progress and adapt plans when the status changes. Sets of projects may affect each others’ execution, e.g., one project might need to precede another or it may be impossible to execute simultaneously two projects that require the same resource. In this case, in addition to orchestrating a multi-project plan, the portfolio manager must monitor the fleet of projects and make adjustments to keep them in concordance over time with respect to resource use and product release, etc. One important event that can occur at the project level is failure. When projects really do fail, the portfolio is better off if they are quickly abandoned. At the project level, this requires incentives to not hide failure and to move on (as was embodied in Johnson & Johnson’s value of “Freedom to Fail,” Bessant, 2003). All these information flows between projects and between project and portfolio managers benefit if the portfolio decision process has organizational legitimacy – if it is transparent and perceived as fair (Matheson and Matheson, 1998). 2.4.7 Interacting levels of analysis Thus, portfolio decision quality is determined at the project level, the portfolio level, and in both directions of the interface between those two levels, as shown in this partial listing. Table 2.1 Some determinants of portfolio decision quality Framing Portfolio level Interchange between project and portfolio levels Resources and Which projects are in Mapping issues to deciwhich portfolio (scoping) sions budgets (bounding) Alternatives Subsets of candidate projects, portfolio strategies Information Project level Projects suitably decom- Well-specified plans for posed multiple funding levels Specifying synergies, dy- Consistency namic dependency, correlations between projects Probability distribution Values Utility function, balance Summary statistics Attributes & measures Logical Optimization Dealing with dependencies Decision tree synthesis over outcomes between projects Implementation Alignment, monitoring Ensuring resource availa- Project management, bility incentives, buy-in, etc. and correcting Since the required level of each element of decision quality depends on the value added by that element as well as its costs (we can think of this terms of bounded rationality), it would help to have a way to measure the value and cost. Cost of efforts to create decision quality is a subject that has not been much studied, and we shall only consider it in abstract terms in this chapter, but it is not so difficult to think about – if specific efforts are contemplated, the main cost of those efforts is the time of the individuals involved, and there are many areas of management in which methods are applied to quantify such costs, e.g., software development cost estimation. Value depends on the information and decisions involved, and perhaps other context, which makes it difficult to judge intuitively. We now explore how models of the portfolio decision process can be used to gain insight about the way specific efforts affect portfolio value. 2.5 Valuing portfolio decision quality 2.5.1 Four discrete levels of portfolio decision quality It can be useful in some of these cases to think about four possible levels of information being brought to bear on the decision. Analytic estimates Intuitive estimates Analog data Ignorance or equivalent Fig. 2.2 Four levels of quality resulting from different processes for obtaining and using information The first level is “ignorance” – not that decision makers have no information, but rather that their decisions take a perspective other than that of using the parametric information they do have. For example, in highly politicized situations, information about project value may have little to do with funding. The value of the portfolio under such a process can be used as a baseline for measuring improvement, and the process is modeled as if it selects projects at random. The second level is “analog” portfolio-level information but not about information specific projects. In this case, a decision rule can be developed that takes into account the characteristics of the average project – or even of the distribution of projects. In the pharmaceutical industry, for example, it is common to use “portfolio-wide averages to estimate success probabilities for individual projects where assessments have not been conducted. The third level is information that is intuitive “estimates” that are situation (or project) specific but non-analytic. For example, qualitative descriptions of project-level characteristics that do not allow as fine comparisons and algorithmic manipulations as do quantitative descriptions. We can usually model this level as a noisy version of the true state. The fourth level is “analytic”: complete quantified and vetted information. We can model processes using this level of information as making decisions based on the true facts. That is artificial, of course, as in many cases even the best estimate of a project’s value is itself only a snapshot at one point in time and from one perspective. These levels correspond to different choices made in structuring the decision process (and the role of analysis in it). Thus, variations of these four levels can be incorporated into optimization simulation models allowing us to understand the value added by improved information that arises from different rules in the decision process. We now review studies using models to value four portfolio level analytic choices. 2.5.2 Modeling the value of higher quality information. Keisler (2004) considers the basic question: what is the value of increasing precision of value and cost estimates in PDA. This model assumes that projects are described by their value-to-cost ratios and their costs, that all projects have values for these parameters drawn from a lognormal distribution, and that the decision maker can receive either no information, imperfect information, or perfect information about each. The decision maker solves for the optimal funding decision vector max Fi EV = E((FiriCi) s.t. (FiCi)<=B, where Fi = 1 if project i is funded and 0 otherwise, B is the available budget, the ri are (in the best guess used for this simulation study) log normal and Ci are either constants (in the base case study) or log normal. The partial information state is interpreted as having pre-analysis information about r (and sometimes C), but being disciplined in funding projects in order of their value-to-cost ratio. The no-information state is interpreted as not applying that discipline (with nothing really implied about what information the decision maker has), and the perfect information state is interpreted as obtaining high quality project valuation and maintaining the discipline to fund projects in order of productivity. Partial prioritization is defined as assigning a threshold level for the productivity index and funding projects that exceed this threshold in random order until funds are exhausted. Thus, with reasonable assumptions based on empirical data, a simulated portfolio showed that the added value (the increment from the no-information case) resulting from disciplined prioritization with intuitive value estimates the baseline no-information case to well-disciplined (and being disciplined) is somewhat greater than the additional value attributable to the obtaining high-quality value estimates, but on the same order of magnitude. As seen below, the when the budget is sufficient to fund most projects, the value added by a disciplined process alone is almost as great as that added by full analysis. Depending on other conditions (not shown here), the total value added by analysis can be as low as 0 (if there is no uncertainty) but often as high as 100%+ of the baseline portfolio value, which itself is quite substantial (and is consistent with practitioner experience). 200 180 160 Perfect + Prioritization 140 Partial Information + Prioritization 100 No Information / No prioritization Portfolio benefit 120 80 60 Partial Information + Partial Prioritization 40 Perfect Information + Partial Prioritization 20 0 0 5 10 15 20 25 30 Budget level 35 40 45 50 Fig. 2.3 Value achieved with different levels of information and prioritization in a simple portfolio The implication is that portfolio managers should focus first on creating a culture that supports discipline in sticking to prioritization, and only focus on more precise estimates where there is relatively large uncertainty. Under some circumstances, reasonable shortcuts can yield most of the value added by analysis with proportionally lower cost of analysis than with the brute-force approach in which all projects are analyzed. Specifically, if organizational conditions allow for a well-defined threshold productivity level to be identified before analysis, a project can be funded or not merely based on whether its value-to-cost ratio exceeds the threshold. A modified threshold rule works well if there is large variance among the value-to-cost ratio of different projects: apply triage and fund projects that exceed the productivity threshold by a certain amount, don’t fund projects that fall a certain amount below, and analyze the rest of the projects so that they can be funded if after analysis they are shown to exceed the threshold. This model considers the general question of improving estimates. The three other models essentially take this as a starting point and consider choices about which bases of the value estimate to improve, or, alternatively, what more complex portfolio value measures should be based on the general value estimates. Decision analysts who saw these results found them interesting, but also stressed that analysts do more than obtain precise value estimates. For example, they make the alternatives better – in particular by identifying a range of alternatives for different funding levels. 2.5.3 Modeling the value of higher quality alternatives The next model (Keisler, 2011) compares different tactics for soliciting alternatives for various budget levels for each potential investment. In this model, C denotes cost, and each project is assumed to have an underlying value trajectory (colloquially called a buyup curve): Vi(Ci) = ri[1–EXP(–kiCi/Cimax)]/[1–EXP(–ki)]. Project level parameter values for the return (ri) and curvature (ki) are drawn from random distributions. Funding decisions to maximize E(ΣiVi) s.t. Σi Ci <= B are compared for analytic strategies that result in stronger or weaker information states about k and r. As before, in a baseline situation projects are selected at random, after a firstcut analysis, projects are funded based on their productivity parameter, also essentially as before. At the other extreme is the gold standard analysis in which a full continuum of funding levels and corresponding project values is determined for each project. In between are strategies where a small number of intermediate alternatives are defined for each project as well as “haircut” strategies that trim each project’s funding from its requested level, with the rationale that projects tend to have decreasing returns to scale. If returns to scale are not decreasing, the optimal portfolio allocates projects either 0 or 100% of their requested funds. Otherwise, the relative value added by different analytic strategies varies with the distribution of returns to scale across projects. This study showed value added by generating project level alternatives can be as high as 67% of the value added by getting precise value estimates for the original simple projects as in the first model, as seen in Figure 2.4. Contininuous project funding levels 160000 P o r t f o l i o 140000 Funding proportional to project's productivity index at full funding 120000 100000 Four possible funding levels per project 80000 60000 v a l u e Each project gets sam e portion of request (haircuts) 40000 20000 Yes-no funding decisions 0 0 1000 2000 Budget 3000 4000 Fund projects at random Fig. 2.4 Portfolio value when projects may be funded at intermediate levels Most of the additional value added can be obtained by either generating an additional one or two intermediate funding level alternatives per project. In some circumstances, similar gains are possible from applying a formula that utilizes project-specific information about k along with a portfolio-wide estimate of r, for a sophisticated type of haircut (layered). The results were sensitive to various parameters. For example, at high budget levels where most of the funding requested by projects is available, there is little benefit to having additional alternatives between full and no funding. 2.5.4 Modeling the value of higher quality on values Another PDA technique is scoring projects on multiple attributes, computing total project value as a weighted average of these scores (e.g., Kleinmuntz and Kleinmuntz, 1999); weights themselves may be derived from views of multiple stakeholders. To understand the value added by this approach, Keisler (2008) simulates portfolios in which for each portfolio, a set of weights (randomly generated) applies and each project has randomly generated scores on each attribute. In this model, the portfolio value is the sum of project values, ∑i Vi. Individual k uses attribute weights Wk, for attributes j = 1 to n. As before, there is a funding constraint ∑i FiCi <= B. There is a “true weight” wj0 for each attribute, and wjk = wj0bjk, where b represents an individual bias factor. wj0 are log normal, bjk are log normal, with parameter values calibrated to available data. The baseline strategy again funds projects at random, and the gold standard uses correct weights for each attribute. In between are strategies that may use a subset of the attributes, and may use equal weights or an imperfect approximation of the correct weights (such as rank-based weights). Figure 2.5 shows results of a simulation in which one approach identified multiple attributes and estimated their weights. 50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% No info One Indiv's top Top attr attribute attr All attrs, Top attr All attrs All attrs, Top attr All attrs, Weights Fig. 2.5 Percent increase in portfolio value from identifying attributes and weights Those steps added almost 50% to the baseline portfolio value – as before, a noteworthy level of improvement. But this study also showed that merely identifying all attributes and giving them equal weighting achieves 70% of the value added from perfect weighting. It also found that using a single individual’s imperfectly assessed weights across all attributes does not add much value beyond that, and that the frugal step of identifying the most important attribute adds substantial value to the selected portfolio regardless of how many other attributes are identified. 2.5.5 Modeling the value of considering synergies In each of the previous models, projects interact only through competition for resources. We now consider the importance of this assumption. In general, projects may also interact by affecting each other’s value or cost directly. For example, two new products may share a common base technology and thus they could share the cost of that element if both were pursued, i.e., they have cost synergy. Alternatively, projects can have value synergy, e.g., while each product has a target market, new markets can be pursued only if multiple products are present (e.g., peanut butter and chocolate). In these cases where portfolio cost and value is not actually the sum of individual project cost and value PDA can add value by identifying such synergies prior to funding decisions. A decentralized PDA will not identify synergies, but where the process has a specific step built in to identify synergies, it will most likely identify those that exist. This is not trivial, as the elements from which potential synergies emerge are not labeled as such – in the example above, it would be necessary to identify the peanut butter cup market and this would require creative interaction involving the chocolate and peanut butter product teams. The model in Keisler (2005) compares analytic strategies that evaluate all synergies, cost synergies or value synergies against the simplest strategy that considers no synergies and again, a baseline in which projects are not funded or are funded at random. (This model does not include possible dissynergies, e.g., in weapon selection problems where if an enemy is going to be killed by one weapon, there is no added value in another weapon that is also capable of killing the enemy). In this model, each project can require successful completion of one or more of atomic cost elements, where Sik = 1 or 0 depending on whether or not cost element k is required to complete project i. Similarly, Rij = 1 or 0 depending on whether project i is required to achieve value element j. The jth value element is worth Vj, and the kth cost element costs Ck. Portfolio value is ∑j Vj ∏i Fi Rij, and portfolio cost is ∑k Ck Maxi FiSik. In the simulation, V and C are drawn from known distributions, and Rij and Sik have randomly generated values of 0 or 1 with known probability. The relative value added by strategies that comprehend synergies compared to myopic strategies depends on the munificence of the environment (Lawerence and Lorsch, 1967). At low levels of actual synergy, value added is small because little is worth funding, and above a saturation point, value added is small because many projects are already worth funding. At a sweet spot in the middle, completely considering synergies can increase portfolio value more than 100% and in some cases over 300%. In certain cases, comprehension of the possibility of synergies is a substantial improvement over the case where each project is evaluated myopically. Figure _ shows results from a less-extreme cluster of projects simulated in this study, where cross-project cost synergies were assumed to be present of 30% of the cost elements in the cluster, and cross-project value synergies were assumed to be present in 10% of the value elements. In this example, the value of identifying synergies is substantial, but far more substantial if both cost and value synergies are both identified. Most important in determining the value added at each level are the prevalence of synergies between value elements and cost elements of different projects, as well as the relative size of value elements to cost elements (and thus the likelihood that projects will merit funding even in isolation). 32.8 35 30 Value 25 20 21.2 20.0 21.6 16.2 15 10 5 0 NoValue NoCost NoValue PerfectCost PerfectValue NoCost PartialValue PerfectCost PerfectValue PerfectCost Information on synergy Fig. 2.6 Value added by obtaining information about cost and value synergies among projects 2.5.6 Discussion of model results In each of the above models, typical analytic strategies observed in practice were characterized as attempts to obtain and structure a specific set of parameters, the same set for each project. Each of the models also considered certain frugal strategies that were not necessarily observed in practice, but which could be characterized in terms of the same type of structuring method. The frugal strategies typically involved making optimal use of pre-existing analog-level knowledge about the portfolio in some areas, while actively acquiring it either through estimates or analysis in other areas. e.g., project cost, project value, returns to scale, strength of synergies. It would not be realistic to work toward a comprehensive model to calculate value of all sorts of possible analyses in real situations. But a portfolio manager could make judgments about the salient characteristics of the portfolio. Pre-analysis, we might ask whether the mean values, the spread of values, or the uncertainty of values for some aspect of the portfolio (levels of synergy, project costs, etc.) are especially large or small. We would then note the implications this has for the value of different analytic thrusts. Organizations that regularly make decisions about their portfolios might explore their archival data to help characterize the portfolio (see Stonebraker and Keisler, in this volume). 2.6 Cost of analysis and organizational requirements A final set of considerations in choosing a direction for PDA is its cost to the organization (especially if we go by the guideline that 100% decision quality is where the marginal value of improved quality no longer exceeds its marginal cost). This section does not present research results, but does provide guidelines for understanding what drives the cost of a process. Assessment costs ought to vary systematically with the number of assessments that must be done of each type. This way of thinking about decision process costs is similar to the way that computation times for algorithms are estimated – considering how often various steps are deployed as a function of the dimensions of the problem and then accounting for the cost of each step (and in simpler decision contexts, psychologists (Payne et al, 2003) have similarly modeled the cost of computation. Beyond assessments, there are organizational hurdles to successfully implementing some approaches. The treatment of both issues below is speculative, but in practice, consultants use similar methods to define the budget and scope for their engagements. Estimating assessment costs A: number of projects analyzed, B: cost of analysis per project C: number of alternatives generated per project D: cost per alternative generated/evaluated E: number of attributes in value function F: cost of assessing an attribute’s weight G: cost of scoring a project on an attribute H: projects per cluster I: number of value elements J: number of cost elements K: cost per synergy possibility checked Analysis for estimation of project productivity: AxB For example, with a portfolio with 30 (=A) projects for which productivity must be estimated at a cost of $5000 (B) of analysis per project, the cost of analysis would be $150,000. Analysis for estimation of intermediate alternatives: AxCxD For example, with a portfolio with 30 (=A) projects, each having a total of 3 (=C) non-zero funding levels whose cases must be detailed at a cost of $10,000 (=D) per case, the cost of analysis would be $900,000. Analysis involving use of multiple criteria: ExF+ExAxG For example, with a portfolio with 30 (=A) projects being evaluated on 8 (=E) attributes with a cost of $5000 (=F) per attribute to judge its importance (e.g., translate to a dollar scale), and $1000 (= G, e.g., the cost 1 hour of a facilitated group meeting with 5 managers) to score a single attribute on a single project, the cost of analysis would be $40,000 + $240,000 = $280,000. Analysis to identify synergy: (A / H) x H! x (I +J) K For example, with a portfolio of 30 (=A) projects divided into clusters of size 5 (=H), searching for potential synergies among 7 (I) value elements and 8 (J) potential value elements at an average cost of $50 per synergy checked (=K, some short interviews and checked with some brief spreadsheet analysis), the cost of analysis would be 6 * 5! * (7+8) * $50 = $540,000. These costs apply only to the parts of analysis that are at the most complete level. Frugal strategies replace some of the multiplicative cost factors here with one-time characterizations, and it is also possible to omit some of the analysis entirely. Excluded are certain fixed costs of analysis. 2.7 Observations from practice and from other research Both the PDQ framework and the associated value-of-analysis as value-ofinformation approach have, in my experience, been useful lenses for examining organizations’ portfolio decision processes. At one company (Stonebraker and Keisler, in this volume) this framework facilitated analysis of the process used at a major pharmaceutical corporation. In general, the data showed (at a coarse level) that the organization was putting more effort in where the value added was higher. We were able to identify specific areas in which it may have spent too much or too little effort developing analytic detail. This led to discussion about the reasons for such inconsistencies – which were intentional in some cases but not in others –and whether the organization could improve its effectiveness in managing those sub-portfolios. At another major company (Keisler and Townson, 2006), analysis of the data from a recent round of portfolio planning revealed that adding more alternatives at the project level would add substantial value to some of the portfolio by allowing important tasks on relatively low-priority projects to go ahead, and that this would be an important consideration for management of certain specific portions of the portfolio. We were able to identify some simple steps to gain some of this potential value with minimal disruption. At this same company, we also considered the measures used for project evaluation and were able to find a simpler set of criteria that could lead to the same value as the existing approach – or better if some of the measures were better specified. In another application (described at the end of Keisler, 2008), a quick look at the portfolio characteristics showed the way to a satisfactory approach involving a modest amount detail on criteria and weights, to successfully recover from an unwieldy design for a portfolio decision support tool. Finally, on another PDA effort that covered multiple business units at a major pharmaceutical company, involved many projects and products with a substantial amount of interaction in terms of value and cost, and it looked like it would be very challenging to handle such a volume of high-quality analysis. At the outset of the project, the engagement manager and I discussed the value of identifying potential synergies in PDA. In setting up the effort, the analysis team divided up to consider sub-portfolios within which synergies were considered most likely. The resulting engagement went efficiently and was considered a strong success. 2.8 Research agenda Within this framework, we can discuss certain broad directions for research. Only a few pieces of the longer list of decision quality elements were modeled. More such optimization simulation models could enrich our understanding of what drives the value of portfolio decision quality. Additionally, the description here of the elements of PDQ can be fleshed out and used to organize lessons about prac- tices that have been successful in various situations. As we consider where to develop more effective analytic techniques, we can focus on aspects of portfolio decisions where the value from increased quality would be highest – that is, as it becomes easier to make a decision process perform at a high level, the “100% quality level” will be higher as will the total value added by the decision process. Alternatively, in areas where the value added by greater precision etc. is minimal, research can focus on finding techniques – shortcuts – that are simpler to apply. The relative value of PDQ depends on the situation, and it may be that simple and efficient approaches ought to be refined for one area in one application setting, while more comprehensive approaches could be developed for another setting. 2.9 Conclusions Interpreting DQ in the context of portfolios allows us to use it for the same purposes as in other decision contexts. We aim to improve decisions as much as is practical by ensuring that all the different aspects of the decision are adequately considered. But we also recognize – especially with portfolio decisions – that many parts of the decision process are themselves costly due to the number of elements involved, e.g., eliciting probability judgments. Resources for decision making (as opposed to resources allocated as a result of the decision) are generally quite limited, and time may be limited as well. Therefore, we can use the DQ checklist to test whether the resources applied to the decision process match the requirements of the situation. To the extent we can think in concrete terms of the drivers of the value added by a decision process, we can use the DQ framework more skillfully. To the extent that portfolio decisions have common characteristics that are not shared by other classes of decisions, more detailed descriptions of their DQ elements help ensure that attention goes to the right parts of the process. As organizations formally integrate PDA into their planning processes, choices about what is centralized or decentralized, what is done in parallel, etc. translate in a clear way to the overhead cost of the process and to the levels of quality and hence the value added by the process. References Argyris N, Figueira JR, Morton A (2011) Interactive Multicriteria Methods in Portfolio Decision Analysis. In: Advances in Portfolio Decision Analysis, Salo, A, Keisler, J, Morton, A (eds.) Springer, New York Bessant J (2003) Challenges in Innovation Management. In: The international handbook on innovation (761-774), LV Shavinina, ed. Elsevier Cooper RJ, Edgett S, Kleinschmidt E (2001) Portfolio Management for New Products, 2nd edition, Perseus, Cambridge, MA Farquhar PH, RaoVR (1976) A Balance Model for Evaluating Subsets of Multiattributed Items. Management Science 22(5):528-539 Fasolo B, Morton A, von Winterfeldt D (2011) Behavioural issues in Portfolio Decision Analysis. In: Advances in Portfolio Decision Analysis, Salo A, Keisler J, Morton A (eds.) Springer, New York Howard RA (1988) Decision Analysis: Practice and Promise. Management Science 34(6): 679695 Howard RA (2007) The Foundations of Decision Analysis Revisited. In: Advances in Decision Analysis From Foundations to Applications (32-56), Edwards W, Miles RF, von Winterfeldt D (eds.) Cambridge University Press, Cambridge UK Keeney RL (1996) Value Focused Thinking. Harvard University Press, Boston, MA Keisler J, Townson D (2006) Granular portfolio analysis at Pfizer. Presented at POMS Conference. Boston, MA Keisler J (2008) The value of assessing weights in multi-criteria portfolio decision analysis. Journal of Multi-Criteria Decision Analysis 15(5-6): 111-123 Keisler, J (2011) The value of refining buyup alternatives in portfolio decision analysis. UMass Boston College of Management Working Paper UMBCMWP 1048 Keisler J (2004) Value of information in portfolio decision analysis. Decision Analysis 1(3): 177-189 Keisler J (2005) When to consider synergies in project portfolio decision analysis, UMass Boston College of Management Working Paper Kiesling E, Gettinger J, Stummer C, Vetschera V (2011) An Experimental Comparison of Two Interactive Visualization Methods for Multi-Criteria Portfolio Selection. In: Advances in Portfolio Decision Analysis, Salo A, Keisler J, MortonA (eds.) Springer, New York Kleinmuntz CE, Kleinmuntz DN (1999) A strategic approach to allocating capital in healthcare organizations. Healthcare Financial Management 53(4):52-58 Lawerence P, Lorsch D (1967) Organization and Environment. Harvard Business School Press, Boston, MA Manganelli R, Hagen BW (2003) Solving the Corporate Value Enigma. American Management Association, New York Matheson D, Matheson JE (1998) The Smart Organization. Harvard Business School Press. Boston, MA McDaniels TL, Gregory RS, Fields D (1999) Democratizing Risk Management: Successful Public Involvement in Local Water Management Decisions. Risk Analysis 19(3): 497-510 McNamee J, Celona P (2001) Decision Analysis for the Practitioner 4th Ed. SmartOrg, Inc., Menlo Park, CA Montibeller G, Franco LA, Lord E, Iglesias A (2009) Structuring resource allocation decisions: A framework for building multi-criteria portfolio models with area-grouped options. European Journal of Operational Research 199(3):846-856 Nissinen J (2007) The Impact of Evaluation Information in Project Portfolio Selection. Masters Thesis Helsinki University of Technology Owen DL (1984) Selecting Projects to Obtain a Balanced Research Portfolio. In: Readings on the Principles and Applications of Decision Analysis, Howard RA, Matheson, JE (eds.) Strategic Decisions Group, Menlo Park, CA Payne JW, BettmanJR, Johnson JR (2003) The Adaptive Decision Maker. Cambridge University Press, Cambridge, UK Poland WB (1999) Simple probabilistic evaluation of portfolio strategies. Interfaces29: 75–83 Sharpe PT, Keelin T (1998) How SmithKline Beecham makes better re-source-allocation decisions. Harvard Business Review 76(2): 45-57 Spradlin CT, Kutoloski DM (1999). Action Oriented Portfolio Management. Research Technology Management 42(2): 26-32 Stonebraker J, Keisler J (2011) Characterizing the analytical process used in evaluating the commercial potential of drug development projects in the portfolio for a large pharmaceutical company. In: Advances in Portfolio Decision Analysis, Salo A, Keisler J, Morton A (eds.) Springer, New York Tversky A, Kahneman D (1981) The framing of decisions and the psychology of choice. Science 30 211 (4481): 453-458 Watson SR, Brown RV (1978) The Valuation of Decision Analysis. Journal of the Royal Statistical Society A-141: 69-78