Uncertainty
Applying risk and uncertainty into decision making Max Henrion Energy Collaborative Analysis Workshop in Washington DC June 27-28, 2007
Copyright © 2007 Lumina Decision Systems, Inc.
Guidance from OMB: How to conduct regulatory analysis •
•
•
•
“For major rules … you should present a formal quantitative analysis of the relevant uncertainties about benefits and costs.” “… expert solicitation is a useful way … to quantify the probability distributions of key parameters.” “These … can be combined in Monte Carlo simulations to derive a probability distribution of benefits and costs.” “Use a numerical sensitivity analysis to examine how the results vary with plausible changes in assumptions, choices of input data.” [Emphases added]
OMB Circular A-4, John Graham, OIRA Administrator, 17 Sep 2003 http://www.whitehouse.gov/omb/circulars/a004/a-4.html Copyright © 2007 Lumina Decision Systems, Inc.
Probabilistic simulation for prospective projections
1. Express uncertainty by eliciting probability distributions from experts
2. Use Monte Carlo simulation to propagate probability distributions through the model.
3. View uncertainty on key results 5. Make a decision
4. Use sensitivity analysis to compare effects of uncertain assumptions on results Copyright © 2007 Lumina Decision Systems, Inc.
A personal decision under uncertainty:
When to leave for the airport? • On average, it takes about 40 minutes to drive from my home to San Jose International Airport, plus 30 minutes to park, get through security and walk to the gate; I’m supposed to be at the gate 20 minutes before departure. 40+30+20 = 90
• So, I should leave 90 minutes before departure, right? • Umm, no. That way I would miss my plane about half the time. Copyright © 2007 Lumina Decision Systems, Inc.
Value function for plane catching 400 350
EVIU Expected value of including uncertainty
300
Value Value
250 200
EViu expected value, ignoring uncertainty diu decision ignoring uncertainty
150 100 50 0 40
60 60
80 80
100 100
120 120
140 140
EVev maximum expected value dev decision to maximize expected value 160 160
Time av e be fore plane parts (minutes) Time to to le leave before planede departs (minutes)
View
Pr obability Density
Value ignoring uncertainty
Expected value
0 .0 2
0 .0 1 4
For more, see chapter 12 of Uncertainty: A Guide to Dealing with Uncertainty in Risk and Policy Analysis. M Granger Morgan & Max Henrion, Cambridge UP, 1990
0 .0 1
4m 0 40
60
80
100
120
Tim e n e e d e d (m in u te s )
140
160 Copyright © 2007 Lumina Decision Systems, Inc.
Sample decisions under uncertainty for an energy consumer • Should we retrofit buildings to reduce energy usage? To what level? • Do we need backup batteries or generators in case of power outage? What capacity is cost-effective? • Should we install photovoltaics now, when CA offers a large subsidy, or wait a few years until PV is cheaper, but lower subsidy? • Should we purchase long-term energy supply contracts or hedges to protect against price volatility? Copyright © 2007 Lumina Decision Systems, Inc.
Sample decisions under uncertainty for an energy R&D organization • Deep or wide? Should we spend most funds on a few, promising projects, or spread funds over a wider range? • R or D? How should we balance early-stage seed research vs. late-stage commercial development? • When to start: Should we start funding when early research indicates technical success is conceivable, or wait until commercial success is likely – or somewhere in between? • When to stop: How soon should we abandon a project when it starts to look like it may not succeed?
Copyright © 2007 Lumina Decision Systems, Inc.
1. How to express uncertainty as probability distributions • • • • • •
•
Probability is the clearest, most widely used language for expressing uncertainty. Statistics helps us understand the uncertainty in historical data The quantity we want is not usually one for which we have data Judgment is unavoidable in extrapolating from what we have to what we want. Let’s be explicit about it Obtaining probability distributions from a range of experts is the best way to quantify the current state of knowledge (and lack thereof) There are well-developed methods for obtaining expert judgment as probability distributions Uncertainty: A Guide to Dealing with Uncertainty in Risk and Policy Analysis. M Granger Morgan & Max Henrion, Cambridge UP, 1990 Copyright © 2007 Lumina Decision Systems, Inc.
4. Sensitivity analysis: Which uncertainties matter? When? Why? • Sensitivity and uncertainty analysis quantify relative contribution of each input to uncertainty in output • A potent source of insights. • Suggests priorities for further research
Copyright © 2007 Lumina Decision Systems, Inc.
5. Making decisions under uncertainty •
•
• • •
Virtually, all important decisions are made under uncertainty – whether we acknowledge it or not. Usually, we select the decision with the maximum expected value (net social benefit) If net benefits are large relative to the uncertainty, we can act now If not, we can weigh expected benefits of awaiting better information We can assess the value of more research using the expected value of information
Copyright © 2007 Lumina Decision Systems, Inc.
3. How to display uncertainties to decision makers Probability density function
600u
Probability Density
Numerical percentiles
700u
500u 400u 300u 200u 100u 0 $0
$1000
$2000
$3000
$4000
$5000
$6000
$7000
Unit cost of PV in 2010 ($/Kw) 1 0.9 0.8
Cumulative Probability
Cumulative probability function Box plot
0.1 0 $0
$1000 $2000 $3000 $4000 $5000 $6000 $7000
Unit cost of PV in 2010 ($/Kw) 2000
600T 500T 400T 300T
1800
100T 0 $0
$1000 $2000 $3000 $4000 $5000 $6000 $7000
U n it c o s t o f P V in 2 0 1 0 ( $ /K w )
Scatter plot Monte Carlo sample Iteration (Run)
1Q 900T 800T
0.6 0.5 0.4 0.3
1600 1400 1200 1000 800 600 400 200 0 $1000
For more, see chapter 9 of Uncertainty: A Guide to Dealing with Uncertainty in Risk and Policy Analysis. M Granger Morgan & Max Henrion, Cambridge UP, 1990
$2000
$3000
$4000
$5000
$6000
U n it c o s t o f P V in 2 0 1 0 ( $ /K w )
Copyright © 2007 Lumina Decision Systems, Inc.
How to assess the uncertainty in projections from models • Probabilistic simulation for prospective projections: Assess uncertainties on all key inputs, and propagate them through the model with Monte Carlo • Retrospective evaluation: Compare results from past projections with what actually happened Copyright © 2007 Lumina Decision Systems, Inc.
Retrospective assessment:
Measured speed of light (km/sec)
Reported uncertainty in measurements of the speed of light 1900 to 1984 299,810
Currently accepted value 1984 value
299,800
299,790
299,780
Henrion, M & Fischhoff, B, “Assessing uncertainty in physical constants”, American J. Physics, 54 (9), 1986
299,770
299,760
299,750 1900
1910
1920
1930
1940
1950
1960
Year of experiment Copyright © 2007 Lumina Decision Systems, Inc.
Retrospective assessment: US Primary energy use in 2000 from 1970s
Projections of total US primary energy use from the 1970s From “What can history teach us? A Retrospective from Examination of Long-Term Energy Forecasts for the United States” PP Craig, A Gadgil, and JG Koomey, Ann. Review Energy Environ. 2002. 27. Redrawn from US Dep. Energy. 1979. Energy Demands 1972 to 2000. Rep. HCP/R4024-01. Washington, DC: DOE. Copyright © 2007 Lumina Decision Systems, Inc.
Retrospective assessment:
Coal production (million tons) Coal Production (million short tons)
1300 1250
1150
AEO 1985
1100 1050
AEO 2000
AEO 1990
1200
AEO 1982
AEO 1995
1000
Actual
950 900 850 1985
1990
1995
2000
2005
Forecast year Selected AEO years and actual AEO 1982
AEO 1985
AEO 1990
AEO 1995
AEO 2000
Actual
Data from Annual Energy Outlook: Retrospective Review 2007. Copyright © 2007 Lumina Decision Systems, Inc.
Retrospective assessment:
World oil price ($/barrel) Compare sel forecast and actual
60
AEO 1982
50
Actual
AEO 1985
40
AEO 1990
30
AEO 1995
20
AEO 2000
Actual
10
0 1985
1990
1995
2000
2005
Forecast year Selected AEO years and actual AEO 1982
AEO 1985
AEO 1990
AEO 1995
AEO 2000
Actual
Data from Annual Energy Outlook: Retrospective Review 2007. Copyright © 2007 Lumina Decision Systems, Inc.
Retrospective assessment:
Error frequency distributions Error frequency
For 12 energy quantities 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 -50%
-40%
-30%
-20%
-10%
0%
10%
20%
30%
40%
50%
Error percent
For 4 energy prices Error frequency
0.1 0.08 0.06 0.04 0.02 0 -100%
-50%
0%
50%
100%
150%
200%
250%
300%
Error percent
Data from Annual Energy Outlook: Retrospective Review 2007. Copyright © 2007 Lumina Decision Systems, Inc.
Retrospective assessment:
Error widths by forecast period for projections of 12 energy quantities Error percentage
35%
95%ile
30% 25% 20% 15%
80%ile
10% 5%
50%ile
0% -5%
20%ile 5%ile
-10% -15% 5 1 to 5
6 10 to 10
11 to1515
Forecast Forecastperiod period(Years) (years) Percentiles
Data from Energy Outlook: Review 2007. 5%Annual 20% 50% Retrospective 80% 95% Copyright © 2007 Lumina Decision Systems, Inc.
Retrospective assessment:
Some observations • You need a long history of projections for useful results. • Some types of quantity (e.g. prices) are less predictable than others (e.g. energy flows). • Error distributions have long tails (not normal). Alexander I. Shlyakhter, Daniel M. Kammen, Claire L. Broido and Richard Wilson : The credibility ofenergy projections from trends in past data: The US energy sector, Energy Policy, Feb 1994
• Large errors are often due to rare events, outside and beyond the model. •
The Black Swan: The Impact of the Highly Improbable, Nassim Taleb, Random House, 2007
Copyright © 2007 Lumina Decision Systems, Inc.
Comparing ways to assess uncertainty in model projections Retrospective assessment Pros: • Easy to do for past years. • Interesting and informative. Cons: • Requires judgment to apply to the future: New models, and the world they represent will be different. 60
Compare sel forecast and actual
Prospective probabilistic simulation Pros: • Works for new models. Cons: • Liable to omit important sources of uncertainty.
50
40
30
20
10
0 1985
1990
1995
2000
2005
Forecast year Selected AEO years and actual AEO 1982
AEO 1985
AEO 1990
AEO 1995
AEO 2000
Actual
Copyright © 2007 Lumina Decision Systems, Inc.
Understanding probabilistic assessments of uncertainty • A degree of judgment is unavoidable: Prospective simulation: To assess input uncertainties and to judge missing sources of uncertainty Retrospective evaluation: To apply results for prospective projections
• Assessment of uncertainties are lower bounds on calibrated uncertainty Copyright © 2007 Lumina Decision Systems, Inc.
For more…
Uncertainty: A Guide to Dealing with Uncertainty in Risk and Policy Analysis. M Granger Morgan & Max Henrion, Cambridge University Press, 1990 Copyright © 2007 Lumina Decision Systems, Inc.
Summary • It is well worth the effort to quantify explicitly the uncertainties in model projections. • Quantifying uncertainties unavoidably involves judgment. Better make it explicit. • Retrospective assessments of error distributions in past projections are a valuable complement to prospective: We should do more of it. • Error distributions are long-tailed – not normal – because rare events are not so rare: we can’t model everything. • We and decision makers should understand that probabilistic projections are really lower bounds on uncertainty. Copyright © 2007 Lumina Decision Systems, Inc.
An Influence diagram for R & D decision analysis Key Decision
R&D funding
Uncertainty
Commercialization
R&D value
Value
Influence
R&D results
Market success
Copyright © 2007 Lumina Decision Systems, Inc.
A decision tree for R&D decision analysis Market value
Market success Commercialization R&D results R&D funding
Stop
EV=$210M
Succeed p=0.3
Fail p=0.7
$500M
p=0.3
Moderate p=0.5
$200M
Poor p=0.2
Stop
Go $10M R&D cost EV=$63M
Market
Good
-$200M $0 $0 $0
Copyright © 2007 Lumina Decision Systems, Inc.
Making decisions with limited information: Cost-effectiveness vs. cost-benefit analysis •
•
•
Coping with the threat of terrorism on power transmission system: How should we design a security system? For cost-benefit analysis, we need to estimate the probability of terrorist attack and the cost if successful to compare with cost of security system For cost-effectiveness: Choose the security system with maximum effectiveness in reducing vulnerability given budget available
Copyright © 2007 Lumina Decision Systems, Inc.
Rating projects on “soft” objectives: Beyond hard NPV revenues
• Don’t let “hard” numbers (monetized objectives) drive out the soft numbers • Actually, all the numbers are soft to a degree Copyright © 2007 Lumina Decision Systems, Inc.
• Don’t let hard numbers drive out soft criteria • Analyzing cost-effectiveness under a budget can let you make meaningful decisions even when some factors are too hard to quantify.
Copyright © 2007 Lumina Decision Systems, Inc.
Copyright © 2007 Lumina Decision Systems, Inc.
Guidance from OMB: How to conduct regulatory analysis •
•
•
•
“For major rules … you should present a formal quantitative analysis of the relevant uncertainties about benefits and costs.” “… expert solicitation is a useful way … to quantify the probability distributions of key parameters.” “These … can be combined in Monte Carlo simulations to derive a probability distribution of benefits and costs.” “Use a numerical sensitivity analysis to examine how the results vary with plausible changes in assumptions, choices of input data.” [Emphases added]
OMB Circular A-4, John Graham, OIRA Administrator, 17 Sep 2003 http://www.whitehouse.gov/omb/circulars/a004/a-4.html Copyright © 2007 Lumina Decision Systems, Inc.
Probabilistic simulation for prospective projections
1. Express uncertainty by eliciting probability distributions from experts
2. Use Monte Carlo simulation to propagate probability distributions through the model.
3. View uncertainty on key results 5. Make a decision
4. Use sensitivity analysis to compare effects of uncertain assumptions on results Copyright © 2007 Lumina Decision Systems, Inc.
A personal decision under uncertainty:
When to leave for the airport? • On average, it takes about 40 minutes to drive from my home to San Jose International Airport, plus 30 minutes to park, get through security and walk to the gate; I’m supposed to be at the gate 20 minutes before departure. 40+30+20 = 90
• So, I should leave 90 minutes before departure, right? • Umm, no. That way I would miss my plane about half the time. Copyright © 2007 Lumina Decision Systems, Inc.
Value function for plane catching 400 350
EVIU Expected value of including uncertainty
300
Value Value
250 200
EViu expected value, ignoring uncertainty diu decision ignoring uncertainty
150 100 50 0 40
60 60
80 80
100 100
120 120
140 140
EVev maximum expected value dev decision to maximize expected value 160 160
Time av e be fore plane parts (minutes) Time to to le leave before planede departs (minutes)
View
Pr obability Density
Value ignoring uncertainty
Expected value
0 .0 2
0 .0 1 4
For more, see chapter 12 of Uncertainty: A Guide to Dealing with Uncertainty in Risk and Policy Analysis. M Granger Morgan & Max Henrion, Cambridge UP, 1990
0 .0 1
4m 0 40
60
80
100
120
Tim e n e e d e d (m in u te s )
140
160 Copyright © 2007 Lumina Decision Systems, Inc.
Sample decisions under uncertainty for an energy consumer • Should we retrofit buildings to reduce energy usage? To what level? • Do we need backup batteries or generators in case of power outage? What capacity is cost-effective? • Should we install photovoltaics now, when CA offers a large subsidy, or wait a few years until PV is cheaper, but lower subsidy? • Should we purchase long-term energy supply contracts or hedges to protect against price volatility? Copyright © 2007 Lumina Decision Systems, Inc.
Sample decisions under uncertainty for an energy R&D organization • Deep or wide? Should we spend most funds on a few, promising projects, or spread funds over a wider range? • R or D? How should we balance early-stage seed research vs. late-stage commercial development? • When to start: Should we start funding when early research indicates technical success is conceivable, or wait until commercial success is likely – or somewhere in between? • When to stop: How soon should we abandon a project when it starts to look like it may not succeed?
Copyright © 2007 Lumina Decision Systems, Inc.
1. How to express uncertainty as probability distributions • • • • • •
•
Probability is the clearest, most widely used language for expressing uncertainty. Statistics helps us understand the uncertainty in historical data The quantity we want is not usually one for which we have data Judgment is unavoidable in extrapolating from what we have to what we want. Let’s be explicit about it Obtaining probability distributions from a range of experts is the best way to quantify the current state of knowledge (and lack thereof) There are well-developed methods for obtaining expert judgment as probability distributions Uncertainty: A Guide to Dealing with Uncertainty in Risk and Policy Analysis. M Granger Morgan & Max Henrion, Cambridge UP, 1990 Copyright © 2007 Lumina Decision Systems, Inc.
4. Sensitivity analysis: Which uncertainties matter? When? Why? • Sensitivity and uncertainty analysis quantify relative contribution of each input to uncertainty in output • A potent source of insights. • Suggests priorities for further research
Copyright © 2007 Lumina Decision Systems, Inc.
5. Making decisions under uncertainty •
•
• • •
Virtually, all important decisions are made under uncertainty – whether we acknowledge it or not. Usually, we select the decision with the maximum expected value (net social benefit) If net benefits are large relative to the uncertainty, we can act now If not, we can weigh expected benefits of awaiting better information We can assess the value of more research using the expected value of information
Copyright © 2007 Lumina Decision Systems, Inc.
3. How to display uncertainties to decision makers Probability density function
600u
Probability Density
Numerical percentiles
700u
500u 400u 300u 200u 100u 0 $0
$1000
$2000
$3000
$4000
$5000
$6000
$7000
Unit cost of PV in 2010 ($/Kw) 1 0.9 0.8
Cumulative Probability
Cumulative probability function Box plot
0.1 0 $0
$1000 $2000 $3000 $4000 $5000 $6000 $7000
Unit cost of PV in 2010 ($/Kw) 2000
600T 500T 400T 300T
1800
100T 0 $0
$1000 $2000 $3000 $4000 $5000 $6000 $7000
U n it c o s t o f P V in 2 0 1 0 ( $ /K w )
Scatter plot Monte Carlo sample Iteration (Run)
1Q 900T 800T
0.6 0.5 0.4 0.3
1600 1400 1200 1000 800 600 400 200 0 $1000
For more, see chapter 9 of Uncertainty: A Guide to Dealing with Uncertainty in Risk and Policy Analysis. M Granger Morgan & Max Henrion, Cambridge UP, 1990
$2000
$3000
$4000
$5000
$6000
U n it c o s t o f P V in 2 0 1 0 ( $ /K w )
Copyright © 2007 Lumina Decision Systems, Inc.
How to assess the uncertainty in projections from models • Probabilistic simulation for prospective projections: Assess uncertainties on all key inputs, and propagate them through the model with Monte Carlo • Retrospective evaluation: Compare results from past projections with what actually happened Copyright © 2007 Lumina Decision Systems, Inc.
Retrospective assessment:
Measured speed of light (km/sec)
Reported uncertainty in measurements of the speed of light 1900 to 1984 299,810
Currently accepted value 1984 value
299,800
299,790
299,780
Henrion, M & Fischhoff, B, “Assessing uncertainty in physical constants”, American J. Physics, 54 (9), 1986
299,770
299,760
299,750 1900
1910
1920
1930
1940
1950
1960
Year of experiment Copyright © 2007 Lumina Decision Systems, Inc.
Retrospective assessment: US Primary energy use in 2000 from 1970s
Projections of total US primary energy use from the 1970s From “What can history teach us? A Retrospective from Examination of Long-Term Energy Forecasts for the United States” PP Craig, A Gadgil, and JG Koomey, Ann. Review Energy Environ. 2002. 27. Redrawn from US Dep. Energy. 1979. Energy Demands 1972 to 2000. Rep. HCP/R4024-01. Washington, DC: DOE. Copyright © 2007 Lumina Decision Systems, Inc.
Retrospective assessment:
Coal production (million tons) Coal Production (million short tons)
1300 1250
1150
AEO 1985
1100 1050
AEO 2000
AEO 1990
1200
AEO 1982
AEO 1995
1000
Actual
950 900 850 1985
1990
1995
2000
2005
Forecast year Selected AEO years and actual AEO 1982
AEO 1985
AEO 1990
AEO 1995
AEO 2000
Actual
Data from Annual Energy Outlook: Retrospective Review 2007. Copyright © 2007 Lumina Decision Systems, Inc.
Retrospective assessment:
World oil price ($/barrel) Compare sel forecast and actual
60
AEO 1982
50
Actual
AEO 1985
40
AEO 1990
30
AEO 1995
20
AEO 2000
Actual
10
0 1985
1990
1995
2000
2005
Forecast year Selected AEO years and actual AEO 1982
AEO 1985
AEO 1990
AEO 1995
AEO 2000
Actual
Data from Annual Energy Outlook: Retrospective Review 2007. Copyright © 2007 Lumina Decision Systems, Inc.
Retrospective assessment:
Error frequency distributions Error frequency
For 12 energy quantities 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 -50%
-40%
-30%
-20%
-10%
0%
10%
20%
30%
40%
50%
Error percent
For 4 energy prices Error frequency
0.1 0.08 0.06 0.04 0.02 0 -100%
-50%
0%
50%
100%
150%
200%
250%
300%
Error percent
Data from Annual Energy Outlook: Retrospective Review 2007. Copyright © 2007 Lumina Decision Systems, Inc.
Retrospective assessment:
Error widths by forecast period for projections of 12 energy quantities Error percentage
35%
95%ile
30% 25% 20% 15%
80%ile
10% 5%
50%ile
0% -5%
20%ile 5%ile
-10% -15% 5 1 to 5
6 10 to 10
11 to1515
Forecast Forecastperiod period(Years) (years) Percentiles
Data from Energy Outlook: Review 2007. 5%Annual 20% 50% Retrospective 80% 95% Copyright © 2007 Lumina Decision Systems, Inc.
Retrospective assessment:
Some observations • You need a long history of projections for useful results. • Some types of quantity (e.g. prices) are less predictable than others (e.g. energy flows). • Error distributions have long tails (not normal). Alexander I. Shlyakhter, Daniel M. Kammen, Claire L. Broido and Richard Wilson : The credibility ofenergy projections from trends in past data: The US energy sector, Energy Policy, Feb 1994
• Large errors are often due to rare events, outside and beyond the model. •
The Black Swan: The Impact of the Highly Improbable, Nassim Taleb, Random House, 2007
Copyright © 2007 Lumina Decision Systems, Inc.
Comparing ways to assess uncertainty in model projections Retrospective assessment Pros: • Easy to do for past years. • Interesting and informative. Cons: • Requires judgment to apply to the future: New models, and the world they represent will be different. 60
Compare sel forecast and actual
Prospective probabilistic simulation Pros: • Works for new models. Cons: • Liable to omit important sources of uncertainty.
50
40
30
20
10
0 1985
1990
1995
2000
2005
Forecast year Selected AEO years and actual AEO 1982
AEO 1985
AEO 1990
AEO 1995
AEO 2000
Actual
Copyright © 2007 Lumina Decision Systems, Inc.
Understanding probabilistic assessments of uncertainty • A degree of judgment is unavoidable: Prospective simulation: To assess input uncertainties and to judge missing sources of uncertainty Retrospective evaluation: To apply results for prospective projections
• Assessment of uncertainties are lower bounds on calibrated uncertainty Copyright © 2007 Lumina Decision Systems, Inc.
For more…
Uncertainty: A Guide to Dealing with Uncertainty in Risk and Policy Analysis. M Granger Morgan & Max Henrion, Cambridge University Press, 1990 Copyright © 2007 Lumina Decision Systems, Inc.
Summary • It is well worth the effort to quantify explicitly the uncertainties in model projections. • Quantifying uncertainties unavoidably involves judgment. Better make it explicit. • Retrospective assessments of error distributions in past projections are a valuable complement to prospective: We should do more of it. • Error distributions are long-tailed – not normal – because rare events are not so rare: we can’t model everything. • We and decision makers should understand that probabilistic projections are really lower bounds on uncertainty. Copyright © 2007 Lumina Decision Systems, Inc.
An Influence diagram for R & D decision analysis Key Decision
R&D funding
Uncertainty
Commercialization
R&D value
Value
Influence
R&D results
Market success
Copyright © 2007 Lumina Decision Systems, Inc.
A decision tree for R&D decision analysis Market value
Market success Commercialization R&D results R&D funding
Stop
EV=$210M
Succeed p=0.3
Fail p=0.7
$500M
p=0.3
Moderate p=0.5
$200M
Poor p=0.2
Stop
Go $10M R&D cost EV=$63M
Market
Good
-$200M $0 $0 $0
Copyright © 2007 Lumina Decision Systems, Inc.
Making decisions with limited information: Cost-effectiveness vs. cost-benefit analysis •
•
•
Coping with the threat of terrorism on power transmission system: How should we design a security system? For cost-benefit analysis, we need to estimate the probability of terrorist attack and the cost if successful to compare with cost of security system For cost-effectiveness: Choose the security system with maximum effectiveness in reducing vulnerability given budget available
Copyright © 2007 Lumina Decision Systems, Inc.
Rating projects on “soft” objectives: Beyond hard NPV revenues
• Don’t let “hard” numbers (monetized objectives) drive out the soft numbers • Actually, all the numbers are soft to a degree Copyright © 2007 Lumina Decision Systems, Inc.
• Don’t let hard numbers drive out soft criteria • Analyzing cost-effectiveness under a budget can let you make meaningful decisions even when some factors are too hard to quantify.
Copyright © 2007 Lumina Decision Systems, Inc.
