Nassim Taleb's The Black Swan
ALGA 2015 Annual Conference
photo credit: Sergis V. Duarte, 11/23/08, via wikimedia.org
Nassim Taleb’s The Black Swan
2
Outsized impact Outside of expectations (viral)
Gaga - Gangnam
3
Cognitive dissonance
see-no-evil
4
5
6
Recommendation
7
8
• Unforeseen • According to the plan • Major impact
9
10
…and not surprised
11
Outline: Nassim Taleb’s The Black Swan
12
Outline: Risk, complex systems, failure Nassim Taleb’s The Black Swan
13
Outline: Extreme Value Analysis Nassim Taleb’s The Black Swan Risk, complex systems, failure
14
Outline: EVA in program auditing I: quantifying exposure & p Nassim Taleb’s The Black Swan Risk, complex systems, failure Extreme Value Analysis
15
Outline: 5
EVA in program auditing: II risk mitigation Nassim Taleb’s The Black Swan Risk, complex systems, failure Extreme Value Analysis EVA in practice, program auditing I
16
Outline: 6
Wrap and Q & A Nassim Taleb’s The Black Swan Risk, complex systems, failure Extreme Value Analysis EVA in practice, program auditing I EVA in practice, program auditing II
17
Acknowledgements County Auditor Peter Morris John Hutzler (my boss)
18
Risk, complex systems, failure
19
20
Simple
Complicated
Complex
21
complex
simple 22
Complex operations
Goal Fail 23
Swiss Cheese model (Orlandella/Reason)
24
https://doctordeparture.files.wordpress.com/2011/07/swisscheese.png
Drift into failure
Failure
Intended operating range 25
James Clerk Maxwell (1839‐1879) 26
Black Swans & auditing
27
Extreme Value Analysis
28
29
The good ol’ normal distribution… …often doesn’t work
30
Benoit Mandelbrot
• Cotton prices exhibit power‐law behavior • 80‐20 rule of thumb • Fractal phenomena
31
Vilfredo Pareto, 1848‐1923
Generalized Pareto Distribution
80‐20 rule 32
seven years of records: Frequency distribution: self‐insurance claims 2006‐2012
33
seven years of records:
Percentiles 100.0% maximum 99.5% 90.0% 75.0% quartile 50.0% median 25.0% quartile 10.0% 0.5%
Observed 194,897 54,340 11,514 4,150 1,451 401 130 19 34
normal distribution fits poorly...
Percentiles 100.0% maximum 99.5% 90.0% 75.0% quartile 50.0% median 25.0% quartile 10.0% 0.5%
Observed 194,897 54,340 11,514 4,150 1,451 401 130 19
Normal 47,000 34,000 19,300 12,400 4,700 ‐3,000 ‐9,900 ‐24,700
35
exponential distribution, the same... Fit of Exponential distribution 200,000 150,000 100,000 50,000
Percentiles 100.0% maximum 99.5% 90.0% 75.0% quartile 50.0% median 25.0% quartile 10.0% 0.5%
Observed Exponential 194,897 53,900 54,340 24,800 11,514 10,782 4,150 6,500 1,451 3,200 401 1,300 R² = 0.976 130 500 19 25
0
observed Series1
exponential fit Expon. (Series1)
36
Generalized Pareto Distribution three variables:
Vilfredo Pareto 1848‐1923
µ = location σ = scale k = shape
37
peaks‐over‐threshold (PoT): all records above a threshold
38
Key idea: the tail has its own “child” distribution
• No assumptions are made non‐extreme observations 39 http://www.r‐bloggers.com/playing‐with‐quantiles‐part‐1/
EVA in practice, part I quantifying exposure & probability
40
EVA in practice, part I • Is EVA suitable for the data? • What threshold is best? • What values for location, scale, shape? • What losses for range of probabilities?
41
Threshold selection:
42
Shape
Scale
Location
Threshold selection:
43
Diagnose fit:
7 6 5 3
4
empirical
8
9
Quantile Plot
3
4
5
6
7
8
9
model http://www.sldinfo.com/wp‐content/uploads/2011/09/bigstock_Black_Swan_63382.jpg 44
Best‐fit results: > current1 $320k 50
How well did it work?
51
…compared to what?
52
Tips: • The numbers only tell part of the story • Focus on the confidence interval • Be mindful of the black swans • Don’t over‐extend • Be conservative • Trust the method 53
6
Outline: Wrap and Q & A Nassim Taleb’s The Black Swan Extreme Value Analysis, theory I Extreme Value Analysis, theory II EVA in practice, program auditing I EVA in practice, program auditing II 54
Don’t use old‐&‐busted methods...
55 credit: animalworld.tumblr.com/post/7488869041
Recommended text:
credit: Sergis V. Duarte, via wikimedia.org 12/2006
56
References Bensalah Y (2000) Steps in Applying Extreme Value Theory to Finance: A Review. Research and Risk Management Section, Financial Markets Department, Bank of Canada. Ottawa, Ontario, Canada p 9 Cook RI (2000) How Complex Systems Fail, a Short Treatise. Cognitive Technologies Laboratory, University of Chicago 5 pps Dekker S (2011) Drift Into Failure: From Hunting Broken Components to Understanding Complex Systems. Ashgate Publishing Company 234 pps Dorner, D (1997) The Logic of Failure: Recognizing and Avoiding Errors in Complex Situations. Basic Books 240 pps Gall J (1977) Sytemantics: How Sytems Work, and Especially How They Fail. Pocket Books 111 pps Mohtadi H, Murshid A (2005) Assessing the Risk of Terrorism Using Extreme Value Statistics. Proceedings of the Institute of Food Technologists’ First Annual Food Protection and Defense Conference, October 27, 2005 57
References (cont’d) Perrow C (1999) Normal Accidents: Living with High Risk Technologies. Princeton University Press 464 pps Taleb N (2007) The Black Swan: The Impact of the Highly Improbable. Random House, NY 366 pps Taleb N (2010) The Fourth Quadrant: A Map of the Limits of Statistics. edge.org http://www.slideshare.net/MediaCorpInternational/130221‐0850‐david‐bond‐thiess www.nytimes.com/2007/04/22/books/chapters/0422‐1st‐tale.html www.thefullwiki.org/Black_swan_theory www.insead.edu/facultyresearch/faculty/documents/4_Quadrants‐w.pdf www.nakedcapitalism.com www.coverclock.blogspot.com www.zoonek2.free.fr/UNIX/48_R/07.html www2.etf.com/publications/journalofindexes/joi‐articles/24230‐what‐is‐ risk.html?showall=&fullart=1 Graphics: Unless noted otherwise, all graphics from via wikimedia.org 58
6
Outline: Wrap and Q & A Nassim Taleb’s The Black Swan Extreme Value Analysis, theory I Extreme Value Analysis, theory II EVA in practice, program auditing I EVA in practice, program auditing II 59
http://www.tylervigen.com 60
61 http://www.tylervigen.com
62 http://www.tylervigen.com
63 http://www.tylervigen.com
Enjoy the rest of the conference!
64
photo credit: Sergis V. Duarte, 11/23/08, via wikimedia.org
Nassim Taleb’s The Black Swan
2
Outsized impact Outside of expectations (viral)
Gaga - Gangnam
3
Cognitive dissonance
see-no-evil
4
5
6
Recommendation
7
8
• Unforeseen • According to the plan • Major impact
9
10
…and not surprised
11
Outline: Nassim Taleb’s The Black Swan
12
Outline: Risk, complex systems, failure Nassim Taleb’s The Black Swan
13
Outline: Extreme Value Analysis Nassim Taleb’s The Black Swan Risk, complex systems, failure
14
Outline: EVA in program auditing I: quantifying exposure & p Nassim Taleb’s The Black Swan Risk, complex systems, failure Extreme Value Analysis
15
Outline: 5
EVA in program auditing: II risk mitigation Nassim Taleb’s The Black Swan Risk, complex systems, failure Extreme Value Analysis EVA in practice, program auditing I
16
Outline: 6
Wrap and Q & A Nassim Taleb’s The Black Swan Risk, complex systems, failure Extreme Value Analysis EVA in practice, program auditing I EVA in practice, program auditing II
17
Acknowledgements County Auditor Peter Morris John Hutzler (my boss)
18
Risk, complex systems, failure
19
20
Simple
Complicated
Complex
21
complex
simple 22
Complex operations
Goal Fail 23
Swiss Cheese model (Orlandella/Reason)
24
https://doctordeparture.files.wordpress.com/2011/07/swisscheese.png
Drift into failure
Failure
Intended operating range 25
James Clerk Maxwell (1839‐1879) 26
Black Swans & auditing
27
Extreme Value Analysis
28
29
The good ol’ normal distribution… …often doesn’t work
30
Benoit Mandelbrot
• Cotton prices exhibit power‐law behavior • 80‐20 rule of thumb • Fractal phenomena
31
Vilfredo Pareto, 1848‐1923
Generalized Pareto Distribution
80‐20 rule 32
seven years of records: Frequency distribution: self‐insurance claims 2006‐2012
33
seven years of records:
Percentiles 100.0% maximum 99.5% 90.0% 75.0% quartile 50.0% median 25.0% quartile 10.0% 0.5%
Observed 194,897 54,340 11,514 4,150 1,451 401 130 19 34
normal distribution fits poorly...
Percentiles 100.0% maximum 99.5% 90.0% 75.0% quartile 50.0% median 25.0% quartile 10.0% 0.5%
Observed 194,897 54,340 11,514 4,150 1,451 401 130 19
Normal 47,000 34,000 19,300 12,400 4,700 ‐3,000 ‐9,900 ‐24,700
35
exponential distribution, the same... Fit of Exponential distribution 200,000 150,000 100,000 50,000
Percentiles 100.0% maximum 99.5% 90.0% 75.0% quartile 50.0% median 25.0% quartile 10.0% 0.5%
Observed Exponential 194,897 53,900 54,340 24,800 11,514 10,782 4,150 6,500 1,451 3,200 401 1,300 R² = 0.976 130 500 19 25
0
observed Series1
exponential fit Expon. (Series1)
36
Generalized Pareto Distribution three variables:
Vilfredo Pareto 1848‐1923
µ = location σ = scale k = shape
37
peaks‐over‐threshold (PoT): all records above a threshold
38
Key idea: the tail has its own “child” distribution
• No assumptions are made non‐extreme observations 39 http://www.r‐bloggers.com/playing‐with‐quantiles‐part‐1/
EVA in practice, part I quantifying exposure & probability
40
EVA in practice, part I • Is EVA suitable for the data? • What threshold is best? • What values for location, scale, shape? • What losses for range of probabilities?
41
Threshold selection:
42
Shape
Scale
Location
Threshold selection:
43
Diagnose fit:
7 6 5 3
4
empirical
8
9
Quantile Plot
3
4
5
6
7
8
9
model http://www.sldinfo.com/wp‐content/uploads/2011/09/bigstock_Black_Swan_63382.jpg 44
Best‐fit results: > current1 $320k 50
How well did it work?
51
…compared to what?
52
Tips: • The numbers only tell part of the story • Focus on the confidence interval • Be mindful of the black swans • Don’t over‐extend • Be conservative • Trust the method 53
6
Outline: Wrap and Q & A Nassim Taleb’s The Black Swan Extreme Value Analysis, theory I Extreme Value Analysis, theory II EVA in practice, program auditing I EVA in practice, program auditing II 54
Don’t use old‐&‐busted methods...
55 credit: animalworld.tumblr.com/post/7488869041
Recommended text:
credit: Sergis V. Duarte, via wikimedia.org 12/2006
56
References Bensalah Y (2000) Steps in Applying Extreme Value Theory to Finance: A Review. Research and Risk Management Section, Financial Markets Department, Bank of Canada. Ottawa, Ontario, Canada p 9 Cook RI (2000) How Complex Systems Fail, a Short Treatise. Cognitive Technologies Laboratory, University of Chicago 5 pps Dekker S (2011) Drift Into Failure: From Hunting Broken Components to Understanding Complex Systems. Ashgate Publishing Company 234 pps Dorner, D (1997) The Logic of Failure: Recognizing and Avoiding Errors in Complex Situations. Basic Books 240 pps Gall J (1977) Sytemantics: How Sytems Work, and Especially How They Fail. Pocket Books 111 pps Mohtadi H, Murshid A (2005) Assessing the Risk of Terrorism Using Extreme Value Statistics. Proceedings of the Institute of Food Technologists’ First Annual Food Protection and Defense Conference, October 27, 2005 57
References (cont’d) Perrow C (1999) Normal Accidents: Living with High Risk Technologies. Princeton University Press 464 pps Taleb N (2007) The Black Swan: The Impact of the Highly Improbable. Random House, NY 366 pps Taleb N (2010) The Fourth Quadrant: A Map of the Limits of Statistics. edge.org http://www.slideshare.net/MediaCorpInternational/130221‐0850‐david‐bond‐thiess www.nytimes.com/2007/04/22/books/chapters/0422‐1st‐tale.html www.thefullwiki.org/Black_swan_theory www.insead.edu/facultyresearch/faculty/documents/4_Quadrants‐w.pdf www.nakedcapitalism.com www.coverclock.blogspot.com www.zoonek2.free.fr/UNIX/48_R/07.html www2.etf.com/publications/journalofindexes/joi‐articles/24230‐what‐is‐ risk.html?showall=&fullart=1 Graphics: Unless noted otherwise, all graphics from via wikimedia.org 58
6
Outline: Wrap and Q & A Nassim Taleb’s The Black Swan Extreme Value Analysis, theory I Extreme Value Analysis, theory II EVA in practice, program auditing I EVA in practice, program auditing II 59
http://www.tylervigen.com 60
61 http://www.tylervigen.com
62 http://www.tylervigen.com
63 http://www.tylervigen.com
Enjoy the rest of the conference!
64
