The Signal and the Noise
The Signal and the Noise
Why Most Predictions Fail but Some Don’t
By Nate Silver
When a presidential election is underway, most of us have an opinion on who’s going to win. And if we really want to go out on a limb, we might even take a stab at predicting the margin of victory. Sometimes we even get it right. But if we get it wrong, it’s no big deal; it’s not like our livelihood depends on making accurate guesses. Unless we’re Nate Silver, that is. Nate Silver – arguably America’s most influential political forecaster – makes his living by predicting the outcomes of elections (and also by writing about them). Along with a whole bunch of other game-changing political and economic events over the last decade or so, Silver correctly predicted the outcomes of the 2008 and 2012 U.S. presidential elections down to the decimal point. For Silver, prediction is more than a game. It is a science; or something very close to it anyway. You may have come across Nate Silver’s column in the political blog FiveThirtyEight. com, which is published by the New York Times. Or, if you’re a sports fan, you may be familiar with his earlier work that forecasted the performance of Major League Baseball players. Last year, Silver gave his loyal fans something new. This time, instead of a blog, it’s a book. In The Signal and the Noise, Silver takes us on a whirlwind tour of the success and failure of predictions and teaches us how to systematically improve our own forecasting skills. Drawing on the latest research from psychology, economics and neuroscience, Silver examines the world of prediction, investigating how we can distinguish a true “signal” (i.e. a reliable indicator that something important is about to happen) from a universe of noisy data. Silver interviews some of America’s most successful forecasters in a diverse range of fields, from Capitol Hill to the NBA. He explains and evaluates how these forecasters think and what traits they share. What lies behind their success? Are they good – or just lucky? “Predictions fail,” writes Silver, “and often at great cost to society. This is because most of us have a poor understanding of probability and uncertainty.” Silver points out that the most accurate forecasters tend to have a superior command of probability, and they tend to be both humble and hardworking. They distinguish the predictable from the
1
To purchase a personal subscription or corporate license, please visit us at www.TheBusinessSource.com or fill out the simple request form at www.TheBusinessSource.com/contact-us. All Rights reserved. Reproduction or redistribution in whole or in part without prior written permission of The Business Source is strictly prohibited.
The Signal and the Noise
unpredictable, and they notice a thousand little details that lead them closer to the truth. In other words, they distinguish the signal from the noise. After reading this summary, and absorbing a few of Silver’s key lessons along the way, it’s our hope that you’ll be that much closer to being able to do the same.
The Problem with Prediction Most of us aren’t very good at making accurate predictions. For the most part, the consequences we suffer when our predictions go wrong are not all that bad – we end up getting rained on, for instance, because we neglected to pack an umbrella. But sometimes the consequences are far more dire. Consider the attacks of September 11, 2001, for example. The problem with our (failed) predictions leading up to that fateful day was not a lack of information. As had been the case in the Pearl Harbor attacks six decades earlier, all the signals were there. But we had not put them together. Lacking a proper theory for how terrorists might behave, we were blind to the data and the events of 9/11 completely blindsided us. To add insult to injury, there were also the widespread failures of prediction that accompanied the 2008 global financial crisis. As Silver points out, our naïve trust in unproven models – and our failure to realize how fragile they were given the shaky underlying assumptions – yielded disastrous results. And then came the calamitous Japanese nuclear incident a few years later. As we now know, the failed Fukushima nuclear reactor had originally been designed to handle a magnitude 8.6 earthquake, in part because the “expert” seismologists concluded that anything larger was impossible. Yet the March 2011 earthquake registered a 9.1 on the Richter scale. Why are we so lousy at making accurate predictions? It comes down to the way our brains are hardwired. Biologically, we aren’t very different from our stone-age ancestors. “Human beings do not have very many natural defenses,” writes Silver. “Instead, we survive by means of our wits. Our minds are very quick. We are wired to detect patterns and respond to opportunities and threats without much hesitation.” Tens of thousands of years ago, our natural instinct to look for patterns and respond instantly served us very well. But some of our stone-age strengths have become information-age weaknesses. Today, we’re living in a data-rich society (an era of “Big Data”). According to best estimates, the quantity of raw information is increasing by 2.5 quintillion bytes per day. But that doesn’t necessarily mean that the amount of useful information is increasing at the same pace. “Most of the new data that’s being generated is just noise,” writes Silver, “and the noise is increasing faster than the signal. There are so many hypotheses to test, so many data sets to mine – but there’s still a relatively constant amount of objective truth.” Unless we work very hard to become aware of the evolutionary biases that naturally cloud our thinking, the societal returns from all of this additional information may be minimal, or even diminishing. In other words, Big Data may just be making things worse.
2
TheBusinessSource.com
All Rights Reserved
The Signal and the Noise
The Prediction Solution “Prediction is indispensable to our lives,” writes Silver. “Every time we choose a route to work, decide whether to go on a second date, or set money aside for a rainy day, we are making a forecast about how the future will proceed – and how our plans will affect the odds for a favorable outcome.” As such, it’s impossible to avoid making predictions altogether; we simply wouldn’t be able to function without them. We just need to learn to make better ones. The good news is, the most calamitous failures of prediction usually have a lot in common. This means we can consciously look for these potential blind-spots and try to correct for them. The most common prediction failures are: (1) We focus on those signals that tell a story about the world as we would like it to be, not how it really is; (2) We ignore the risks that are hardest to measure, even when they pose the greatest threats to our well-being; (3) We make assumptions about the world that are much cruder than we realize; and (4) We abhor uncertainty, even when it is an irreducible part of the problem we are trying to solve. To see how these four fatal flaws in prediction can play out in practice, let’s consider what happened during the recent global financial crisis. In the years leading up to the 2008 meltdown, our financial system had become so highly levered that a single lax assumption in the credit ratings agencies’ models played a huge role in bringing down the whole global financial system. Specifically, credit rating agencies like Standard & Poor’s and Moody’s made the mistake of assigning AAA blue-chip credit ratings to bundled packages of residential mortgage debt, which were then sold to investors by the big Wall Street brokerage firms. These bundled Collateralized Debt Obligations, or CDOs, represented a fairly novel financial innovation that was not well understood. The ratings firms may have thought they had a pretty good handle on the risks associated with mortgage-backed CDOs. But according to Silver, the ratings agencies’ were clearly unable to appreciate the distinction between risk and uncertainty, which ties back to two of the four most common prediction failures outlined above. The concepts of risk and uncertainty can be tricky ones to grasp. Risk, as first articulated by the economist Frank H. Knight in 1921, is something that you can put a price on. “Imagine that you’ll win a poker hand unless your opponent draws to an inside straight,” explains Silver. “In this case the chances of that happening are exactly 1 chance in 11.” Risk is inherently measurable. Needless to say, it’s not a pleasant feeling when you take a “bad beat” in poker. But at least you know the odds of it and can account for it ahead of time. Uncertainty, on the other hand, is risk that is hard, or impossible, to measure. “You might have some vague awareness of the demons lurking out there,” writes Silver. “And you might even be acutely concerned about them. But you have no real idea how many of them there are or when they might strike.” Consequently, any “back-of-the-envelope” estimate you come up with might well be off by a factor of 100, or even 1,000. Really, there is just no good way to know.
3
TheBusinessSource.com
All Rights Reserved
The Signal and the Noise
So, if uncertainty is all around us, and if it defies any type of measurement, how can we account for the problem of uncertainty in our predictions? By applying the following three principles:
Three Principles for Making Better Predictions Principle 1: Think Probabilistically Silver is very careful to articulate a range of possible outcomes and assign a probability to each one. For example, if you play enough poker hands, chances are you’ll get beaten badly by someone who’s holding a royal flush when you only have a full house. Now, the chances of your opponent catching a royal flush and beating your full house are very small – let’s say less than one percent. So you probably don’t have to spend a lot of time worrying about this potential outcome. The chances of it happening are very small. But they’re not exactly zero, and this is Silver’s point. In his view, far too many forecasters fail to account for the possibility of low-probability events occurring. The importance of communicating uncertainty by speaking in probabilistic (as opposed to dead certain) language is particularly acute when public safety may be at risk. To illustrate this point, Silver reminds us of the devastating 1997 Red River flood in Grand Forks, North Dakota. Nearly 50,000 residents had to be evacuated, clean-up costs ran into the billions, and 75% of the city’s homes were damaged or destroyed. Silver argues that Grand Forks may have been a preventable disaster had the city’s floodwalls been properly reinforced using sandbags or other means. The residents of Grand Forks had been aware of the flood threat for months. Snowfall had been especially heavy in the Great Plains that winter, and the National Weather Service, anticipating runoff as the snow melted, had predicted the waters of the Red River would crest to forty-nine feet, close to the all-time record. But no one was panicking as the snow started melting, because the levees in Grand Forks had been built to handle a flood of fifty-one feet. So the residents felt safe. The problem was, even a small miss in the forty-nine-foot prediction would prove catastrophic to the whole town. And in fact, the river ended up cresting to fifty-four feet. The margin of error on the Weather Service’s forecast – based on how well their forecasts had done in the past – was about plus or minus nine feet. That implied about a 35 percent chance of the levees being overtopped. But the Weather Service had explicitly avoided communicating the uncertainty in their forecast to the public, emphasizing only the forty-nine-foot prediction. The weather forecasters later told researchers that they were afraid the public might lose confidence in the forecast if they had conveyed any uncertainty in the outlook. Instead, it would have made the public much better prepared; and possibly able to prevent the flooding by reinforcing the levees or diverting the river flow. Left to their own devices, many residents became convinced they didn’t have anything to worry about. (Very few of them bought flood insurance.) A prediction of a forty-nine-foot crest in the river, expressed without any
4
TheBusinessSource.com
All Rights Reserved
The Signal and the Noise
reservation, seemed to imply that the flood would hit forty-nine feet exactly; the fifty-one-foot levees would be just enough to keep them safe. Some residents even interpreted the forecast of forty-nine feet as representing the maximum possible extent of the flood. Fortunately, the National Weather Service has since come to recognize the importance of communicating the uncertainty in their forecasts accurately and honestly to the public. But according to Silver, this sort of attitude is still relatively rare among other kinds of forecasters.
Principle 2: Today’s Forecast Is the First Forecast of the Rest of Your Life Predictions should never be carved in stone because the world is a dynamic, ever-changing place. This is especially true when we’re speaking about the economy. The global economy, much like the global atmosphere, is a dynamic system. Everything affects everything else and the systems are perpetually in motion. In meteorology, this problem is quite literal; since the weather is subject to “chaos theory” (e.g. a butterfly flapping its wings in Brazil can theoretically cause a tornado in Texas). But in loosely the same way, “A tsunami in Japan or a longshoreman’s strike in Long Beach can affect whether someone in Texas finds a job.” For Silver, the best way to deal with a constantly changing environment is to adopt a humble attitude. Make the best forecast possible today – regardless of what you said last week, last month, or last year. Making a new forecast doesn’t mean that the old one just disappears. (Ideally, you should keep a record of it and let people evaluate how well you did over the whole course of predicting an event.) But if you have reason to think that yesterday’s forecast was wrong, there is no glory in sticking to it. “When the facts change, I change my mind,” the economist John Maynard Keynes famously quipped.
Principle 3: Look for Consensus According to Silver, every person who makes their living by predicting outcomes secretly fantasizes that they will become famous one day by making a daring, audacious, outside-thebox prediction – one that differs radically from the consensus view on a subject. In their fantasy, they imagine themselves on the front page of the Wall Street Journal, or maybe even sitting on Jay Leno’s couch, singled-out as a bold and brave pioneer in their chosen field. “Every now and then, it might be correct to make a bold forecast,” writes Silver. The expert consensus can be wrong. For example, someone who had forecasted the collapse of the Soviet Union would have deserved most of the kudos that came to him. But the fantasy scenario is hugely unlikely, and plenty of historical evidence suggests that aggregate or group forecasts are more accurate than individual ones. In fact, consensusbased forecasts are typically somewhere between 15 and 20 percent more accurate,
5
TheBusinessSource.com
All Rights Reserved
The Signal and the Noise
depending on the discipline.
Why Context Matters “Data is useless without context,” says Nate Silver. With the exponential growth in the availability of information and computing technologies, we can now measure millions of potentially interesting variables. “For instance, the U.S. government now publishes data on about 45,000 economic statistics. If you want to test for relationships between all combinations of two pairs of these statistics (e.g. is there a causal relationship between the bank prime loan rate and the unemployment rate in Alabama?) that gives you literally one billion hypotheses to test.” Clearly, the number of meaningful relationships in all of this economic data – i.e. those that speak to causality rather than mere correlation – is orders of magnitude smaller. Or, to put it differently, most of the economic and other data we collect and disseminate is just noise. In order to cut through all the clutter and find the “signal” in the noise, we need context. To explain this point, Silver tells the story of the famous Vegas sports bettor Haralabos “Bob” Voulgaris. In a bad year, Voulgaris probably makes about a million dollars profit, give or take. In a good year, he might make three or four times that. “Voulgaris doesn’t fit the stereotype of the cigar-chomping gambler in a leisure suit. He does not depend on crooked insider tips, or other sorts of hustles to make his bets. Nor does he have a ‘system’ of any kind. He uses computer simulations, but does not rely upon them exclusively.” Instead of looking for more data, Voulgaris spends much of his waking life looking for more “context.” During basketball season, Voulgaris spends every single night from November through June watching the NBA. He has five games on at a time, on five Samsung flat screens. What makes Voulgaris so successful is the way that he analyzes information. He isn’t just hunting for patterns. Instead, he combines his knowledge of statistics with his knowledge of basketball in order to identify meaningful relationships in the data. In short, Voulgaris’ big secret is that he doesn’t have a big secret. Instead, he has a thousand little secrets. He watches virtually every NBA game – some live, some on tape – and develops his own opinions about which teams are playing up to their talent and which aren’t. He looks for subtleties. For example, he knows which star players may be coming up on free agency, and thus may be more motivated to go out and score a whole bunch of points, versus those that just signed big, fat contracts and consequently may feel like they have nothing to prove. And for all his knowledge of basketball, and all his “hard work” watching so many games every night, he still gets only about 57 percent of his bets right. It is just exceptionally difficult to do much better than that. Nevertheless, Voulgaris is easily among the top 10 sports bettors in the world. His predictions have made him wealthy because he understands the power of context.
6
TheBusinessSource.com
All Rights Reserved
The Signal and the Noise
A Word on “Bayesian” Forecasting If we aspire to become better forecasters, Silver urges us to become “Bayesians.” The word Bayesian is derived from the Eighteenth Century English mathematician, Thomas Bayes, who created an elegant mathematical rule to account for two or more interrelated variables in the calculation of probabilistic outcomes. In most Bayesian calculations, at least one of those variables is based on a solid, irrefutable fact, and the other is based on a more subjective personal belief (which involves some degree of uncertainty). Since we’d probably all prefer not to have to break out our slide rules at this point, we’ll try to illustrate Bayes Theorem by using a simple, everyday example … Suppose a co-worker told you they had a really nice conversation with someone on an airplane this past weekend. Not knowing anything else about this conversation, the probability that your colleague was speaking with a woman is 50% (based on the fact that roughly 50% of the general population is female). Now suppose your co-worker also told you that this person on the plane had really long hair. Based on your experience, it now seems more likely they were speaking to a woman, since subjectively you believe that most longhaired people are women. Bayes’ theorem can then be used to calculate the probability that the person is female. 18:44 To see how this is done in practice, for the purposes of the mathematical equation for determining probability (“P”) that follows below, we’ll let the letter “W” represent the chance that the conversation on the plane was held with a woman, and the letter “L” denote the chance that it was held with a long-haired person. To populate the equation, you’ll search your mind for relevant facts. As you know, women constitute roughly half the population. So, if you knew nothing else about the conversation in question, it’s safe to say that the probability that the person on the plane was a woman is 50%. But you do have one other bit of relevant information – the length of the person’s hair. Suppose you believe, subjectively, that 75% of women have long hair. You might then say that the probability of the person on the plane being female is 75%. You’d have no hard evidence at your disposal to support this belief, but since, in your experience, you’re far more accustomed to meeting women with long hair as opposed to men, you feel pretty comfortable with your 75% “guesstimate.” Since your end goal is to calculate the probability that the conversation was held with a woman, based on these two sets of variables, you would then plug the numbers into the equation for Bayes Theorem, which reads as follows:
When you go ahead and run the numbers, you’ll find that the probability that the conversation was held with a woman – given that the person also had long hair – is approximately 70%.
7
TheBusinessSource.com
All Rights Reserved
The Signal and the Noise
This result probably shouldn’t be surprising because 70% is essentially a middle-ground number between the factual 50-50 population split between men and women, and your more subjective belief that 75% of people with long hair are women. It seems like a pretty credible prediction. In addition to its obvious mathematical potency, the real beauty of Bayes Theorem from a forecasting point of view is the requirement to explicitly state one’s subjective assumptions (which can be quite revealing about the inner thinking of the forecaster herself). Silver knows that we humans all have natural biases which can be very hard, if not impossible, to overcome. And so, perhaps the best we can do is be upfront about our biases so that others can make their own determinations as to whether or not our predictions are sound.
Beating the Competition Naturally, we all aspire to be perfect in our predictions. Nobody enjoys getting things wrong and looking silly in front of their friends, clients or colleagues. But since no one is actually perfect, the true test of how good you are as a forecaster is your track-record relative to the competition. In other words, it’s less important to see how accurate your predictions are in an absolute sense, but rather how good they are relative to other people’s predictions. Nowhere is this principle more true than in online Texas Hold’em poker. In Texas Hold’em, you can make 95 percent of your decisions correctly and still lose your shirt at a table full of players who are making the right move 99 percent of the time. Likewise, beating the stock market requires out-maneuvering teams of well-paid investors with MBAs and state-of-the-art computer systems at their disposal. There’s a learning curve that applies to poker and stock-picking and any other competitive field that involves some type of prediction. The key thing about a learning curve is that it really is a curve: the progress we make performing the task is not linear. The name for this curve comes from the well-known business maxim called the Pareto Principle or 80-20 rule (as in: 80 percent of your profits come from 20 percent of your customers). As Silver applies this rule in the context of making relatively good predictions, it posits that getting a few basic things right can go a long way. In poker, for instance, simply learning to consistently fold your worst hands, bet your best ones, and consider what your opponent holds will substantially improve your game. If you’re willing to do this, then perhaps 80 percent of the time you’ll be making the same good decisions as the very best poker players in the game. In most fields, if your predictions are accurate 80% of the time, then you’ll probably be doing well enough to hold onto your job and maybe even outshine much of your competition. But in some fields – and online poker is surely one of them – it can require a lot of extra effort to beat the competition. You may work hard to improve your game and come to be accurate with your predictions 80% of the time, but you still won’t be beating your competition, which is very intense. So at that point, you’d be left with essentially two choices: You could continue to work really hard to improve your game, or you could drop out and find a different form of
8
TheBusinessSource.com
All Rights Reserved
The Signal and the Noise
difference in your outcomes. You’ll have to work twice as hard to reach 85% proficiency as you did to reach 80% proficiency in the first instance, and you may still find yourself losing money. So consider Silver’s advice: If you have strong analytical skills (and good context knowledge) that might be applicable in a number of other disciplines, it is very much worth considering the strength of your competition. “It is often possible to make a profit by being pretty good at prediction in fields where the competition often succumbs to bad habits, or blind adherence to tradition,” he writes.
Conclusion In North America, we live in a very results-oriented society. If someone is rich or famous or beautiful, we tend to assume they somehow deserve to be those things. As an empirical matter, however, success is determined by some combination of hard work, natural talent, and a person’s environment – in other words, some combination of noise and signal. In North America, we tend to emphasize the signal component most of the time – we’re too quick to point to some sort of casual relationship between a person’s talent, or hard work and their success. In some cases, they just happened to be in the right place at the right time. When it comes to assessing other people’s predictions, we’re similarly results-oriented. “The investor who called the stock market bottom must be a genius,” writes Silver, “even if he had some buggy statistical model that just happened to get it right.” Truly, the guy just got lucky. The solution here is to focus more on the prediction process than on its results. This can be hard to do, but it pays huge dividends. Consider again the example of competitive poker players who tend to understand that process trumps results, if only because they experience the ups-and-downs in such an immediate, visceral way. If you play a lot of poker, you know that sometimes you’ll play well and win. But you’ll also play well and lose. And you’ll play badly and lose, and even sometimes play badly and win. Every serious poker player has experienced each of these conditions so many times that they know there is a difference between process and results. But playing well, regardless of the short-term outcomes, is what will make you profitable over the long-term. For example, if you correctly detect an opponent’s bluff and call him on it with the best hand, but he gets a lucky card “on the river” and wins the hand anyway, Silver says you should be actually pleased with yourself rather than angry, because you played as well as you could. The main thing is to be disciplined about the process you take to prediction; and to also be humble about your chances of being right. The results you seek will surely follow.
9
TheBusinessSource.com
All Rights Reserved
Why Most Predictions Fail but Some Don’t
By Nate Silver
When a presidential election is underway, most of us have an opinion on who’s going to win. And if we really want to go out on a limb, we might even take a stab at predicting the margin of victory. Sometimes we even get it right. But if we get it wrong, it’s no big deal; it’s not like our livelihood depends on making accurate guesses. Unless we’re Nate Silver, that is. Nate Silver – arguably America’s most influential political forecaster – makes his living by predicting the outcomes of elections (and also by writing about them). Along with a whole bunch of other game-changing political and economic events over the last decade or so, Silver correctly predicted the outcomes of the 2008 and 2012 U.S. presidential elections down to the decimal point. For Silver, prediction is more than a game. It is a science; or something very close to it anyway. You may have come across Nate Silver’s column in the political blog FiveThirtyEight. com, which is published by the New York Times. Or, if you’re a sports fan, you may be familiar with his earlier work that forecasted the performance of Major League Baseball players. Last year, Silver gave his loyal fans something new. This time, instead of a blog, it’s a book. In The Signal and the Noise, Silver takes us on a whirlwind tour of the success and failure of predictions and teaches us how to systematically improve our own forecasting skills. Drawing on the latest research from psychology, economics and neuroscience, Silver examines the world of prediction, investigating how we can distinguish a true “signal” (i.e. a reliable indicator that something important is about to happen) from a universe of noisy data. Silver interviews some of America’s most successful forecasters in a diverse range of fields, from Capitol Hill to the NBA. He explains and evaluates how these forecasters think and what traits they share. What lies behind their success? Are they good – or just lucky? “Predictions fail,” writes Silver, “and often at great cost to society. This is because most of us have a poor understanding of probability and uncertainty.” Silver points out that the most accurate forecasters tend to have a superior command of probability, and they tend to be both humble and hardworking. They distinguish the predictable from the
1
To purchase a personal subscription or corporate license, please visit us at www.TheBusinessSource.com or fill out the simple request form at www.TheBusinessSource.com/contact-us. All Rights reserved. Reproduction or redistribution in whole or in part without prior written permission of The Business Source is strictly prohibited.
The Signal and the Noise
unpredictable, and they notice a thousand little details that lead them closer to the truth. In other words, they distinguish the signal from the noise. After reading this summary, and absorbing a few of Silver’s key lessons along the way, it’s our hope that you’ll be that much closer to being able to do the same.
The Problem with Prediction Most of us aren’t very good at making accurate predictions. For the most part, the consequences we suffer when our predictions go wrong are not all that bad – we end up getting rained on, for instance, because we neglected to pack an umbrella. But sometimes the consequences are far more dire. Consider the attacks of September 11, 2001, for example. The problem with our (failed) predictions leading up to that fateful day was not a lack of information. As had been the case in the Pearl Harbor attacks six decades earlier, all the signals were there. But we had not put them together. Lacking a proper theory for how terrorists might behave, we were blind to the data and the events of 9/11 completely blindsided us. To add insult to injury, there were also the widespread failures of prediction that accompanied the 2008 global financial crisis. As Silver points out, our naïve trust in unproven models – and our failure to realize how fragile they were given the shaky underlying assumptions – yielded disastrous results. And then came the calamitous Japanese nuclear incident a few years later. As we now know, the failed Fukushima nuclear reactor had originally been designed to handle a magnitude 8.6 earthquake, in part because the “expert” seismologists concluded that anything larger was impossible. Yet the March 2011 earthquake registered a 9.1 on the Richter scale. Why are we so lousy at making accurate predictions? It comes down to the way our brains are hardwired. Biologically, we aren’t very different from our stone-age ancestors. “Human beings do not have very many natural defenses,” writes Silver. “Instead, we survive by means of our wits. Our minds are very quick. We are wired to detect patterns and respond to opportunities and threats without much hesitation.” Tens of thousands of years ago, our natural instinct to look for patterns and respond instantly served us very well. But some of our stone-age strengths have become information-age weaknesses. Today, we’re living in a data-rich society (an era of “Big Data”). According to best estimates, the quantity of raw information is increasing by 2.5 quintillion bytes per day. But that doesn’t necessarily mean that the amount of useful information is increasing at the same pace. “Most of the new data that’s being generated is just noise,” writes Silver, “and the noise is increasing faster than the signal. There are so many hypotheses to test, so many data sets to mine – but there’s still a relatively constant amount of objective truth.” Unless we work very hard to become aware of the evolutionary biases that naturally cloud our thinking, the societal returns from all of this additional information may be minimal, or even diminishing. In other words, Big Data may just be making things worse.
2
TheBusinessSource.com
All Rights Reserved
The Signal and the Noise
The Prediction Solution “Prediction is indispensable to our lives,” writes Silver. “Every time we choose a route to work, decide whether to go on a second date, or set money aside for a rainy day, we are making a forecast about how the future will proceed – and how our plans will affect the odds for a favorable outcome.” As such, it’s impossible to avoid making predictions altogether; we simply wouldn’t be able to function without them. We just need to learn to make better ones. The good news is, the most calamitous failures of prediction usually have a lot in common. This means we can consciously look for these potential blind-spots and try to correct for them. The most common prediction failures are: (1) We focus on those signals that tell a story about the world as we would like it to be, not how it really is; (2) We ignore the risks that are hardest to measure, even when they pose the greatest threats to our well-being; (3) We make assumptions about the world that are much cruder than we realize; and (4) We abhor uncertainty, even when it is an irreducible part of the problem we are trying to solve. To see how these four fatal flaws in prediction can play out in practice, let’s consider what happened during the recent global financial crisis. In the years leading up to the 2008 meltdown, our financial system had become so highly levered that a single lax assumption in the credit ratings agencies’ models played a huge role in bringing down the whole global financial system. Specifically, credit rating agencies like Standard & Poor’s and Moody’s made the mistake of assigning AAA blue-chip credit ratings to bundled packages of residential mortgage debt, which were then sold to investors by the big Wall Street brokerage firms. These bundled Collateralized Debt Obligations, or CDOs, represented a fairly novel financial innovation that was not well understood. The ratings firms may have thought they had a pretty good handle on the risks associated with mortgage-backed CDOs. But according to Silver, the ratings agencies’ were clearly unable to appreciate the distinction between risk and uncertainty, which ties back to two of the four most common prediction failures outlined above. The concepts of risk and uncertainty can be tricky ones to grasp. Risk, as first articulated by the economist Frank H. Knight in 1921, is something that you can put a price on. “Imagine that you’ll win a poker hand unless your opponent draws to an inside straight,” explains Silver. “In this case the chances of that happening are exactly 1 chance in 11.” Risk is inherently measurable. Needless to say, it’s not a pleasant feeling when you take a “bad beat” in poker. But at least you know the odds of it and can account for it ahead of time. Uncertainty, on the other hand, is risk that is hard, or impossible, to measure. “You might have some vague awareness of the demons lurking out there,” writes Silver. “And you might even be acutely concerned about them. But you have no real idea how many of them there are or when they might strike.” Consequently, any “back-of-the-envelope” estimate you come up with might well be off by a factor of 100, or even 1,000. Really, there is just no good way to know.
3
TheBusinessSource.com
All Rights Reserved
The Signal and the Noise
So, if uncertainty is all around us, and if it defies any type of measurement, how can we account for the problem of uncertainty in our predictions? By applying the following three principles:
Three Principles for Making Better Predictions Principle 1: Think Probabilistically Silver is very careful to articulate a range of possible outcomes and assign a probability to each one. For example, if you play enough poker hands, chances are you’ll get beaten badly by someone who’s holding a royal flush when you only have a full house. Now, the chances of your opponent catching a royal flush and beating your full house are very small – let’s say less than one percent. So you probably don’t have to spend a lot of time worrying about this potential outcome. The chances of it happening are very small. But they’re not exactly zero, and this is Silver’s point. In his view, far too many forecasters fail to account for the possibility of low-probability events occurring. The importance of communicating uncertainty by speaking in probabilistic (as opposed to dead certain) language is particularly acute when public safety may be at risk. To illustrate this point, Silver reminds us of the devastating 1997 Red River flood in Grand Forks, North Dakota. Nearly 50,000 residents had to be evacuated, clean-up costs ran into the billions, and 75% of the city’s homes were damaged or destroyed. Silver argues that Grand Forks may have been a preventable disaster had the city’s floodwalls been properly reinforced using sandbags or other means. The residents of Grand Forks had been aware of the flood threat for months. Snowfall had been especially heavy in the Great Plains that winter, and the National Weather Service, anticipating runoff as the snow melted, had predicted the waters of the Red River would crest to forty-nine feet, close to the all-time record. But no one was panicking as the snow started melting, because the levees in Grand Forks had been built to handle a flood of fifty-one feet. So the residents felt safe. The problem was, even a small miss in the forty-nine-foot prediction would prove catastrophic to the whole town. And in fact, the river ended up cresting to fifty-four feet. The margin of error on the Weather Service’s forecast – based on how well their forecasts had done in the past – was about plus or minus nine feet. That implied about a 35 percent chance of the levees being overtopped. But the Weather Service had explicitly avoided communicating the uncertainty in their forecast to the public, emphasizing only the forty-nine-foot prediction. The weather forecasters later told researchers that they were afraid the public might lose confidence in the forecast if they had conveyed any uncertainty in the outlook. Instead, it would have made the public much better prepared; and possibly able to prevent the flooding by reinforcing the levees or diverting the river flow. Left to their own devices, many residents became convinced they didn’t have anything to worry about. (Very few of them bought flood insurance.) A prediction of a forty-nine-foot crest in the river, expressed without any
4
TheBusinessSource.com
All Rights Reserved
The Signal and the Noise
reservation, seemed to imply that the flood would hit forty-nine feet exactly; the fifty-one-foot levees would be just enough to keep them safe. Some residents even interpreted the forecast of forty-nine feet as representing the maximum possible extent of the flood. Fortunately, the National Weather Service has since come to recognize the importance of communicating the uncertainty in their forecasts accurately and honestly to the public. But according to Silver, this sort of attitude is still relatively rare among other kinds of forecasters.
Principle 2: Today’s Forecast Is the First Forecast of the Rest of Your Life Predictions should never be carved in stone because the world is a dynamic, ever-changing place. This is especially true when we’re speaking about the economy. The global economy, much like the global atmosphere, is a dynamic system. Everything affects everything else and the systems are perpetually in motion. In meteorology, this problem is quite literal; since the weather is subject to “chaos theory” (e.g. a butterfly flapping its wings in Brazil can theoretically cause a tornado in Texas). But in loosely the same way, “A tsunami in Japan or a longshoreman’s strike in Long Beach can affect whether someone in Texas finds a job.” For Silver, the best way to deal with a constantly changing environment is to adopt a humble attitude. Make the best forecast possible today – regardless of what you said last week, last month, or last year. Making a new forecast doesn’t mean that the old one just disappears. (Ideally, you should keep a record of it and let people evaluate how well you did over the whole course of predicting an event.) But if you have reason to think that yesterday’s forecast was wrong, there is no glory in sticking to it. “When the facts change, I change my mind,” the economist John Maynard Keynes famously quipped.
Principle 3: Look for Consensus According to Silver, every person who makes their living by predicting outcomes secretly fantasizes that they will become famous one day by making a daring, audacious, outside-thebox prediction – one that differs radically from the consensus view on a subject. In their fantasy, they imagine themselves on the front page of the Wall Street Journal, or maybe even sitting on Jay Leno’s couch, singled-out as a bold and brave pioneer in their chosen field. “Every now and then, it might be correct to make a bold forecast,” writes Silver. The expert consensus can be wrong. For example, someone who had forecasted the collapse of the Soviet Union would have deserved most of the kudos that came to him. But the fantasy scenario is hugely unlikely, and plenty of historical evidence suggests that aggregate or group forecasts are more accurate than individual ones. In fact, consensusbased forecasts are typically somewhere between 15 and 20 percent more accurate,
5
TheBusinessSource.com
All Rights Reserved
The Signal and the Noise
depending on the discipline.
Why Context Matters “Data is useless without context,” says Nate Silver. With the exponential growth in the availability of information and computing technologies, we can now measure millions of potentially interesting variables. “For instance, the U.S. government now publishes data on about 45,000 economic statistics. If you want to test for relationships between all combinations of two pairs of these statistics (e.g. is there a causal relationship between the bank prime loan rate and the unemployment rate in Alabama?) that gives you literally one billion hypotheses to test.” Clearly, the number of meaningful relationships in all of this economic data – i.e. those that speak to causality rather than mere correlation – is orders of magnitude smaller. Or, to put it differently, most of the economic and other data we collect and disseminate is just noise. In order to cut through all the clutter and find the “signal” in the noise, we need context. To explain this point, Silver tells the story of the famous Vegas sports bettor Haralabos “Bob” Voulgaris. In a bad year, Voulgaris probably makes about a million dollars profit, give or take. In a good year, he might make three or four times that. “Voulgaris doesn’t fit the stereotype of the cigar-chomping gambler in a leisure suit. He does not depend on crooked insider tips, or other sorts of hustles to make his bets. Nor does he have a ‘system’ of any kind. He uses computer simulations, but does not rely upon them exclusively.” Instead of looking for more data, Voulgaris spends much of his waking life looking for more “context.” During basketball season, Voulgaris spends every single night from November through June watching the NBA. He has five games on at a time, on five Samsung flat screens. What makes Voulgaris so successful is the way that he analyzes information. He isn’t just hunting for patterns. Instead, he combines his knowledge of statistics with his knowledge of basketball in order to identify meaningful relationships in the data. In short, Voulgaris’ big secret is that he doesn’t have a big secret. Instead, he has a thousand little secrets. He watches virtually every NBA game – some live, some on tape – and develops his own opinions about which teams are playing up to their talent and which aren’t. He looks for subtleties. For example, he knows which star players may be coming up on free agency, and thus may be more motivated to go out and score a whole bunch of points, versus those that just signed big, fat contracts and consequently may feel like they have nothing to prove. And for all his knowledge of basketball, and all his “hard work” watching so many games every night, he still gets only about 57 percent of his bets right. It is just exceptionally difficult to do much better than that. Nevertheless, Voulgaris is easily among the top 10 sports bettors in the world. His predictions have made him wealthy because he understands the power of context.
6
TheBusinessSource.com
All Rights Reserved
The Signal and the Noise
A Word on “Bayesian” Forecasting If we aspire to become better forecasters, Silver urges us to become “Bayesians.” The word Bayesian is derived from the Eighteenth Century English mathematician, Thomas Bayes, who created an elegant mathematical rule to account for two or more interrelated variables in the calculation of probabilistic outcomes. In most Bayesian calculations, at least one of those variables is based on a solid, irrefutable fact, and the other is based on a more subjective personal belief (which involves some degree of uncertainty). Since we’d probably all prefer not to have to break out our slide rules at this point, we’ll try to illustrate Bayes Theorem by using a simple, everyday example … Suppose a co-worker told you they had a really nice conversation with someone on an airplane this past weekend. Not knowing anything else about this conversation, the probability that your colleague was speaking with a woman is 50% (based on the fact that roughly 50% of the general population is female). Now suppose your co-worker also told you that this person on the plane had really long hair. Based on your experience, it now seems more likely they were speaking to a woman, since subjectively you believe that most longhaired people are women. Bayes’ theorem can then be used to calculate the probability that the person is female. 18:44 To see how this is done in practice, for the purposes of the mathematical equation for determining probability (“P”) that follows below, we’ll let the letter “W” represent the chance that the conversation on the plane was held with a woman, and the letter “L” denote the chance that it was held with a long-haired person. To populate the equation, you’ll search your mind for relevant facts. As you know, women constitute roughly half the population. So, if you knew nothing else about the conversation in question, it’s safe to say that the probability that the person on the plane was a woman is 50%. But you do have one other bit of relevant information – the length of the person’s hair. Suppose you believe, subjectively, that 75% of women have long hair. You might then say that the probability of the person on the plane being female is 75%. You’d have no hard evidence at your disposal to support this belief, but since, in your experience, you’re far more accustomed to meeting women with long hair as opposed to men, you feel pretty comfortable with your 75% “guesstimate.” Since your end goal is to calculate the probability that the conversation was held with a woman, based on these two sets of variables, you would then plug the numbers into the equation for Bayes Theorem, which reads as follows:
When you go ahead and run the numbers, you’ll find that the probability that the conversation was held with a woman – given that the person also had long hair – is approximately 70%.
7
TheBusinessSource.com
All Rights Reserved
The Signal and the Noise
This result probably shouldn’t be surprising because 70% is essentially a middle-ground number between the factual 50-50 population split between men and women, and your more subjective belief that 75% of people with long hair are women. It seems like a pretty credible prediction. In addition to its obvious mathematical potency, the real beauty of Bayes Theorem from a forecasting point of view is the requirement to explicitly state one’s subjective assumptions (which can be quite revealing about the inner thinking of the forecaster herself). Silver knows that we humans all have natural biases which can be very hard, if not impossible, to overcome. And so, perhaps the best we can do is be upfront about our biases so that others can make their own determinations as to whether or not our predictions are sound.
Beating the Competition Naturally, we all aspire to be perfect in our predictions. Nobody enjoys getting things wrong and looking silly in front of their friends, clients or colleagues. But since no one is actually perfect, the true test of how good you are as a forecaster is your track-record relative to the competition. In other words, it’s less important to see how accurate your predictions are in an absolute sense, but rather how good they are relative to other people’s predictions. Nowhere is this principle more true than in online Texas Hold’em poker. In Texas Hold’em, you can make 95 percent of your decisions correctly and still lose your shirt at a table full of players who are making the right move 99 percent of the time. Likewise, beating the stock market requires out-maneuvering teams of well-paid investors with MBAs and state-of-the-art computer systems at their disposal. There’s a learning curve that applies to poker and stock-picking and any other competitive field that involves some type of prediction. The key thing about a learning curve is that it really is a curve: the progress we make performing the task is not linear. The name for this curve comes from the well-known business maxim called the Pareto Principle or 80-20 rule (as in: 80 percent of your profits come from 20 percent of your customers). As Silver applies this rule in the context of making relatively good predictions, it posits that getting a few basic things right can go a long way. In poker, for instance, simply learning to consistently fold your worst hands, bet your best ones, and consider what your opponent holds will substantially improve your game. If you’re willing to do this, then perhaps 80 percent of the time you’ll be making the same good decisions as the very best poker players in the game. In most fields, if your predictions are accurate 80% of the time, then you’ll probably be doing well enough to hold onto your job and maybe even outshine much of your competition. But in some fields – and online poker is surely one of them – it can require a lot of extra effort to beat the competition. You may work hard to improve your game and come to be accurate with your predictions 80% of the time, but you still won’t be beating your competition, which is very intense. So at that point, you’d be left with essentially two choices: You could continue to work really hard to improve your game, or you could drop out and find a different form of
8
TheBusinessSource.com
All Rights Reserved
The Signal and the Noise
difference in your outcomes. You’ll have to work twice as hard to reach 85% proficiency as you did to reach 80% proficiency in the first instance, and you may still find yourself losing money. So consider Silver’s advice: If you have strong analytical skills (and good context knowledge) that might be applicable in a number of other disciplines, it is very much worth considering the strength of your competition. “It is often possible to make a profit by being pretty good at prediction in fields where the competition often succumbs to bad habits, or blind adherence to tradition,” he writes.
Conclusion In North America, we live in a very results-oriented society. If someone is rich or famous or beautiful, we tend to assume they somehow deserve to be those things. As an empirical matter, however, success is determined by some combination of hard work, natural talent, and a person’s environment – in other words, some combination of noise and signal. In North America, we tend to emphasize the signal component most of the time – we’re too quick to point to some sort of casual relationship between a person’s talent, or hard work and their success. In some cases, they just happened to be in the right place at the right time. When it comes to assessing other people’s predictions, we’re similarly results-oriented. “The investor who called the stock market bottom must be a genius,” writes Silver, “even if he had some buggy statistical model that just happened to get it right.” Truly, the guy just got lucky. The solution here is to focus more on the prediction process than on its results. This can be hard to do, but it pays huge dividends. Consider again the example of competitive poker players who tend to understand that process trumps results, if only because they experience the ups-and-downs in such an immediate, visceral way. If you play a lot of poker, you know that sometimes you’ll play well and win. But you’ll also play well and lose. And you’ll play badly and lose, and even sometimes play badly and win. Every serious poker player has experienced each of these conditions so many times that they know there is a difference between process and results. But playing well, regardless of the short-term outcomes, is what will make you profitable over the long-term. For example, if you correctly detect an opponent’s bluff and call him on it with the best hand, but he gets a lucky card “on the river” and wins the hand anyway, Silver says you should be actually pleased with yourself rather than angry, because you played as well as you could. The main thing is to be disciplined about the process you take to prediction; and to also be humble about your chances of being right. The results you seek will surely follow.
9
TheBusinessSource.com
All Rights Reserved
