Nate Silver is the new hero of the liberal left in the US. This mathematician and statistician correctly forecast Obama’s victory in the presidential election and in the Senate and the result for the electoral college in all 50 states. On the morning of the 6 November 2012, the final update of Silver’s model gave President Barack Obama a 90.9% chance of winning a majority of the 538 electoral votes. Both in summary tables and in an electoral map, Silver forecast the winner of each state. Silver’s model correctly predicted the winner of every one of the 50 states. In contrast, individual pollsters were less successful. For example, Rasmussen Reports, widely quoted by the right-wing “missed on six of its nine swing-state polls”.
Silver has now published a new book that is already a best seller and he now regularly appears on TV talk shows. Silver brilliantly exposed the biased commentaries of the right-wing TV channels and papers whose pundits regularly appeared on screen or in print to say that they ‘had a hunch’ that Romney would win or that the polls were ‘biased’ against the Republican candidates. Silver, in the meantime, quietly presented a statistical analysis of the polls and concluded the probability of Obama winning was over 80% and rising. His forecast was dead right. On November 12th, his new book, The Signal and the Noise (print edition) was named Amazon’s Best Book of the Year for 2012.
The evidence is that statistical analysis is way better at forecasting things than ‘hunches’ or human intuition. Indeed, out of the one hundred studies comparing the accuracy of actuarial statistics (probability analysis) and intuition, there has not been one case humans doing better (Stuart Sutherland, Irrationality, p200). Indeed, in most studies, actuarial analysis was way better. Take bank loans, nowadays 90% of loan applications are reviewed by computers taking into account client details against aggregate evidence on bank accounts, jobs etc to gauge risk. Loans granted by computer using statistical probabilities turn out to have far less defaults than those borrowers chosen by bankers on their own judgement. Insurance companies have applied to risk in life expectancy and accidents for many years. So when somebody tells you that their intuition delivers better results, they are talking out of their hats. Why would you not choose statistical methods to raise your chances of getting things right even if nothing is 100% certain?
Take the stock market. We are continually told in investment adverts by expensive investment advisers that they can make your money work for you more than just tracking a stock index, like the S&P-500. In other words, they can beat the market. But a host of statistical studies prove the opposite. Sure, some advisers can do better than the index for a few years, but eventually, they all come a cropper. It’s just so much snake oil voodoo investing.
But everything is not entirely random. If you were to read Nicholas Taleb’s book, Black Swan (see my book, The Great Recession, chapter 31), you would think that it was. Or to be more exact, even the most unlikely can happen under the law of chance. It was assumed that there were only white swans until Europeans got to Australia and found black ones. It was the ‘unknown unknowns’, to quote Bush’s neo-con Secretary of State, Donald Rumsfeld. The most unlikely can happen but you cannot know everything. For Taleb, the Great Recession was one such event that could not have been predicted and therefore bankers, politicians and above all, economists are not at fault. This was the excuse used by bankers when giving evidence to the US Congress and to the UK parliament.
But modern statistical methods do have predictive power – all is not random. In his book, Silver offers detailed case studies from baseball, elections, climate change, the financial crash, poker and weather forecasting. Using as much data as possible, statistical techniques can provide degrees of probability, like “the probability of Obama winning the electoral college is 83% and the probability of him winning the popular vote is 50.1%”. This is different from much statistical method in colleges and universities today that rely on idealized modelling assumptions that rarely hold true. Often such models reduce complex questions to overly simple “hypothesis tests” using arbitrary “significance levels” to “accept or reject” a single parameter value. In contrast, the practical statistician needs a sound understanding of how baseball, poker, elections or other uncertain processes work, what measures are reliable and which not, what scales of aggregation are useful, and then to utilize the statistical tool kit as well as possible. You need extensive data sets, preferably collected over long periods of time, from which one can then use statistical techniques to incrementally change probabilities up or down relative to prior data.
This is the modern form of what is called the Bayesian approach, named after the 18th century minister Thomas Bayes who discovered a simple formula for updating probabilities using new data. The essence of the Bayesian approach is to provide a mathematical rule explaining how you should change your existing beliefs in the light of new evidence. In other words, it allows scientists to combine new data with their existing knowledge or expertise.
What constitutes Bayes approach that led to Nate Silver’s accurate forecasts? Let me try and explain as best I can, using the help of examples provided by Eliezer Yudkowsky in his excellent blog (http://yudkowsky.net/).
Suppose it is an established fact through other studies that 1% of women at age forty who participate in routine screening have breast cancer. Second, 80% of women with breast cancer will get positive mammographies. But 9.6% of women without breast cancer will also get positive mammographies. A woman in this age group has a positive mammography in a routine screening. What is the probability that she actually has breast cancer? The correct answer is 7.8%, obtained as follows: out of 10,000 women, 100 have breast cancer; 80 of those 100 have positive mammographies. From the same 10,000 women, 9,900 will not have breast cancer and of those 9,900 women, 950 will also get positive mammographies. This makes the total number of women with positive mammographies 950+80 or 1,030. Of those 1,030 women with positive mammographies, 80 will have cancer. Expressed as a proportion, this is 80/1,030 or 0.07767 or 7.8%. So the answer is not 1% who do have cancer or the 80% with a positive mammo.
The original proportion of patients with breast cancer is known as the prior probability. The chance that a patient with breast cancer gets a positive mammography and the chance that a patient without breast cancer gets a positive mammography are known as the two conditional probabilities. Collectively, this initial information is known as the priors. The final answer – the estimated probability that a patient has breast cancer, given that we know she has a positive result on her mammography – is known as the revised probability or the posterior probability. The mammography doesn’t increase the probability that a positive-testing woman has breast cancer by increasing the number of women with breast cancer – of course not; if mammography increased the number of women with breast cancer, no one would ever take the test! However, requiring a positive mammography is a membership test that eliminates many more women without breast cancer than women with cancer. The number of women without breast cancer diminishes by a factor of more than ten, from 9,900 to 950, while the number of women with breast cancer is diminished only from 100 to 80. Thus, the proportion of 80 within 1,030 is much larger than the proportion of 100 within 10,000. The evidence of the positive mammography slides the prior probability of 1% to the posterior probability of 7.8%.
Actually, priors are true or false just like the final answer – they reflect reality and can be judged by comparing them against reality. For example, if you think that 920 out of 10,000 women in a sample have breast cancer and the actual number is 100 out of 10,000, then your priors are wrong. In this case, the priors might have been established by three studies – a study on the case histories of women with breast cancer to see how many of them tested positive on a mammography, a study on women without breast cancer to see how many of them test positive on a mammography, and an epidemiological study on the prevalence of breast cancer in some specific demographic.
Let’s say you’re a woman who’s just undergone a mammography. Previously, you figured that you had a very small chance of having breast cancer; we’ll suppose that you read the statistics somewhere and so you know the chance is 1%. When the positive mammography comes in, your estimated chance should now shift to 7.8%. There is no room to say something like, “Oh, well, a positive mammography isn’t definite evidence, some healthy women get positive mammographies too. I don’t want to despair too early, and I’m not going to revise my probability until more evidence comes in. Why? Because I’m an optimist.” And there is similarly no room for saying, “Well, a positive mammography may not be definite evidence, but I’m going to assume the worst until I find otherwise. Why? Because I’m a pessimist.” Your revised probability should go to 7.8%, no more, no less.
What’s so great about Bayes’ theorem is that it can be used for reasoning about the physical universe. But I think Bayes law also shows two other things that are useful to remember in economic analysis. The first is the power of data or facts over theory and models. Neoclassical mainstream economics is not just voodoo economics because it is ideologically biased, an apology for the capitalist mode of production. But in making assumptions about individual consumer behaviour, about the inherent equilibrium of capitalist production etc, it is also based on theoretical models that bear no relation to reality: the known facts or priors. In contrast, a scientific approach would aim to test theory against the evidence on a continual basis, not just to falsify it (as Karl Popper would have it) but also to strengthen its explanatory power – unless a better explanation of the facts comes along. Newton’s theory of gravity explained very much about the universe and was tested by the evidence, but then Einstein’s theory of relativity came along and better explained the facts (or widened our understanding to things that could not be explained by Newton’s laws). In this sense, Marxist method is also scientific. Marx went from the abstract (theory) to the concrete (facts). The facts would strengthen the explanatory power of the theory or modify it.
This approach using statistical methods like Bayes law is what mainstream economics does not do. Here is what Dan Kervick said in his blog recently (http://neweconomicperspectives.org/2012/09/shamanistic-economics.html), in a brilliant post on mainstream economics:
You guys in economics are supposed to be empirical scientists, not philosophers. You are supposed to develop the a priori elements of your science only so that you can produce empirically testable models of the real world, and then bring those models to bear on the world we actually live in. You are also supposed to help develop techniques that are relevant to decision-making and government policy in having predictable outcomes. You need to map the terrain of the actual world in detail, so you can help others navigate through it. To the extent you want to give policy advice that deserves to be taken seriously, your focus needs to be on contingent reality, not a priori possibility.
My criticism is that an awful lot of the policy advice we are getting lately is from theorists who are lost in the clouds of a priori models, and who don’t have a clear understanding of the structure of the actual economic order we live in, based on the functioning of actual, highly contingent and specific economic and political institutions.
If you are trying to navigate your way through a mountain range, you don’t ask a geologist; you ask a guide who has explored the mountain range in detail. If the guide has geological knowledge that can definitely help, but the geological knowledge itself is not sufficient to guide people through the terrain. If you want to fix a broken airplane engine, you don’t ask a theoretical thermodynamicist, you ask an engineer. The engineer’s knowledge of thermodynamics can help, but the thermodynamical knowledge itself is not sufficient to know how to fix an airplane engine.
It is not enough for you to describe logically coherent possible worlds with possible sets of beliefs about possible equilibria and possible time paths to those equilibria, where possible statements, and possible actions have possible effects as a result. You need to show we live in such a world – and this is a task for which you don’t seem to have much patience. When challenged on the score of institutional facts, you have repeatedly retreated back into the construction of other models and thought experiments.
The second thing we can glean from the use of Bayes law and Nate Silver’s results is the power of the aggregate. The best economic theory and explanation comes from looking at the aggregate, the average and its outliers. Data based on a few studies or data points provide no explanatory power. That may sound obvious but it seems that many political pundits were prepared to forecast the result of the US election based on virtually no aggregated evidence. It’s the same with much of economic forecasting. Sure, what happened in the past is no certain guide to what may happen in the future, but aggregated evidence over time is a hell of sight better than ignoring history.
9 thoughts on “Bayes law, Nate Silver and voodoo economics”
Time for you to run us through the pitfalls of Capital 2 and equilibrium, Mike 😉
As for Nate and Bayes, I went hunting online and found the following interesting stuff for diving into probability and statistics:
Al Drake’s Book
(Part of the materials for a free online MIT course in probability and statistics), and
a clear graphical explanation of the Bayes theorem
My favourite is still MJ Moroney’s Pelican ‘Facts From Figures’ (3rd ed) 1960, five bob, but then that’s me…
I might add that when I was working as a teacher I found the statistics of standard deviation and the bell curve invaluable to me as a grading tool. Working in Sweden at the time made it very easy to rank the kids by performance in relation to one another and to all the kids of their age in the country. “Your marks place you here, Jimmy’s marks place him here. He’s in a clearly different group from you – this big group between mark X and mark Y, see That’s why he gets a C and you get a D. Janey is right on the borderline between you, so we’ve got to take a long, hard look at her work before we can sort her out.” But it can still help even in counter-intuitive shit systems like the English, though more for your own peace of mind than anything else.
Once statistics (ie scientific investigation of numerical (quantified) behaviour patterns) is freed from bourgeois enchainment it’ll come into its own in all kinds of areas – economics and education, of course, but also everyday life when we start using it for fun. Numerate people having fun with numbers will be almost as entertaining as sexually enlightened people having fun with sex 😉
But we’re not there yet, to put it mildly.
The deliberate, malevolent obscurantism of present-day bourgeois economists is very similar to the irrationality propagated by the geo-centric astronomers of the Middle Ages and Counter-Reformation. They were backed up by excommunication and the torture chambers of the Inquisition, our crowd are backed up by (academic and professional) excommunication and whatever degree of physical/mental injury, they can get away with.
As you stress, Bayesian statistical methods are helpful when lots of data are available – which leads to a precaution. Looking for plentiful, comparable data points should not narrow your vision. How often is a mode of production overthrown by revolution? Statistical evidence can help show that a revolution is historically imminent, not so much by comparing disparate revolutions as by establishing the current motion of the mode of production. A leap out of everyday events needs to be considered.
Probability and Bayes’ theorem also play an important role in ecology and population biology. I think there are many insights from these disciplines that would be useful to understand social issues.
You may already be familiar with this literature. I think that one of the most insightful discussions about the role of modeling in scientific understanding can be found in an old paper by Richard Levins, “The strategy of model building in population biology” (American Scientist 1966). It can easily be found online along with interesting commentaries, some of which also deal with Bayes’ theorem.
Levins starts from what he sees as the three main characteristics of a model – generality, realism, precision – and maintains that you cannot have all three at the same time. What you sacrifice depends on the task you are trying to accomplish. Levins also developed the loop analysis approach for transforming qualitative models into quantitative ones and, as any good Marxist, also applied this approach to social issues (agriculture, health).
a new documentary
As far as I can tell Silver more or less averaged* a bunch of polls and is now a ‘genius’. His results were basically the same a RCP literal averaging.
He basically exploited others’ work – the work of various survey companies – and made a name for himself. His triumph belongs more in Marketing than Mathematics. Truly an American triumph.
* yes, not exactly averaged, but the math is not complex at all
I’m a bit confused by this. As a Marxist and a student of (mainstream) economics, pretty much every mainstream journal article I read starts off (mostly micro ones so far, admittedly) with theory then moves to empirics by trying to fit data to the best model possible. What exactly is it that they are doing wrong? Or are you mainly aiming criticism at macroeconomists?
Also, in my experience, both Marxian and Austrian economists seem to use more theory and less stats in their articles than mainstream economists.
Or have I misunderstood? PS thanks for the always-interesting blog: it’s been a big influence on me as I fight my way through neoclassical paradigms…
This is a big subject but it seems that mainstream economics generally prefers models over stats. And when their models dont fit the facts, they dont tend to drop them (for ideological reasons) – like economies tend to equilibrium or markets always clear or there are no credit bubbles except those artificially created by central banks or through unexpected ‘shocks’ etc. I agree that Marxist and Austrian economists also tend to avoid evidence too! We need the scientific method: theory to evidence and back to modifying theory or developing new ones that better fit the facts. As you can see from my blog, I want evidence and stats. Marx did too.
The problem in “predicting” The Great Recession (and some did, in fact) is that it was caused by changing the economic platform, in medias res. Specifically, the increase in the ratio (median house / median income) guaranteed failure. Normal macro models don’t look at data outside the time series of interest. The assumption is that all relevant information is embedded in the data stream. MCMC is then used to gin up new numbers, assuming the platform is stable.
Unless one used economic “intuition”, one would not bother to look beyond the macro variables of interest. The field, after all, is truly Political Economics; the politics was what caused The Great Recession. And the signal event was Greenspan’s crashing of interest rates in 2002. That event propelled Coupon Clipping Money out of Treasuries into Something Else Risk Free. Of course, there is no such instrument, but, historically, US housing was nearly so.
In order to satisfy the demand for such instruments, and to provide “wealth” to mortgage holders with stagnant wages, residential (and commercial, but less so) mortgages had to be manufactured. Existing protocols couldn’t provide sufficient volume, so protocols were eased. And so it was.
In the end: The Great Recession is the product of increasing wealth concentration. The (unearned) property appreciation provided an income stream to tapped out consumers, whose wages were, at best, stagnant. Without this artifice, Dubya would have no chance of re-election; the distributional problem couldn’t be avoided.
So, in audio circuit terms, it was a push-pull amplifier: the push of crashed interest rates demanding an alternate “risk free” instrument (in volume) and the pull of income starved wage earners looking for supplemental moolah.
And so it was. Until it wasn’t.
This article lauds statistics, downplaying intuition. Yet intuition is the basis of Bayes’ Theorem in determining the prior probability. Bayesian methods are derived assuming rationality, much like neoclassical microeconomics. The methods of neoclassical economics and Bayes are very similar, assuming rational utility and profit maximizing agents, with intuitions of theory built in. If microeconomics is biased pseudo science, so is Bayesian analysis, for the same reasons. Both just use cherry picked data to back up their largely preconceived intuitions. Nate Silver got 50 states right in the 2012 election. That’s misleading, as everyone else also got every state right, except the closest swing states which were just toss ups in many cases. Many people could flip coins and get those close swing states right. Biased intuition formed the Silver prior probability in Bayes Theorem, just as it did in right wing predictions, whether they used Bayes or not. Also, political predictions can be self-fulfilling, as money donors watch polls and statistical predictions, moving more money to perceived winners, to gain influence. Silver’s prediction, seemingly based on statistics, when published, helped move some money towards Obama. If we wanted a fair unbiased election, we should ban publishing polls and poll results, much as we ban betting on elections. Exit polls are useful to determine if the official results were rigged. We do that in foreign countries, but when exit polls showed Bush stole the election from Kerry, for some reason few noticed.