Ditch the 'Experts'
Grading pundits and prognosticators: More famous = less accurate.
By Geoffrey Colvin

(FORTUNE Magazine) - You have been a world-class sap for years. Why? For listening to the economic and political forecasts of experts. We in the media have been irresponsible fools for reporting those forecasts. And the experts themselves? Delusional egomaniacs--and maybe even con artists.

I didn't always think this way. But I've been reading a book that marshals powerful evidence to make this case. For all of us in the world of business, economics, and capital markets--a world that often turns on the judgments of experts--the question is whether we're brave enough to face these uncomfortable facts.

The book is Expert Political Judgment: How Good Is It? How Can We Know? by Philip E. Tetlock, a professor at the University of California at Berkeley. It summarizes the results of a truly amazing research project: Over seven years Tetlock got a wide range of experts and nonexperts to answer carefully constructed questions about the likelihood of specific future events. He ended up with a staggering 82,361 forecasts, expressed in quantifiable form and thus able to be analyzed deeply. His definition of "political judgment" included plenty of topics that you and I would call economic, such as government spending and national economic performance.

Tetlock then cranked all those numbers through every kind of statistical thresher, flail, and grinder you can imagine, and the result was clear: Experts don't actually exist. Specifically, experts were no better than nonexperts at predicting the future. They weren't even as good as computer programs that merely extrapolate the past. The best experts could not explain more than 20% of the variability in outcomes, but crude algorithms could explain 25% to 30%, and sophisticated algorithms could explain 47%. Consider what this means. On all sorts of questions you care about--Where will the Dow be in two years? Will the federal deficit balloon as baby-boomers retire?--your judgment is as good as the experts'. Not almost as good. Every bit as good.

Which is not to say that experts are no different from you and me. They're very different. For example, they're much more confident in their predictions than nonexperts are, though they obviously have no reason to be. For example, the members of the American Political Science Association predicted in August 2000 that a Gore victory was a slam dunk.

Experts can also give far more reasons for their predictions than nonexperts can. Their vast erudition lets them explain at daunting length why something will or won't happen. Not that all those reasons make the forecasts one bit better.

The question that screams out from the data is why the world keeps believing that "experts" exist at all. In large part, the answer is human nature. We desperately want to believe the world is not just a big game of dice, that things happen for good reasons and wise people can figure it all out. It may not be so; a school of researchers known as radical skeptics presents impressive evidence that the world is totally random, or at least that we humans are eternally unable to figure it out. But most of us can't bear to believe that, so we cling to the notion of experts.

Another part of the answer is especially troubling for the media. The awfulness of Tetlock's experts was almost uniform whether they had doctorates or bachelor's degrees, lots of experience or little, access to classified data or none. He found but one consistent differentiator: fame. The more famous the experts, the worse they performed. And of course it's those of us in TV, radio, newspapers, magazines, and on the web who bestow that fame. We're now reaching a deeper answer to the question of why experts persist. The media like experts, the more confident the better, though Tetlock found that more confident means less reliable. The media like them because you, the consumers of media, like them. Experts like to appear in the media because it pays. Tetlock's conclusion: "The three principals--authoritative-sounding experts, the ratings-conscious media, and the attentive public--may thus be locked in a symbiotic triangle."

It's never too late for a good New Year's resolution, so I propose that in the coming year, whenever any of us read or hear an expert opinion, or publish one, or issue one, we ask ourselves whether Tetlock might be onto something with this quietly withering conclusion about all of us in that triangle: "It is tempting to say they need each other too much to terminate a relationship merely because it is based on an illusion."

FEEDBACK gcolvin@fortunemail.com Top of page