Why you should ignore market forecasts.
Answer the ten questions below by writing down the range within which you are 90% confident the correct answer lies. You need a lower limit, and a higher limit. And don't scroll too far down the page or you'll see the answers!
Here goes:
- Martin Luther King's age at death?
- Length of the Nile in miles?
- Number of countries in OPEC?
- Number of books in the Old Testament?
- Diameter of the Moon in miles?
- Weight of an empty Boeing 747 in pounds?
- Year of Wolfgang Amadeus Mozart's birth?
- Gestation period in days of an Asian elephant?
- Air distance from London to Tokyo?
- Deepest (known) point of the ocean in feet?
(Source: Russo and Schoemaker 1989)
Got your confidence intervals written down? If you actually know all of these facts then your general knowledge is freakishly good. You'll miss the point of the test, though. For everyone who isn't a Mastermind contestant, you should now have an upper and lower estimate for each number.
Here are the answers. You score a point if the correct number lies within the boundaries of your estimates:
- 39 years
- 4,187 miles
- 13 countries
- 39 books
- 2,160 miles
- 390,000lbs
- 1,756
- 645 days
- 5,959 miles
- 36,198 ft
This isn't about testing how many random facts you know; it's about what statisticians call your 'level of calibration'. In other words, do you know how much you don't know?
Being 'Well Calibrated'
If you scored nine out of ten then well done: you are a 'well calibrated' estimator. Faced with uncertainty, you are aware of the true range of possible answers and you allow plenty of margin for error in your guesses. Very few people are 'well calibrated'; the typical score for this kind of test is between four and seven.
Most people set quite narrow intervals, suggesting that they are far too confident in their ability to make estimates. And this is where tests like this become relevant to investors: not only will you spend a lot of time reading other people's predictions, but you might also make a few of your own.
Doctors and Weathermen
The record of professionals from all disciplines who 'predict' for a living is mixed. Weathermen do surprisingly well (Michael Fish in 1987 excepted), but doctors are terrifyingly poor.
A 1991 study gave both groups some data relevant to their own work: the weathermen received weather patterns and were asked to make a forecast; the doctors were presented with case notes and asked to make a diagnosis. When a weatherman was 90% sure of his prediction, he was right about 80% of the time; the doctors scored only 15%.
Is this because doctors are all frauds and meteorologists are not? Hardly.
The weather is known to follow predictable, consistent patterns that respond well to mathematical modeling. Short-term forecasts are extremely accurate.
The human body is a less predictable beast; the doctors in this study were asked to interpret subjective case notes written by an unknown person, and described patients they had never met. It isn't surprising that their forecast accuracy was lower.
But why were the doctors so poorly calibrated? Surely their extensive training should have taught them that 90% certainty is difficult to achieve? It's ok to be wrong again and again when dealing with the unpredictable; it is less forgivable to be so certain of your incorrect conclusion.
Calling the Market
Is a stock market forecaster more like a weatherman, or a doctor? Some extremely smart people spend a lot of time trying to model the market using maths to spot exploitable patterns; the industry would probably like us to view them as closer to weathermen.
The truth, however, is that market forecasters are even worse than doctors: in tests, they display huge self-confidence and dreadful accuracy.
Self Interest
The business of making stock market forecasts is also infected by self interest. Many pundits are in the pay of a fund management firm, or other business, with an interest in maintaining a sense of optimism. These forecasters tend to give positive, but modest, predictions to the public regardless of their true feelings. This makes sense from their point of view: they will never risk their reputation by calling a large move up or down. But it also goes against the interests of their employer to make a gently pessimistic forecast like "the FTSE will fall by around 8% this year"; that's an open invitation not to invest just now.
Phrases like "low double-digit returns" are a typical market forecast. This maintains enough optimism to encourage new investors, but doesn't expose the speaker to serious ridicule if the market does something else. It's much easier on one's credibility to fail to predict something dramatic than it is to warn of 1929 Mk 2 and be utterly wrong. A market pundit's career is based not on always being right, but on never being embarrassingly wrong.
Above Average?
James Montier asked 200 fund managers this question: "Are you above average at your job?"
75% of respondents said "Yes". Something is clearly amiss with the 'calibration' of the average fund manager; they just don't know what they don't know.
But don't laugh too hard at the professionals. Do you think you're an above average investor, too? Are you sure? Research seems to suggest that the more confident the investor, the worse their decisions will be.
More from R J Johnson:
