FOOL'S EYE VIEW
Problems With Past Performance

By James Carlisle
March 4, 2002

Carburton Street, London -- It is often said that past performance is no guide to the future but, taken literally, that's rubbish. The past is all we have and, over history, it's provided an excellent guide to the future. After an inebriated Saturday evening, I drank two pints of water before bed and woke up rosy on Sunday. Past performance has shown that it's a good idea to do that (although it also shows that I generally forget).

There are countless other examples of where past performance tells us something. Even when it apparently tells us nothing, that, in itself might be useful information. In a game of roulette, assuming the wheel is fairly balanced, you might say that the past results show that you can't tell what's coming next. In fact, it's more subtle that that. If you look at enough of it, what the data will show is that it is random. It tells you that there's no reason to suppose that any particular pattern will emerge.

Patterns in data

But just because there is no reason to expect a given pattern to emerge in the future, does not mean that there will not be patterns. The most extraordinary thing would be if there weren't any. In fact, the complete lack of a pattern would have to be a pattern in itself. If you spilt some salt on the floor, how would you define a totally random pattern? More importantly, how could you tell if that brief swirly line of salt over on the left is likely to repeat (perhaps because of the structure of the salt cellar), or whether it's just one of those patterns from nowhere.

The answer is that you can't tell. Well not absolutely anyway. Statisticans are better than sparrows at sitting on a fence. You won't catch them saying that this is that or that that is this. Instead you'll hear them express things as probabilities. So 'that will have perhaps a 90% chance of being this, given my assumptions' is about the best you'll get.

Patterns in coin tosses

Let's say we were asked to check to see if a coin is fair or not. So we set off and toss it 1,000 times. How many heads would we need to be sure that the coin was fair (ie that the results were random)? It all depends on how sure you need to be. For a coin toss, the sums are relatively straitforward. In fact, there's a function in Microsoft Excel (called BINOMDIST) that will work it out for you. It turns out that the chances of getting between 490 and 510 heads (and tails) are 51%, so you wouldn't attribute too much significance to getting around the 490 or 510 mark. As you move away from the middle (of 500), however, you will increasingly begin to smell a rat.

The chances of getting between 450 and 550 heads are apparently 99.87% (it seems very high, but that's what Microsoft says), so you'd be stunned if you came outside this range. The chances of getting below 400 or above 600 are a bit less than one in a billion (assuming I've counted up the noughts up correctly) so, if you fell outside of that range, you'd express very considerable confidence (that's about as much as you'll get from a statistician) that you had a dodgy coin. Either that, or you'd just seen a leprechaun kissing a unicorn on their way past a blue moon.

'Backtesting' investment strategies

So, if you're looking at some form of share selection strategy and you decide to back test it over the last few months, don't be surprised to see a pattern. In fact, every strategy will show a divergence from the market average over the last few months (unfortunately, the perfect index tracker has not been invented). More than that, you'd expect half of them to show a profit and, let's say, 10% to show a very healthy profit.

But finding these situations doesn't, in itself, demonstrate any non-randomness. They must be there so, if you keep looking for them, then you will find them. Not only that, but you'd expect half of these to show a profit over the next few months and about 10%, let's say, to provide a healthy profit. So, 1% of our original sample has provided healthy profits over a period of a few months and then continued to do so for another few months. Yet, there's no reason, in itself, for this to make you think that you've found a successful strategy. This presents significant dangers to investors because it can provide confidence when there's no reason for any. That, in turn, might lead to inappropriate investment decision making.

Past performance of investment funds

The same goes, of course, for managed investment funds. Suppose you start out with 1,000 investment funds and that there's a 50% chance of a fund outperforming the market average in each year. After ten years, the odds are that you'd have one fund that had outperformed in every single year. Experts and punters alike would be trumpeting the fund as the greatest, most consistent performer of all time, but the truth is that it's nothing more than we should have expected.

The situation with share selection strategies and investment funds is more complicated than coin tosses because there are so many different factors at work. For instance, you would expect costs, all other things being equal, to affect a strategy or investment fund's performance adversely. If you take the view that risky assets, such as shares, are priced more cheaply (and therefore to provide greater returns) than less risky assets, such as cash, then you might also expect strategies focusing on the risky end of the spectrum to display higher average returns (as well as more volatility). In short, there are factors that you might expect to show persistency (ie non-randomness) overlaying the randomness.

Survivorship bias

To make matters worse, you have to deal with 'survivorship bias'. This is the name for a process by which you only tend to hear about the success stories. Investment funds tend to get disbanded or gobbled up by rivals if they perform very badly, so whenever you're looking at the past performance of a group of funds in existence today, you're automatically ignoring the performance of the worst ones. The same applies to share selection strategies. Once they go off the boil, they tend to stop being talked about. So, you shouldn't be surprised if people seem to be talking about a bunch of apparently very successful approaches.

These are traps for the unwary investor. You can get sucked in by apparently very seductive data. Such and such a fund might have the best performance over the last x years or such and such a trading strategy might have made you a return of y% over the last z years. When you hear about this type of thing, remember that there has to be a fund that does the best over five years and there has to be a trading strategy that returns y% over the last z years. It would be most peculiar if there hadn't been.

By itself, this sort of data doesn't tell you anything useful. What matters is whether there is any reason to expect to see patterns persisting into the future. The only factors that have consistently been shown to demonstrate this over the long term, to my knowledge, are:

(a) that costs tend to undermine your performance;

(b) that shares, as a class, tend to outperform other types of investment; and

(c) that a diversified pool of assets reduces your risks.

As a statistician might say, that would lead us to have a reasonable level of confidence, for long-term investment, in a low-cost method of investing in a broad spread of shares. Sound familiar?