If you are looking for trading futures, commodities, forex or stocks, you are likely to encounter “backtest” results. These results literally show what a trading method would do in the past, if you follow it. One reason why the US government’s phrase “past performance is no guarantee of future results” is necessary when discussing trading systems or approaches.
Backtests are speculative, and may or may not actually be traded. It is even possible that the trades shown in the backtest are impossible to do.
As an experienced system developer, with my Tradestation testing software in 5 minutes, I can create a system in any market with a backtest that will blow you away – it would be nice to see. But, it will fail to move forward – practically sure! Believe me, many developers build systems this way and then try to sell you their “secrets”.
So, why is the backtest so unreliable? Four reasons come to mind.
1) Optimization – Most testing software has an optimization feature, which will select the best set of parameters based on past data. Most developers abuse this feature. What works best in the past is less likely to work in the future. As a result, reading this extra optimized backtest, you think you are buying a Mercedes, but in reality you get a Yugo.
2) Hindsite Bias – It is difficult to create a system without “picking” the data. Since no one can rise to the top in future data, a developer that does this during development is, in fact, a fraud. A lot of times, people don’t even realize they’re doing it – it can be a subtle mistake.
3) Software Limitations – Software has limitations that allow unrealistic or unattainable fulfillment. For example, a close order system most likely includes an unrealistic fill in the system with the market, since the order may be sent (and never fulfilled) after the market closes, but the software still thinks it has been filled.
4) There is no real-time performance – developers post their own backtest results, show them as real, and have no independent real-time verification of their results. Can you really believe the backstage of the same person who is trying to sell you the system?
What is the solution to this backtest dilemma? Simple, if you are working with a CTA (product trading advisor), hedge fund or mutual fund, make sure you see real, monitored real time records. If you’re dealing with a system developer, make sure the results you see are independently checked and verified, especially with results based on real money (not demo) accounts. I list two of these websites below.
Easy to be seduced by the extraordinary looking backtest results. Just keep in mind that these results may not be realistic.
Websites such as worldcupadvisor.com or collective2.com are just two of the many websites that offer third party verification of trading results.