** DRAFT ** draft
Usually a blog post with such a provocative claim start with a disclaimer.
But, unfortunately, if you are testing an algorithmic strategy, it is not possible to do proper backtesting, as defined by the statistical sciences for validating predicting models for time series.
So, why almost algo trader (miss) uses the term? On this post, I'll develop an argument around the fact that different people mean different things when they employ the term backtest and, for those who attempt to do a proper backtests, will always fail to preserve the cause and effect relations. Later, I'll propose a more realistic and scientific approach to measuring an algorithm's ability to perform reliably.
----
First, lets realize that what quants and algo traders may mean when they say they backtested an algorithm isn't always the formal meaning of backtest, but a simple test based on real data -- or not so real, as we'll see.
The formal meaning of backtest is, as defined by wikipedia:
"Backtesting is a term used in modeling to refer to testing a predictive model on historical data. Backtesting is a type of retrodiction, and a special type of cross-validation applied to previous time period(s)."
Lets comment the highlighted terms:
The issue is with retrodiction: this is a scientific process that uses past events to estimate the patterns for future occurrences. Since it is a process, you may constantly refer back to the same data, with an improved hipotesis, to test if your prognosis matches subsequent untested data. ... and this is not possible for Algorithmic Trading.
---
First, lets give an example where backtests may fully use the retrodiction proccess: weather forecast. On algorithmic trading, when you run your model, you are changing the orders on the book; on weather forecasting, when you run your model, you don't change the weather nor anything that influences on it -- at least, not on the prediction times involved.
The purpose of backtesting an algo trading is, utterly, to check that the orders it issues will be profitable. One should remind that, for a deal to be closed, there must be a buyer and a seller. Since we are backtesting, thus working on past data, closed deals were always made between two parties that weren't you. This leaves two choices: to ignore or not the book of orders, as detailed below.