How hard is to do proper backtests in Algorithmic Trading?

** DRAFT ** draft


Usually a blog post with such a provocative claim start with a disclaimer.

But, unfortunately, if you are testing an algorithmic strategy, it is not possible to do proper backtesting, as defined by the statistical sciences for validating predicting models for time series.

So, why almost algo trader (miss) uses the term? On this post, I'll develop an argument around the fact that different people mean different things when they employ the term backtest and, for those who attempt to do a proper backtests, will always fail to preserve the cause and effect relations. Later, I'll propose a more realistic and scientific approach to measuring an algorithm's ability to perform reliably.

----

First, lets realize that what quants and algo traders may mean when they say they backtested an algorithm isn't always the formal meaning of backtest, but a simple test based on real data -- or not so real, as we'll see.

The formal meaning of backtest is, as defined by wikipedia:

"Backtesting is a term used in modeling to refer to testing a predictive model on historical data. Backtesting is a type of retrodiction, and a special type of cross-validation applied to previous time period(s)."

Lets comment the highlighted terms:

    • predictive model: no issues here. Most algo tradings attempt to predict the future based on past behavior;
    • historical data / previous time periods: most "backtests" are done using D1, T1... (day data, tick data). This is not optimal, even for long term trading. Please read "The case for Order Book Analysis: don't let your Algorithms go lazy by trading on D1 or T1 data models: use the real think";
    • cross-validation: there is also no issues here: one should always test with "out of sample data" to guarantee your model is not just able to perform well on the tested data -- what is called overfitting.

The issue is with retrodiction: this is a scientific process that uses past events to estimate the patterns for future occurrences. Since it is a process, you may constantly refer back to the same data, with an improved hipotesis, to test if your prognosis matches subsequent untested data. ... and this is not possible for Algorithmic Trading.

---

First, lets give an example where backtests may fully use the retrodiction proccess: weather forecast. On algorithmic trading, when you run your model, you are changing the orders on the book; on weather forecasting, when you run your model, you don't change the weather nor anything that influences on it -- at least, not on the prediction times involved.

The purpose of backtesting an algo trading is, utterly, to check that the orders it issues will be profitable. One should remind that, for a deal to be closed, there must be a buyer and a seller. Since we are backtesting, thus working on past data, closed deals were always made between two parties that weren't you. This leaves two choices: to ignore or not the book of orders, as detailed below.

  • Conciliation strategies attempting to cope with the book of orders:
    1. Introduce a VIRTUAL BUYER or SELLER that will repeat (but, this time, with you) the deal you are trying to mimic;
    2. Steal the deal from the BUYER or SELLER
  • Conciliation strategies that simply ignore the book of orders:
    1. Fix on a price point, considering you may BUY or SELL any quantity you want at a past real deal's prince point