AIalgo.si
Guides

How to Detect Overfitting in Your Backtest

TKTomaž KosOctober 2, 20258 min read

Quick answer

A backtest that looks perfect is usually lying. Six diagnostic checks — from out-of-sample splits to parameter sensitivity — that expose curve-fitted strategies.

Why perfect backtests are a red flag

A backtest is an optimization problem: with enough parameters, any strategy can be curved to fit past prices. If you try 500 combinations of moving-average periods, one of them will look brilliant by pure chance. That is overfitting — the strategy memorized noise instead of learning signal.

The tell is simple: brilliant in-sample performance that evaporates the moment you change a parameter by 10% or move the test window forward a year.

Six diagnostic checks

One: hold out the last 20-30% of data and never touch it until the final evaluation. Two: run a parameter sweep — if performance decays smoothly away from your chosen parameters, the edge is probably real; if it spikes at one point, it is noise. Three: check the number of trades — a backtest with 12 trades proves nothing statistically.

Four: include realistic costs. Slippage and fees kill most high-frequency edges; a strategy that only works with zero costs is fiction. Five: test across regimes — a trend strategy validated only on the 2020-2021 bull run has not been tested. Six: shuffle the entry signals as a placebo; a strategy that still profits on random entries has a data leak.

Metrics that overfitting inflates

Overfitted strategies show extreme Sharpe ratios (above 3 in a backtest usually means curve-fit), unrealistically smooth equity curves, and win rates above 80% with tight stops. Healthy backtests show Sharpe between 0.5 and 2, drawdowns of 10-30%, and win rates of 40-60%.

The AIalgo.si Strategy backtester defaults to honest parameters and always displays drawdown next to return, because a return figure without its drawdown is marketing, not analysis.

The one-sentence summary

If your backtest cannot survive a 10% parameter change, a cost model, and an out-of-sample window, you have not found an edge — you have found a coincidence.

Cite this page

AIalgo.si Editorial (2025). How to Detect Overfitting in Your Backtest. AIalgo.si Resources. https://aialgo.si/resources/how-to-detect-overfitting-in-your-backtest

#backtesting#trading#overfitting

Keep reading