How to Detect Overfitting in Your Backtest
Quick answer
A backtest that looks perfect is usually lying. Six diagnostic checks — from out-of-sample splits to parameter sensitivity — that expose curve-fitted strategies.
Why perfect backtests are a red flag
A backtest is an optimization problem: with enough parameters, any strategy can be curved to fit past prices. If you try 500 combinations of moving-average periods, one of them will look brilliant by pure chance. That is overfitting — the strategy memorized noise instead of learning signal.
The tell is simple: brilliant in-sample performance that evaporates the moment you change a parameter by 10% or move the test window forward a year.
Six diagnostic checks
One: hold out the last 20-30% of data and never touch it until the final evaluation. Two: run a parameter sweep — if performance decays smoothly away from your chosen parameters, the edge is probably real; if it spikes at one point, it is noise. Three: check the number of trades — a backtest with 12 trades proves nothing statistically.
Four: include realistic costs. Slippage and fees kill most high-frequency edges; a strategy that only works with zero costs is fiction. Five: test across regimes — a trend strategy validated only on the 2020-2021 bull run has not been tested. Six: shuffle the entry signals as a placebo; a strategy that still profits on random entries has a data leak.
Metrics that overfitting inflates
Overfitted strategies show extreme Sharpe ratios (above 3 in a backtest usually means curve-fit), unrealistically smooth equity curves, and win rates above 80% with tight stops. Healthy backtests show Sharpe between 0.5 and 2, drawdowns of 10-30%, and win rates of 40-60%.
The AIalgo.si Strategy backtester defaults to honest parameters and always displays drawdown next to return, because a return figure without its drawdown is marketing, not analysis.
The one-sentence summary
If your backtest cannot survive a 10% parameter change, a cost model, and an out-of-sample window, you have not found an edge — you have found a coincidence.
Cite this page
AIalgo.si Editorial (2025). How to Detect Overfitting in Your Backtest. AIalgo.si Resources. https://aialgo.si/resources/how-to-detect-overfitting-in-your-backtest