Walk-forward backtesting: what chronology fixes and what it cannot

Walk-forward backtesting repeatedly develops or fits a strategy using earlier observations, then evaluates it on a later interval. The object of evaluation is the whole selection procedure: candidate rules, preprocessing, parameter search, objective and retraining schedule. It is not merely the final code attached to a good historical curve.

Chronological splitting addresses a real problem. The scikit-learn time-series splitter keeps training observations before test observations; it also documents expanding training sets and controls for maximum training size, test size and an optional gap. Its equal-duration comparison assumes equally spaced samples. [1]

For trading, those indices are a starting point. The analytical framework here audits three clocks: when an event occurred, when its information was available and when the researcher selected the rule. A clean-looking split can still violate any of them.

Define the universe, data source and version, training length, test length, feature construction, candidate space, optimisation objective, costs and capital convention before reading the next test result. Record whether positions carry across boundaries and how warm-up observations are handled.

An expanding window preserves more history; a rolling window discards older observations. Neither is universally superior. The choice expresses a hypothesis about how much older evidence remains relevant. Selecting the better-looking window after examining the full evaluation history is another research decision that needs its own later evaluation.

For an illustrative moving-average study, freezing the procedure means fixing which lookbacks may be compared and how the winner is selected. It does not mean keeping one parameter forever. A legitimate walk-forward procedure can change parameters at each scheduled refit, provided each change uses only information available at that time.

The first clock is observation time. The second is availability time. A macroeconomic release can refer to an earlier month but become tradable information later. A revised series downloaded today may contain values unavailable to the historical strategy. A bar's eventual high and close cannot inform a decision made before that bar completed.

The same principle applies to transformations. Scikit-learn's leakage guidance warns against fitting preprocessing or feature selection using test observations; pipelines help keep fitted operations inside the appropriate subset. [2]

Inspect each feature's dependency chain rather than checking one final timestamp. A correctly shifted feature can still inherit a full-sample normalisation. A point-in-time company universe can still use a classification revised later. Reject ambiguous inputs or state their limitations explicitly. The market-memory article explains why storing provenance and revisions matters.

A training example observed before a test boundary may have a label determined by prices after that boundary. If the label looks ahead, the training set can already contain part of the test period's information. A gap between index ranges helps only when it actually covers the information horizon; choosing an arbitrary number of rows is not proof.

The splitter's documented gap excludes observations at the training end. It does not inspect the label definition or trading lifecycle for you. [1]

Also define whether a position opened during training remains open into the test window, whether the test starts flat and how costs are charged. These are different experiments. Write an audit table for the last training decision and the first test decision: observation time, latest input availability, label end, parameter version and first permitted execution time. That small table often exposes a mismatch hidden by an equity curve.

The third clock is the researcher's decision time. If a disappointing test leads you to change the universe, stop rule, cost model or training window, the next reported result is part of development. Calling the reused interval “out of sample” does not restore its independence from selection.

Bailey and colleagues' backtest-overfitting framework addresses selection among candidate strategies and challenges reliance on a simple holdout. It formalises a different quantity from a single strategy's profitability probability. Its author manuscript is methodological evidence, not an estimate of overfitting in VC Trader AI. [3]

Keep a trial ledger that includes failed variants and editorial choices, not only optimiser runs. An AI assistant can accelerate candidate generation; that makes the selection record more consequential. A fresh final holdout can help evaluate the frozen procedure, but repeatedly opening it turns it into development data too.

For independent tests under true null hypotheses, each with false-positive probability 0.05, the probability of at least one false positive is 1 − 0.95^N. At 20 tests it is approximately 64.2%. This is an exact calculation under stated assumptions, not a market dataset, backtest result or measured probability that a chosen strategy is overfit.

Real candidate strategies often share features, trades and regimes, so independence is usually an unrealistic literal description. The calculation remains useful as a transparent demonstration of why the number of opportunities for selection matters. It cannot tell you the effective number of independent trials in a particular research programme. The collection's chart code and CSV make the arithmetic reproducible.

Do not apply a generic haircut to a Sharpe ratio and declare the problem solved. The inference depends on the experiment, the dependence structure, the selection history and the metric distribution. Preserve those details before choosing a statistical method.

Chronology does not make fills realistic. Include the costs and timing relevant to the strategy: spread, commission, financing, delays, available liquidity and an explicit rule for ambiguous intrabar sequences. If the stop and target are both touched in one bar, OHLC data alone may not establish which occurred first. A conservative sensitivity or higher-resolution data is more defensible than silently choosing the favourable path.

Report performance by evaluation window as well as in aggregate. Include trade count, exposure, turnover, drawdown and cost sensitivity. Explain denominators and whether returns refer to fixed or changing capital. Adjacent trades and returns may be dependent; do not present a naive confidence interval as if every trade were an independent observation.

An adverse window is evidence worth understanding, not an inconvenience to remove. State exclusions before interpretation and retain failed runs. This article provides no strategy-performance experiment, confidence interval or tradable-alpha claim.

The defensible conclusion is conditional: this procedure behaved this way over these later intervals, under these data and execution assumptions. It is useful evidence about a research process. It does not establish future returns or operational readiness. Before connecting that process to an account, apply the separate Python-to-MT5 deployment contract.