Hypothetical vs. Live: The Performance Gap

Almost every trading product advertises a backtest. Almost none survive contact with live markets unchanged. The gap is not bad luck — it is a predictable consequence of how backtests are built and selected.

Overfitting is the default, not the exception

When you try many strategy variants and keep the best, you are selecting on noise. Formal work on the probability of backtest overfitting^[1] shows that with enough trials, an outstanding in-sample result is expected even when the true edge is zero — and that such strategies tend to underperform out of sample.

Why regulators single this out

The asymmetry is well understood by regulators. CFTC Regulation 4.41^[2] requires that hypothetical performance carry a prominent disclosure precisely because it “is prepared with the benefit of hindsight” and does not reflect real execution, liquidity, or the financial and emotional pressure of live trading.

A backtest tells you what would have worked. Only a live, cost-adjusted record tells you what does.

What an honest record requires

The academic standard for asset-return claims is strict about out-of-sample evidence and multiple-testing corrections^[3]. Translated into product terms, an honest track record has to do four things:

Test out of sample. Judge a strategy on data it never saw in design. Cost-adjust. Net of modeled commissions and slippage. Gate on significance. No claim until the sample is large enough to mean something. Reconcile to live. Compare hypothetical to realized fills.

This is the methodology behind Niro’s Proof Engine: it tests strategies on real data, reports results net of costs, and withholds any marketing claim until a strategy clears a significance threshold — so what reaches you is evidence, not an overfit highlight reel.