Backtesting Without Fooling Yourself: A Plain-English Checklist

Backtesting sounds like a solved problem: run your strategy against historical data, look at the profit curve, decide if it's good enough. In practice, most backtests that traders run on their own strategies are quietly wrong in one of a handful of predictable ways — and the strategy that looked great on paper performs nothing like that once real money and real conditions are involved.


None of this requires advanced statistics to fix. It requires knowing where the common mistakes hide, and building a habit of checking for them every single time, rather than only when a backtest result looks suspiciously good.

Mistake One: Testing With Zero Friction

Most backtesting tools default to zero commission and zero slippage — the assumption that every trade fills instantly at the exact price shown, with no fee. That assumption is fine for a first rough pass, and wrong for any decision involving real capital. Add realistic transaction costs for your actual broker and instrument, and the apparent profitability of most strategies drops meaningfully — often by a large enough margin to turn a winning strategy into a losing one. Before trusting any backtest result, re-run it with your real commission structure and a slippage estimate appropriate to how liquid the instrument is. A strategy that only looks profitable at zero cost was never actually profitable.

Estimating Slippage Realistically

Slippage isn't a single fixed number — it depends on the instrument's typical bid-ask spread, how fast price is moving at the moment of entry or exit, and the size of your order relative to the available liquidity at that price level. A reasonable starting approach is to check the instrument's typical spread during the session you trade, and add a conservative buffer on top of that spread for market orders entered during fast-moving conditions, since the price can move an extra tick or two between the moment your order is sent and the moment it fills. For strategies that trade frequently, even a seemingly small per-trade slippage assumption compounds into a meaningfully different result across dozens or hundreds of trades.

Mistake Two: Not Enough Trades to Mean Anything

A strategy that generated eight trades over the test period, even if seven of them won, hasn't told you much. Small samples are noisy by nature — a handful of lucky trades can make a mediocre strategy look brilliant, and a handful of unlucky ones can sink a genuinely good one. As a rough guide, fewer than thirty trades is close to statistically meaningless on its own. A credible read needs closer to a hundred trades or more, spread across at least two or three years that include different kinds of market conditions — trending periods, choppy periods, high and low volatility. A strategy tested only during one calm uptrend hasn't been tested against anything difficult.

Why Trade Count Alone Isn't the Full Story

A hundred trades clustered entirely within a single three-month period of unusually high volatility is a different kind of sample than a hundred trades spread evenly across three years with a healthy mix of calm and volatile stretches. When reviewing trade count, also glance at the distribution across time — are trades reasonably spread out, or did most of them cluster into one or two unusual periods that may not represent typical future conditions? A strategy that only generated meaningful trade volume during one historically unusual stretch (a sharp crash, an unusually quiet year) deserves extra skepticism about whether that stretch will repeat.

Mistake Three: Letting the Indicator Repaint

If the signal generator behind your strategy can rewrite its own past signals after the fact — a behavior called repainting — then any backtest built on it is testing a version of history that never existed in real time. The backtest "knew" information at each signal that a live trader wouldn't have had yet. This produces a results curve that cannot be reproduced going forward, no matter how carefully everything else about the test was done. Always confirm the signal source doesn't repaint before trusting any backtest built on top of it — step through the chart candle by candle using a replay feature and watch whether markers move after they first appear.

How Repainting Specifically Corrupts a Backtest Engine

It's worth understanding exactly why this is so damaging rather than just accepting it as a rule to follow. A backtesting engine, whether it's a manual bar-replay review or an automated strategy tester, calculates results by applying the strategy's logic to each historical bar in sequence. If the underlying signal recalculates using data from bars that, chronologically, come after the bar the signal claims to fire on, the engine is effectively given information from the future at the moment it decides to enter a trade. No amount of careful cost modeling, trade-count discipline, or out-of-sample testing fixes this specific problem, because the corruption happens at the data level, before any of those other safeguards even get applied.

Mistake Four: Only Testing on the Period You Optimized

If you tuned a strategy's parameters to maximize profit on a specific stretch of historical data, and then only tested it on that same stretch, you've measured how well you can curve-fit — not how well the strategy generalizes. This is called overfitting, and it's the most common reason a strategy that looked flawless in testing falls apart live. The fix is a strict split: tune parameters only on one period ("in-sample"), then test the final, locked parameters on a separate period you never touched during tuning ("out-of-sample"). If performance collapses on the untouched data, the strategy was fit to noise, not to a real edge.

A Simple Three-Way Split for More Confidence

Beyond the basic in-sample/out-of-sample split, a more rigorous approach splits historical data three ways: one period for initial parameter tuning, a second separate period used only to pick between a handful of finalist parameter sets (preventing you from simply re-tuning on the "out-of-sample" data by trial and error until it looks good), and a third, final period touched only once, at the very end, as a last honest check before considering the strategy validated. This extra step costs more historical data and more patience, but it closes a common loophole where traders think they're doing honest out-of-sample testing while actually iterating their way to a result that's still overfit to that supposedly untouched period.

Signs a Strategy Was Overfit Even If It Passed a Basic Split

A few warning signs are worth checking even after a strategy clears a basic in-sample/out-of-sample split. Does the strategy rely on an unusually specific parameter value — for example, a moving average length of 47 rather than a round number like 20 or 50 — with no logical reason for that exact number? Highly specific, seemingly arbitrary parameter values are a common fingerprint of a parameter search that found a lucky fit rather than a genuine pattern. Does performance change dramatically with a small nudge to one parameter, say shifting a threshold by just one or two units? A strategy whose results are fragile to tiny parameter changes is more likely to have been fit to noise than one whose results stay reasonably stable across a range of nearby parameter values.

Mistake Five: Judging Only by Total Profit

Total profit alone hides the path taken to get there. Two strategies can end with identical total returns while one survives a 10% drawdown along the way and the other survives a 60% drawdown — the second one is far riskier to actually hold through, regardless of the final number. Two practical filters worth applying as a baseline: profit factor (gross profit divided by gross loss) above roughly 1.5, and maximum drawdown kept under a level you could genuinely tolerate without abandoning the strategy mid-drawdown. A strategy that's technically profitable but requires nerve you don't have isn't a strategy you'll actually stick with.

Other Metrics Worth Checking Alongside Total Profit

Beyond profit factor and max drawdown, a few additional figures round out a fuller picture. Average win versus average loss size tells you whether the strategy depends on a small number of large wins offsetting many small losses, or the reverse — both can work, but they require very different psychological tolerance to trade live. The longest losing streak, measured in consecutive losing trades rather than just drawdown in dollar terms, tells you how many losses in a row you need to be mentally prepared to sit through without abandoning the plan. Time spent in drawdown — not just the depth of the drawdown but how long it typically took to recover a new equity high — matters for anyone who needs some predictability around when a strategy is "working" again after a rough stretch.

Manual Bar-Replay Testing vs Automated Strategy-Tester Testing

Most charting platforms offer two distinct ways to backtest: stepping through historical bars manually using a replay feature, or running an automated strategy tester that applies coded entry/exit rules across the entire historical dataset at once. Each has a distinct strength. Manual replay forces you to make real-time decisions candle by candle, which surfaces psychological and discretionary factors an automated test can't capture — useful if your strategy has any judgment-based component at all, however small. Automated testing covers vastly more historical data in far less time and removes the risk of unconscious hindsight bias creeping into a manual review, which is a real risk even for disciplined traders, since the human eye naturally notices winning setups more easily once it already knows they won.
A strategy with any discretionary component at all should be validated with both approaches where possible — automated testing for the fully mechanical portion of the rules, and a manual replay pass specifically to pressure-test the discretionary parts under genuine, real-time-feeling conditions.

Walk-Forward Testing: A Step Beyond a Single Out-of-Sample Split

A single in-sample/out-of-sample split is a meaningful improvement over testing only on the tuned period, but it still only validates the strategy against one specific out-of-sample stretch. Walk-forward testing extends the idea: instead of one split, the historical data is divided into several sequential windows. The strategy is tuned on window one and tested on window two, then re-tuned (allowing parameters to adapt) on windows one and two combined and tested on window three, and so on, rolling forward through the entire dataset.
This approach has two advantages over a single split. First, it produces several out-of-sample results instead of one, giving a better sense of how consistent the strategy's edge is across different stretches rather than relying on a single period's outcome, which could itself have been lucky or unlucky. Second, it mimics how a strategy would actually be used in practice — most traders do periodically revisit and adjust parameters as more data comes in, and walk-forward testing explicitly models that process rather than pretending parameters are frozen forever the day testing ends.

What a Walk-Forward Result Should Look Like

A strategy with a genuine edge should show broadly similar performance characteristics across most of the walk-forward windows — not identical, since market conditions differ window to window, but without one or two windows that are wildly profitable propping up several windows of outright losses. If performance is wildly inconsistent window to window, that's a sign the original edge may have been specific to a particular historical regime rather than something durable enough to adapt across changing conditions.

Platform-Specific Pitfalls Worth Knowing

Strategy Tester Defaults That Quietly Distort Results

Automated strategy testers on most charting platforms come with a handful of default settings that are easy to overlook and that can meaningfully distort results if left unchanged. Order execution settings sometimes default to filling at the exact open or close price with zero delay, rather than modeling any realistic fill behavior. Backtest date ranges sometimes default to "all available history," which can silently include a period with a completely different market structure (a different market-maker regime, a period before a relevant regulatory change, or simply a period with far thinner liquidity) than the conditions you actually intend to trade in. Before trusting any automated test result, explicitly check and set these defaults rather than assuming the platform's out-of-the-box configuration matches a realistic trading scenario.

Indicator Recalculation Settings

Separately from the repaint issue covered above, some charting platforms offer a setting that controls whether an indicator or strategy recalculates on every price tick within a still-forming bar, or only once per bar close. Leaving this set to "calculate on every tick" during backtesting, when your actual live trading plan is to act only on confirmed bar closes, can produce a backtest that looks different from how the strategy will actually behave live — not because of a repaint in the strict sense, but because the backtest and the live plan are evaluating the strategy's logic at different points within each bar. Matching this setting to your actual intended live execution behavior is a small, easy-to-miss detail that measurably affects result accuracy.

Documenting a Backtest So You Can Trust It Later

A backtest that isn't documented is hard to trust months later, when you've forgotten the exact settings used. A simple record worth keeping for every serious backtest includes: the exact date range tested, the commission and slippage assumptions used, the parameter values (and whether they were tuned in-sample or held fixed from a prior test), the total trade count and its distribution across time, and the key risk metrics (profit factor, max drawdown, longest losing streak) alongside the headline profit figure. This record does two things — it lets you honestly compare a new backtest against an older one using the same assumptions, and it forces you to actually produce all of the numbers covered in this checklist rather than skipping straight to the headline profit figure, which is the easiest number to report and the least informative one on its own.

A Short Pre-Trust Checklist

• Re-ran with realistic commission and slippage, not the default zero-cost setting.
• At least 100 trades, across multiple years and multiple market conditions, reasonably spread across time rather than clustered in one unusual stretch.
• Confirmed the underlying signal doesn't repaint, using a bar-by-bar replay check.
• Tested on an out-of-sample period the parameters were never tuned against, ideally with a third, final hold-out period touched only once.
• Checked profit factor and max drawdown, not just the final profit number, plus losing-streak length and time spent in drawdown.
• Checked whether the strategy's parameters are suspiciously specific or fragile to small changes.

Frequently Asked Questions

How much slippage should I assume if I don't know my exact fill history?

A conservative starting point is the instrument's typical bid-ask spread plus a small additional buffer for fast-moving conditions. It's better to overestimate slightly during testing and be pleasantly surprised live than to underestimate and discover the strategy's real edge was thinner than the backtest suggested.

Is 100 trades always enough to trust a backtest?

It's a reasonable rough threshold, not a guarantee. A hundred trades concentrated in one unusual historical stretch tells you less than a hundred trades spread evenly across several different kinds of market conditions. Check the distribution, not just the count.

What's the single most damaging mistake on this list?

Repainting is arguably the most damaging, because it corrupts the backtest at the data level — no amount of careful cost modeling or sample-size discipline fixes a result built on signals that used future information. Overfitting is a close second, because it's the easiest mistake to make without realizing it, especially when tuning parameters feels like careful, diligent work rather than a risk in itself.

Do I need coding skills to backtest properly?

No. Manual bar-replay testing requires no coding at all and is genuinely rigorous if done carefully and honestly. Automated strategy testers make covering large datasets faster, but the core discipline — realistic costs, adequate sample size, no repainting, proper out-of-sample validation — applies identically to both approaches.

Is walk-forward testing necessary for every strategy, or only more advanced ones?

It's most valuable for strategies with several tunable parameters, where overfitting risk is higher. A very simple strategy with one or two parameters and an intuitive, non-arbitrary logic behind them carries less overfitting risk, and a single honest out-of-sample split may be sufficient. As parameter count and tuning effort increase, so does the value of walk-forward testing as an extra safeguard.

Why does the "calculate on every tick" setting matter if I'm not worried about repainting?

Because it changes which exact price point the strategy's logic evaluates within each bar, even for an indicator that doesn't repaint in the strict sense. If your live plan is to act only on confirmed, closed bars, but your backtest evaluated intrabar prices throughout, the two aren't measuring the same decision process — which can quietly produce a backtest result that doesn't match live behavior even without any repaint bug involved.

Where to Go Deeper

For a step-by-step walkthrough of the two main ways to actually run these tests on a charting platform — a manual bar-replay approach and an automated strategy-tester approach — this guide to backtesting an indicator on TradingView covers both methods in detail, including how to set realistic cost assumptions in each.
Backtesting isn't a guarantee of future performance, and no process makes it one. What it can do, done carefully, is rule out the strategies that were never going to work in the first place — which is most of what a backtest is actually good for. Treat every one of the five mistakes above as a standing checklist item, not a one-time lesson, and apply it the same way to every backtest you ever run, including the ones that already look good.
The discipline compounds over time. A trader who runs this checklist consistently, on every strategy idea, builds an increasingly reliable intuition for which setups are worth pursuing further and which are quietly repeating one of these five mistakes. That intuition, built from consistent process rather than from any single lucky or unlucky result, is ultimately more valuable than any individual backtest number.