How to backtest a stock strategy and avoid common biases

Historical scan, event study and performance backtest

Three kinds of test are often all called backtests, and they answer different questions. A historical scan evaluates a rule as of an earlier session and lists what matched. An event study measures what happened to those matches over the following sessions. A performance backtest simulates trading the rule over many dates, with entries, exits, position sizes and costs.

TradeClever’s Backtest Scan selects the stocks that matched a rule on a chosen start date, keeps that group to a chosen end date and reports each stock’s price return with their equal-weight average and the counts of winners and losers. It is an event study, not a trading simulator: it has no position sizing, costs or exits. A historical scan or an event study can show that a rule selected stocks that later rose; neither proves that trading the rule would have been profitable.

How to backtest a stock trading strategy

A useful test starts with a written hypothesis and with dates chosen before any result is seen. The steps below work with a historical scan and an event study, and carry over to a full performance backtest in other software.

1. Write the hypothesis in one sentence, with its horizon: pullbacks in established uptrends beat the universe median over the next 21 sessions.
2. Write the rule in CleverScan and save it; do not change it during the test.
3. Choose test dates in advance, spread across rising, falling and sideways markets, and set the latest third aside.
4. For each date, run Backtest Scan from that session over the horizon and record the matches and their period returns; run the same dates with a broad rule such as close > 0, whose sorted Period Return column gives the universe median.
5. Combine the dates: count the matches and compare their median and average with the universe median.
6. Check the result for look-ahead bias, survivorship bias and overfitting.
7. Subtract realistic costs for the holding period you intend.
8. Run the dates you set aside once, and accept what they show.

securityType = "Stock" and close > sma200 and sma50 > sma200 and distSma50Pct >= -3 and distSma50Pct <= 1

Read the results: averages, medians and sample size

An event study produces a list of forward returns, and a few numbers summarize it. The median is less sensitive to one extreme result than the average, and the comparison with the universe median separates the rule from the market’s move. The hypothetical sample below shows why both matter, and why ten matches settle very little.

One stock lifts the average from 1.97% to 4.17%, while the median moves from 1.40% to 2.15%. The standard error, the sample standard deviation divided by the square root of the number of matches, is 2.71 percentage points for all ten, so the average’s lead of 2.97 points over the universe median is about one standard error: well within what chance alone can produce. With the same spread of returns, the standard error shrinks only with the square root of the sample.

Matches from one session move with the same market, and forward windows of nearby dates overlap, so the returns are not independent and the true uncertainty is larger than these figures. More dates, spaced at least one horizon apart, add more information than more matches on a single date.

Look-ahead bias and survivorship bias

Look-ahead bias enters a test when a rule uses information that was not known when the decision was made: a swing point confirmed sessions later, earnings figures released after the close, or today’s company data applied to an earlier date. Survivorship bias enters when the test universe holds only companies that still exist, leaving out those that were delisted, acquired or failed, which tends to flatter the results.

• Time-stamp every input and ask whether it was known before the decision.
• Enter no earlier than the close that completes the signal; starting from the next session is stricter.
• Use point-in-time company data where you can, since sector, market capitalization and index membership change over time.
• Build the universe as it was on each test date, including companies that later delisted.

Two cautions apply to tests in TradeClever. With Close entry, Backtest Scan measures returns from the close of the session the rule was evaluated on, so it assumes you could act at that close; Open entry evaluates the rule on the session before and enters at the start date’s open. And in a historical scan some company metadata uses current values, so a condition on such a field reflects today’s classification rather than the one known on the earlier date; run a scan for an earlier session explains how to read those results. Comparing the matches with the median of the same universe shares the universe’s biases, which limits their effect on the comparison without removing it.

Overfitting and data snooping

Overfitting is tuning a rule so closely to past data that it describes the noise of that sample rather than a repeatable effect. Data snooping is the same problem across a whole research process: trying many rules, or many thresholds of one rule, and keeping whichever looked best. Warning signs include oddly precise thresholds such as an RSI below 31.5, results that collapse when a threshold moves slightly, and many conditions supported by few matches.

The arithmetic of many attempts is unforgiving. If a rule has no real edge and each variant has a 5% chance of passing a significance test by luck, the chance that at least one of k independent variants passes is 1 − 0.95^k.

• Keep rules short and thresholds round, and fix them before testing.
• Record every variant you try, not only the one you keep.
• Check the neighbors: if RSI below 30 looks good, RSI below 28 and below 32 should not look bad.
• Judge the final version only on data it was not tuned on.

Counter-example: a rule with seven conditions that selected 14 stocks across five test dates, each condition added after looking at the losers, can show an impressive average. It mostly describes those 14 stocks.

In-sample, out-of-sample and walk-forward testing

In-sample data is the history used to develop a rule; out-of-sample data is history the rule has never seen. A rule developed on one period and confirmed on a later, untouched period is more credible than one tested only where it was built. Walk-forward testing repeats that split: develop on one window, test on the next, roll both forward, and judge the rule only by the test windows together.

Each development window holds 504 sessions, about two years, and each test window 126 sessions, about six months. The six test windows cover sessions 505 to 1,260 without overlap, so every out-of-sample result comes from sessions its rule version had not seen. With TradeClever’s Backtest Scan, the same idea means choosing development dates and later test dates in advance, and not editing the rule after seeing the test dates.

Test strategy performance across market conditions

A rule that worked only in a rising market may simply have held stocks during a rally. Classify each test date by the condition of the broad market before looking at results, for example by whether a broad index fund closed above or below its 200-session average and whether that average was rising, falling or flat. Then compare the rule with the universe median separately in each group.

• Rising market: the lead over the universe median matters more than the raw return, since most stocks rose.
• Falling market: check whether the matches fell less than the universe, or simply fell with it.
• Sideways market: ranges produce many closes above a level that do not last, the failure a breakout rule is exposed to, while mean-reversion rules are designed for them.
• Volatile periods: note the typical atrPct14 of the matches; a rule that works only when volatility is high has a narrower use than it seems.

Comparing each date with the universe median of the same date, from the same Backtest Scan dates over a broad rule, removes part of the market’s move. Recording the market condition next to each date shows whether the lead holds across conditions or comes from one of them.

Costs, slippage and why live results differ

An event study measures close-to-close price changes, while real trading pays commissions, the bid-ask spread and slippage, the difference between the expected price and the fill. Costs are paid on every round trip, so they weigh most on rules that trade often and hold briefly.

The 0.30% cost is hypothetical, for example half the bid-ask spread plus some slippage on both the entry and the exit. Spreads are wider in small, thinly traded stocks, so estimate costs for the stocks your rule actually selects.

• Overfitting: the rule described its test sample, not a repeatable effect.
• Costs and slippage: the test assumed fills at the close.
• Changing conditions: the effect belonged to the market of the test period.
• Execution: signals arrive after the close, and gaps open beyond planned exits.
• Behavior: trades are skipped after a loss, or held after the exit level is reached.

How to account for earnings gaps in strategy testing

Earnings releases often arrive outside trading hours, so the next session can open far from the previous close. A close-to-close forward return includes that gap, and an exit order cannot protect against it: an exit planned below the open may be filled at the open instead. A rule whose results depend on a few earnings gaps measures earnings reactions, not the pattern it describes.

1. For each match, note whether an earnings release fell inside its forward window, using the company’s investor-relations page.
2. Report the results with and without those matches.
3. Model exits that gap through the exit level at the open, not at the planned price.
4. Check entries made on gap sessions separately, with a gapPct condition as below.

securityType = "Stock" and close > sma200 and sma50 > sma200 and distSma50Pct >= -3 and distSma50Pct <= 1 and gapPct > -2 and gapPct < 2

The earnings dates guide and the stock price gaps guide cover the dates and the gaps themselves.

Evaluate breakout, pullback and mean-reversion strategies

Each style has its own horizon and its own way of failing, so measure what would undermine it, not only its average return. Write the measure down with the rule, before the test.

Some of these measures need the chart rather than the Backtest Scan summary: whether a breakout failed, or whether a pullback undercut its last higher low, has to be checked symbol by symbol for at least a sample of the matches. The guides on breakouts and pullbacks and stock market structure define those levels.