Escaping the Backtesting Illusion

Abstract

Two tests can help asset managers to develop more robust investment strategies: an impact test and a survival test. Both tests complement the backtest, in which one checks how a proposed investment strategy would have performed in the past. The impact test considers the performance of the strategy when assets under management grow (crowdedness), and it checks the impact that growth in assets under management in competing strategies has on the proposed strategy (cross impact). The survival test considers the effect of the long-term evolution of assets under management in competition for market capital. Using Shiller’s S&P 500 index and bond market data, we show that time-series momentum (relative strength) performs best in the backtest and the impact test but that an expected relative cash-flow rule (relative dividend yield) has the best long-term survival properties.

TOPICS: Statistical methods, simulations

Key Findings

• • Robustness checks of investment strategies require more than backtesting.

• • An impact test is proposed to measure the effects of crowdedness and cross impact on investment strategies.

• • A survival test is proposed to assess the long-term effects of investment strategies’ competition for market capital.