Backtesting at Theme

Share:
Indexes
  1. Off-the-shelf Solution
  2. Writing Our Own Backtester
  3. Components and Events
  4. Components and Dependency Injection
  5. Drawbacks
  6. Benefits

Backtesting is a critical part of developing a trading strategy. The predictive model of a trading strategy is first run on the historical data, simulating how the strategy would have performed in the past.

At Theme, we are building an agile process to quickly research, test and productionize trading strategies. We want to make quick and sound decisions about every new strategy that we develop and backtesting is a crucial part of this process. It’s the first indication of the potential and outlook of a strategy, but it is also one of the principal analysis tools during a strategy’s development and optimization. Ensuring that we’ve developed a good backtesting system fundamentally helps us make data-driven decisions.

Goals of Backtesting

When we set out to build our backtesting platform, we had a few goals in mind:

  1. Accuracy: Our backtester has to be a realistic simulator of the market and avoid common backtesting pitfalls, like look-ahead bias.
  2. Development speed: We want to quickly develop and backtest strategies. We don’t want to write a new backtester every time we want to test a new strategy. A feature implemented once, such as an order execution algo, should be available to all future backtesting.
  3. Performance: Backtests need to run fast. Our goal was to minimize the overhead of fetching and parsing market data.
  4. Reproducibility: Results for every backtest run should be reproducible. We need to be able to recall the exact version of the strategy, model and data that we used for a particular backtest.
  5. Result analysis: We need to be in full control of the backtest result reporting. We need to have parameters of each backtest stored alongside its results for easy comparative analysis.
  6. Time-to-trading: There should be minimal differences between the code for backtesting and live trading.

Options for Backtesting

Off-the-shelf Solution

There are several–though not too many–existing backtesting solutions. Some solutions are full fledged, like Backtrader, while others are simpler libraries like backtesting.py. The advantage of an off-the-shelf solution would be quicker time to writing strategies.

However, learning the intricacies of these frameworks would take time and we would lack confidence in results until we became experts in them. We also felt that there would be limited flexibility in ready-made solutions; each optimization and modification could potentially become a time-consuming issue.

Writing Our Own Backtester

In order to have complete control over the code and full confidence in our backtesting, we settled on developing our own.

The easiest and quickest approach to writing a backtester is to write code that loops over market data. The code loads Open High Low Close price aggregations (“bars”), one by one, and decides how the strategy should react to each price change.

While this approach is simple, each backtest becomes a bespoke solution for only one strategy. It also does not promote code reusability, which allows for easy repetition of mistakes and bugs as new strategies are developed.

A better alternative is an event-driven backtester. In an event-driven architecture, different parts of the code communicate with each other by emitting and consuming events that represent some change in state. For example, a change in symbol price, placement of an order, or an update in an account’s cash value are represented by an event.

Note on language: In software engineering terminology, an “event” in an “event-driven architecture” is different from an “event” in an “event-driven trading strategy.” Events in software engineering serve as messages between loosely coupled components and are often used in development of modular software, like a micro-service architecture. Ironically, and at the risk of jargon conflict, this type of software architecture lends itself nicely to event-driven trading strategies that create instructions based on real-world catalysts.

Event-driven architecture also promotes design-by-contract programming where it is easy to expand capabilities, like creating a new order type or changing a marking data source, without having to rewrite any of the code or worry that previously written backtests would break.

Architecture

Quantstart published an excellent series about developing an event-driven backtester, which we recommend reading to get a more thorough explanation of event-driven architecture.

In our solution, components emit events that are saved in a queue and an event loop continuously reads that queue. When there is an event waiting, the event loop dispatches it to all components that are registered to receive that event. In turn, components themselves add events to the queue. When the queue is empty, the event loop loads new market data and moves the internal clock forward.

Components and Events

Components in an event-driven backtesting architecture
Components in our event-driven backtesting framework

The core of our backtester is an abstract Strategy class that we extend for every new strategy to implement the core business logic. If strategy needs to create an order when the price of a symbol reaches a certain point, Strategy class will listen for a price change signified by a MarketEvent, and when the price is right, it will add an OrderEvent to the queue. If the strategy needs to do something at a particular time, it will create a cron job to send an event at that time.

As context, cron is a job scheduler used to execute tasks at specific times. Of course, in backtests, if something needs to happen in 15 minutes, for example, we don’t want our code to pause execution for 15 minutes like we would in live trading; rather, it should just skip ahead by 15 minutes. So, we wrote a custom Cron component that is responsible for scheduling tasks and simulating time flow.

StrategyWorker implements the event loop that repeatedly looks to see if there are any events waiting to be read from the queue. It also consults the Cron to see if it has some events to add to the queue. In backtesting, if the event queue is empty and no cron jobs are waiting to be executed, Cron moves its internal clock forward.

DataProvider is a component that simulates price movement in the market. Whenever Cron moves the clock forward, DataProvider loads the next bar (in our case, a 1-minute bar) and emits a MarketEvent, signaling that new prices are available.

Broker represents the broker with whom we place orders. If it receives an OrderEvent, it will create an object representing the order and store it in its list of orders. Broker will send a FillEvent when the order has been filled. We’ve also designed Broker to handle the particularities of different order types. For example, with market orders, we can simulate slippage, and with limit orders, we create fill events only if a price condition is met sometime in the future.

Portfolio listens to events from other components. It stores the current cash and information about open positions and is responsible for generating reports with details about all trades and cumulative performance.

Components and Dependency Injection

A big advantage of our architecture is that the same code can run the trading strategy in both a backtest and a live trading environment, which dramatically decreases the time between strategy exploration and trading. We call this KPI the “time to trading.”

To achieve this, the abstract class Cron has two implementations: BacktestCron and LiveCron. As mentioned above, the BacktestCron simulates the passage of time and makes sure that all cron jobs are executed sequentially. LiveCron ensures that cron jobs are executed at precisely the time when they were scheduled and, of course, does not have to simulate the passage of time.

Similarly, most other components have a backtest and a live implementation, too. For example, BacktestBroker simulates price filling or holding limit orders, while the LiveBroker actually sends orders to the broker and lets them handle execution.

To pull it all together, StrategyFactory is the class that creates StrategyWorker instances and injects them with appropriate dependencies. StrategyWorker for a backtest will be injected with components for simulated trading, whereas StrategyWorker for live trading will have components that are designed for live trading, for example, a live market data feed and integration with an actual broker.

Writing New Strategies

One of the goals of this architecture was development speed: we want to be able to quickly develop and backtest new strategies. Since the Strategy class defines its cron jobs and callbacks for market events, any strategy of medium complexity can be implemented in a few lines of code without any boilerplate. Strategy purely contains the business logic of a trading strategy, while different order types and algos are already implemented in reusable chunks of loosely coupled code.

Backtests require a number of parameters that we add to the Strategy code. Since live trading reuses the same strategy class, we can simply enable or disable any parameter, depending on how well it performed in backtests.

Conclusion

Drawbacks

The biggest drawback of our homemade event-driven backtesting system is the initial investment in development time. It’s not easy to simplify and prototype the backtesting solution, so we had to have all components in place in order to test it.

Another drawback is that when multiple strategies use the same backtesting framework, any change to backtesting code will affect all backtests. This drawback is a blessing in disguise, though. It forces us to be vigilant about reproducibility of backtests. For each backtest run, we need to store model version and strategy parameters, as well as the version of the data used for training the model and running the backtest.

Benefits

There are many more positive things about the framework we have developed.

First, since we use the same code for both backtesting and live trading, each backtesting run is a quality assurance mechanism for live trading. We never have to write the same functionality in two different places, so we reduce the chance of miscommunication or bugs.

This approach is also good for our discipline: it’s impossible to make a change to the live strategy without also implementing it for backtesting.

Next, our framework is easy to extend. If we want to change a live data provider, all we need to do is create a new LiveDataProvider class and inject it into the StrategyWorker. Similarly, if we want to backtest using quotes instead of minute bars, it’s only a matter of swapping in a new QuoteBacktestDataProvider implementation.

Finally, the component-based design of our backtester and live strategy is also very suitable for the test driven development that we nurture at Theme. Each component and event is a naturally testable unit of code.

Our architecture gives us a lot of confidence in the results of backtests and never gets in our way when we are analyzing those results. It is a flexible framework in which we can easily model new strategies and quickly deploy them to trading.

If you have any questions or would like to learn more, feel free to reach out to us! We’d love to hear from you.