Don't Miss
Home » Articles » How to build a professional simulation environment – On the cheap

How to build a professional simulation environment – On the cheap

Do you want to learn how to build a professional environment for trading simulations without paying an arm and a leg?

Building a proper environment for developing and testing trading strategies can be very expensive. It doesn’t have to be though, as long as you’re willing to put in some work. I work with both cheap and expensive tools and if need be, I could make due with just the cheap ones.

The most common question I’m asked is how a retail trader or a semi pro can get started with strategy modeling. In this guide, I’m going to go ahead and assume that you’re comfortable with technology but lack budget to buy expensive software and data. If you’re not comfortable with the tech side, you really should learn it. This is one thing you just can’t get around. In this business, you do need to have a solid understanding of computers.
ClenowFuturesIntelligenceReport

Getting the Data

The first thing you need to do is get hold of the data you need. I use multiple vendors, but in the lower cost segment I prefer CSI Data. In fact, when it comes to futures data, I prefer them over the more expensive ones as well. I’ve been using Reuters terminals for as long as I can remember, but despite the steep price of those I would rather use CSI for futures data.

CSI has all the data coverage you’ll need in the futures space and their software will adjust the time series to your specifications. Adjusting the data is crucial, as I’ve discussed in previous articles.

CSI Data

CSI Data

I’m using the World Futures Gold Package for about 700 USD per year. That’s about as cheap as you can get for quality futures data. Oh wait, you can get slightly cheaper… If you enter coupon code Tradersplace, you’ll get a 15% discount. Yes, I get a commission on that too. I’ve been recommending, and using, CSI Data since long before I had that discount deal in place though and I like to think that making some pocket money on referrals does not violate my integrity. They’re simply good and I like them.

You can see all the different packages and options here.

I set the CSI software to dump daily flat files on disk. In the interface, I’ve selected all the fields I want, such as date, open, high, low, close, volume, open interest, current delivery, unadjusted close and a few other things. The program will fetch this data daily and just dump the files on disk.

 

CSI Data Settings

CSI also have equity data, which I’m told is of good quality but I can’t speak for it myself. I haven’t used their equity data yet, though I plan to give it a try. On equity strategies, I tend to look at broader geographical universes, and they seem quite focused on the US markets.

Storing the Data

I’m not a fan of working directly against flat files. It’s possible, but I like to have a little stronger data environment than that. It’s just much more convenient to work against real databases. My preference here is the completely free MySql Server. Set up the tables as you like them and as it makes sense to you.

I suggest to make a meta table and a history table. The meta table would be a lookup table, containing things like name, currency, point value, sector and any other metadata you might need. The history table would, well, contain the historical data you get from CSI.

My own solution for getting the data into the MySql is to have a little C# program I made myself do the job. This little program autostarts once a day and simply reads data from the CSI flat files and pushes them into the DB. To make sure the database is always in complete sync with the flat files, I simply flush the old data from the MySql daily and re-populate with the CSI Data.

MySql Tables

MySql Tables

And how do you set up the database and build a program to push the data from text files to db-tables? That you’ll have to do on your own. It’s very basic stuff and there’s plenty of guides on the internet on how to learn these kind of tasks. Database skills can be quite useful for quantitative traders.

Simulation Platform

I find it almost comical that I’ve been using so many platforms with extremely varying price points, and in the end my favorite one is also the cheapest. My strategy development is concentrated on one single platform which costs around 500 USD, once off. Yes, that’s right. No yearly subscription fees. In the light of this, I’m amazed that people pay thousands of dollars for competing retail level platforms.

The platform that I use is RightEdge. It’s a CLR based software capable of massive portfolio simulations and it can be customized without limits. That last part is very important to me. Almost all simulation platforms I’ve used hit a wall sooner or later. You reach a point where you want to do something that they just can’t do. For retail level platforms, you hit that wall one day one but even the more advanced platforms tend to have serious flaws.

RightEdge can do pretty much anything you’ll ever need to do. Sure, you might have to rebuild it and extend the functionality if you’re looking to do something exotic, but that’s part of the job. Even out of the box, it can do considerably more than most competing, and much more expensive, software packages.

You can write your simulations in C#, VB.Net or any other CLR language. The environment of RightEdge is very similar to Visual Studio, so if you’re comfortable with a compiler you should feel right at home. You can even write your strategies directly in Visual Studio if you so prefer.

If you’d like RightEdge to do something over the usual, just build it yourself. You can make your own DLLs in VS and change or extend the functionality as you like. My version is heavily modified.

Oh, and you get another discount here as well, by using the same coupon code: Tradersplace. That’ll lower the price by 10%, saving you $50. Again, I’ve been using and recommending this  platform for years and any referral fee I might get is no where near enough for me to recommend something I don’t use and like.

Now there is one downside with RightEdge that you should be aware of. This is a product for those who understand at least rudimentary programming or are willing to learn it. While RightEdge has a so called ‘drag-and-drop-system-building’, I wouldn’t recommend using it. Never use such a thing, in any environment. Anyone who tells you that you can build great trading systems that way is lying to you or doesn’t understand any better.

RightEdge Environment

RightEdge Environment

Getting the Data into RigthEdge

RightEdge, like most similar platforms, comes with a set of data adapters for various common sources. If you’re using a my recommended solution above however, you’ll need to make a data provider for it. Fear not, that’s not a terribly difficult task. If of course, you’re familiar with programming or willing to learn it. If you’re not, you’re probably better off investing with a good trader than doing it yourself anyhow. Programming is an essential skill for a quantitative trader.

The product comes with examples of how to build an adapter, and it’s quite straight forward. All you need to do is to make a little dll which implements some RightEdge interfaces, fetches data from your MySql database when asked to and returns it in the format that RightEdge expects. Dump this dll in the RightEdge folder, and you’re ready to start building your systems.

RightEdge Data Adapter

RightEdge Data Adapter

Your New Professional Simulation Environment

If you already understand basic programming and databases, you can have this setup up and running in under a day. If you’re not familiar with C#, I strongly recommend you to learn it. This book is a great place to start.

The environment shown here would make a strong, professional setup. It will allow you to construct serious model simulations without the limitations of retail level software. Best of all, it’s dirt cheap.

The downside is that it requires hard work. Very few things worth while can be done without hard work.

This article of course just brushes the surface how what do to and how to do it. It’s the first of this type of article I’ve published, and it’s in a way a test. If this kind of topic generates interest, I’ll go deeper and describe how the details work and how to more practically go about implementing this type of environment. Let me know if you want to see more how-to articles.

Shameless Plug

A month ago I started making my weekly futures research paper available for subscription. This report is designed to be a guiding tool for both systematic and discretionary futures traders, helping you to find opportunities and to identify risks. If you’re tired of sell side, broker style newsletters, this one is made by professionals for professionals. Sign up for a free month trial here.
ClenowFuturesIntelligenceReport

19 comments

  1. Good stuff. I’d love to read more from you on this topic.

  2. I really appreciate your guidance on all issues. Your term structure and building a simulation platform how to’s are amazing. Please keep it up. Thanks

  3. Thanks a lot for sharing this Andreas

  4. This type of guidance is rarely seen and incredibly useful. Let me know what I need to do to get you to write more of these articles. Thank you!

  5. Paulius Gedrimas

    My main coding experience lies in creating Databases on C#.NET. However apart from your blog and your book; trying to find books/websites to learn the basics of back-testing and indicator programming is more difficult than trying to teach my soviet-born grandparents how to use Skype. As mentioned, I do have programming experience, so I don’t need to learn from scratch; but are there any good books/websites which hold your hand to program your first indicators/strategies and import data? Any useful resources would be greatly appreciated.

  6. Thanks for sharing, Andreas! Very useful info!

  7. Definiitely interested in futher articles along this theme…

  8. Hello,
    This is a very nice article.
    I’m trying to build my own platforms from scratch (one purely for backtesting purposes, one for sending orders to my online broker), using C# and MySQL. I download futures data from quandl.com. I am a professional developer (but not in the same field).
    Have you ever tried to do it yourself ? Would you have some recommendation ?
    Many thanks in advance

    • If you do it for fun, then fine. Else, do you really need to build from scratch? Since you use C#, I would just start off with RightEdge and modify what you want to be different.

      I’ve never seen the need to build my own environment from scratch, but we do heavy modifications to existing platforms.

    • Thanks !!
      In fact, I want to do it by myself primilarly for fun, and also to master the whole thing, not being dependent on anything/anyone else. I’ve found an opensource project : “Tradelink” I ‘ll look at it, see if I can use their ideas.
      Thanks again !

    • Sounds like fun. If I had the time, I might look at doing the same. It’s the sort of thing I would do 10-15 years ago, but now I’m just looking to solve whatever problem is in front of me as effectively as possible.

      It would be interesting to hear your conclusions. Would be great if you want to report back later on about how you’re getting on with the project.

      I think there are plenty of things that would be great to have in simulation/trading software that no one seems to be doing. The developers are usually just copying what everyone else is doing and adding some improvements. It would be neat to see someone really start from scratch and build what makes sense. If you’re getting serious about this and if you’re interested, contact me for some ideas..

  9. Ok !
    This is just a project right now, there is really nothing at the time to show !
    I’ve traded stocks for years, never been successful … then I began selling weekly options on one ETF, this has worked a lot better – but there is too much emotion involved and two many decisions to make (adjust ? close ? roll ?…)
    So, why not let a computer perform all the trading decisions ? for that, I want to fully backtest my system, and understand **everything**- thus my desire to build from scratch (and because it is fun)

    So now, I’m focusing first on the “backtesting” solution.
    I have two main concerns right now :
    1. How to build a continuous series from the several contracts. Which rules make more sense ? I’m leaning on the Panama system, used by Ed Seykota, but doing it backwards to always have the right price now, and not in the past – but on which day perform the roll ? And there’s something that bothers me : in actual trading, prices won’t be continuous … if I’ll roll into another contract, I’ll suffer a gap, and backtesting won’t show it …
    2. Where can I store the indicators values (various EMAs, highest high or lowest low for the X last days…) : in database, or in RAM … in database it will take a long time to recalculate the past if I want to change a parameter – in RAM my PC might just freeze !

    This is where I am (at the begining you can see). Any help, or advice would be welcome!

    • You’re onto something here, Jerome.

      I’ve always found it stupid that simulation environments require a continues series. That adds an extra layer of complexity and source of error for the user, not to mention extra cost. It takes away flexibility and creates all sort of problems.

      What would make sense is that the simulation environment itself is aware of the issue. That you can feed the original contract data, each individual series with the real prices, volume, oi etc. The platform itself should have various ways of rolling built in, handled realistically with roll costs and all.

      Such logic would also open up for calendar strategies, analyzing term structure etc. I’m guessing that the only reason this is not done today is that application developers don’t understand the issue.

      As for how to store large volumes of data effectively, you’ll have to talk to better programmers than me. I’m a very pragmatic coder. I aim to get a task done, not to do it in the most efficient way. When you make software for other people, you would of course need to care more about such things.

      Are you planning on an open source project or a for-profit venture? Either approach is fine of course.

      If you’re serious about building something neat and if you’re interested, I could host a forum here for sharing ideas and helping out in the project. There might be others who can help as well, and especially if you’re going the open source route that might be of interest.

    • Concerning the rolling issue, I think the best solution is to generate trading signals with the continuous series, to avoid gaps, but, once a signal is given, trade the actual contract price, and when a contract ends, suffer the gap + commissions – this should and could be done in actual trading : it is the best solution I can think of, and is fully programmable in real time (if open interest is higher in the next contract and you’re long, sell your position, and buy the next contract, at the opening prices tomorrow)

      Concerning the data storage : I’ll put everything in the database – I don’t mind if a calculation takes time for backtesting …. I’ll guess I’ll have high-low-open-close for continuous data, high-low-open-close for the active contract traded, and volume and open interest, and reserve a dozen fields for indicator values. This would also enable me to backtest spread strategies (although I’m not familiar with them right now)

      Open source ?? I really haven’t thought about that. Being a husband and father, I need to go at my own pace (slowly) … going open source and make the code public surely adds some pressure. My original idea was to do it for strictly personal purposes : a simple program, with a large database, able to do a lot of “completely customized” backtest calculations, including position size, trailing stops, to simulate completely actual trading … but not as “easy to use” and user friendly as one would expect from a packaged solution… I’ll see how this whole thing goes ! Thanks again for your interest !

  10. Hi Andreas.

    What would be nice is if RightEdge was able to properly support the idea of a Bi-temporal FutureChain and could use the right data depending on what you were doing.

    ie. I add the LeanHogs FutureChain (as a single item not one for each future) to my Watchlist and when I am running my strategy I am either able to access the “top-level” continuous contract as currently OR I can query from this symbol the full future chain with all the details (openInt, volume etc) of each contract (as-of the date the strategy is trading on at that point). So if I am running back in 2010 it only returns me the list of futures that were available at that time. And when I choose to execute a trade I should be able to choose whether to use the top-level continuous contract (as we are all doing today) or the specific underlying contract and have the PL correctly calculated either way by using the correct timeseries. I would guess the best way to make this work in terms of loading data would be that you don’t load the data before you run your strategy, you have ability to load it in on the fly depending on which timeseries you want at any point.

    The problem I see currently is that I think in order for RightEdge to be aware and allow you to trade each individual maturity future you have to load all of them into the watchlist first and all of them with their own timeseries as well which could be automated but would become pretty unwieldy if you are backtesting 20+ years on 50 different products.

    I can think of ways to modify RightEdge to make parts of this work but Im not sure if you can do it properly without full access to the RightEdge source code. Is it possible to override in the symbol class the way that it loads price data to re-direct and have the idea of multiple different timeseries for the same “product” (ie. continuous and one for each maturity)?

    Is this something you have ever looked into?

    It would make trading the rolls far more realistic and allow you to code your roll-trading strategy yourself. As you also point out it would also open up term structure trades.

    I would be interested/happy to code this up and make it work if it was possible.

    Tks

    Dan