Don't Miss
Home » Articles » Trading Evolved – Errata and Updates

Trading Evolved – Errata and Updates

A book like Trading Evolved will without a doubt have errors. On top of errors, updates may also be needed as new versions of dependent software packages are released. I will keep a running log on this site of issues found, and suggested solutions. After this list grows long enough, I will update the book itself, and put a version number on it.

This article will be continuously updated.

 

Book Version 1.0

Version 1.0 of the book was released on August 8, 2019.

Unable to Install Zipline with Conda version 4.7.10

Conda version 4.7 seems to break some dependencies and may result in an inability to install Zipline using the syntax “conda install zipline -c quantopian”.

Workaround 1 – Downgrade Conda

https://github.com/quantopian/zipline/issues/2514

Workaround 2 – Install using pip

https://www.zipline.io/install.html#installing-with-pip

 

Typo in “Patching the Framework”

On page 96 there’s a typo and the wrong file name is stated. This was the very last thing I did on the book before shipping it, as the issue that needed patching appeared at the last moment.

The file to be changed is “benchmarks.py”, not “loader.py”.

Here is the complete, corrected version.

Zipline, like most software in Python world is not only free, but open source. You can see and edit all the code as you like, and one big advantage of this is that we can make changes and even fix bugs when needed. You may wonder why you would ever want to go and edit someone else’s code, but it can be very useful to have this possibility.

While writing this book, an interesting example of such a situation came up. One day, all the backtests suddenly stopped working, with an error message about failing to load benchmark data from some web address. As it turns out, there is a web call executed during backtests, where Zipline tries to pull data from an online source. This source, much like what happened to many other free data sources, suddenly stopped working without warning.

As of writing, this issue has not yet been addressed. But no worries, we can do it ourselves easily. We don’t actually need this benchmark data for anything that we will do in this book, and the easiest fix is to simply remove the web call and just return empty data.

My version of Zipline is 1.3, and if you’re working with the same version, you likely need to do this fix manually. It won’t take long.

First locate a file called benchmarks.py, which should be in your Zipline installation folder. You can use the file system search tools to locate it. If you are on Windows, the path will probably be similar to this.

C:\ProgramData\Anaconda3\envs\zip35\Lib\site-packages\zipline\data

It’s always a good idea to keep a backup of the original code, so rename this file to keep a copy. Then make a new file, with the name benchmarks.py, and put the following code in it before saving.

This code will simply return a data series from 1930 to 2030 with all zero values, which will effectively sidestep this web call issue, with no negative impact.

Addition to Patch

You may also need to bypass the cache in loader.py, in the same folder:

https://github.com/quantopian/zipline/issues/2480#issuecomment-504387554

 

Zipline Installation: failed with current_repodata.json

Error:

Collecting package metadata (current_repodata.json): done
Solving environment: failed with current_repodata.json, will retry with next repodata source.

Solution:

1) Update Conda to the latest version:

2) Install the free channel:

3) install Zipline:

 

Trouble Ingesting Quandl

Error:

“ValueError: Boolean array expected for the condition, not float64”

Solution:

Downgrading Pandas to 0.18.1

 

Jupyter version 5.0.0 not starting

Solution:

Downgrading “tornado” from 5.1.1 to 4.5.3

 

Trouble ingesting Quandl on Mac

Try

 

Typo Chapter 23 – Bundle Name

In chapter 23, you’re advised to ingest the random stocks bundle using

But that’s not the name of the bundle that we just created a few lines above. The name in used in the bundle was random_stock_data, so we need to use that name when ingesting.

Similar typo in regards to the futures bundle. The name of the file in the sample code is random_futures_data:

And

Note that you can call your bundles whatever you like, but the file names need to match the references, and the registered bundle name needs to match the ingest call.

338 comments

  1. Steven Petersen

    Good new book so far. The following are no big deal but though I’d mention them.

    “If the volatility is exactly half in Microsoft, you would theoretically achieve the same risk level if you invest twice as much in Tesla.” should be “…twice as much AS in Tesla” ?

    “You can create both a folder and a new Python file through that dropdown on the right side, which you can see in Figure 5‑2.” should be “…a new Python FOLDER” ? There is an option to create a file so I got confused.

    • Yes, that first one is unfortunate. Leaving out ‘as’ gave the sentence the opposite meaning…

      But the second sentence seems fine to me, unless I’m still missing something. It’s not a Python folder, it’s a regular folder that you can create. Or ‘directory’ if you’re so inclined.

      So you can create folders in Jupyter, as well as Jupyter Notebook python files. Click the dropdown, and you’ll see.

  2. hi Andreas, thank you for the book. Great book. Hope to be even better at backtesting in time to come. By the way, the sp500 downloaded did not show sp500 values but it showed all the components within instead. My understanding from the book is 2 columns, 1 date and the other would be values under sp500. Am i right to say that? Thank you for taking the time to read this.

  3. Zipline Installation: failed with current_repodata.json

    Missing a dash led to another error. For step 2 it should be:

    conda config –set restore_free_channel true

    I’m loving the book so far. I’m no coder, but have found my interest in investing a useful tool to motivate me to solve problems and tinker.

  4. https://github.com/ContinuumIO/anaconda-issues/issues/1423

    I had an issue trying to install the nb_conda package (pg.101) and while I didn’t save the error message it was something to do with nb_conda not being compatible with Python 3.5. Anyway I used the following command in the terminal from the link above to get it to install
    conda install -c conda-forge nb_conda

  5. Jupyter notebook not downloaded in my zip35 environment, open in jupyter greyed out, and cannot download notebook==6.0

    also can’t ingest quandl already downgraded pandas, or at least i think i did, just used

    conda install pandas=0.18.1

  6. which version of conda I should use to install zipline.

    First part of the message says
    “Workaround 1 – Downgrade Conda” , conda install conda=4.6.11

    The last part of the message says
    1) Update Conda to the latest version:

    What should I do?

  7. Just acquired “Trading Evolved” and want to thank you for your efforts in writing the book. I am finding it extremely useful in my quest to become proficient in using Python for financial analysis and equities trading.

    In the course of following along with the book in Jupyter Notebook I encountered an error in running “first Zipline backtest” as follows:

    ValueError: SQLite file ‘C:\\Users\\r1100/.zipline\\data\\quandl\\2019-08-21T20;57;11.306361\\assets-6.sqlite’ doesn’t exist

    So my question is what is this file and what is required to bring it into existence? Thank you.

    • That sounds like an error during bundle ingest. If you interrupt the ingest process, or if it for any reason fails, this error shows up. Try ingesting it again.

      Also, the Quandl bundle is very limited and I would very much recommend making your own bundle, for your own data, as explained in the later chapters in the book.

    • I don’t recall doing anything that would cause the ingest process to be interrupted. I will certainly try the ingest process again but with the verbose mode on (note I mostly use iPython) to get a better idea of what is going on should the error repeat. I think I will start making my own bundles – I prefer having more control over the overall process.

    • Oh, and be sure to start the kernel in the notebook after ingesting. In case you had the notebook running while doing the ingest, you’ll need to restart the kernel.

  8. Stephen Hankinson

    Great book Andreas. I’ve enjoyed all of your books to date.

    I noticed an issue with the dividend code when trying to import data with dividends. The divs.append method inside the process_stocks function is creating a local copy of the divs DataFrame which doesn’t apply the changes to the divs DataFrame in the random_stock_data function. You can confirm this by adding a print(divs) right before the adjustment_writer.write call, if using a dataset with dividends, and you’ll see the DataFrame being written is empty.

    A simple fix is to change the dividend check to the following:


    # If there's dividend data, add that to the dividend DataFrame
    if 'dividend' in df.columns:

    # Get the current highest index
    idx = len(divs)

    # Append this stock's dividends to the list of all dividends
    tmp = df[df['dividend'] != 0.0]['dividend']
    for ex_date, amount in tmp.iteritems():
    divs.loc[idx] = [sid, amount, ex_date, pd.NaT, pd.NaT, pd.NaT]
    idx = idx + 1

    Another thing I noticed when importing data from alphavantage is that the if your data is ordered newest to oldest you’ll get an error when importing it. This can be fixed by adding df.sort_index(inplace=True) right after the pd.read_csv call.

  9. https://github.com/quantopian/zipline/issues/1591

    im trapped here, please help, already changed the zipline-scrypt.py file, did not work, getting this

    Traceback (most recent call last):
    File “C:\Users\JAY\Anaconda3\envs\zip35\Scripts\zipline-script.py”, line 11, in
    load_entry_point(‘zipline==1.3.0’, ‘console_scripts’, ‘zipline’)()
    File “C:\Users\JAY\Anaconda3\envs\zip35\lib\site-packages\pkg_resources\__init__.py”, line 484, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
    File “C:\Users\JAY\Anaconda3\envs\zip35\lib\site-packages\pkg_resources\__init__.py”, line 2707, in load_entry_point
    return ep.load()
    File “C:\Users\JAY\Anaconda3\envs\zip35\lib\site-packages\pkg_resources\__init__.py”, line 2325, in load
    return self.resolve()
    File “C:\Users\JAY\Anaconda3\envs\zip35\lib\site-packages\pkg_resources\__init__.py”, line 2331, in resolve
    module = __import__(self.module_name, fromlist=[‘__name__’], level=0)
    File “C:\Users\JAY\Anaconda3\envs\zip35\lib\site-packages\zipline\__init__.py”, line 23, in
    from . import data
    File “C:\Users\JAY\Anaconda3\envs\zip35\lib\site-packages\zipline\data\__init__.py”, line 1, in
    from . import loader
    File “C:\Users\JAY\Anaconda3\envs\zip35\lib\site-packages\zipline\data\loader.py”, line 22, in
    from .benchmarks import get_benchmark_returns
    File “C:\Users\JAY\Anaconda3\envs\zip35\lib\site-packages\zipline\data\benchmarks.py”, line 23
    data = data[‘close’] return data.sort_index().iloc[1:]
    ^
    SyntaxError: invalid syntax.

    I’ve been stuck with setting up the environment for quite a while now

    • Hello Jay,

      It looks like a syntax error on line number 23 of the file benchmarks.py:

      File “C:\Users\JAY\Anaconda3\envs\zip35\lib\site-packages\zipline\data\benchmarks.py”, line 23
      data = data[‘close’] return data.sort_index().iloc[1:]
      ^
      SyntaxError: invalid syntax.

      I think you have the return statement in the same line as data = data[‘close’]

      Please change the content of this file “benchmark.py” exactly as follows:

      import pandas as pd
      from datetime import datetime
      from trading_calendars import get_calendar

      def get_benchmark_returns(symbol):
      cal = get_calendar(‘NYSE’)
      first_date = datetime(1930,1,1).date()
      last_date = datetime(2030,1,1).date()
      dates = cal.sessions_in_range(first_date,last_date)
      data = pd.DataFrame(0.0,index=dates,columns=[‘close’])
      data = data[‘close’]
      return data.sort_index().iloc[1:]

      • I put down the book for two months after not being able to crack this, now it got fixed on its own, but now the kernel in my jupyter died(for reasons i cant understand), had to delete the whole environment and now I’m stuck here again, frustrating, I’ve tried everything in this forum no solution, I’m stuck

      • I’m getting the same syntax error, but it has not resolved with the code you provided Nilesh. Could it be that one or more lines of code need to be indented?

      • Please disregard my last message (unfortunately cannot rescind it). I have solved the problem, as follows:

        Clenow’s instruction in Chapter 7 to revise the file benchmarks.py is no longer necessary

        Using the original benchmarks.py I was able to have the data ingested without a problem

        Hopefully this helps others who have been having this same issue as Jay Hastings and I have!

  10. Hi Andreas,

    Thank you very much for writing this book.

    When attempting to run the code in “Your First Zipline Backtest”, I get an error when trying to run the moving average window that says “NameError: name ‘context’ is not defined”.

    I’m sure this is something basic that I am missing, though it stops the whole thing from working. Would you happen to know what I am missing in this case.

    Thanks,
    James

  11. Working on a mac.

    After ingesting the quandl bundle, I ran the first zipline backtest and got the following error:

    ValueError: No JSON object could be decoded

    • I too am working on a mac, Mac OS X. Did you ever find a solution to this problem and were there any other obstacles that you encountered with installation or implementation on a Mac?

      I know from past experience trying to follow tutorials that use a Windows machine by trying to implement the same steps on my Mac can be challenging. So, any insight into this would go a long way for me.

      Thank you in advance.
      Lee

    • Hi Caleb – were you ever able to fix this ValueError: No JSON object could be decoded issue with the first backtest? I am encountering the same issue on Windows 10

      • Yes, I was, but I don’t recall exactly what was causing it. If you follow exactly the steps that Kristof Friedrich outlines below in his installation, I believe you should be able to get it running.

      • I had the same issue but got fixed by changing benchmark.py and loader.py files as mentioned in the thread https://github.com/quantopian/zipline/issues/2480

        First, replace benchmarks.py with:

        import pandas as pd
        from trading_calendars import get_calendar

        def get_benchmark_returns(symbol, first_date, last_date):
        cal = get_calendar(‘NYSE’)

        dates = cal.sessions_in_range(first_date, last_date)

        data = pd.DataFrame(0.0, index=dates, columns=[‘close’])
        data = data[‘close’]

        return data.sort_index().iloc[1:]
        Then in loaders.py, replace
        data = get_benchmark_returns(symbol)
        with
        data = get_benchmark_returns(symbol, first_date, last_date)

        At the end don’t forget to restart anaconda and Jupyter notebook

  12. Hey Andreas, great book so far.

    In regard to the ‘Making a correlation Graph’, my correlation graph came out a bit different compared to the one showed on the book. (more spikes)

    I used the set of data you provided which is ‘indexes’ in chapter 6 pandas folder.

    Don’t know if it’s the matplotlib format/version or its the way the dataset in computed. Should we expect similar differences throughout the book?

    Thank you.

  13. Hi Andreas, great book and i am digging my way through it. Right stuff at the right time so to speak. Got environment set up on desktop – check! However, laptop is posing and issue re Quandl – despite all same as desktop, i am getting the dreaded “failed to create process”. i tried your suggested remedy of apply a double does of “” to the relevant file but alas…still same result ie “failed t0 create process”. Do you have other good ideas of where to look/what to do? Thanks in advance.

    Michael

    • Hi Michael,

      This is most likely an issue regarding installation path and probably there are spaces in your path. There’s a thread somewhere in the Zipline GitHub about this, and the solution likely involves modding the path with double quotes. I understand you tried this, but it’s probably something highly related.

      Sorry I can’t be more specific, but it’s difficult without actually seeing your exact setup. I’m pretty sure you’re on the right track though. This is a classic error relating to Windows and spaces in the path to the interpreter.

  14. Hello Andreas thanks for the book,

    On page 96 what do you mean “You may also need to bypass the cache in loader.py”?
    Do you mean override the whole file or just the section of the code between the triple quotes?

    Thanks

  15. Hey Andreas,

    You are commendable for writing this book, it is well written, clear, and exactly what young python developers are looking for!

    Was wondering if you, or anyone else watching this thread would be able to assist here. I am having the same issue as mentioned on this github post (https://github.com/quantopian/zipline/issues/2532), and as insult to injury I am unable to downgrade Anaconda to 4.6.11. It looks like this issue has arisen fairly recently based on what the community is saying.

    Has anyone else had this issue?

  16. I can not get past this error, JSONDecodeError: Expecting value: line 1 column 1 (char 0). I’ve read all the github threads, and have tried changing the benchmarks.py and loaders.py files, but to no avail.

    Any suggestions, on mac BTW.

    Thanks

    • Same problem here and I’m using Windows. This happened while trying the first zipline backtest:

      JSONDecodeError Traceback (most recent call last)
      in ()
      53 capital_base = 10000,
      54 data_frequency = ‘daily’,
      —> 55 bundle = ‘quandl’)

    • HELP!!!

      —————————————————————————
      JSONDecodeError Traceback (most recent call last)
      in
      53 capital_base = 10000,
      54 data_frequency = ‘daily’,
      —> 55 bundle = ‘quandl’)

      ~\Anaconda3\envs\zpline35\lib\site-packages\zipline\utils\run_algo.py in run_algorithm(start, end, initialize, capital_base, handle_data, before_trading_start, analyze, data_frequency, data, bundle, bundle_timestamp, trading_calendar, metrics_set, default_extension, extensions, strict_extensions, environ, blotter)
      428 local_namespace=False,
      429 environ=environ,
      –> 430 blotter=blotter,
      431 )

      ~\Anaconda3\envs\zpline35\lib\site-packages\zipline\utils\run_algo.py in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, data, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, environ, blotter)
      157 trading_calendar=trading_calendar,
      158 trading_day=trading_calendar.day,
      –> 159 trading_days=trading_calendar.schedule[start:end].index,
      160 )
      161 first_trading_day =\

      ~\Anaconda3\envs\zpline35\lib\site-packages\zipline\finance\trading.py in __init__(self, load, bm_symbol, exchange_tz, trading_calendar, trading_day, trading_days, asset_db_path, future_chain_predicates, environ)
      101 trading_day,
      102 trading_days,
      –> 103 self.bm_symbol,
      104 )
      105

      ~\Anaconda3\envs\zpline35\lib\site-packages\zipline\data\loader.py in load_market_data(trading_day, trading_days, bm_symbol, environ)
      147 # date so that we can compute returns for the first date.
      148 trading_day,
      –> 149 environ,
      150 )
      151 tc = ensure_treasury_data(

      ~\Anaconda3\envs\zpline35\lib\site-packages\zipline\data\loader.py in ensure_benchmark_data(symbol, first_date, last_date, now, trading_day, environ)
      214
      215 try:
      –> 216 data = get_benchmark_returns(symbol)
      217 data.to_csv(get_data_filepath(filename, environ))
      218 except (OSError, IOError, HTTPError):

      ~\Anaconda3\envs\zpline35\lib\site-packages\zipline\data\benchmarks.py in get_benchmark_returns(symbol)
      33 ‘https://api.iextrading.com/1.0/stock/{}/chart/5y’.format(symbol)
      34 )
      —> 35 data = r.json()
      36
      37 df = pd.DataFrame(data)

      ~\Anaconda3\envs\zpline35\lib\site-packages\requests\models.py in json(self, **kwargs)
      895 # used.
      896 pass
      –> 897 return complexjson.loads(self.text, **kwargs)
      898
      899 @property

      ~\Anaconda3\envs\zpline35\lib\json\__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
      317 parse_int is None and parse_float is None and
      318 parse_constant is None and object_pairs_hook is None and not kw):
      –> 319 return _default_decoder.decode(s)
      320 if cls is None:
      321 cls = JSONDecoder

      ~\Anaconda3\envs\zpline35\lib\json\decoder.py in decode(self, s, _w)
      337
      338 “””
      –> 339 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      340 end = _w(s, end).end()
      341 if end != len(s):

      ~\Anaconda3\envs\zpline35\lib\json\decoder.py in raw_decode(self, s, idx)
      355 obj, end = self.scan_once(s, idx)
      356 except StopIteration as err:
      –> 357 raise JSONDecodeError(“Expecting value”, s, err.value) from None
      358 return obj, end

      JSONDecodeError: Expecting value: line 1 column 1 (char 0)

      • This is the error if u did not applied the patch for zipline Andreas mentioned in the book. Also it is in erratas ib this site.
        Still not sure loader.py needs comment out patch since it seems redundant but I do not know zipline much so I applied.
        Once you apply patch, it should be fine.

    • Hi Stephen.

      I finally managed to get passed that problem by also adding:

      from trading_calendars import get_calendar
      from datetime import datetime

      to the benchmarks.py fix that Andreas provided. So the full code would become:

      import pandas as pd
      import requests
      from trading_calendars import get_calendar
      from datetime import datetime

      def get_benchmark_returns(symbol):
      cal = get_calendar(‘NYSE’)
      first_date = datetime(1930,1,1)
      last_date = datetime(2030,1,1)
      dates = cal.sessions_in_range(first_date, last_date)
      data = pd.DataFrame(0.0, index=dates, columns=[‘close’])
      data = data[‘close’]
      return data.sort_index().iloc[1:]

      Moreover, please note that the loader.py contains the following statement twice, so make sure you find them both and apply the triple quotes to bypass them.
      “””
      if data is not None:
      return data
      “””

      I hope that works for you too.

      • Dear Friends,
        I have a similar problem with First Zipline Backtest:

        —————————————————————————
        JSONDecodeError Traceback (most recent call last)
        in ()
        71 handle_data=handle_data,
        72 capital_base=10000,
        —> 73 data_frequency = ‘daily’, bundle=’quandl’
        74 )

        ~\AppData\Local\Continuum\anaconda3\envs\zip35\lib\site-packages\zipline\utils\run_algo.py in run_algorithm(start, end, initialize, capital_base, handle_data, before_trading_start, analyze, data_frequency, data, bundle, bundle_timestamp, trading_calendar, metrics_set, default_extension, extensions, strict_extensions, environ, blotter)
        428 local_namespace=False,
        429 environ=environ,
        –> 430 blotter=blotter,
        431 )

        ~\AppData\Local\Continuum\anaconda3\envs\zip35\lib\site-packages\zipline\utils\run_algo.py in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, data, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, environ, blotter)
        157 trading_calendar=trading_calendar,
        158 trading_day=trading_calendar.day,
        –> 159 trading_days=trading_calendar.schedule[start:end].index,
        160 )
        161 first_trading_day =\

        ~\AppData\Local\Continuum\anaconda3\envs\zip35\lib\site-packages\zipline\finance\trading.py in __init__(self, load, bm_symbol, exchange_tz, trading_calendar, trading_day, trading_days, asset_db_path, future_chain_predicates, environ)
        101 trading_day,
        102 trading_days,
        –> 103 self.bm_symbol,
        104 )
        105

        ~\AppData\Local\Continuum\anaconda3\envs\zip35\lib\site-packages\zipline\data\loader.py in load_market_data(trading_day, trading_days, bm_symbol, environ)
        147 # date so that we can compute returns for the first date.
        148 trading_day,
        –> 149 environ,
        150 )
        151 tc = ensure_treasury_data(

        ~\AppData\Local\Continuum\anaconda3\envs\zip35\lib\site-packages\zipline\data\loader.py in ensure_benchmark_data(symbol, first_date, last_date, now, trading_day, environ)
        215
        216 try:
        –> 217 data = get_benchmark_returns(symbol)
        218 data.to_csv(get_data_filepath(filename, environ))
        219 except (OSError, IOError, HTTPError):

        ~\AppData\Local\Continuum\anaconda3\envs\zip35\lib\site-packages\zipline\data\benchmarks.py in get_benchmark_returns(symbol)

        ~\AppData\Local\Continuum\anaconda3\envs\zip35\lib\site-packages\requests\models.py in json(self, **kwargs)
        895 # used.
        896 pass
        –> 897 return complexjson.loads(self.text, **kwargs)
        898
        899 @property

        ~\AppData\Local\Continuum\anaconda3\envs\zip35\lib\json\__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
        317 parse_int is None and parse_float is None and
        318 parse_constant is None and object_pairs_hook is None and not kw):
        –> 319 return _default_decoder.decode(s)
        320 if cls is None:
        321 cls = JSONDecoder

        ~\AppData\Local\Continuum\anaconda3\envs\zip35\lib\json\decoder.py in decode(self, s, _w)
        337
        338 “””
        –> 339 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
        340 end = _w(s, end).end()
        341 if end != len(s):

        ~\AppData\Local\Continuum\anaconda3\envs\zip35\lib\json\decoder.py in raw_decode(self, s, idx)
        355 obj, end = self.scan_once(s, idx)
        356 except StopIteration as err:
        –> 357 raise JSONDecodeError(“Expecting value”, s, err.value) from None
        358 return obj, end

        JSONDecodeError: Expecting value: line 1 column 1 (char 0)

        Benchmarks.py:
        import pandas as pd
        from trading_calendars import get_calendar
        from datetime import datetime

        def get_benchmark_returns(symbol):
        cal = get_calendar(‘NYSE’)
        first_date = datetime(1930,1,1)
        last_date = datetime(2030,1,1)
        dates = cal.sessions_in_range(first_date, last_date)
        data = pd.DataFrame(0.0, index=dates, columns=[‘close’])
        data = data[‘close’]
        return data.sort_index().iloc[1:]

        I use Windows 10, I’ve read all the github threads, but without succes. Hints are welcome! Many Thanks”

    • Same problem here with the JSON decode error. No fixes seem to work. I’m just testing the First Zipline Backtest code, got it all good including ingesting quandl on Mac Catalina but continue to get:

      ~/opt/anaconda3/envs/zip35/lib/python3.5/json/decoder.py in raw_decode(self, s, idx)
      355 obj, end = self.scan_once(s, idx)
      356 except StopIteration as err:
      –> 357 raise JSONDecodeError(“Expecting value”, s, err.value) from None
      358 return obj, end

      JSONDecodeError: Expecting value: line 1 column 1 (char 0)

      • After adopting changes in benchmarks.py and loader.py as recommended, I am not getting the error:

        File “”, line 3
        cal=get_calendar(‘NYSE’)
        ^
        SyntaxError: invalid character in identifier

        Which seems impossible.

      • Alas there ARE hidden characters in benchmarks.py. Clean them up and things work fine. Thank you — me — for solving!

        fwiw

      • @tdetz: Were you able to find a work around? I am also getting SyntaxError: invalid character in identifier with some gibberish character

      • open benchmarks.py in any code editor and remove all spaces manually and then indent properly ,this helped me
        if still u have any issue you can mail me
        rockey.pandit98@gmail.com

  17. Thank you for this community here. I am currently facing a zipline issue. Used to work okay when I followed the instructions in chapter 7. Ingested the customized bundle as mentioned in chapter 23 but it did not work. Was unsure why so I uninstall anaconda and reinstalled everything. The same issue remains, zipline does not seem to be available/working.

    (zip35) C:\Users\chuam>zipline ingest -b quandl
    ‘zipline’ is not recognized as an internal or external command,
    operable program or batch file.

    The above is what I was given from the terminal. Not sure what is going on. Appreciate your advice. I have browsed through various forums the past one week but to no avail. Look forward to overcoming this. Thank you once again Andreas!

  18. Hello Andreas,
    Thank you very much for the book!
    I have a question for you, I have an API which gives me access to intraday data for futures and equities and I would like to have zipline connected to my API (which is from TradeStation).
    Could you please show me where I could find a guide for doing that? I am not a programmer and the chapter “Importing your data” in the book does not report the example.
    Thank you very much.

    Kind regards,
    Mark

  19. For the code in chapter 15, I keep getting AttributeError: ‘BundleData’ object has no attribute ‘equity_bar_reader’. This is after ingesting the random_futures_data bundle, editing the extensions.py file to register the bundle, and then patching the framework.

  20. Trying to use norgate data-zipline to implement your momentum 50 . but when I tried to get symbol(‘SPY’) or symbol(‘TLT’). it shows SymbolNotFound: Symbol ‘SPY’ was not found.

  21. Hi Andreas,

    Enjoying the book! I am using csi data, but having trouble reproducing results from the futures models (focusing on Time Return model since it is the simplest). Equity models are working fine.

    One issue at least is I have not been successful in creating a bundle with correct price_multiplier information. I have adapted the random_futures_bundle.py to include this in the metadata df and included a print(metadata) so I can see the info when I ingest. The print provides the correct price_multiplier. But for some reason running the Time Return model, the algo uses default price_multiplier = 1.0.

    Any idea how where the issue might be?

  22. Hi Anreas,

    Great book.
    Something I have always wanted to do. (Test systems with python)
    Now I am much much closer.
    Just wondering if you used norgate index constituents in back testing any models?
    I’m having some troubles but will try figure it out.

    • The Norgate Python API wasn’t ready in time, so i didn’t use that for the book. Their data is good though and I’m sure their support can figure it out for you.

    • Sam,

      Have you figured this one out yet? I’m having the same issues where the Norgate index constituents filters load an emply list into the function, and therefore return false for everything.

      • If you want worked examples from Trading Evoved that use Norgate Data’s capabilities (index constituents etc.) just contact us (Norgate).

        Happy to provide a zip file with Jupyter notebooks that show how it all works.

        It’s really quite neat.

  23. Im on page 106 – Your First Zipline Backtest

    When running the complete source code i downloaded, i run into these errors:

    Any one can point me to the direction to fix this? Thanks!

    —————————————————————————
    NameError Traceback (most recent call last)
    in ()
    71 handle_data=handle_data,
    72 capital_base=10000,
    —> 73 data_frequency = ‘daily’, bundle=’quandl’
    74 )

    C:\Users\Anaconda3\envs\zip35v2\lib\site-packages\zipline\utils\run_algo.py in run_algorithm(start, end, initialize, capital_base, handle_data, before_trading_start, analyze, data_frequency, data, bundle, bundle_timestamp, trading_calendar, metrics_set, default_extension, extensions, strict_extensions, environ, blotter)
    428 local_namespace=False,
    429 environ=environ,
    –> 430 blotter=blotter,
    431 )

    C:\Users\Anaconda3\envs\zip35v2\lib\site-packages\zipline\utils\run_algo.py in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, data, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, environ, blotter)
    157 trading_calendar=trading_calendar,
    158 trading_day=trading_calendar.day,
    –> 159 trading_days=trading_calendar.schedule[start:end].index,
    160 )
    161 first_trading_day =\

    C:\Users\Anaconda3\envs\zip35v2\lib\site-packages\zipline\finance\trading.py in __init__(self, load, bm_symbol, exchange_tz, trading_calendar, trading_day, trading_days, asset_db_path, future_chain_predicates, environ)
    101 trading_day,
    102 trading_days,
    –> 103 self.bm_symbol,
    104 )
    105

    C:\Users\Anaconda3\envs\zip35v2\lib\site-packages\zipline\data\loader.py in load_market_data(trading_day, trading_days, bm_symbol, environ)
    147 # date so that we can compute returns for the first date.
    148 trading_day,
    –> 149 environ,
    150 )
    151 tc = ensure_treasury_data(

    C:\Users\Anaconda3\envs\zip35v2\lib\site-packages\zipline\data\loader.py in ensure_benchmark_data(symbol, first_date, last_date, now, trading_day, environ)
    215
    216 try:
    –> 217 data = get_benchmark_returns(symbol)
    218 data.to_csv(get_data_filepath(filename, environ))
    219 except (OSError, IOError, HTTPError):

    C:\Users\Anaconda3\envs\zip35v2\lib\site-packages\zipline\data\benchmarks.py in get_benchmark_returns(symbol)
    21 Andreas Solution to side step the problem
    22 “””
    —> 23 cal = get_calendar(‘NYSE’)
    24 first_date = datetime(1930,1,1)
    25 last_date = datetime(2030,1,1)

    NameError: name ‘get_calendar’ is not defined

    • i managed to troubleshoot the issue. i’ve forgotten to import get_calendar when patching the benchmarks.py

      silly me.

  24. Hi, Andreas. I purchased your kindle book. It is a great one. Really like it. Appreciate your effort putting on the book.
    I have a question on charter 12. For calculating annual slope, your code is ann_slope = (np.power(np.exp(slope), 252) -1)*100. Should it be ann_slope = (np.power(1+np.exp(slope)), 252) -1)*100?
    I will appreciate your response.

  25. Jose Andres Riveros

    Hi Andreas and everybody, Im currently now installing as it says in page 100, Matplotlib and nb_conda in my zip35 enviroment. i was able to install Matplotlib but when i try to do the same with nb_conda im getting this error it seems there is come incompatibility about the versions but in im doing it in the zip35 enviroment as supposed to, no clue how to solve this one appreciate in advance any help regarding this issue.

    UnsatisfiableError: The following specifications were found
    to be incompatible with the existing python installation in your environment:

    – nb_conda -> python[version=’>=3.6,=3.7, nb_conda_kernels[version=’>=2.0.0′] -> notebook[version=’>=4.2.0′] -> terminado[version=’>=0.8.1′] -> pywinpty -> python[version=’>=3.6,=3.7, pip -> wheel -> setuptools
    python=3.5 -> pip -> wheel -> setuptools
    Package wheel conflicts for:
    python=3.5 -> pip -> wheel
    nb_conda -> nb_conda_kernels[version=’>=2.0.0′] -> notebook[version=’>=4.2.0′] -> terminado[version=’>=0.8.1′] -> pywinpty -> python[version=’>=3.6,=3.7, pip -> wheel
    Package pip conflicts for:
    python=3.5 -> pip
    nb_conda -> nb_conda_kernels[version=’>=2.0.0′] -> notebook[version=’>=4.2.0′] -> terminado[version=’>=0.8.1′] -> pywinpty -> python[version=’>=3.6,=3.7, pip
    Package colorama conflicts for:
    python=3.5 -> pip -> colorama
    nb_conda -> nb_conda_kernels[version=’>=2.0.0′] -> notebook[version=’>=4.2.0′] -> terminado[version=’>=0.8.1′] -> pywinpty -> python[version=’>=3.6,=3.7, pip -> colorama
    Package wincertstore conflicts for:
    python=3.5 -> pip -> wheel -> setuptools -> wincertstore[version=’>=0.2′]
    nb_conda -> nb_conda_kernels[version=’>=2.0.0′] -> notebook[version=’>=4.2.0′] -> terminado[version=’>=0.8.1′] -> pywinpty -> python[version=’>=3.6,=3.7, pip -> wheel -> setuptools -> wincertstore[version=’>=0.2′]
    Package certifi conflicts for:
    nb_conda -> nb_conda_kernels[version=’>=2.0.0′] -> notebook[version=’>=4.2.0′] -> terminado[version=’>=0.8.1′] -> pywinpty -> python[version=’>=3.6,=3.7, pip -> cachecontrol -> requests -> urllib3[version=’>=1.21.1,=1.21.1,=1.21.1, certifi
    python=3.5 -> pip -> cachecontrol -> requests -> urllib3[version=’>=1.21.1,=1.21.1,=1.21.1, certifi
    Package six conflicts for:
    nb_conda -> nb_conda_kernels[version=’>=2.0.0′] -> notebook[version=’>=4.2.0′] -> terminado[version=’>=0.8.1′] -> pywinpty -> python[version=’>=3.6,=3.7, pip -> cachecontrol -> requests -> urllib3[version=’>=1.21.1,=1.21.1,=1.21.1, pyopenssl[version=’>=0.14′] -> cryptography[version=’>=1.9,>=2.2.1′] -> six[version=’>=1.4.1′]
    python=3.5 -> pip -> cachecontrol -> requests -> urllib3[version=’>=1.21.1,=1.21.1,=1.21.1, pyopenssl[version=’>=0.14′] -> cryptography[version=’>=1.9,>=2.2.1′] -> six[version=’>=1.4.1′]

    not pretty sure what the issue is in here

    • Thomas N. Helget

      Jose:

      I had a similar problem with both matplotlib and nb_conda:

      UnsatisfiableError: The following specifications were found
      to be incompatible with the existing python installation in your environment:

      – matplotlib -> python[version=’>=2.7,=3.5,=3.6,=3.7, python[version=’>=2.7, pip
      nb_conda -> nb_conda_kernels[version=’>=2.0.0′] -> notebook[version=’>=4.2.0′] -> prometheus_client -> twisted -> service_identity[version=’>=18.1.0′] -> attrs[version=’>=16.0.0′] -> hypothesis -> enum34 -> python[version=’>=2.7, pip
      matplotlib -> numpy -> mkl_random[version=’>=1.0.2, mkl-service[version=’>=2, six -> python[version=’>=2.7,=3.5,=3.6,=3.7, pip
      Package wincertstore conflicts for:
      python=3.5 -> pip -> wheel -> setuptools -> wincertstore[version=’>=0.2′]
      nb_conda -> nb_conda_kernels[version=’>=2.0.0′] -> notebook[version=’>=4.2.0′] -> prometheus_client -> twisted -> service_identity[version=’>=18.1.0′] -> attrs[version=’>=16.0.0′] -> hypothesis -> enum34 -> python[version=’>=2.7, pip -> wheel -> setuptools -> wincertstore[version=’>=0.2′]
      matplotlib -> numpy -> mkl_random[version=’>=1.0.2, mkl-service[version=’>=2, six -> python[version=’>=2.7,=3.5,=3.6,=3.7, pip -> wheel -> setuptools -> wincertstore[version=’>=0.2′]
      Package colorama conflicts for:
      python=3.5 -> pip -> colorama
      nb_conda -> nb_conda_kernels[version=’>=2.0.0′] -> notebook[version=’>=4.2.0′] -> prometheus_client -> twisted -> service_identity[version=’>=18.1.0′] -> attrs[version=’>=16.0.0′] -> hypothesis -> enum34 -> python[version=’>=2.7, pip -> colorama
      matplotlib -> numpy -> mkl_random[version=’>=1.0.2, mkl-service[version=’>=2, six -> python[version=’>=2.7,=3.5,=3.6,=3.7, pip -> colorama
      Package six conflicts for:
      nb_conda -> nb_conda_kernels[version=’>=2.0.0′] -> notebook[version=’>=4.2.0′] -> prometheus_client -> twisted -> service_identity[version=’>=18.1.0′] -> attrs[version=’>=16.0.0′] -> hypothesis -> enum34 -> python[version=’>=2.7, pip -> cachecontrol -> requests -> urllib3[version=’>=1.21.1,=1.21.1, pyopenssl[version=’>=0.14′] -> cryptography[version=’>=1.9,>=2.2.1′] -> six[version=’>=1.4.1′]
      matplotlib -> numpy -> mkl_random[version=’>=1.0.2, mkl-service[version=’>=2, six
      python=3.5 -> pip -> cachecontrol -> requests -> urllib3[version=’>=1.21.1,=1.21.1, pyopenssl[version=’>=0.14′] -> cryptography[version=’>=1.9,>=2.2.1′] -> six[version=’>=1.4.1′]
      Package setuptools conflicts for:
      nb_conda -> nb_conda_kernels[version=’>=2.0.0′] -> notebook[version=’>=4.2.0′] -> prometheus_client -> twisted -> service_identity[version=’>=18.1.0′] -> attrs[version=’>=16.0.0′] -> hypothesis -> enum34 -> python[version=’>=2.7, pip -> wheel -> setuptools
      matplotlib -> numpy -> mkl_random[version=’>=1.0.2, mkl-service[version=’>=2, six -> python[version=’>=2.7,=3.5,=3.6,=3.7, pip -> wheel -> setuptools
      python=3.5 -> pip -> wheel -> setuptools
      Package wheel conflicts for:
      python=3.5 -> pip -> wheel
      nb_conda -> nb_conda_kernels[version=’>=2.0.0′] -> notebook[version=’>=4.2.0′] -> prometheus_client -> twisted -> service_identity[version=’>=18.1.0′] -> attrs[version=’>=16.0.0′] -> hypothesis -> enum34 -> python[version=’>=2.7, pip -> wheel
      matplotlib -> numpy -> mkl_random[version=’>=1.0.2, mkl-service[version=’>=2, six -> python[version=’>=2.7,=3.5,=3.6,=3.7, pip -> wheel
      Package certifi conflicts for:
      matplotlib -> numpy -> mkl_random[version=’>=1.0.2, mkl-service[version=’>=2, six -> python[version=’>=2.7,=3.5,=3.6,=3.7, pip -> cachecontrol -> requests -> urllib3[version=’>=1.21.1,=1.21.1, certifi
      python=3.5 -> pip -> cachecontrol -> requests -> urllib3[version=’>=1.21.1,=1.21.1, certifi
      nb_conda -> nb_conda_kernels[version=’>=2.0.0′] -> notebook[version=’>=4.2.0′] -> prometheus_client -> twisted -> service_identity[version=’>=18.1.0′] -> attrs[version=’>=16.0.0′] -> hypothesis -> enum34 -> python[version=’>=2.7, pip -> cachecontrol -> requests -> urllib3[version=’>=1.21.1,=1.21.1, certifi

  26. Thanks for the very informative book and for releasing the kindle version at such a reasonable price.

    Just for info I’ve managed to get the first zipline backtest (Chapter 7) working on Windows 10 using the stock Python 3.5 and pip and avoiding the need for anaconda (in case anyone was thinking of going that way and wondered if it was possible).

    Cheers,
    Jamie.

  27. Dear Andreas,

    Thank you for producing another great book! I have very much enjoyed reading all three of your books.

    I had trouble ingesting futures data into zipline (both your data, as well as my own). No errors were produced during ingestion, but when running an algorithm, switching from contract to contract did not work correctly. Regardless of roll method, Zipline was selecting the next contract in the order that it occurred in my file directory! In my case, for instance, it was rolling from March 2017 to March 2018, instead of to June 2018.

    The zipline docs could be better regarding futures ingestion, so I had to look through the zipline code, which was a bit complicated. As far as I can see, rather than checking all live contracts to determine which to roll to next, Zipline seems to only check the next unexpired SID in the list for a given symbol root and wait to roll that one. So if the contracts are out of order, there will be problems.

    A straightforward solution is to adjust your ingest code to sort the file directory list by a combined string of delivery year and delivery month (assuming they are included in the filename) before passing the list on to the rest of the code as normal. This appears to work as expected, but the Zipline library is quite complex and I could easily be missing something else. If you have any further info regarding custom futures ingestion, please advise!

    I hope this may help some of you other readers and save them some grief!

    Thanks again for producing such great books!.

    All the Best,
    Paul

    • Hey Paul, I think I might be running into a similar (the same) issue as you. I am trying to replicate the sample futures backtests that Andreas has in the book. With all of them, I am error’ing out with an issue stitching the continuous futures together, specially in the “roll_finder.py”.
      At the top of the error output:
      —–

      Exception ignored in: ‘zipline.assets.continuous_futures.OrderedContracts.contract_before_auto_close’
      AttributeError: ‘NoneType’ object has no attribute ‘next’

      —–
      But, specifically, at the end of the output:
      —–

      39 session = self.trading_calendar.minute_to_session_label(dt)
      40 front = oc.contract_before_auto_close(session.value)
      —>41 back = oc.contract_at_offset(front, 1, dt.value)
      42 if back is None:
      43 return front
      with the additional trailing detail:
      ~\zipline\assets\continuous_futures.pyx in zipline.assets.continuous_futures.OrderedContracts.contract_at_offset (zipline/assets\continuous_futures.c:7309)()

      ~\zipline\assets\continuous_futures.pyx in zipline.assets.continuous_futures.OrderedContracts.contract_at_offset (zipline/assets\continuous_futures.c:7061)()

      KeyError: 0

      —–
      Is this what you are encountering? I am completely stymied at this point and am wondering if I have found someone with the same problem! I’m continuing to work on this and will of course report any progress… would welcome anyone else’s too! Cheers

      • Hi Devon,

        I’m not sure if I got that error specifically, but I expect we had the same problem.

        When digging through that part of the Zipline code I couldn’t see where OrderedContracts actually did any ordering! It seemed to me that OrderedContracts expected the contracts to already be in order! But I could have missed something.

        Everything seems to be OK for me now once I made sure that I ingested the futures contracts in expiration order. Prior to that, most all of the functionality related to futures and continuous futures was not working properly. Of course, one must make all the changes that Andreas talks about in his book (constants.py for instance).

        If you have ordered the contracts and you are still having problems, feel free to let me know as I did make one or two other (presumably) minor changes to the ingest code during the process of solving this issue which seemingly didn’t make a difference to the outcome at the time, but may have actually played a role in fixing the issue.

        Best of luck!
        Paul

      • Hello Devon,

        I have the same issue as you, both in the Chapter 15 and Chapter 18 in the book. I tried much time to fix it according to the related methods here. But still can’t solve it.
        Could you provide the method if you make it clear of this problem ? Thank you very much.

        Best regards,
        Yong
        fluid@126.com

      • Did anybody ever figure out what was causing the KeyError: 0 issues when trying to run the futures models? I’m stuck, I can usually figure these out with enough googling and trial and error but this one has me at a standstill.

  28. Hi Paul, That’s interesting. I did find one minor issue in my setup (was calling up futures contracts that weren’t included in Andreas’ “random futures data” set (are you using this data source or another futures data source?) but after fixing that I am still at an impasse that is precluding me from completing my first futures backtest.
    For what it’s worth, at the top of the error output:
    —–

    IndexError Traceback (most recent call last)
    in ()
    218 capital_base=initial_portfolio_millions * 1000000,
    219 data_frequency = ‘daily’,
    –> 220 bundle=’random_futures_data’ )

    —–
    But, specifically, at the end of the output:
    —–

    ~\Anaconda3\envs\env_zipline\lib\site-packages\pandas\core\indexing.py in _getitem_axis(self, key, axis)
    1828
    1829 # validate the location
    -> 1830 self._is_valid_integer(key, axis)
    1831
    1832 return self._get_loc(key, axis=axis)

    ~\Anaconda3\envs\env_zipline\lib\site-packages\pandas\core\indexing.py in _is_valid_integer(self, key, axis)
    1711 l = len(ax)
    1712 if key >= l or key 1713 raise IndexError(“single positional indexer is out-of-bounds”)
    1714 return True
    1715

    IndexError: single positional indexer is out-of-bounds

    —–
    At this point, I’d be interested in your “minor changes” that you made. On a broader scale, I wonder if there is any interest in setting up a Google Group or something similar to allow for broader idea swapping… Anyways, cheers! Devon

  29. Hey Paul, I’m still stuck on my issue but, to your point re: ordered contracts, check out this thread, starting with the comment I’m posting here: https://github.com/quantopian/zipline/issues/2293#issuecomment-424337172

  30. Hello Andreas,

    first thank you for your books and contstant attention to everyone writing to you in this site.

    Iwanted to ask for your opinion on the following issue. I have done a 18 yr backtest on the strategy explained in “Stocks on the move” and as I read your latest book, “Trading Evolved”, you recall something that i saw while doing my backtest but could not find the way of fixing it on that moment, which is the following: how to deal with already nonexistent stocks. My backtest was done over more than 700 european stocks, so I am not limited to an index with the risk of trading only the current stocks and not the historical. But still I figure my backtest is “not-trading” some stocks that are no currently in the market anymore. How do you thta could be fixed or, do you think on such an ample stock base, would have had a very significant impact?

    thank you,

    Javier

  31. Hey Paul, would you mind sharing with the group the amendment you made to the code to get the futures bundle to properly ingest the underlying futures files in the appropriate (chronological) order? Many thanks!

  32. Hi Andreas, great book with a lot of insights! I am having however some problems populating the mysql database as per Chapter 24. I have a problem for the vals part of the insert statement where I get an error message saying that symbol and df are defined. Also get an error in the process symbol function but managed to fix it by changing data_location to data_path and importing tqdm_notebook from tqdm. Any help would be greatly appreciated. Thanks in advance

  33. Hi Andreas,

    In chapter 10, “Constructing ETF Models”, there is a bundle called ‘ac_equities_db’ in the code sample without further explanation. The name of the bundle suggests that this is a self-made Andreas C. bundle. I wonder what bundle the reader should use to run this code.

    Thanks for your help! Seb

    • Hi Sebastian,

      The ac_equities_db bundle is just the local bundle I used to run the code. It’s a custom bundle, based on my own local data. In the later chapters, you’ll read about how to make your own custom data to fit your local data.

      • Joakim Arvidsson

        Hi Andreas,

        Does this mean we can’t run the earlier code in chapter 10, 12, etc that uses the ‘ac_equities_db’ bundle, until we learn how to create our own db and data bundle in chapter 23-24?

        Thanks for a great book!

        Joakim

      • Well, you’ll need to hook up data to the backtester one way or another. If you just want to learn the process, you can use the random data I provided for download. You’ll get absurd results of course, with the data being randomly generated, but you’ll learn the process. As for real data, I’m not a data provider and can’t distribute proper time series without fending off a myriad of lawsuits…

  34. Hello everyone, I am trying to run the trend model from chapter 15. Managed to ingest the random_futures_data bundle as per chapter 23. However when running the model I get the following error. Any help would be greatly appreciated. Thanks in advance

    —————————————————————————
    KeyError Traceback (most recent call last)
    in ()
    311 capital_base=starting_portfolio,
    312 data_frequency = ‘daily’,
    –> 313 bundle=’random_futures_data’ )

    /Users/Redwall/anaconda3/envs/Zipline/lib/python3.5/site-packages/zipline/utils/run_algo.py in run_algorithm(start, end, initialize, capital_base, handle_data, before_trading_start, analyze, data_frequency, data, bundle, bundle_timestamp, trading_calendar, metrics_set, default_extension, extensions, strict_extensions, environ, blotter)
    428 local_namespace=False,
    429 environ=environ,
    –> 430 blotter=blotter,
    431 )

    /Users/Redwall/anaconda3/envs/Zipline/lib/python3.5/site-packages/zipline/utils/run_algo.py in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, data, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, environ, blotter)
    227 ).run(
    228 data,
    –> 229 overwrite_sim_params=False,
    230 )
    231

    /Users/Redwall/anaconda3/envs/Zipline/lib/python3.5/site-packages/zipline/algorithm.py in run(self, data, overwrite_sim_params)
    754 try:
    755 perfs = []
    –> 756 for perf in self.get_generator():
    757 perfs.append(perf)
    758

    /Users/Redwall/anaconda3/envs/Zipline/lib/python3.5/site-packages/zipline/gens/tradesimulation.py in transform(self)
    204 for dt, action in self.clock:
    205 if action == BAR:
    –> 206 for capital_change_packet in every_bar(dt):
    207 yield capital_change_packet
    208 elif action == SESSION_START:

    /Users/Redwall/anaconda3/envs/Zipline/lib/python3.5/site-packages/zipline/gens/tradesimulation.py in every_bar(dt_to_use, current_data, handle_data)
    132 metrics_tracker.process_commission(commission)
    133
    –> 134 handle_data(algo, current_data, dt_to_use)
    135
    136 # grab any new orders from the blotter, then clear the list.

    /Users/Redwall/anaconda3/envs/Zipline/lib/python3.5/site-packages/zipline/utils/events.py in handle_data(self, context, data, dt)
    214 context,
    215 data,
    –> 216 dt,
    217 )
    218

    /Users/Redwall/anaconda3/envs/Zipline/lib/python3.5/site-packages/zipline/utils/events.py in handle_data(self, context, data, dt)
    233 “””
    234 if self.rule.should_trigger(dt):
    –> 235 self.callback(context, data)
    236
    237

    in daily_trade(context, data)
    197 fields=[‘close’,’volume’],
    198 frequency=’1d’,
    –> 199 bar_count=250,
    200 )
    201

    /Users/Redwall/anaconda3/envs/Zipline/lib/python3.5/site-packages/zipline/_protocol.pyx in zipline._protocol.check_parameters.__call__.assert_keywords_and_call (zipline/_protocol.c:3747)()

    /Users/Redwall/anaconda3/envs/Zipline/lib/python3.5/site-packages/zipline/_protocol.pyx in zipline._protocol.BarData.history (zipline/_protocol.c:10190)()

    /Users/Redwall/anaconda3/envs/Zipline/lib/python3.5/site-packages/zipline/data/data_portal.py in get_history_window(self, assets, end_dt, bar_count, frequency, field, data_frequency, ffill)
    965 else:
    966 df = self._get_history_daily_window(assets, end_dt, bar_count,
    –> 967 field, data_frequency)
    968 elif frequency == “1m”:
    969 if field == “price”:

    /Users/Redwall/anaconda3/envs/Zipline/lib/python3.5/site-packages/zipline/data/data_portal.py in _get_history_daily_window(self, assets, end_dt, bar_count, field_to_use, data_frequency)
    804
    805 data = self._get_history_daily_window_data(
    –> 806 assets, days_for_window, end_dt, field_to_use, data_frequency
    807 )
    808 return pd.DataFrame(

    /Users/Redwall/anaconda3/envs/Zipline/lib/python3.5/site-packages/zipline/data/data_portal.py in _get_history_daily_window_data(self, assets, days_for_window, end_dt, field_to_use, data_frequency)
    827 field_to_use,
    828 days_for_window,
    –> 829 extra_slot=False
    830 )
    831 else:

    /Users/Redwall/anaconda3/envs/Zipline/lib/python3.5/site-packages/zipline/data/data_portal.py in _get_daily_window_data(self, assets, field, days_in_window, extra_slot)
    1115 days_in_window,
    1116 field,
    -> 1117 extra_slot)
    1118 if extra_slot:
    1119 return_array[:len(return_array) – 1, :] = data

    /Users/Redwall/anaconda3/envs/Zipline/lib/python3.5/site-packages/zipline/data/history_loader.py in history(self, assets, dts, field, is_perspective_after)
    547 dts,
    548 field,
    –> 549 is_perspective_after)
    550 end_ix = self._calendar.searchsorted(dts[-1])
    551

    /Users/Redwall/anaconda3/envs/Zipline/lib/python3.5/site-packages/zipline/data/history_loader.py in _ensure_sliding_windows(self, assets, dts, field, is_perspective_after)
    429 adj_dts = prefetch_dts
    430 prefetch_len = len(prefetch_dts)
    –> 431 array = self._array(prefetch_dts, needed_assets, field)
    432
    433 if field == ‘sid’:

    /Users/Redwall/anaconda3/envs/Zipline/lib/python3.5/site-packages/zipline/data/history_loader.py in _array(self, dts, assets, field)
    571 dts[0],
    572 dts[-1],
    –> 573 assets,
    574 )[0]
    575

    /Users/Redwall/anaconda3/envs/Zipline/lib/python3.5/site-packages/zipline/data/dispatch_bar_reader.py in load_raw_arrays(self, fields, start_dt, end_dt, sids)
    110 for i, asset in enumerate(assets):
    111 t = type(asset)
    –> 112 sid_groups[t].append(asset)
    113 out_pos[t].append(i)
    114

    KeyError:

  35. Hello, theres a kernel error in my Jupyter

    Traceback (most recent call last):
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\notebook\base\handlers.py”, line 516, in wrapper
    result = yield gen.maybe_future(method(self, *args, **kwargs))
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\tornado\gen.py”, line 1055, in run
    value = future.result()
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\tornado\concurrent.py”, line 238, in result
    raise_exc_info(self._exc_info)
    File “”, line 4, in raise_exc_info
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\tornado\gen.py”, line 1063, in run
    yielded = self.gen.throw(*exc_info)
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\notebook\services\sessions\handlers.py”, line 75, in post
    type=mtype))
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\tornado\gen.py”, line 1055, in run
    value = future.result()
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\tornado\concurrent.py”, line 238, in result
    raise_exc_info(self._exc_info)
    File “”, line 4, in raise_exc_info
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\tornado\gen.py”, line 1063, in run
    yielded = self.gen.throw(*exc_info)
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\notebook\services\sessions\sessionmanager.py”, line 79, in create_session
    kernel_id = yield self.start_kernel_for_session(session_id, path, name, type, kernel_name)
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\tornado\gen.py”, line 1055, in run
    value = future.result()
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\tornado\concurrent.py”, line 238, in result
    raise_exc_info(self._exc_info)
    File “”, line 4, in raise_exc_info
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\tornado\gen.py”, line 1063, in run
    yielded = self.gen.throw(*exc_info)
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\notebook\services\sessions\sessionmanager.py”, line 92, in start_kernel_for_session
    self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_name)
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\tornado\gen.py”, line 1055, in run
    value = future.result()
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\tornado\concurrent.py”, line 238, in result
    raise_exc_info(self._exc_info)
    File “”, line 4, in raise_exc_info
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\tornado\gen.py”, line 307, in wrapper
    yielded = next(result)
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\notebook\services\kernels\kernelmanager.py”, line 94, in start_kernel
    super(MappingKernelManager, self).start_kernel(**kwargs)
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\jupyter_client\multikernelmanager.py”, line 110, in start_kernel
    km.start_kernel(**kwargs)
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\jupyter_client\manager.py”, line 240, in start_kernel
    self.write_connection_file()
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\jupyter_client\connect.py”, line 547, in write_connection_file
    kernel_name=self.kernel_name
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\jupyter_client\connect.py”, line 212, in write_connection_file
    with secure_write(fname) as f:
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\contextlib.py”, line 59, in __enter__
    return next(self.gen)
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\jupyter_client\connect.py”, line 100, in secure_write
    win32_restrict_file_to_user(fname)
    File “C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\jupyter_client\connect.py”, line 53, in win32_restrict_file_to_user
    import win32api
    ImportError: No module named ‘win32api’

  36. Hi Andreas,

    Excellent book, thank you a lot for your effort to write this book as a practical manual, where readers can get hands dirty and code. This is the best way to learn!

    It’s very good you have written this book despite possible problems with installing zipline. There are not so many practical books and I think just because you decided to write about problematic topic you will get more readers.

    If there will be the next revision I would suggest:

    1) To introduce to readers more about open source community and how they can contribute and raising zipline relevant problems in github or asking help in stackoverflow. If they understand open source idea, instead of critics on your address, they can ask community to help and if they know where to ask for help, learning process will be less frustrating.

    2) To introduce code repo for the book to speed up learning and avoid rewriting all the code. Some books include links with password to the code repo. At the moment it takes a lot of time to rewrite the code from the book.

    3) I think there is not full code on the page 83-84 “The full code in one go would look like this right below.” I think “data = get_data(‘indexes’)” will not work…

    I’m just on the beginning of the book and I’m looking forward to read it further. Very good book and I definitely recommend the book to readers.

    Rg
    D

  37. Hi Andreas,

    I’ve just noticed in the book there is the link to zipline github so please edit my previous text and remove my comment about githug, but maybe for less experienced readers who have no idea about coding, stackoverflow and how to contribute could be usefull to mention…

    Rg
    D

  38. Hi Andreas,

    do you have an overview somewhere what your futures accronyms are? I’m interested in the commodity ones foe example [CL, HO, RB, NG, GC, LC, _C, _S, _W, SB, HG, CT, KC] ?

    • Hi Henning,

      All standard codes, with an underscore in front of single character codes.

      CL Crude
      HO Heating oil
      RB RBOB gasoline
      GC Gold
      LC Live cattle
      _C Corn
      _S Soybeans
      _W Wheat
      SB #11 sugar
      HG High grade copper
      CT Cotton
      KC Arabica coffee

      • Andreas,

        Please provide the description for the other markets. I’m having difficulty identifying them.

        thanks,
        Farrell

  39. Wong Man Chun Manfred

    Dear Andreas,

    I am your reader for many years since your first book (Following the Trend) as well as your website got published years ago. You are the people who aspired me about systematic investment and fortunately I now have become an institutional fund manager for a Chicago based asset managers.

    I have recent bought your latest book and faced issue when installing zipline: Zipline Installation: failed with current_repodata.json. I fixed it by your provided solution (although I still want to know why we had to change the set_free_channel thing?)

    My current question is: As I finished installing zipline as well as creating the python 3.5 environment in Anaconda, as I tried installing Spyder (which is my preferred IDE) by directly clicking the “install” button in Anaconda Navigator, I am faced with the following warning that I do not how to solve:

    “python==3.3.6 cannot be installed on this environment. Do you want to install the package in an existing environment or….”

    I was expecting the Spyder to be installed and launched smoothly just like how I launched in the root environment (python 3.7).

    Please kindly advise and I very much look forward to your help after failing to fix it for a week.

    • Hi Manfred,

      I’ve seen this before with Anaconda.

      I think there’s a problem with Anaconda not resolving the environment correctly and forcing an upgrade of Python 3.5.

      Try doing:
      conda install spyder
      from a terminal in your Python 3.5 environment instead of clicking the GUI “Install” button.

      When I tried this a few weeks ago, it didn’t upgrade Python and seemed to work (i.e. I had Spyder, Zipline and Python 3.5 all operational in one environment).

      • Richard,

        It is so nice of you giving me a help on that. I read that as a solution from somewhere too. I tried yet it produced some weird messages about having some bugs or inconsistency. Although I could then launch spyder after that step, as spyder gets launched it has an error window (with a red cross). So this seems not perfect either….

        I know the book recommends using Jupyter. Yet even installing Jupyter from Anaconda in the python 3.5 environment still have the same problem. Have you heard of before? And which IDE do you use in following the book’s example if so under python 3.5?

      • Hi Manfred,

        It’s quite little sensitive to installation order in my experience.

        I’ve documented the environment exact environment I and installation steps plus patches I used here:
        https://pypi.org/project/norgatedata/

        scroll down to Zipline Limitations/Quirks.

  40. Any one with this issue in the first backtest? I’ve already ingested both quandl and the random_stock_data and neither are letting me backtest any ideas?

    AttributeError Traceback (most recent call last)
    in ()
    27 handle_data=handle_data,
    28 capital_base=10000,
    —> 29 data_frequency=’daily’,bundle=’quandl’
    30 )
    31

    C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\zipline\utils\run_algo.py in run_algorithm(start, end, initialize, capital_base, handle_data, before_trading_start, analyze, data_frequency, data, bundle, bundle_timestamp, trading_calendar, metrics_set, default_extension, extensions, strict_extensions, environ, blotter)
    428 local_namespace=False,
    429 environ=environ,
    –> 430 blotter=blotter,
    431 )

    C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\zipline\utils\run_algo.py in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, data, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, environ, blotter)
    157 trading_calendar=trading_calendar,
    158 trading_day=trading_calendar.day,
    –> 159 trading_days=trading_calendar.schedule[start:end].index,
    160 )
    161 first_trading_day =\

    C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\zipline\finance\trading.py in __init__(self, load, bm_symbol, exchange_tz, trading_calendar, trading_day, trading_days, asset_db_path, future_chain_predicates, environ)
    101 trading_day,
    102 trading_days,
    –> 103 self.bm_symbol,
    104 )
    105

    C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\zipline\data\loader.py in load_market_data(trading_day, trading_days, bm_symbol, environ)
    147 # date so that we can compute returns for the first date.
    148 trading_day,
    –> 149 environ,
    150 )
    151 tc = ensure_treasury_data(

    C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\zipline\data\loader.py in ensure_benchmark_data(symbol, first_date, last_date, now, trading_day, environ)
    215
    216 try:
    –> 217 data = get_benchmark_returns(symbol)
    218 data.to_csv(get_data_filepath(filename, environ))
    219 except (OSError, IOError, HTTPError):

    C:\Users\JAVIER\Anaconda3\envs\zipline_35\lib\site-packages\zipline\data\benchmarks.py in get_benchmark_returns(symbol)
    26 data=pd.DataFrame(0.0,index=dates,columns=[‘close’])
    27 data=[‘close’]
    —> 28 return data.sort_index().iloc[1:]
    29

    AttributeError: ‘list’ object has no attribute ‘sort_index’

  41. Hi Andreas. Thanks for publishing this book. I am stuck at page 102. When I launch Jupyter Notebook from my zip35 environment I get the following message in the terminal window:

    “Native kernel (python3) is not available”.

    Then when I go into Jupyter and try to do anything I get a series of error messages saying that the kernel has died and that automatic restart failed.

    Can you help? I’ve searched online without success.

    Thank you, Mike

  42. Hello everyone, I was wondering whether anybody has experienced the issue I detail below –

    My futures bundle ingested correctly from my database following was detailed in the book. I checked the sqlite files for how everything ended up mapping in zipline and all seemed to have worked fine.

    I have been following this post between Andreas and Jonathan Larkin (https://github.com/quantopian/zipline/issues/2293) as a guide for a possible solution to this problem as it was flagged by Andreas briefly in that post.

    My issue is as follows

    I ran a test backtest with a single root symbol [“AD”] over a 1 and a half year timeframe. The algo did nothing, but the setup seemed to be okay as the performance stats were all generated.

    I then delved into trying to access the data directly like Andreas and Jonathan have, and noticed that I am experiencing the same blank data in the continuous future object. I can access individual symbols via both their symbol and sid, but when the continuous contract object is instantiated it returns a blank dataframe with NaNs for OHLC data.

    Has anyone experienced this/found out why it may be occurring?

    • Jason,

      Incorrect symbology is the most likely culprit.

      • Thanks Richard,

        In particular do you mean symbology in terms of incorrect symbol format (i.e. not adhering to “xxMYY”), or something like a root symbol name creating an error or an incorrect expiry month? Or do you mean an incorrect ingest in terms of a field in one of the tables being left out?
        or all of the above?

        In any case I will test the data in my ingest script for all of those cases and anything else I can think of

  43. Hi Jason,

    Yes – I’m referring to the symbology xxMYY format that it uses to decode what to do. You might also want to ensure that all of the metadata fields are also populated too (asset_type, notice_date, expiration_date, auto_close_date, exchange, exchange_full, root_symbol, multiplier, start_date, end_date, first_traded too. I think ingestion order might also be important too – make sure the securities are ordered by delivery date.

    It took me a lot of frustrating effort (dozens of hours) with difficult-to-understand error messages and a lot of wild goose chases to write the zipline_norgatedata package, but I’m happy with the results now. With a few small changes I’m now futures trading systems testing back to 1970 too.

    Cheers,
    Richard.

    • excellent thank you Richard, massive help and extremely valuable to know where to focus my time. I took your advice to look at symbols and have had one success.

      For anyone following and future troubleshooters, I realized that I was importing the metadata table with my entire dataset worth of symbols (back to ~1960something) whereas the actual price data table was following a rule of ‘include in the SQL query if your end date is >= 2001-01-01.’

      I made the metadata table – and each symbol query when the individual OHLC lookup occurs – abide by the same logic and now my continuous futures contract for “AD” has populated itself with data. I have not yet checked whether it has rolled correctly. I suspect not as there are days with 0 volume toward the end of the dataset.. (set to roll=’volume’, adjustment=’mul’).

      I simply dumped a csv on my desktop of the data.history object to view this.

      This shows the value in having a community forum, I would absolutely have overlooked this issue..

  44. Hi, complete newbie here. I am up to page 96 on ‘Patching the Framework’ And I have no idea what to do. I can find the file Benchmark.py in my windows folders but I am confused as of what to do after that. Where do I create a new file? And where do I put the code in?
    This may be really obvious but I am stuck. If someone could explain this simply I would be very grateful…

    Thanks!
    Alan

    • Hi Alan,

      The objective of that page is to fabricate a DataFrame for the S&P500 benchmark index that zipline defaults to. The DataFrame that we’re creating spans from 1930-2030 and has 0s for prices. This is to circumvent a broken web data retrieval that is still programmed into Zipline. So open the file and replace the code that is currently in there with what Andreas has written on page 98. Then save it and close it, that’s all you need to do.

      The loader.py file is in the same folder as benchmarks, and there are a few lines that read as follows:

      if data is not None:
      return data.

      just ctrl + f and search for part of that code to find them as the doc is a few hundred lines long.

      these need to be commented out to read:
      “””if data is not None:
      return data”””

      This is how mine are currently set. Again save and close. They are small modifications.

      As for finding the Benchmarks.py file, mine is here: D:\DefaultLocation\Anaconda\envs\env_zipline\Lib\site-packages\zipline\data\Benchmarks.py. If it is not there, my suggestion would be to reinstall the Zipline library as it auto generates. Hope this help a bit.

  45. Hi all,

    Just wondering what is known regarding the order of ingest data for futures bundles and how this affects continuous futures objects. I have amended several strange behaviours such as data.history DataFrames with NaNs, incorrect prices for symbols, and strange rolling logic (0 or low volumes, etc.).

    I have fixed the obvious things and have cleaned up my ‘symbology’ as per what was mentioned above (lots of SQL queries to ensure my database has accurate and correct data, correct month codes/RRMYY convention, and making sure that my ingest script is formatting the data correctly and in the right tables), however cannot seem to return a data.history DataFrame that is 100% correct.

    My problems include unusual continuous futures rolling logic, and NaNs for part, but not all of the data.history dataframe.

    what has worked in my favour is ordering the data in my metadata and price data tables with the following logic in my SQL query prior to ingesting:

    ORDER BY blah.Root_Symbol ASC, blah.auto_close_date ASC;

    This has my E-mini S&P contract (crash test dummy) spitting out correct prices, volumes, and appearing to finally roll with the correct adjustment behaviour. Yet there are several months of price data missing from the dataframe.

    It is possible this is due to the following – the root symbol data only having quarterly contracts, or I’m still ordering something incorrectly, or there are likely a bunch of things I’m still doing wrong…

    has anyone encountered this before?

  46. I got the following error while trying to run the file Chapter 24 – Data and Databases/Populating Database.ipynb:
    NotSupportedError: (mysql.connector.errors.NotSupportedError) Authentication plugin ‘caching_sha2_password’ is not supported (Background on this error at: http://sqlalche.me/e/tw8g)

    Can anybody help me out of this?

    PS : While installing MySQL Community Edition the Python Connectors had failed to install. So I pip installed mysql-connector-python separately.

  47. I think the data in equity_history table is being inserted incorrectly. TICKER column is set as a unique ticker & because of the duplicate key update function the data is overwritten, finally inserting only single row for the ticker in the table.

    • For now I have dropped the idea of using a database. I am directly using csv files to ingest the bundle, as shown in chapter 23. Till now I am successfully able to run backtests. However I am wondering how do I periodically update the database say every day/week.

      For example, I have uploaded data till 25th Nov, 2019 & I want to now update data for 26th Nov.

      The obvious way is to append the csv files & run the zipline ingest -b random_stock_data function. But that is tricky & time consuming.
      Is there an easier way where I can just upload data for 26th Nov & not worry about the data that is uploaded till 25th Nov?
      How do you update your data bundle sir? @Andreas Clenow

      • You could always run a batch script scheduled to update as often as you like or at times when it’s convenient. I’m not aware of a simple way to do incremental ingest.

  48. Hi

    After a lot of effort i installed zipline since I managed to downgrade anaconda to 4.6.11.
    It was impressive for me that the version of zipline is 0.7 with no option for upgrade.

    After taking a look at the respective website i realised that version 1.3 works only in 64 bit OS.

    By the way i work in a Windows 7, 32 bit system.

    So no commands seem to work for me regarding zipline, like “”zipline ingest – b quandl”” where it is not recognised at all!

    Is there any solution for me or i have to format my pc to a 64 bit OS?

    *I am really disappointed because i have to stop reading waiting for an answer.

    Thanks!

    • Sorry you’re having issues, Chris. It’s impossible to predict and support all kinds of obsolete tech environments and I would strongly suggest to upgrade to modern software. Win7 is a decade old and even Microsoft stopped supporting it now, so it’s even a major security issue to run it at all. A 32 bit OS will limit you in all sort of ways, not the least that you’ll only be able to access 4gb memory.

  49. Panagiotis Troxidis

    While i have installed everything according to Trading.Evolved book i execute the command

    zipline ingest ? b quandl

    and the result is this:

    (zipline35) C:\Windows\system32>zipline ingest ? b quandl
    Traceback (most recent call last):
    File “c:\programdata\anaconda3\envs\zipline35\lib\runpy.py”, line 170, in _run
    module_as_main
    “main”, mod_spec)
    File “c:\programdata\anaconda3\envs\zipline35\lib\runpy.py”, line 85, in run
    code
    exec(code, run_globals)
    File “C:\ProgramData\Anaconda3\envs\zipline35\Scripts\zipline.exe_main.py”
    , line 5, in
    File “c:\programdata\anaconda3\envs\zipline35\lib\site-packages\zipline__init
    _.py”, line 23, in
    from . import data
    File “c:\programdata\anaconda3\envs\zipline35\lib\site-packages\zipline\data_
    init.py”, line 1, in
    from . import loader
    File “c:\programdata\anaconda3\envs\zipline35\lib\site-packages\zipline\data\l
    oader.py”, line 22, in
    from .benchmarks import get_benchmark_returns
    File “c:\programdata\anaconda3\envs\zipline35\lib\site-packages\zipline\data\b
    enchmarks.py”, line 16
    data = data[‘close’] return data.sort_index().iloc[1:]
    ^
    SyntaxError: invalid syntax

    Any idea on how can i get out of this?

    Thank you.

    • My guess here is that there’s a missing line break.

      This row looks odd:
      data = data[‘close’] return data.sort_index().iloc[1:]

      Should probably be two rows:
      data = data[‘close’]
      return data.sort_index().iloc[1:]

  50. Hello, I am really enjoying the book. I have worked through most of the errors and bugs so far but I am stuck on this one. I am getting the follow error:

    ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

    After googling this and trying a few different things I thought it could be a pandas compatibility issue but I am afraid to roll back my pandas and would not even know which version to roll back to. Any suggestions would be much appreciated. thanks so much

    2015-01-02 – Last Month Result: 0.00%
    C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\numpy\lib\function_base.py:3103: RuntimeWarning: invalid value encountered in subtract
    X -= avg[:, None]
    C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\scipy\stats\_distn_infrastructure.py:879: RuntimeWarning: invalid value encountered in greater
    return (self.a < x) & (x < self.b)
    C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\scipy\stats\_distn_infrastructure.py:879: RuntimeWarning: invalid value encountered in less
    return (self.a < x) & (x < self.b)
    C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\scipy\stats\_distn_infrastructure.py:1821: RuntimeWarning: invalid value encountered in less_equal
    cond2 = cond0 & (x <= self.a)
    Traceback (most recent call last):

    File "”, line 180, in
    bundle=’custom-csvdir-bundle’ )

    File “C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\zipline\utils\run_algo.py”, line 430, in run_algorithm
    blotter=blotter,

    File “C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\zipline\utils\run_algo.py”, line 229, in _run
    overwrite_sim_params=False,

    File “C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\zipline\algorithm.py”, line 756, in run
    for perf in self.get_generator():

    File “C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\zipline\gens\tradesimulation.py”, line 206, in transform
    for capital_change_packet in every_bar(dt):

    File “C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\zipline\gens\tradesimulation.py”, line 134, in every_bar
    handle_data(algo, current_data, dt_to_use)

    File “C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\zipline\utils\events.py”, line 216, in handle_data
    dt,

    File “C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\zipline\utils\events.py”, line 235, in handle_data
    self.callback(context, data)

    File “”, line 161, in rebalance
    if ranking_table[security] > minimum_momentum:

    File “C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\pandas\core\generic.py”, line 1121, in __nonzero__
    .format(self.__class__.__name__))

    ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

  51. Hello,
    It appears I am the only one who can’t get past page 93, “Be sure to select Python version 3.5 in the dropdown, as in figure 7-.” My only choice in the dropdown is Python 3.7.
    How do I get 3.5?
    Thank you, and great book so far.

  52. Hello,
    Having an error on attempting to ingest Quandl.

    Error:

    ImportError: ‘cannot import name RLock’

    File “zip35\lib\site-packages\logbook\concurrency.py”, line 31, in
    from gevent_threading import <Lock as Threadlock,

  53. Here is the full text of the error on attempting to ingest quandl:

    (zip35) C:\Users\Brad>zipline ingest -b quandl
    Traceback (most recent call last):
    File “C:\Users\Brad\Anaconda3\envs\zip35\Scripts\zipline-script.py”, line 11,
    in
    load_entry_point(‘zipline==1.3.0’, ‘console_scripts’, ‘zipline’)()
    File “C:\Users\Brad\Anaconda3\envs\zip35\lib\site-packages\pkg_resources\__ini
    t__.py”, line 480, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
    File “C:\Users\Brad\Anaconda3\envs\zip35\lib\site-packages\pkg_resources\__ini
    t__.py”, line 2691, in load_entry_point
    return ep.load()
    File “C:\Users\Brad\Anaconda3\envs\zip35\lib\site-packages\pkg_resources\__ini
    t__.py”, line 2322, in load
    return self.resolve()
    File “C:\Users\Brad\Anaconda3\envs\zip35\lib\site-packages\pkg_resources\__ini
    t__.py”, line 2328, in resolve
    module = __import__(self.module_name, fromlist=[‘__name__’], level=0)
    File “C:\Users\Brad\Anaconda3\envs\zip35\lib\site-packages\zipline\__init__.py
    “, line 23, in
    from . import data
    File “C:\Users\Brad\Anaconda3\envs\zip35\lib\site-packages\zipline\data\__init
    __.py”, line 1, in
    from . import loader
    File “C:\Users\Brad\Anaconda3\envs\zip35\lib\site-packages\zipline\data\loader
    .py”, line 17, in
    import logbook
    File “C:\Users\Brad\Anaconda3\envs\zip35\lib\site-packages\logbook\__init__.py
    “, line 14, in
    from .base import (
    File “C:\Users\Brad\Anaconda3\envs\zip35\lib\site-packages\logbook\base.py”, l
    ine 17, in
    from logbook.concurrency import (
    File “C:\Users\Brad\Anaconda3\envs\zip35\lib\site-packages\logbook\concurrency
    .py”, line 31, in
    from gevent._threading import (Lock as ThreadLock,
    ImportError: cannot import name ‘RLock’

    I looked in concurrency.py and found the reference. It is in an ‘if’ block:
    “if has_gevent:
    from gevent._threading import (Lock as ThreadLock,
    RLock as ThreadRLock,
    get_ident as thread_get_ident,
    local as thread_local)
    from gevent.thread import get_ident as greenlet_get_ident
    from gevent.local import local as greenlet_local
    from gevent.lock import BoundedSemaphore
    from gevent.threading import __threading__

    There is much more within this if block, but I have only copied the part where it is attempting to import RLock. I had found a reference somewhere to something using CRLock instead of RLock, but I cannot find that now.
    Any ideas?

    • Question is if you really need Quandl anyhow. It’s nice for a quick demo on free data, but I wouldn’t base anything of importance on that data source. Better to move on and get a real source instead.

  54. Andreas, thanks for writing the book I really enjoyed the examples and learnt alot. I’ve reached the futures trend following and have already loaded in the random futures data but got an error when I rand the example on the ingested random futures data. I’ve done the recommeded patches. I understand there is some workaround.

    Exception ignored in: ‘zipline.assets.continuous_futures.OrderedContracts.contract_before_auto_close’
    AttributeError: ‘NoneType’ object has no attribute ‘next’
    KeyError: 0

  55. Andreas;

    I am enjoying “Trading Evolved”. Since I am not a programmer it has been a start and stop affair but recently was making good progress until page 102 (after successfully installing zipline and ingesting the QUANDL bundle) trying to install nb_conda. I receive the following error: (The actual error MESSAGES were considerably longer and showed other incompatablities.)

    Specifications:

    – nb_conda -> python[version=’>=2.7,=3.6,=3.7, python[version=’>=3.6′]
    Your python: python=3.5

    I am about to uninstall and reinstall Anaconda but before doing so thought I would reach out to you. Also, when I open a Jupyter Notebook I show a “Kernal Error”. Thoughts?

    Thanks

    Chip

    • Chip,

      I (and others) on this page had the same problem. Our workaround was to install nb_conda using the terminal rather than through the GUI of Anaconda Navigator. From your environment in Anaconda Navigator, open up the terminal. Then paste in:

      conda install -c conda-forge nb_conda

      Cheers!

      Tom

      • Thanks Tom, I will try your suggestion although I did finally get throught he examples in Chapter 7 without nb_conda. I am now experiencing the same problems as reported below by Deepak Gupta which seems to be related to the installation of pyfolio. I will keep you posted.

        Thanks

        Chip

      • I ran into the same dependency issues when installing nb_conda and running the above command worked for me.

        Thanks
        Deepinder

    • Hi Chip,

      I fixed both the Kernel error and the nb_conda incompatibility by doing the following:

      1. in your ‘root’ environment: conda install nb_conda
      2. in your ‘zip35’ enviroment:
      – conda install ipykernel
      – conda install tornado=4.5.3 (compatibility downgrade)
      – conda install pywin32

      3. I also had to later fix the ‘font_manager.py’ file in matplotlib, as it failed to get the proper info from pywin32 package when building graphs. Maybe if you install pywin32 first, you don’t need this later step… but it is just a guess, i haven’t tried it out.

      With all these, you can now open Jupyter in your ‘root’ enviroment and chose to create new notebooks under the zip35 environment (thats whats nb_conda is all about). And by installing ipykernel to your zip35 environment, you should no longer get the kernel error.

      Obs: If kernel error still persists, switch to zip35 environment, open jupyter notebook and see if kernel runs. Most likely it will not run, but at least you will get a proper error message to try and debug it.

      Hope it helps, best of luck

  56. Dear Andreas,

    Thank you for writing this book. I am learning a lot from it.
    I tried running the “backtest analysis” program in Chapter 8. I am getting the following error:
    ——————————————————————————————————————————————
    ValueError Traceback (most recent call last)
    in ()
    5 from datetime import datetime
    6 import pytz
    —-> 7 import pyfolio as pf
    8
    9 def initialize(context):

    ~\Anaconda3\envs\ziptest\lib\site-packages\pyfolio\__init__.py in ()
    2
    3 from . import utils
    —-> 4 from . import timeseries
    5 from . import pos
    6 from . import txn

    ~\Anaconda3\envs\ziptest\lib\site-packages\pyfolio\timeseries.py in ()
    23 import scipy as sp
    24 import scipy.stats as stats
    —> 25 from sklearn import linear_model
    26
    27 from .deprecate import deprecated

    ~\Anaconda3\envs\ziptest\lib\site-packages\sklearn\__init__.py in ()
    72 else:
    73 from . import __check_build
    —> 74 from .base import clone
    75 from .utils._show_versions import show_versions
    76

    ~\Anaconda3\envs\ziptest\lib\site-packages\sklearn\base.py in ()
    18
    19 from . import __version__
    —> 20 from .utils import _IS_32BIT
    21
    22 _DEFAULT_TAGS = {

    ~\Anaconda3\envs\ziptest\lib\site-packages\sklearn\utils\__init__.py in ()
    18 from scipy.sparse import issparse
    19
    —> 20 from .murmurhash import murmurhash3_32
    21 from .class_weight import compute_class_weight, compute_sample_weight
    22 from . import _joblib

    __init__.pxd in init sklearn.utils.murmurhash()

    ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject

    It is probably an issue related to numpy. Some people have reported this error and solved it by upgrading numpy. I have numpy version 1.14.2 running. I tried upgrading numpy, but this appears to be the latest version for my configuration (python 3.5).

    This is proving to be a roadblock. I will be grateful if you could advise on the issue.

    Thanks in advance,
    Deepak Gupta

    • Did you find a solution to this issue?

    • Same error here, all solutions on the web seems to request an update to numpy==1.16.1, which is not compatible with python 3.5.

      Any help is appreciated.

      • Kristof Friedrich

        Dear all,

        I have found a solution for the above problem and I am now running PyFolio together with Jupyter exactly like it is described in the book.
        Basically, all the problems boil down to jupyter-server and numpy and the json version they use: “jupyterlab-server 1.0.0 has requirement jsonschema =3.0.1” has been the error message at the root of the problem.

        First, some general info: I’m running Win 7 Ultimate 64bit.

        Here’s what I did to solve the issue:

        I downgraded jupyterlab-server to version v0.2.0.
        I donwgraded jupyterlab to version v0.35.0.
        I then upgraded numpy to version v1.16.1.

        You can use the following commands to do so:

        pip install –user numpy==1.16.1
        This will automatically search for the specified version. Same goes for the other packages:

        pip install jupyterlab-server==0.2.0
        pip install jupyterlab==0.35.0

        Make sure to restart both Jupyter and Anaconda.
        I had to play around with the versions a lot so I probably forgot some steps. That’s why exported all the info for the enviroment that I set up (see below). If everything fails, then try the following:

        -Copy everything from below and save it as “enviroment.yml”. Make sure to modify the last line to fit your particular installation path on your computer.
        -Run the following command in the terminal: “conda env create -f environment.yml”
        Important: You’ll either have to give the directory or use “cd” to open the terminal in the folder, where the file is located!

        This can take a while. In theory, you should be left with an exact copy of my working enviroment. It should be shown as “zip35” below the “root” enviroment.

        Here’s the text you’ll need to copy:

        name: zip35
        channels:
        – anaconda
        – conda-forge
        – Quantopian
        – defaults
        dependencies:
        – alembic=0.7.7=py35_0
        – asn1crypto=0.24.0=py35_0
        – bcolz=0.12.1=np114py35_0
        – blas=1.0=mkl
        – bleach=3.1.0=py_0
        – blosc=1.16.3=h7bd577a_0
        – bottleneck=1.2.1=py35h452e1ab_1
        – bzip2=1.0.8=he774522_0
        – ca-certificates=2019.11.28=hecc5488_0
        – certifi=2018.8.24=py35_1
        – cffi=1.11.5=py35h74b6da3_1
        – chardet=3.0.4=py35_1
        – click=7.0=py_0
        – colorama=0.4.3=py_0
        – contextlib2=0.6.0.post1=py_0
        – cryptography=2.3.1=py35h74b6da3_0
        – cycler=0.10.0=py35hcc71164_0
        – cyordereddict=0.2.2=py35_0
        – cython=0.28.5=py35h6538335_0
        – decorator=4.4.1=py_0
        – defusedxml=0.6.0=py_0
        – empyrical=0.5.0=py35_0
        – entrypoints=0.2.3=py35_2
        – freetype=2.9.1=ha9979f8_1
        – hdf5=1.10.2=hac2f561_1
        – icc_rt=2019.0.0=h0cc432a_1
        – icu=58.2=vc14hc45fdbb_0
        – idna=2.7=py35_0
        – intel-openmp=2019.4=245
        – intervaltree=2.1.0=py35_0
        – ipykernel=4.10.0=py35_0
        – ipython=5.8.0=py35_0
        – ipython_genutils=0.2.0=py35ha709e79_0
        – jinja2=2.10.3=py_0
        – jpeg=9b=vc14h4d7706e_1
        – json5=0.8.5=py_0
        – kiwisolver=1.0.1=py35h6538335_0
        – libiconv=1.15=h1df5818_7
        – libpng=1.6.37=h2a8f88b_0
        – libsodium=1.0.16=h9d3ae62_0
        – libxml2=2.9.9=h464c3ec_0
        – libxslt=1.1.33=h579f668_0
        – logbook=0.12.5=py35_0
        – lru-dict=1.1.4=py35_0
        – lxml=4.2.5=py35hef2cd61_0
        – lz4-c=1.8.1.2=h2fa13f4_0
        – lzo=2.10=h6df0209_2
        – m2w64-gcc-libgfortran=5.3.0=6
        – m2w64-gcc-libs=5.3.0=7
        – m2w64-gcc-libs-core=5.3.0=7
        – m2w64-gmp=6.1.0=2
        – m2w64-libwinpthread-git=5.0.0.4634.697f757=2
        – mako=1.1.0=py_0
        – markupsafe=1.0=py35hfa6e2cd_1
        – matplotlib=3.0.0=py35hd159220_0
        – mistune=0.8.3=py35hfa6e2cd_1
        – mkl=2018.0.3=1
        – msys2-conda-epoch=20160418=1
        – multipledispatch=0.6.0=py35_0
        – nb_conda=2.2.1=py35_0
        – nb_conda_kernels=2.1.0=py35_0
        – nbconvert=5.5.0=py_0
        – nbformat=4.4.0=py35h908c9d9_0
        – networkx=1.11=py35_1
        – notebook=5.6.0=py35_0
        – numexpr=2.6.1=np114py35_0
        – numpy=1.14.2=py35h5c71026_0
        – openssl=1.0.2u=hfa6e2cd_0
        – pandas=0.22.0=py35h6538335_0
        – pandas-datareader=0.8.1=py_0
        – pandoc=2.2.3.2=0
        – pandocfilters=1.4.2=py35_1
        – patsy=0.5.0=py35_0
        – pickleshare=0.7.4=py35h2f9f535_0
        – pip=10.0.1=py35_0
        – prometheus_client=0.7.1=py_0
        – prompt_toolkit=1.0.15=py35h89c7cb4_0
        – pycparser=2.19=py35_0
        – pygments=2.5.2=py_0
        – pyopenssl=18.0.0=py35_0
        – pyparsing=2.4.6=py_0
        – pyqt=5.9.2=py35ha878b3d_0
        – pysocks=1.6.8=py35_0
        – pytables=3.4.4=py35he6f6034_0
        – python=3.5.6=he025d50_0
        – python-dateutil=2.8.1=py_0
        – pytz=2019.3=py_0
        – pywinpty=0.5.4=py35_0
        – pyzmq=17.1.2=py35hfa6e2cd_0
        – qt=5.9.6=vc14h1e9a669_2
        – requests=2.20.1=py35_0
        – requests-file=1.4.3=py35_0
        – scipy=1.1.0=py35hc28095f_0
        – send2trash=1.5.0=py35_0
        – setuptools=40.2.0=py35_0
        – simplegeneric=0.8.1=py35_2
        – sip=4.19.12=py35h6538335_0
        – six=1.11.0=py35_1
        – snappy=1.1.7=h777316e_3
        – sortedcontainers=1.4.4=py35_0
        – sqlalchemy=1.2.11=py35hfa6e2cd_0
        – sqlite=3.30.1=he774522_0
        – statsmodels=0.9.0=py35h452e1ab_0
        – terminado=0.8.1=py35_1
        – testpath=0.4.4=py_0
        – toolz=0.10.0=py_0
        – tornado=5.1.1=py35hfa6e2cd_0
        – trading-calendars=1.11.1=py35_0
        – traitlets=4.3.2=py35h09b975b_0
        – urllib3=1.23=py35_0
        – vc=14.1=h0510ff6_4
        – vs2015_runtime=14.16.27012=hf0eaf9b_1
        – wcwidth=0.1.7=py35h6e80d8a_0
        – webencodings=0.5.1=py35_1
        – wheel=0.31.1=py35_0
        – win_inet_pton=1.0.1=py35_1
        – win_unicode_console=0.5=py35h56988b5_0
        – wincertstore=0.2=py35hfebbdb8_0
        – winpty=0.4.3=4
        – xz=5.2.4=h2fa13f4_4
        – zeromq=4.2.5=he025d50_1
        – zipline=1.3.0=np114py35_0
        – zlib=1.2.11=h62dcd97_3
        – zstd=1.3.7=h508b16e_0
        – pip:
        – attrs==19.3.0
        – importlib-metadata==1.4.0
        – inflection==0.3.1
        – joblib==0.14.1
        – jsonschema==3.2.0
        – jupyter-client==5.3.4
        – jupyter-core==4.6.1
        – jupyterlab==0.35.0
        – jupyterlab-server==0.2.0
        – more-itertools==8.1.0
        – pyfolio==0.9.2
        – pyrsistent==0.15.7
        – pywin32==227
        – quandl==3.5.0
        – scikit-learn==0.22.1
        – seaborn==0.9.0
        – zipp==1.0.0
        prefix: F:\Anaconda\envs\zip35

        Hopefully this helps. Feel free to reach out to me if you get stuck at some point!

  57. Here are the steps I took on a Windows 10 machine to get the environment set up that Andreas is shooting for in Chapter 7. Hopefully this will help some of you.

    (Note, if you tried and failed to create a zip35 environment, even if you deleted that one, start with a fresh environment with a name that you haven’t tried to use before.)

    In Anaconda Navigator, click on Environments. While in the base (root) environment, click on the triangle just to the right of “base (root)” and select Open Terminal.

    Once the terminal comes up type in the following commands. (Be sure to substitute something different for “yourenvname” and insert your own Quandl API key where I have written “your_own_api_key”):

    conda update conda
    conda create -n yourenvname python=3.5 anaconda
    conda activate yourenvname
    conda install -c Quantopian zipline
    (If–while installing zipline–you get an error stating “ImportError: cannot import name ‘RLock’ “, then try the following command:
    conda uninstall gevent
    and then redo the “conda install -c Quantopian zipline” command).
    (Also, if you get an error at this point saying the zipline package appears corrupted, just re-enter the “conda install -c Quantopian zipline” command again.)
    Set QUANDL_API_KEY = your_own_api_key
    zipline ingest –b quandl
    conda install -c conda-forge nb_conda
    conda install -c conda-forge matplotlib

    Now, take the exact ten lines of code that Andreas provides you above on this errata page as a substitute for the code in the benchmarks.py file and paste those ten lines of code into the benchmarks.py file on your computer. Search in the C drive to find that file. You can delete everything in that file and replace it with those ten lines of code.

    Next, for the loader.py file (located in the same folder as benchmarks.py), don’t replace everything in that file with the text that Andreas provides above. Instead, find the two lines of code in the loader.py file that say “if data is not None:” and “return data.” Leave those lines exactly as they are. You need to simply surround those two lines (exactly as pictured above by Andreas) with “”” immediately above and below the text in the loader.py file. This process is called “commenting out” code and will change the file to fix a bug.

    Now go back to Anaconda Navigator. Go to environments. Click on your new environment. Once it is ready, click the triangle to the right of the name and click Open With Jupyter Notebook. Once in the notebook, click on upload and upload the sp500.csv file you can find here:

    https://www.dropbox.com/s/tj85sufbsi820ya/Trading%20Evolved.zip?dl=0&file_subpath=%2FChapter+6+Pandas

    Now upload the First Zipline Backtest document that you can find here:

    https://www.dropbox.com/s/tj85sufbsi820ya/Trading%20Evolved.zip?dl=0&file_subpath=%2FChapter+7+-+Backtesting+Trading+Strategies

    Now open up First Zipline Backtest file, click on the code somewhere, and then hit control enter. Note the hour glass running at the very top of the page showing that the code is running. If you get an initial error, but the hourglass is still running, just continue to wait.

    A few additional points:

    –Sometimes after making all these fixes, you still get an error message for a while when you try to run the code in Jupyter Notebook. Get out of all terminals. Get out of Anaconda Navigator. Then restart and try to run the code in the Jupyter Notebook. It just might work :).

    –Make sure in the upper right hand corner of the Jupyter Notebook that you are trying to run that it is showing that your code it running on the environment you created. If it isn’t, you can stop the kernel on Jupyter Notebook page and start it using the correct environment.

    –Also, some “errors” when you run the code are more like warnings. If you just wait a little longer, the charts will eventually come up.

    –Jupyter Lab (also available on Anaconda Navigator) is the next generation of Jupyter Notebook. I personally find it a lot more intuitive and easy to use than Jupyter Notebook.

    I hope this helps a few of you to avoid the two days it took me to get the environment up and running. Best of luck.

    • Thomas,
      Thank you for posting this solution, it has worked and you have helped me to get through a four-day struggle to get my zipline environment functioning. Very much appreciated. I will add two comments in case anybody else runs across it, I was able to install zipline but when I got to the ingest function, that is where I got the error “ImportError: cannot import name ‘RLock’ “, your solution to ‘conda uninstall gevent’ worked fine at that point. Also, if anybody runs into this error “importerror: no module named ‘win32api'” when trying to start the kernel in Jupyter Notebook, just go back to your terminal (in your new environment” and ‘conda install pywin32’ will solve that issue. Thank you again.

    • Thank you. The steps provided were very accurarate and helped me to the moon and back 🙂

    • Thanks Thomas, very helpful and appreciate the time you took to post this, solved the issue I was having with nb_conda

  58. Hi Andreas,

    Thank you for writing this awesome book. However I have ran into 2 questions so far
    1) I used your random data to populate my database and it took about 3 hours to do so. Is it suppose to take that long?

    2) when querying the database and trying to create my first graph it only outputs a big grey box or it gives me an error when changing the datas in plot_data = hist[]… I did a straight copy of your code but no luck

    I replaced your password with mine and changed the schema name to match mine as well

    %matplotlib inline
    import pandas as pd
    from matplotlib import pyplot as plt, rc
    from sqlalchemy import create_engine

    engine = create_engine(‘mysql+mysqlconnector://root:root@localhost/mimisbrunnr’)

    # Chart formatting, used for book image
    font = {‘family’ : ‘eurostile’,
    ‘weight’ : ‘normal’,
    ‘size’ : 16}
    rc(‘font’, **font)

    # Function to fetch history from db
    def history(symbol):
    query = “””select trade_date, close, volume
    from equity_history where ticker='{}’
    “””.format(symbol)
    print(‘This is the SQL statement we send: \n {}’.format(query))
    df = pd.read_sql_query(query, engine, index_col=’trade_date’, parse_dates=[‘trade_date’])
    return df

    # Fetch data for the stock
    ticker = ‘AAPL’
    hist = history(ticker)

    # Set up the chart
    fig = plt.figure(figsize=(15, 8))
    ax = fig.add_subplot(111)
    ax.grid(True)
    ax.set_title(‘Chart for {}’.format(ticker))

    # Note how you can use date ranges to slice the time-series
    plot_data = hist[‘2014-01-01′:’2015-01-01’]

    # Plot the close data
    ax.plot(plot_data.close, linestyle=’-‘, label=ticker, linewidth=3.0, color=’black’)
    ax.set_ylabel(“Price”)

    # Make a second Y axis, sharing the same X
    ax2 = ax.twinx()
    ax2.set_ylabel(“Volume”)

    # Plot volume as bars
    ax2.bar(plot_data.index, plot_data.volume, color=’grey’)

    ERROR MESSAGE
    TypeError Traceback (most recent call last)
    in
    43
    44 # Plot volume as bars
    —> 45 ax2.bar(plot_data.index, plot_data.volume, color=’grey’)
    46

    c:\users\seanp\anaconda3\envs\zipline\lib\site-packages\matplotlib\__init__.py in inner(ax, data, *args, **kwargs)
    1808 “the Matplotlib list!)” % (label_namer, func.__name__),
    1809 RuntimeWarning, stacklevel=2)
    -> 1810 return func(ax, *args, **kwargs)
    1811
    1812 inner.__doc__ = _add_data_doc(inner.__doc__,

    c:\users\seanp\anaconda3\envs\zipline\lib\site-packages\matplotlib\axes\_axes.py in bar(self, x, height, width, bottom, align, **kwargs)
    2275 if align == ‘center’:
    2276 if orientation == ‘vertical’:
    -> 2277 left = x – width / 2
    2278 bottom = y
    2279 elif orientation == ‘horizontal’:

    TypeError: ufunc subtract cannot use operands with types dtype(‘<M8[ns]') and dtype('float64')

    AND IF I TRY THIS
    plot_data = fetch['1996-01-01':'2014-01-01']

    It will plot a big grey box

    Thank you

    • Andreas,
      As Sean mentioned earlier when querying the Database using the code from page 413 it appears that I am only getting one row. Then, when I plot I get a big grey box. Is there something in the itterows function
      on page 411 that is not looping through the rest of the days (rows)? Thanks for your help on this matter. I want to move on to making a bundle.

  59. Andreas – thanks for another fantastic book!

    Sorry for high-jacking this forum, but I came across a strange behavior, that maybe someone else has seen and fixed?

    I ingested a custom bundle for sector ETFs, trying to implement a rotational strategy.
    The the backtest runs fine, until at some point in time, order_target_percent doesn’t see the specified asset anymore. The next loop of handle_data (which is my rebalancing function) shows an unchanged portfolio (context.portfolio.positions).

    Has anyone seen something like this before?
    Thank you in advance!
    Dirk

    • Sorry, typo in my message — should have read:
      “order_target_percent doesn’t *SELL* the specified asset anymore”

    • Ok, so I figured out what the problem is. Basic math failure on my end.
      This is the troublesome code:

      # How much to hold of each equity
      target_percent = 100 / context.topn

      It should have been target_percent context.topn / 100 instead.
      I’m assuming this leads to situations in which orders aren’t filled properly, leading to the described behavior.

      Lesson learned:
      – Check for open orders and cancel them, if needed
      – Keep an eye on leverage and position sizes and check against restrictions during the algo run

  60. Hi Andreas – Really enjoying the book. I am running into the following error when ingesting the Quandl Bundle. Any idea on how I can get past it? Really appreciate it

    (zip35) C:\Users\Edmond>zipline ingest -b quandl
    Traceback (most recent call last):
    File “C:\Users\Edmond\Anaconda3\envs\zip35\Scripts\zipline-script.py”, line 11, in
    load_entry_point(‘zipline==1.3.0’, ‘console_scripts’, ‘zipline’)()
    File “C:\Users\Edmond\Anaconda3\envs\zip35\lib\site-packages\pkg_resources\__init__.py”, line 484, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
    File “C:\Users\Edmond\Anaconda3\envs\zip35\lib\site-packages\pkg_resources\__init__.py”, line 2707, in load_entry_point
    return ep.load()
    File “C:\Users\Edmond\Anaconda3\envs\zip35\lib\site-packages\pkg_resources\__init__.py”, line 2325, in load
    return self.resolve()
    File “C:\Users\Edmond\Anaconda3\envs\zip35\lib\site-packages\pkg_resources\__init__.py”, line 2331, in resolve
    module = __import__(self.module_name, fromlist=[‘__name__’], level=0)
    File “C:\Users\Edmond\Anaconda3\envs\zip35\lib\site-packages\zipline\__init__.py”, line 23, in
    from . import data
    File “C:\Users\Edmond\Anaconda3\envs\zip35\lib\site-packages\zipline\data\__init__.py”, line 2, in
    from .loader import (
    ImportError: cannot import name ‘load_prices_from_csv’

  61. Hi Andreas,

    I am trying to run the first backtest and I get the following error –

    FileNotFoundError: [Errno 2] No such file or directory: ‘C:\\Users\\haloc\\.matplotlib\\fontlist-v300.json’

    I have uninstallled matplotlib and reinstalled it in zip35

    • After 3 days of installation problems and now this runtime problem I have given up on making this environment work on Windows 10. It is just not worth it. There are too many hacks to make it work.

  62. Chapter 8: Analyzing Backtesting Results

    Has anyone ran/replicated the example in chapter 8 recently? My results do not match the book.

    Start date 2003-01-02
    End date 2017-12-29
    Total months 179
    Backtest
    Annual return 4.3%
    Cumulative returns 89.0%
    Annual volatility 18.0%
    Sharpe ratio 0.33
    Calmar ratio 0.07
    Stability 0.05
    Max drawdown -64.6%
    Omega ratio 1.06
    Sortino ratio 0.46
    Skew -0.12
    Kurtosis 9.49
    Tail ratio 0.93
    Daily value at risk -2.2%

  63. Dana;

    I had many frustrating experiences trying to get a new environment setup correctly. It was solved by a post in Quantopian’s Forum here:

    https://www.quantopian.com/posts/new-book-on-quantopian-slash-zipline-backtesting-and-modeling

    Look for a post by Edward Hayman that list the steps to setup a new enviorment. It was posted 6 days ago. I am on a Win 10-64 computer.

    Chip

  64. Hi Andreas,

    I really enjoyed the book. When trying to ingest either the random stock data or the random futures data I get the following error in the prompt:

    Traceback (most recent call last):
    File “…Anaconda3\envs\zip35\lib\site-packages\pkg_resources\__init__.py”, line 578, in _build_master
    ws.require(__requires__)
    File “…Anaconda3\envs\zip35\lib\site-packages\pkg_resources\__init__.py”, line 895, in require
    needed = self.resolve(parse_requirements(requirements))
    File “…Anaconda3\envs\zip35\lib\site-packages\pkg_resources\__init__.py”, line 786, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
    pkg_resources.ContextualVersionConflict: (sortedcontainers 1.4.4 (…anaconda3\envs\zip35\lib\site-packages\sortedcontainers-1.4.4-py3.5.egg), Requirement.parse(‘sortedcontainers=2.0’), {‘intervaltree’})

    Do you have any idea what the problem could be? I have used the code downloaded from here. Any help would be greatly appreciated!

    Nigel

  65. Chapter 23 – using your own data
    Andreas, great book so far – thanks.

    For backtesting it’s important to use correct data including stock splits. Has anyone implemented this yet? Chapter 23 covers dividend but I did not see anything on stock splits?

    • It’s quite unusual for historical data not to already be adjusted for splits. Pretty much anything you see from regular sources are already adjusted. If your data is not adjusted for splits, simply supply the split ratios when you ingest the data and you’re good to go.

  66. Hi Andreas, Hi everybody

    during the last 4 weeks I build a security master database, because I didn’t found anything in GitHub which worked.

    It has separate tables for exchange, vendor, ticker symbols, price, fundamental and so on.
    I should be able to hold futures as well.

    Was well I build 2 codes to fill it and to make a bundle. The fill code can as well do updates.
    Im using the Quandl for price and fundamental, as they offer 1 Month around 20years for a good price.

    I would like to share the database plus the two codes to everybody.

    @ Andreas, I just (obviously) parts of your code, would this be ok for you?

    I believe I could be a good starting point for others to build on it and improve it.

    Downloading 20 Years price and fundamental data for 14000 stocks takes around 24 house which is quite long but just need to be done one time.

    The ingesting is around 30min, but I believe if one wants to update daily it could be faster.

    Now my problem : I don’t / can’t host it at GitHub, because I have a quite busy job and are not able to answer questions regarding issues which will surely appear.

    Somebody interesting to host and maintain this?

    Best
    Carsten

    • Hi Carsten,

      Sure, feel free to use the code.

      I don’t mind hosting the code, but I suppose the problem is more on maintaining and addressing questions. If competent people are willing to spend time on in, I could set something up.

      Andreas

      • Hi Andreas

        Ok, thanks.

        …and yes, it’s regarding answering questions and maintenance.

        On the other side, everyone which understands your codes from the book would immediately understand them.
        There are not much different. Instead of query one table, they need to query several and combine them.

        For the database I just build a model. I this way its easy to read it into a MySQL dashboard and inject it.

        It should be possible to host them without comments and maintenance.

        My ideas is just to offer a nucleus and than other people could build on it.

        Next weekend I’m going to comment them and send them to you. If you would host them we could reach more people.

        …the storage is quite an underestimated issues.

        Thanks
        Carsten

  67. Hi Andreas,

    Thank you for writing such a fantastic book and helping us transition from tools like excel to python.

    While going through the chapter-23 I am facing the following issue. I am trying to ingest the random data you have provided in the zip35 environment, but am not able to proceed beyond this point. It would be helpful if you can direct me to the solution or where I could get some help on this.

    (zip35) C:\Users\Admin>zipline ingest -b random_stock_data
    Traceback (most recent call last):
    File “C:\Users\Admin\Anaconda3\envs\zip35\Scripts\zipline-script.py”, line 11, in
    load_entry_point(‘zipline==1.3.0’, ‘console_scripts’, ‘zipline’)()
    File “C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\pkg_resources\__init__.py”, line 484, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
    File “C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\pkg_resources\__init__.py”, line 2707, in load_entry_point
    return ep.load()
    File “C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\pkg_resources\__init__.py”, line 2325, in load
    return self.resolve()
    File “C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\pkg_resources\__init__.py”, line 2331, in resolve
    module = __import__(self.module_name, fromlist=[‘__name__’], level=0)
    File “C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\zipline\__init__.py”, line 29, in
    from .utils.run_algo import run_algorithm
    File “C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\zipline\utils\run_algo.py”, line 27, in
    from zipline.algorithm import TradingAlgorithm
    File “C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\zipline\algorithm.py”, line 68, in
    from zipline.finance.blotter import SimulationBlotter
    File “C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\zipline\finance\blotter\__init__.py”, line 16, in
    from .simulation_blotter import SimulationBlotter
    File “C:\Users\Admin\Anaconda3\envs\zip35\lib\site-packages\zipline\finance\blotter\simulation_blotter.py”, line 43, in
    class SimulationBlotter(Blotter):
    TypeError: ‘str’ object is not callable

    • Impossible to know without seeing the code, but my first guess is that you made a variable called str, and thereby redefined it from a build-in function to a named variable. If so, change the name of your str variable.

      • Ok. I just used the code you have provided for random_stock_data function and added the path of my folder. Then made changes to the random_stock_data.py file and extensions.py file and then tried to ingest it. Incidently when I am trying to ingest the quandl bundle I am getting the same error. Should I reinstall the anaconda3 and try again?

  68. Regarding setting the API key on page 71 …

    “Run the following command.
    Set QUANDL_API_KEY=your_own_api_key”

    When I run this I keep getting …
    ‘QUANDL_API_KEY’ is not recognized as an internal or external command, operable program or batch file.

  69. Carter Fitzpatrick

    I bought this book a few weeks ago and I have had limited success. I successfully created an environment and ingested the ‘quandl’ bundle. Then, I was able to run the first two examples in Chapter 7 with the benchmark data showing zero’s. However, Chapter 8 has given me problems and I can’t fix the “Backtest Analysis”. I get various errors like numpy.unfunc, unpacking and indexing. So, I tried reworking the installation of the environment multiple times to no success . I really want to work this out. Is there currently an environment with pyfolio and zipline fully working that someone can pass along? I have too much time in this to quit. Please offer some guidance.

  70. tried that and now I get …

    ‘export’ is not recognized as an internal or external command, operable program or batch file.

  71. Hi Andreas,

    Thanks for the great book! Totally enjoyed it!

    One question does come to mind is that if one creates his own data bundles for futures across different exchanges (and register corresponding calendars), would one be able to backtest them in one go? i.e. would Zipline recognize the calendars in the data bundles then trade accordingly? If not, it would be quite a process to run multiple backtests & combine the results.. I think…

  72. I had spent days trying to make the random futures data work with no success. I get this message when I run a dummy test (Just using one future and getting data)

    “Exception ignored in: ‘zipline.assets.continuous_futures.OrderedContracts.contract_before_auto_close’
    AttributeError: ‘NoneType’ object has no attribute ‘next'”

    After this I get many lines of error code. Did anyone get this error before?

  73. Hi Andres,
    Very nice book. I have a question regarding ingestion.
    zipline ingest -b random_stock_data: works fine
    zipline ingest -b random_stock_data: returns the following error:

    zipline ingest -b random_futures_data
    /Users/fatemesadatemami/opt/anaconda3/envs/zipline/lib/python3.5/site-packages/zipline/__main__.py:60: UserWarning: Failed to load extension: ‘/Users/fatemesadatemami/.zipline/extension.py’
    unexpected indent (random_futures_data.py, line 51)
    os.environ,
    Error: No bundle registered with the name ‘random_futures_data’
    (zipline) ➜ zipline git:(master) ✗

    Could you please elaborate?

    • Hi Amir,

      It would be impossible for me to provide tech support on these things. It would take not only all of my time, but I would need a staff just for that. For the vast majority of readers, this process worked just fine. Where it fails, it’s likely some peculiarity in the local installation, the setup of the machine, the combination of libraries installed etc.

      It sounds like something is wrong in the futures bundle, causing it not to be properly registered. Check the code and how that bundle is registered.

      • Thanks Andreas. I finally managed to solve it. It was the “confounded” path that caused this issue. Now resolved 🙂

      • For posterity, I also ran into this same exact error while going through the random_stock_data bundle exercise. I had created a conda environment for this exercise and it turns out I had placed the random_stock_data.py file into the bundles folder that was in the global zipline/data/bundles folder.

        Putting random_stock_data.py into the zipline/data/bundles folder that was in my current python environment path solved the issue. So basically, if you’re using a conda environment make sure your bundle is in the envs path! Hope this helps anyone running into the same issue!

        Also, thanks Andreas for writing this book. I am learning a ton and thoroughly enjoying it!

  74. I have an exception handling question for anyone who’s working on the curve model. I’m using a bundle with data that’s up to date for today. But I can only run the model with an end date of October, otherwise it returns a ZeroDivisionError. How would you tell it to ignore contracts for which it can’t calculate the annualized carry?

    in weekly_trade(context, data)
    # Calculate the annualized carry
    annualized_carry = (np.power(
    (prices[0] / prices[1]), (365 / days_to_front))
    ) -1

    ZeroDivisionError: division by zero

    • Solved this. Simple solution is not to divide by “days_to_front”, but instead, “days_to_front +1” so that there is never a zero value in the denominator. Maybe not the most elegant solution, but, it works.

  75. Hi Andreas,
    you wrote a great book !
    I’m just having trouble importing “random_stock_data”. Since it throws me an error:

    “ValueError: Boolean array expected for the condition, not float64”.

    If I downgrade my pandas version to 0.18.1 then I have problems with pandas datareader that is not compatible. I running in circles.
    Any hint ?

    thanks

    Raimondo

    • Never mind !
      I solved the issue.
      When you use some columns remember in your file like “dividends”, ecc… rember to fill in their value with zero. blank is not accepted!

  76. Does anyone know how to make a pipeline in zipline locally?

  77. Hi Andreas

    Thanks for the great book !

    How do you access/use the “in_sp500 ” column (from the random_stock_data) in the backtesting environment ? The data.history call only takes OHLC variables as inputs. Also, could you please provide some clarification regarding which method zipline uses to adjust dividends ( ref page.176 in your book) ?

  78. Dear Andreas,

    congratulations on this excellent book! It’s a must read for anyone interested enough to dive deeper into systematic trading. As this year’s CFA exam got postponed due to COVID-19 outbreak I took the chance to freely educate myself and ordered your book on Amazon. Although I’m a slow reader these were the fastest 400 pages I’ve read for a quite a while. Hats off for your idea on how to calculate momentum! In addition to your presentation of some really clever coding concepts, I enjoyed the writing style and even the little picking on the mutual funds industry in chapter 21(working in here..).

    However after being done with the first read, I’d like to take it further and develop my own strategies. I thought about adding the factor “3M EPS Revisions for the current fiscal year” to your momentum factor as it could be a great complement to determine stocks that are doing good, but which are still underestimated.

    I know that you can’t help with technical/ coding problems, but I thought maybe you could give me some advice on which data provider is best to use (in terms of data availability as well as ease of doing the ingest process). I do have access to Bloomberg and it took me almost the whole day to figure out how to access the API and pull closing prices via python outside of the terminal. So I’m not really confident about finalizing the ingest process to zipline with Bloomberg. Do you by any chance have some ‘code examples’ on how to ingest data pulled from Bloomberg – or know which other data provider could help me out with my model in mind?

    Thanks a lot for your time,
    Johannes

    • Hi Johannes,

      Technically you can’t ingest custom fields in Zipline, just OHCLV. I would work around this by creating dummy tickers for fundamental data. E.g. IBM_EPS ticker which holds time series for IBM EPS numbers. Either that, or just store the fundamentals on disk in a CSV, and read it into memory before the simulation starts. It’s not all that much data anyhow, so you can just read it in advance and access when you need.

      As far as I know, Bloomie data apis are built to prevent you from using the data locally. They’re mostly for hosted use, and that means you can’t use it with Zipline. Their export functionality has a built in analyzer to make sure that you can’t really siphon out data as well. It lets you get calculated data, which cannot be reversed back into the original series. Yes, that sucks, but it’s Bloomberg policy. I recently had a demo of their Python API and concluded that it’s not meant for any serious quant work.

      Andreas

      • Hi Andreas,

        Thanks for your reply. I have moved to quantopian for that purpose to accelerate my progress in learning how to research my strategies. Just out of curiousity have you tested your momentum factor (on quantopian) with alphalens? I have copied your momentum factor formulation of this article ( https://www.followingthetrend.com/2017/01/getting-started-with-python-modeling-making-an-equity-momentum-model/ )
        as well as the one presented in Trading Evolved, and for the same time horizon (last 3 years), the backtest for the first performed a bit better than the latter -although it is not based on log returns, which I find quite astonishing..In general, I have the impression that the punishment of the coefficient of determination is not enough to filter out short-term high-flying stocks in times of higher market volatility. To take on further tests I wanted to conduct research with alphalens, but was until now not able to put the function it into the Class Customfactor() structure – any advice?

        Best regards
        Johannes

      • Hi Andreas,

        I finished your book a week ago and have been working on my own backtest since. First I would like to thank your for the book, I really enjoyed it.

        Furthermore, I wonder if you could elaborate on how to get other data than OHCLV in the backtest. Where in the backtesting code should I add the other data from the CSV? I greatly appreciate any help or push in the right direction.

        Best regards,
        Anton

  79. Hi Andreas,
    Thanks for writing this book, it’s definitely one that I will be adding to my gift list this holidays.
    Hopefully you will send out a notification to your email list when you complete the other “different” book you alluded to during your podcast with Meb.
    I’ve now read all of your books and look forward to the next ones.

    A general question to anyone out there that may have an answer or you Andreas if you have a spare moment;
    Is there an better way to output the buy_list and new_portfolio list referenced in Chp 12 Systematic momentum
    I’d like to see the stocks each month that would make it into the portfolio.
    Am currently using the following inserted at the end of the “Stock Selection Logic” section

    df = ranking_table.loc[
    ~ranking_table.index.isin(kept_positions)][:replacement_stocks].copy()

    The original section starts as follows;

    … buy_list = ranking_table.loc[
    ~ranking_table.index.isin(kept_posi…….

    new_portfolio = pd.concat(
    (buy_list,
    ranking_table.loc[ranking_table.index.isin(kep……..

    Regards,
    Tet

    • Hi Tet,

      Once you’ve got your data in a neat DataFrame, you can do anything you like to it. You could .to_csv(), to_clipboard(), make a little function that emails a nicely formatted table to you, or pretty much anything you can think of. The .copy() will just return a copy of the DataFrame, not output it.

  80. Hi Andreas,
    Yes I figured if I could get the buy list to a dataframe I can then start to get fancy.
    For the buy list I have tried all of the following typically resulting in “df not defined” error

    df = buy_list
    df = [[‘buy_list’]]
    df = ranking_table.loc[
    ~ranking_table.index.isin(kept_positions)][:replacement_stocks].copy()

    If I try to get the function “.to_csv as in below it does create the file “buylist1” but this alters the back test as each month now has a 0 return.
    buy_list = ranking_table.loc[
    ~ranking_table.index.isin(kept_positions)][:replacement_stocks].to_csv(‘buylist1’)

    I appreciate any pointers and I’d like to understand rather than have it done for me it at all possible.
    Regards,
    Tet

  81. Hey Andreas,

    Loving the book!

    Anyone workout a solution for mac?

    using:
    export QUANDL_API_KEY=”my api key”

    zipline ingest -b quandl

    setting the API key seems to work but get a “command not found” when trying the ingest line.

    Thanks!
    Mark

  82. Hi Andreas,

    Awesome book, it’s teaching me a lot!

    I’m at the part in the book where I’m trying to ingest data bundles into zipline (Chapter 23). I need your help to figure out where to find the random historical data & the metadata lookup for the futures data part. I’ve browsed this site and looked at the dropbox file @ https://www.dropbox.com/s/tj85sufbsi820ya/Trading%20Evolved.zip and could not find the data.

    I could probably download similar data online but, being new at this game, I’m not sure how those two data files should be setup. The actual files or examples of how those files are structured would be very helpful.

    Best,

  83. Dear Andreas,
    First, I want to extend my sincere gratitude for publishing this book. I’ve never coded before, but bought both Kindle and softcover versions, and have worked through the book cover to cover.

    I’ve set up a MySQL database, started with CSI Data for Futures, and want to make sure I am downloading the appropriate data from CSI.

    My understanding of Zipline (per the book), is that it will back adjust and trade at the prices of the actual futures contract for a given close date.

    If that’s the case, should I omit downloading/importing into MySql the back adjusted time series files from CSI and only load the files for the actual contracts – ie., include ED20J but omit the ED_B & ED_N files?

    It looks like the _B _N files are adjusted series created by CSI.

    I appreciate any help from you and the community.

    Take care,
    Ben

  84. Andreas,
    Book Version 1.2. Some possible errata:
    1. In Ch.16, pg. 279, you pass the average 20-day volume to the position size function, but on pg. 278, the position size function does not use it. I assume you would add the logic from pg. 264 where you take the min of contracts_to_trade and contracts_cap, i.e.,
    contracts = min(target_variation/contract_variation, avg_volume*0.2)
    2. In Ch.17, pg. 293, you mention that you will add an updating chart to show the exposure changes. You have omitted the update_chart function from Ch.16, pg. 277 and the scheduling of it.
    3. In Ch.17, pg. 296, I would suggest replacing ‘if context.bars_held[root] >= 20’ with ‘if context.bars_held[root] >= days_to_hold’ to use the variable defined in Model Settings.

    Hope that helps, Adnan

  85. Dear Andreas,
    As you recommended, I combined ETF (Fund) and Equity prices on Quandl and ingested the data. Also, I have a way to ingest fundamental data. My momentum backtest strategy returns the following error:

    zipline.errors.SymbolNotFound: Symbol ‘SPY’ was not found.

    Could you please elaborate on this?

  86. I did ingest the SPY through the ETF lists from the FUND data in Quandl. Here what is not working:

    history = data.history(symbol(“SPY”), ‘close’, 252, ‘1d’) : NOT working

    history = data.history(sid(8554), ‘close’, 252, ‘1d’): working

    • I was able to address it by changing the ingestion codes in zipline and combining the ETF and equity tables.

  87. Benjamin AUBRY

    Hi Andreas,

    Going through your book, just the beginning of it right – installation part and I am getting errors when trying to install nb_conda – kindly see below

    Any idea how I could resolve this as I am unable to open juypter notebooks in this anaconda env (zip35 created)

    Many thanks for your help !

    Collecting package metadata (current_repodata.json): done
    Solving environment: failed with initial frozen solve. Retrying with flexible solve.
    Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
    Collecting package metadata (repodata.json): done
    Solving environment: failed with initial frozen solve. Retrying with flexible solve.
    Solving environment: /
    Found conflicts! Looking for incompatible packages.
    This can take several minutes. Press CTRL-C to abort.
    failed

    UnsatisfiableError: The following specifications were found
    to be incompatible with the existing python installation in your environment:

    Specifications:

    nb_conda -> python[version=’>=3.8,<3.9.0a0']
    Your python: python=3.5

    If python is on the left-most side of the chain, that's the version you've asked for.
    When python appears to the right, that indicates that the thing on the left is somehow
    not available for the python version you are constrained to. Note that conda will not
    change your python version to a different minor version unless you explicitly specify
    that.

  88. Hi, I’m on chapter 8 and I got the code to run successfully. Quick question:

    How do I adjust the time period for the “Analyzing Backtest Results” backtest code?

    Instead of

    start = datetime(2003, 1, 1, tzinfo=pytz.UTC)
    end = datetime(2017, 12, 31, tzinfo=pytz.UTC)

    I changed it to

    start = datetime(2003, 1, 1, tzinfo=pytz.UTC)
    end = datetime(2019, 12, 31, tzinfo=pytz.UTC)

    and the Returns graphs just have a straight line to the right, after 2017.

    Thanks!

    • Does anyone else have a response to this one?

      I’m just curious if it’s because the Quandl data is limited or if I didn’t change the code correctly.

      Thank you!

      • I have the same issue.
        I see the issue start as of March 28th 2018;
        All positions are closed at that day and nothing happens going forward.

    • The answer is just quite simple. After the year 2017 there is no free data from quandl available. This should be the reason. Without data there are no trading signals, which leads to a flat line.

      Regards, Sebastian

  89. Hi,

    I was able to run examples in chapter 7 with no problem, after some fixes but in chapter 8 and getting these errors and dont find the issue:

    Does someone know what could happen?

    ValueError Traceback (most recent call last)
    in
    104 capital_base=10000,
    105 data_frequency = ‘daily’,
    –> 106 bundle=’quandl’
    107 )

    ~\anaconda3\envs\zip35\lib\site-packages\zipline\utils\run_algo.py in run_algorithm(start, end, initialize, capital_base, handle_data, before_trading_start, analyze, data_frequency, data, bundle, bundle_timestamp, trading_calendar, metrics_set, default_extension, extensions, strict_extensions, environ, blotter)
    428 local_namespace=False,
    429 environ=environ,
    –> 430 blotter=blotter,
    431 )

    ~\anaconda3\envs\zip35\lib\site-packages\zipline\utils\run_algo.py in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, data, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, environ, blotter)
    227 ).run(
    228 data,
    –> 229 overwrite_sim_params=False,
    230 )
    231

    ~\anaconda3\envs\zip35\lib\site-packages\zipline\algorithm.py in run(self, data, overwrite_sim_params)
    760 daily_stats = self._create_daily_stats(perfs)
    761
    –> 762 self.analyze(daily_stats)
    763 finally:
    764 self.data_portal = None

    ~\anaconda3\envs\zip35\lib\site-packages\zipline\algorithm.py in analyze(self, perf)
    474
    475 with ZiplineAPI(self):
    –> 476 self._analyze(self, perf)
    477
    478 def __repr__(self):

    in analyze(context, perf)
    88 def analyze(context, perf):
    89 # Use PyFolio to generate a performance report
    —> 90 returns, positions, transactions = pf.utils.extract_rets_pos_txn_from_zipline(perf)
    91 pf.create_returns_tear_sheet(returns, benchmark_rets=None)
    92

    ValueError: too many values to unpack (expected 3)

    • That rings a bell… Make sure you’ve got the latest version of PyFolio installed. I seem to recall from years ago that you’ll get this error with an older version. Note that if you use conda to install PyFolio, you most likely got the wrong version. They haven’t updated that repo, and you’ll need to install using pip.

    • Use this instead with the fourth argument needed
      returns, positions, transactions, leverage = pf.utils.extract_rets_pos_txn_from_zipline(perf)

  90. Hello, any idea how to fix this wear? On page 100 when I did the zipline ingest – b quandl – I get the following –

    Traceback (most recent call last):
    File “/Library/anaconda3/envs/zip35/bin/zipline”, line 11, in
    load_entry_point(‘zipline==1.3.0’, ‘console_scripts’, ‘zipline’)()
    File “/Library/anaconda3/envs/zip35/lib/python3.5/site-packages/pkg_resources/__init__.py”, line 484, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
    File “/Library/anaconda3/envs/zip35/lib/python3.5/site-packages/pkg_resources/__init__.py”, line 2707, in load_entry_point
    return ep.load()
    File “/Library/anaconda3/envs/zip35/lib/python3.5/site-packages/pkg_resources/__init__.py”, line 2325, in load
    return self.resolve()
    File “/Library/anaconda3/envs/zip35/lib/python3.5/site-packages/pkg_resources/__init__.py”, line 2331, in resolve
    module = __import__(self.module_name, fromlist=[‘__name__’], level=0)
    File “/Library/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/__init__.py”, line 23, in
    from . import data
    File “/Library/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/data/__init__.py”, line 1, in
    from . import loader
    File “/Library/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/data/loader.py”, line 211
    logger.info(
    ^
    IndentationError: unexpected indent

  91. Hi Andreas I encountered this error when trying to run the backtest after successfully ingesting my own futures data into zipline:

    …\dispatch_bar_reader.py in load_raw_arrays(self, fields, start_dt, end_dt, sids)
    112 t = type(asset)
    113 print(t)
    –> 114 sid_groups[t].append(asset)
    115 out_pos[t].append(i)
    116

    KeyError:

    It seems to me that this is because zipline think it is equity rather than continuous futures, does this ring any bell ? how do I fix this ? been trying for days with no luck

  92. I’m getting the same sqlite error in chapter 7. I’ve tried restarting the kernel with no luck. Is there a solution to this?

    —————————————————————————
    ValueError Traceback (most recent call last)
    in ()
    12 handle_data=handle_data,
    13 capital_base=10000,
    —> 14 data_frequency = ‘daily’, bundle=’quandl’
    15 )

    C:\Users\jhersh\anaconda3\envs\env_zipline\lib\site-packages\zipline\utils\run_algo.py in run_algorithm(start, end, initialize, capital_base, handle_data, before_trading_start, analyze, data_frequency, data, bundle, bundle_timestamp, trading_calendar, metrics_set, default_extension, extensions, strict_extensions, environ, blotter)
    428 local_namespace=False,
    429 environ=environ,
    –> 430 blotter=blotter,
    431 )

    C:\Users\jhersh\anaconda3\envs\env_zipline\lib\site-packages\zipline\utils\run_algo.py in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, data, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, environ, blotter)
    139 bundle,
    140 environ,
    –> 141 bundle_timestamp,
    142 )
    143

    C:\Users\jhersh\anaconda3\envs\env_zipline\lib\site-packages\zipline\data\bundles\core.py in load(name, environ, timestamp)
    522 return BundleData(
    523 asset_finder=AssetFinder(
    –> 524 asset_db_path(name, timestr, environ=environ),
    525 ),
    526 equity_minute_bar_reader=BcolzMinuteBarReader(

    C:\Users\jhersh\anaconda3\envs\env_zipline\lib\site-packages\zipline\assets\assets.py in __init__(self, engine, future_chain_predicates)
    265 @preprocess(engine=coerce_string_to_eng(require_exists=True))
    266 def __init__(self, engine, future_chain_predicates=CHAIN_PREDICATES):
    –> 267 self.engine = engine
    268 metadata = sa.MetaData(bind=engine)
    269 metadata.reflect(only=asset_db_table_names)

    C:\Users\jhersh\anaconda3\envs\env_zipline\lib\site-packages\zipline\utils\input_validation.py in preprocessor(func, argname, arg)
    790 def preprocessor(func, argname, arg):
    791 if isinstance(arg, from_):
    –> 792 return to(arg, **to_kwargs)
    793 return arg
    794 return preprocessor

    C:\Users\jhersh\anaconda3\envs\env_zipline\lib\site-packages\zipline\utils\sqlite_utils.py in check_and_create_engine(path, require_exists)
    44 def check_and_create_engine(path, require_exists):
    45 if require_exists:
    —> 46 verify_sqlite_path_exists(path)
    47 return sa.create_engine(‘sqlite:///’ + path)
    48

    C:\Users\jhersh\anaconda3\envs\env_zipline\lib\site-packages\zipline\utils\sqlite_utils.py in verify_sqlite_path_exists(path)
    33 def verify_sqlite_path_exists(path):
    34 if path != ‘:memory:’ and not os.path.exists(path):
    —> 35 raise ValueError(“SQLite file {!r} doesn’t exist.”.format(path))
    36
    37

    ValueError: SQLite file ‘C:\\Users\\jhersh/.zipline\\data\\quandl\\2020-04-30T23;16;14.486779\\assets-6.sqlite’ doesn’t exist.

  93. Reid Hutchins

    Have loved the book.

    I am having an issue when I try to ingest data. I get this error using conda with python=3.5 after a conda install. ValueError: Boolean array expected for the condition, not float64 The errata in the book says to downgrade to 0.18.1. I do that using PIP and then I get the following error. (pandas 0.18.1 (c:\users\rch\anaconda3\envs\zip35_2\lib\site-packages), Requirement.parse(‘pandas>=0.21’), {‘pandas-datareader’})

    So it appears that I need pandas= 0.18.1 and pandas >=0.21 simultaneously. I did see that having NaN’s instead of 0 for dividend values could be a problem. I do believe that my data is NaN free. Any help would be greatly appreciated… Thanks!

  94. Hi Andreas,
    Love the book. Thank you!!

    I am getting following error while creating custom bundle as suggested in Chapter 23. Appreciate your help!

    :\ProgramData\Anaconda3\envs\env_zipline\Lib\site-packages\zipline\data\bundles

    File “C:\ProgramData\Anaconda3\envs\env_zipline\lib\site-packages\pandas\core\frame.py”, line 2881, in _reindex_index
    allow_dups=False)
    File “C:\ProgramData\Anaconda3\envs\env_zipline\lib\site-packages\pandas\core\generic.py”, line 3145, in _reindex_with_indexers
    copy=copy)
    File “C:\ProgramData\Anaconda3\envs\env_zipline\lib\site-packages\pandas\core\internals.py”, line 4139, in reindex_indexer
    self.axes[axis]._can_reindex(indexer)
    File “C:\ProgramData\Anaconda3\envs\env_zipline\lib\site-packages\pandas\core\indexes\base.py”, line 2944, in _can_reindex
    raise ValueError(“cannot reindex from a duplicate axis”)
    ValueError: cannot reindex from a duplicate axis

    • Hi Andreas,
      I love your books and recently started reading ‘Trading Evolved: Anyone can Build Killer Trading Strategies in Python’. Just like your previous work, this one too is a fantastic book to read and learn.
      As far as creating your own custom data bundle for zipline is concerned, if someone has clean, quality US equity data and follows the approach suggested by you in chapter 23, it works well. However, this is a very tedious task when someone has free non-US equity data to work with.
      I struggled a lot when I wanted to use free quandl data to set up India equity bundle for BSE 500 stocks. It requires data clean up to match the XBOM trading calendar dates and padding as additional steps to achieve this. There was some help on how to set up a single asset as a csvdir data bundle but nothing much on multiple stocks.
      I would like to put up an article on how to set up India equities bundle using free quandl data. I plan to include the approach suggested by you plus the additional steps including data download, clean up, matching XBOM calendar, and finally bundle set up. It will have all due credit to you. Please let me know if you are good with this.
      This will be helpful to many people.

    • Hi, I have the same problem. Did you manage to solve it?

  95. Hello Andreas!

    I’ve got a question. Since your book is close to 1 year old and there’s quite a lot of posts on this errata and updates page, are you planning to prepare 2nd edition that will gather all of this info into the actual book? It seems to be a good idea, at least for a Kindle edition which doesn’t have to be printed, distributed and so on…
    Or maybe some other users have some knowledge on this?

    Best regards.

    • Printing and distributing isn’t a big deal these days, when most books, except for dinosaur publishing houses, are printed on demand. The problem is more the time required to make a proper update in relation to the percentage of readers who will see it. Don’t get me wrong, it’s on my to-do list, but that’s a pretty long list at the moment…

      • Ok, it makes a lot of sense. Thanks for reply.

        And if I may suggest something: maybe next time (if it actually ever happens) just prepare a virtual environment (with requirements.txt) for the readers, especially those who are new to Python? Based on this forum I guess it might save you a lot of time and effort by reducing the number of “Help! I’ve got an error!” messages.

        For those who are confused: https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

      • Plenty of lessons learnt from this experience, to be sure. For one, I should have set up a proper discussion forum somewhere, instead of having all these discussions in the comments. Also, I should probably have used Docker and provided a ready-to-go virtual environment.

        Well, It’s like all my books, in a way. I never claim to be the best. I’m not sure I even aim for that. But I hope that I’m better than other authors at explaining and making things accessible to a broader audience.

      • Actually I think you’re pretty unique in this “retail investor book market” and that makes you the best by definition 😉 Jokes aside, I really believe it.

        Sorry for wasting your time, but I need to ask two more questions, about the book content this time.

        1. Page 107, “Your first Zipline backtest”, while creating the 1st system based on 100 day average and close price of the Apple stock: you say “As we requested the
        same amount of data points as the moving average period, all we need to do is to
        calculate the mean of those prices. No need to actually “move” the average, is
        there.” How can it be true if you use the last day close price to calculate the 100 day mean? I’ll paste a quote from you (page 67) that I find adequate: “But, here’s an important part in terms of the logic. At this point in the code, the position column changes the same day as the average crosses over. That is, we trade instantly at the closing price, the same value that we base our average calculation on, and thereby our signal. That’s clearly cheating. To make this even reasonably fair, we need to delay the trade to the day after the signal.”

        2. Page 67 (yes, 67 again), “Simple Python simulation”, while shifting data to overcome cheating mentioned below. I strongly believe that you should shift it by 2 periods – first because of the reason you explained and second because you need one more day to actually produce any daily return. There’s a code with fake data that justifies my opinion – just look at the stock price when you enter and exit the market and compare it to the ‘Strategy’ and ‘Strategy_Andreas’ columns:

        import pandas as pd
        import numpy as np

        stock = [*([1000]*19), 1100, 1200, 1100, 1200, 900, 900, 800, 700, 600, 500]
        index = pd.date_range(‘2020-06-08′, periods=len(stock), freq=’B’)

        df = pd.DataFrame({‘Price’:stock, ‘index’:index})
        df.set_index(‘index’, inplace=True)

        df[‘MA10’] = df[‘Price’].rolling(10).mean()
        df[‘MA20’] = df[‘Price’].rolling(20).mean()

        df[‘Position’] = df[‘MA10’]>df[‘MA20’]
        df[‘Position’] = df[‘Position’].shift(2)

        df[‘Position_Andreas’] = df[‘MA10’]>df[‘MA20’]
        df[‘Position_Andreas’] = df[‘Position_Andreas’].shift()

        df[‘Position_Change’] = ((df[‘Position’]==False) & (df[‘Position’].shift(-1)==True)) | ((df[‘Position’]==True) & (df[‘Position’].shift(-1)==False))

        df[‘StrategyPct’] = df[‘Price’].pct_change(1) * df[‘Position’]
        df[‘Strategy’] = (df[‘StrategyPct’] + 1).cumprod()

        df[‘StrategyPct_Andreas’] = df[‘Price’].pct_change(1) * df[‘Position_Andreas’]
        df[‘Strategy_Andreas’] = (df[‘StrategyPct_Andreas’] + 1).cumprod()

        df[‘BuyHold’] = (df[‘Price’].pct_change(1) + 1).cumprod()

        df

        Best regards,
        Maciek

  96. Hi Andreas,

    I am stuck in chapter 7 with implementation, also I went way forward with reading;
    Implementation of zipline was almost straightforward, didn’t had to downgrade anaconda, just Matplotlib.
    However, when running the algorithm I get the following error:

    SSLError: HTTPSConnectionPool(host=’api.iextrading.com’, port=443): Max retries exceeded with url: /1.0/stock/SPY/chart/5y (Caused by SSLError(SSLError(“bad handshake: Error([(‘SSL routines’, ‘ssl3_get_server_certificate’, ‘certificate verify failed’)],)”,),))

    —> 55 bundle = “quandl”)

    Question:
    if quandle is ingested and everything went well, the data must be saved locally in my machine, why do I have this kind of issues? (SSLError)

    Do you have experience with this issue?

    Thanks for any help
    Domenico

    • That’s a problem in the IEX call, which doesn’t work anymore. For some reason the Zipline code attempts to get data from IEX and as they removed their API, it fails. Odd that Q didn’t fix it after years. I thought I had already published a fix for this. Google ‘zipline iextrading’ and you’ll find a workaround.

  97. Hi Andreas,

    I followed the steps you indicate above. However I have got the following erro message

    Last login: Mon Jun 1 11:28:09 on ttys000
    . /Users/orizonferreira/opt/anaconda3/bin/activate && conda activate /Users/orizonferreira/opt/anaconda3/envs/zip35;

    The default interactive shell is now zsh.
    To update your account to use zsh, please run chsh -s /bin/zsh.
    For more details, please visit https://support.apple.com/kb/HT208050.
    (base) iMac-de-Orizon:~ orizonferreira$ . /Users/orizonferreira/opt/anaconda3/bin/activate && conda activate /Users/orizonferreira/opt/anaconda3/envs/zip35;
    (zip35) iMac-de-Orizon:~ orizonferreira$ export QUANDL_API_KEY=kjFzvzsyaj4GuKJVDgzJ
    (zip35) iMac-de-Orizon:~ orizonferreira$ zipline ingest -b quandl
    Traceback (most recent call last):
    File “/Users/orizonferreira/opt/anaconda3/envs/zip35/bin/zipline”, line 11, in
    load_entry_point(‘zipline==1.3.0’, ‘console_scripts’, ‘zipline’)()
    File “/Users/orizonferreira/opt/anaconda3/envs/zip35/lib/python3.5/site-packages/pkg_resources/__init__.py”, line 484, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
    File “/Users/orizonferreira/opt/anaconda3/envs/zip35/lib/python3.5/site-packages/pkg_resources/__init__.py”, line 2707, in load_entry_point
    return ep.load()
    File “/Users/orizonferreira/opt/anaconda3/envs/zip35/lib/python3.5/site-packages/pkg_resources/__init__.py”, line 2325, in load
    return self.resolve()
    File “/Users/orizonferreira/opt/anaconda3/envs/zip35/lib/python3.5/site-packages/pkg_resources/__init__.py”, line 2331, in resolve
    module = __import__(self.module_name, fromlist=[‘__name__’], level=0)
    File “/Users/orizonferreira/opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/__init__.py”, line 23, in
    from . import data
    File “/Users/orizonferreira/opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/data/__init__.py”, line 1, in
    from . import loader
    File “/Users/orizonferreira/opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/data/loader.py”, line 22, in
    from .benchmarks import get_benchmark_returns
    ImportError: No module named ‘zipline.data.benchmarks’
    (zip35) iMac-de-Orizon:~ orizonferreira$ conda install -c Quantopian zipline
    Collecting package metadata (current_repodata.json): done
    Solving environment: done

    # All requested packages already installed.

    (zip35) iMac-de-Orizon:~ orizonferreira$ export QUANDL_API_KEY=kjFzvzsyaj4GuKJVDgzJ
    (zip35) iMac-de-Orizon:~ orizonferreira$ zipline ingest -b quandl
    Traceback (most recent call last):
    File “/Users/orizonferreira/opt/anaconda3/envs/zip35/bin/zipline”, line 11, in
    load_entry_point(‘zipline==1.3.0’, ‘console_scripts’, ‘zipline’)()
    File “/Users/orizonferreira/opt/anaconda3/envs/zip35/lib/python3.5/site-packages/pkg_resources/__init__.py”, line 484, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
    File “/Users/orizonferreira/opt/anaconda3/envs/zip35/lib/python3.5/site-packages/pkg_resources/__init__.py”, line 2707, in load_entry_point
    return ep.load()
    File “/Users/orizonferreira/opt/anaconda3/envs/zip35/lib/python3.5/site-packages/pkg_resources/__init__.py”, line 2325, in load
    return self.resolve()
    File “/Users/orizonferreira/opt/anaconda3/envs/zip35/lib/python3.5/site-packages/pkg_resources/__init__.py”, line 2331, in resolve
    module = __import__(self.module_name, fromlist=[‘__name__’], level=0)
    File “/Users/orizonferreira/opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/__init__.py”, line 23, in
    from . import data
    File “/Users/orizonferreira/opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/data/__init__.py”, line 1, in
    from . import loader
    File “/Users/orizonferreira/opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/data/loader.py”, line 22, in
    from .benchmarks import get_benchmark_returns
    ImportError: No module named ‘zipline.data.benchmarks’

    Please, could you give me a tip on how to solve it. By the way, I’m using a mac computer.

    Thank you.

    Orizon

  98. Andreas,
    I am going through the First Zipline Backtest in your book and I am getting this error:
    Please help. Thanks
    —————————————————————————
    NameError Traceback (most recent call last)
    in
    20 equities_hist = data.history(context.stock, “close” , context.index_average_window, “1d”)
    21 #check if price is above moving average
    —> 22 if equities_hist[-1] > equities_hist.mean():
    23 stock_weight=1.0
    24 else:

    NameError: name ‘equities_hist’ is not defined

    Here is the code:
    %matplotlib inline

    #import zipline functions
    from zipline import run_algorithm
    from zipline.api import order_target_percent, symbol
    from zipline.data.bundles.quandl import quandl_bundle
    #import date and time libraries
    from datetime import datetime
    import pytz
    import matplotlib.pyplot as plt
    import quandl
    def initialize(context):
    #which stock tp trade
    context.stock=symbol(‘AAPL’)
    #moving average window
    context.index_average_window=100

    def handle_data(context, data):
    #request history for the stock
    equities_hist = data.history(context.stock, “close” , context.index_average_window, “1d”)
    #check if price is above moving average
    if equities_hist[-1] > equities_hist.mean():
    stock_weight=1.0
    else:
    stock_weight=0.0

    order_target_percent(context.stock, stock_weight)

    def analyze(context, perf):
    fig=plt.figure(figsize=(12,8))
    #first chart
    ax=fig.add_subplot(311)
    ax.set_title(‘Srategy Results’)
    ax.semilogy(perf[‘portfolio_value’], linestyle=’-‘, label=’equity curve’,linewidth=3.0)
    ax.legend()
    ax.grid(False)

    #second chart
    ax=fig.add_subplot(312)
    ax.plot(perf[‘gross_leverage’], label=’exposure’,linestyle=’-‘, linewidth=1.0)
    ax.legend()
    ax.grid(True)

    #third chart
    ax=fig.add_subplot(313)
    ax.plot(perf[‘returns’], label=’returns’,linestyle=’-‘, linewidth=1.0)
    ax.legend()
    ax.grid(True)
    #set start and end dates
    start_date=datetime(1996,1,1,tzinfo=pytz.UTC)
    end_date=datetime(2018,12,31,tzinfo=pytz.UTC)

    #backtest
    results=run_algorithm(
    start=start_date,
    end=end_date,
    initialize=initialize,
    analyze=analyze,
    handle_data=handle_data,
    capital_base=1000000,
    data_frequency=’daily’,bundle=’quandl’)

  99. Hello, I get this error when I’m trying to ingest random_stock_data, my data contains .csv files from yahoo,
    my versions of pandas and numpy (0.22.0-1.14.0), does anyone have some ideas? Thank you.

    C:\Windows\system32>zipline ingest -b random_stock_data
    Loading A…
    Traceback (most recent call last):
    File “C:\Users\neloi\anaconda3\envs\Zip35\Scripts\zipline-script.py”, line 11, in
    load_entry_point(‘zipline==1.3.0’, ‘console_scripts’, ‘zipline’)()
    File “C:\Users\neloi\anaconda3\envs\Zip35\lib\site-packages\click\core.py”, line 829, in __call__
    return self.main(*args, **kwargs)
    File “C:\Users\neloi\anaconda3\envs\Zip35\lib\site-packages\click\core.py”, line 782, in main
    rv = self.invoke(ctx)
    File “C:\Users\neloi\anaconda3\envs\Zip35\lib\site-packages\click\core.py”, line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
    File “C:\Users\neloi\anaconda3\envs\Zip35\lib\site-packages\click\core.py”, line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
    File “C:\Users\neloi\anaconda3\envs\Zip35\lib\site-packages\click\core.py”, line 610, in invoke
    return callback(*args, **kwargs)
    File “C:\Users\neloi\anaconda3\envs\Zip35\lib\site-packages\zipline\__main__.py”, line 348, in ingest
    show_progress,
    File “C:\Users\neloi\anaconda3\envs\Zip35\lib\site-packages\zipline\data\bundles\core.py”, line 451, in ingest
    pth.data_path([name, timestr], environ=environ),
    File “C:\Users\neloi\anaconda3\envs\Zip35\lib\site-packages\zipline\data\bundles\random_stock_data.py”, line 63, in random_stock_data
    process_stocks(symbols, sessions, metadata, divs)
    File “C:\Users\neloi\anaconda3\envs\Zip35\lib\site-packages\zipline\data\us_equity_pricing.py”, line 257, in write
    return self._write_internal(it, assets)
    File “C:\Users\neloi\anaconda3\envs\Zip35\lib\site-packages\zipline\data\us_equity_pricing.py”, line 319, in _write_internal
    for asset_id, table in iterator:
    File “C:\Users\neloi\anaconda3\envs\Zip35\lib\site-packages\zipline\data\us_equity_pricing.py”, line 249, in
    for sid, df in data
    File “C:\Users\neloi\anaconda3\envs\Zip35\lib\site-packages\zipline\data\us_equity_pricing.py”, line 414, in to_ctable
    # we already have a ctable so do nothing
    File “C:\Users\neloi\anaconda3\envs\Zip35\lib\site-packages\zipline\data\us_equity_pricing.py”, line 417, in to_ctable
    winsorise_uint32(raw_data, invalid_data_behavior, ‘volume’, *OHLC)
    File “C:\Users\neloi\anaconda3\envs\Zip35\lib\site-packages\zipline\data\us_equity_pricing.py”, line 121, in winsorise_uint32

    File “C:\Users\neloi\anaconda3\envs\Zip35\lib\site-packages\zipline\data\us_equity_pricing.py”, line 137, in winsorise_uint32
    mask = df[columns] > UINT32_MAX
    File “C:\Users\neloi\anaconda3\envs\Zip35\lib\site-packages\pandas\core\frame.py”, line 2133, in __getitem__
    return self._getitem_array(key)
    File “C:\Users\neloi\anaconda3\envs\Zip35\lib\site-packages\pandas\core\frame.py”, line 2177, in _getitem_array
    indexer = self.loc._convert_to_indexer(key, axis=1)
    File “C:\Users\neloi\anaconda3\envs\Zip35\lib\site-packages\pandas\core\indexing.py”, line 1269, in _convert_to_indexer
    .format(mask=objarr[mask]))
    KeyError: “[‘volume’ ‘high’ ‘close’ ‘low’ ‘open’] not in index”

  100. Andreas, your book relies on a model of a chain of interdependent libraries and engines that is going to rapidly get disconnected and obsolete – probably will no longer be able to be installed and set up in a couple of years as it requires a very specific setup. The setup process as of now is pretty convoluted and fragile and it took me a couple of days before I could run my first zipline backtest. For this reason, I think it may be best if we could just code our own backtesting environment in python using the standard libraries. How difficult would that be, in your opinion? Are there any resources we could look up on that? Thank you. Nikolay

    • I understand your concern, Nikolay and I’ve been watching the increasingly negative news around Q the past year or so. Well, the good news is that the code is open source and it’s still one of the best backtesting platforms out there. It won’t stop working and the community is free to fork it.

      Whether or not you want to build your own depends on what you need. Making an all purpose backtester for many asset classes, supporting slippage, commissions, preventing data snooping, dividend handling, etc etc is a substantial undertaking, but perhaps you don’t need all of the fancy stuff.

      If you, as an example, just have an interest in long term ETF asset allocation models, you can build a nice little framework in an afternoon.

      • Since I want to build futures and stock models, this means the choice is simple:

        It’s either to spend the next couple years researching/designing a backtesting engine on my own, or spending some time to research and learn zipline as well as possible (possibly modding it). I guess the second choice is a winner then 🙂

  101. Quantopian are supposed to be working on a new release, but the downside is that there seems to be no real movement since Februrary and they furloughed their development staff (https://github.com/quantopian/zipline/issues/2664)!

    Why a lockdown affects an online trading platform/software development I have no idea, but it does slightly question their long term viability

    • Yes, I’ve been following this development as well. It would be unrealistic to expect further updates from Q at this point. I hope they’ll pull through of course.

      The good part is that the code is open source. Apart from a few weird things, it’s still one of the best Python backtesters out there. Perhaps even the best. I hope the open source community will take over, and I’d be happy to contribute.

      • Looks like Quantopian has folded. They have withdrawn all community access to their algos

      • They are shutting down their community platform, yes.

        For what it’s worth, two interesting points, though:

        1.) Right after the shutdown announcement, they’ve created a new organization on GitHub called “quantopian-enterprise” with forks of all essential repositories including zipline.

        2.) In the days leading to the shutdown announcement, Quantopian developer richafrank has been working on a zipline branch called “new-new-new”, seemingly with the goal of making zipline ready for Python 3.8 and the latest (!) pandas version. It’s not fully completed yet, though, at least the migration away from pandas’ depreciated “panel” data type hasn’t been handled yet (I’ve done a test run with the current state of that branch today).

        This leaves 2 possibilities: Either richafrank didn’t know about the coming shutdown 2 days prior to its announcement, or they will continue development, a possibility (1) points to. What further direction this project will take and if the repositories will remain open source remains to be seen.

      • I hope they have found a new, and working, business model.

        I’ve known the Quantopian guys since around 2015. Great people, many of which I consider personal friends. Their conferences, QuantCon, was probably the best of it’s kind. Their online platform a leap forward in terms of what’s available to ‘civilians’.

        I’ve been worried from the start about the day when they fold. Even though I thought them highly competent, I never bought into the business strategy. They had a great product and many ways of monetizing it, but they rejected them all in favor of the unrealistic idea of running a hedge fund based of thousands of amateur strategies.

        If I had what they had, I’d probably start a brokerage arm instead. Give all the tools, all the infrastructure and knowledge, and let them trade through an in-house brokerage. I brought that idea up with them years ago, but they were quite set on the whole crowdsourced hedge fund idea.

        Well, the writing was on the wall since early 2019 when their staff started leaving. By mid 2019, almost everyone I knew had jumped ship. All oddly unwilling to talk about why they left. After that, I expected the company to cease to exist before end of the year. If they now instead found a workable business model, that’s great news.

        Also, in case someone out there is looking for a new business: The demise of QuantCon has left a vacuum ready to be filled. I believe there’s some money on the table here, for someone to organize similar events in NY, Europe and Asia. They’ll find speaker roster by looking at the old QC events and with a bit of logistical planning it could be quite profitable too. Once the world has returned to something resembling sanity, that is…

  102. I am getting this error while running backtest
    ValueError: ‘Time Period’ is not in list

    Probably some issue with this website giving treasury returns data:
    https://www.federalreserve.gov/datadownload/Output.aspx?

    In case if you are facing the same issue & find a fix, please let me know…

    Thanks in advance!

  103. Hi Andreas, Great book and I bought Leveraged Trading on your recommendation… also a nice one to have near the computer. Quick question regarding the futures bundle (and perhaps this question applies to the equity bundle) that you wrote in Trading Evolved: after ingesting minute-level data, is there a way to see “per minute” results from the output of run_algorithm? Currently my results product only shows per day performance where I would like to see the per minute, intraday, activity of my backtest. In case I’m not clear: in my results output I see one row of data representing a day’s worth of activity when I would like to see approximately 450 rows (number of minutes in a trading day) of data for each day. Thanks

  104. Hi Andreas

    Thank you for making writing this book!

    I am an experienced python user, but getting things going with zipline has turned out to be one of the most trying experiences in recent memory, so I’m hoping to get some help from the community!

    A couple questions:

    1. is anyone able to run the 2 examples in ch-7 with python 3.5.6 and pandas 0.25?

    2. For those two example notebooks, I got the following error msgs, and I am at loss on how to fix it:

    TypeError: Already tz-aware, use tz_convert to convert.

    And I haven’t found pertinent solutions on StackOverflow or Pandas documentation. Any help is appreciated!

    Best,
    SH

  105. Hi Andreas,

    While I am trying to follow your instruction in Chapter 7 to run a simple backtest, the data seems to stop at somewhere in April 2018. I tried to use other stock codes instead of AAPL, it just doesn’t work.
    Is there any update regarding the Quandl bundle or it’s no longer providing data after April 2018?

    Thanks

  106. Thanks for a great book. I am learning a lot and have since purchased your other books to read on the heels of this.

    Like others here, has anyone successfully been able to load and run the Futures models from Chapter 15?

    I also get the error:

    Exception ignored in: ‘zipline.assets.continuous_futures.OrderedContracts.contract_before_auto_close’
    AttributeError: ‘NoneType’ object has no attribute ‘next’

    This is after using the code as offered in the data download, and adding in the sorting the symbols as noted in several posts. It is exceedingly difficult to learn the concepts when the samples don’t work.

  107. Dear Andreas,

    As I was trying the first backtest code, i found myself running into several errors. I used the various comments on this page to sort out them by reinstalling Anaconda and using the env_zipline environment. I also corrected the Patchworking error. however, i am still running into an error which I am not sure of as to how do I correct.

    Absolute beginner with Python so possible that the error is my own doing. Your help will be much appreciated.

    —————————————————————————
    NameError Traceback (most recent call last)
    in ()
    37 #CHARTS
    38 #First subplot
    —> 39 ax=fig.add_subplot(311)
    40 ax.set_title(‘SIMPLE SMA CROSSOVER STRATEGY’)
    41 ax.semilogy(perf[‘portfolio value’],linestyle=’-‘, label=’Equity Curve’, linewidth=3.0)

    NameError: name ‘fig’ is not defined
    ____________________________________________________________________________
    the code is below:

    #this insures that our graphs will be shown properly within the Notebook
    %matplotlib inline

    #Import zipline functions that are needed
    from zipline import run_algorithm
    from zipline.api import order_target_percent, symbol

    #Import date and time zone libraries
    from datetime import datetime
    import pytz

    #Import visualizations
    import matplotlib.pyplot as plt

    def initialize(context):
    #Which instrument to trade
    context.stock=symbol(‘AAPL’)
    #MA Window
    context.index_average_window=100

    def handle_data(context,data):
    #Request data history – this segment specifies how much data to be pulled for each calculation
    equities_hist=data.history(context.stock,”close”,
    context.index_average_window,”1d”)
    #Check if price abobe MA
    if equities_hist[-1] > equities_hist.mean():
    stock_weight = 1.0
    else:
    stock_weight = 0.0
    #Order entry process – target weight based
    order_target_percent(context.stock,stock_weight)

    def analyze(context,perf):
    fig=plt.figure(figsize=(12,8))

    #CHARTS
    #First subplot
    ax=fig.add_subplot(311)
    ax.set_title(‘SIMPLE SMA CROSSOVER STRATEGY’)
    ax.semilogy(perf[‘portfolio value’],linestyle=’-‘, label=’Equity Curve’, linewidth=3.0)
    ax.legend()
    ax.grid(False)

    #Second Chart
    ax = fig.add_subplot(312)
    ax.plot(perf[‘gross leverage’], label=’Exposure’, linestyle=’-‘, linewidth=2.0)
    ax.legend()
    ax.grid(True)

    #Third chart
    ax = fig.add_subplot(313)
    ax.plot(perf[‘returns’], label=’Returns’, linestyle=’-.’, linewidth=1.0)
    ax.legend()
    ax.grid(True)

    #Set start and end date
    start_date=datetime(1996,1,1,tzinfo=pytz.UTC)
    end_date=datetime(2020,7,4,tzinfo=pytz.UTC)

    #Fire off the backtest
    results=run_algorithm(start=start_date,end=end_date, initialize=initialize,analyze=analyze,
    handle_data=handle_data,capital_base=1000000,data_frequency=’daily’, bundle=’quandl’)

    • Looks like you’re missing tabs at the start of the lines for your code in the analyze function. It doesn’t display well here in the comments section, but I can see the characters copy/pasted in the console. In your analyze function, only the first line is probably aligned, the rest lack the initial tab required.

      Demonstrating, using —– to stand in for the tab, as the white space will be tripped by WordPress.

      def analyze(context,perf):
      —–fig=plt.figure(figsize=(12,8)) # << Correct #CHARTS #First subplot ax=fig.add_subplot(311) # << not correct! Python will try to run this right away, and not see it as part of the analyze function, where you define fig. Therefore, fig is not recognized. Solution is to tab this section out, so it will form part of analyze().

      • Thanks very much Andreas. I found my error which as you pointed out had to do with indentation within the analyze function. many thanks for your help.

  108. Hello sir thanks for writing such an awesome and useful book , while implementing first zipline test , i stuck with an error :
    ” JSONDecodeError: Expecting value: line 1 column 1 (char 0) “
    when i searched i got an issue in github that ” Getting benckmark data via IEX API does not work anymore “
    , i haven’t tried any solution as i have my own bought data ,but request you to please post a solution for this in this forum for beginners .

  109. Hi Andreas,

    I am really enjoying the book so far.

    I am getting a problem in chapter 24. I am noticing that the populating database code is actually overwriting over dates, such that there is only one row per symbol left and thus when that particular symbol is called using query, I am only getting one data point per symbol. I am assuming that this is due to the handle duplicates part overwriting values at duplicate keys. Is there a way to avoid overwriting over the dates?

    Thanks in advance!

  110. Hi

    I am having problems on loading all the assets from my custom bundle and could not find any solution to get a list of available assets from the bundle.

    The example in the book takes the asset symbols from a csv file.

    Is there a way to load all the assets available in my custom bundle without having to loading them from a custom csv or from my database which was used to create the bundle?

    Rgds

  111. I followed the instructions in the book under MacOS Catalina without performing any patches or forced downgrades (except for Python 3.5). To get the first zipline example running I had to make a few changes:

    — first_zipline.py 2020-09-18 13:43:14.000000000 +0200
    +++ first_zipline_fixed.py 2020-09-18 13:44:03.000000000 +0200
    @@ -3,7 +3,9 @@

    # Import Zipline functions that we need
    from zipline import run_algorithm
    -from zipline.api import order_target_percent, symbol
    +from zipline.api import order_target_percent, symbol, set_benchmark
    +
    +import pandas as pd

    # Import date and time zone libraries
    from datetime import datetime
    @@ -20,6 +22,8 @@
    # Moving average window
    context.index_average_window = 100

    + set_benchmark(False)
    +
    def handle_data(context, data):
    # Request history for the stock
    equities_hist = data.history(context.stock, “close”,
    @@ -59,8 +63,10 @@
    ax.grid(True)

    # Set start and end date
    -start_date = datetime(1996, 1, 1, tzinfo=pytz.UTC)
    -end_date = datetime(2018, 12, 31, tzinfo=pytz.UTC)
    +start_date = pd.Timestamp(datetime(2000, 1, 1, tzinfo=pytz.UTC))
    +end_date = pd.Timestamp(datetime(2017, 12, 31, tzinfo=pytz.UTC))
    +# start_date = datetime(1996, 1, 1, tzinfo=pytz.UTC)
    +# end_date = datetime(2018, 12, 31, tzinfo=pytz.UTC)

    # Fire off the backtest
    results = run_algorithm(

    For some reason I had to limit the timeframe for zipline to not complain about missing data.

    HTH.

  112. Hello everyone.

    By using the “benchmarks.py” fix zipline version 1.3 the backtest works without any problems.

    However, a new version of zipline 1.4 exists and similar issues regarding the “benchmarks.py” appear.
    Anyone has an idea of what needs to be done in the “benchmarks.py” file to make the all thing work with zipline 1.4?

    Thank you in advance.

    best
    FJ

  113. Hello Andreas,
    Really appreciate the book. It’s been a very long time since I programmed, so the level of detail is great for me. I’m on Chapter 7, Backtesting Trading Strategies, and trying to set up the Zipline environment. You indicate to run Python 3.5, but Anaconda Navigator now only allows me to select Python 3.8 (no drop-down to select 3.5). I did find v3.5.4, v3.5.5 and v3.5.6 available on Anaconda’s website at https://anaconda.org/anaconda/python/files?version=3.5.6. Should one of these work? I’m assuming I can figure out how to get it to work, but I also assume this will take some time, so if you have any recommendations about how to get these working, that also would be greatly appreciated.
    Regards,
    Robert

  114. Andreas,

    In the book, you size positions based on percent of inverse volatility. Have you considered using percent of sharpe ratio instead? I haven’t done extensive testing yet, but it appears that it could produce better returns with similar overall sharpe ratio and larger but still acceptable drawdowns.

  115. Hello,

    My question about My First Backtest is no longer relevant. Changed from Zipline 1.4 to 1.3 and made the “cache changes” in loader.py to two functions and it started to work 🙂

    Best,
    Mikko

  116. I have tried to search for a solution to this, but could not find one. In the jupyter notebook for ch16, the trend model has a list of futures symbols. many of these throw up an error suggesting the symbol isn’t recognised. I am assuming it is something that is happening in zipline.api future_symbol but cant work it out. Is this something you have come across or even better solved?
    Thanks
    Mark

  117. I am trying to do the momentum model example from Chapter 12 and keep running into this error:
    TypeError: Cannot compare type ‘Timestamp’ with type ‘str’

    Related to the code:
    —-> 9 all_prior = context.index_members.loc[context.index_members.index < today]

    I've tried googling and solving it on my own but my python skills are not up to the task. 'today' is of type Timestamp and it appears that when it runs the comparison it converts each date in the index to a 'str'?
    Any help would be appreciated.

  118. Book version 1.2, page 108, “In that segment we don’t actually trade. Not yet. We just set the value of the variable target_weight.” target_weight should be stock_weight per the code snippet just above it.

  119. I’m finding that the chapter 8 example in the book does not match the results I’m seeing. FYI, I’m using zipline 1.4.1 and Python 3.6. I’ve found that zipline 1.4.1 now uses pandas timestamps instead of datetimes. This is the only change that I needed to make to the author’s code to get things working, i.e.:

    start = pd.Timestamp(‘2003-01-01′, tz=’utc’)
    end = pd.Timestamp(‘2017-12-31′, tz=’utc’)

    Up until chapter 8, I’ve had no issues. With chapter 8, I am seeing the following differences:

    – Cumulative returns in book is 295.2%. My results show a slight difference of 295.3%. All other values are exactly the same as in “Table 8.2 PyFolio Key Ratios” in the book
    – Some graphs in the book look slightly different, especially the underwater plot
    – The main difference I’m seeing is in the Day Snapshot for 2009-03-17. The set of top 10 performers for this day are completely different to what’s shown in the book, and the cash position is negative, causing an exception when plotting the pie graph.

    Note that I am using the author’s *exact* code examples, unmodified apart from changing from datetime to pandas timestamps. The only things that I can think of that might be going wrong here are:

    – Data issues – i.e. differences in the data used for the book compared with the data I’m using as of today’s date (still using the quandl bundle, data last ingested 4 Nov 2020 – but still using the same historical range of data as in the book)
    – Code changes or bug fixes (or bugs introduced) sometime between when the book was written and today
    – The author ran code for a different day to what’s shown in the book (highly unlikely)
    – I’ve missed something. Given that my results are extremely similar, apart from 0.1% out for cumulative returns, I don’t see this as being as likely as the first two points – but I’m not ruling it out.

    Has anyone else seen this at all, or able to reproduce it?

    Cheers,
    Dan

  120. Dan,

    It’s the quandl data. Quandl has stopped maintaining the free stock data as of April 2018, and even the existing data prior to this date has somehow managed to deteriorate (at least that’s what I have experienced after ingesting it). Lots of NaNs which tend to derail the position sizing algorithm, sometimes leading to grossly incorrect sizes and even negative ones, resulting in short positions in long-only systems. If you can, use another data source.

    https://www.quandl.com/databases/WIKIP/documentation

  121. Hello,
    Below error returns in Chapter 8 ( Backtest Analysis ) when analyze function is executed. How may I solve this problem ?
    I have env_zip35 installed with latest pyfolio via pip.
    Thank you.

    C:\Users\user\.conda\envs\env_zipline\lib\site-packages\empyrical\utils.py:442: UserWarning: Yahoo Finance read failed:
    Yahoo Actions has been immediately deprecated due to large breaks in the API without the
    introduction of a stable replacement. Pull Requests to re-enable these data
    connectors are welcome.

    See https://github.com/pydata/pandas-datareader/issues
    , falling back to Google
    UserWarning)
    C:\Users\user\.conda\envs\env_zipline\lib\site-packages\pandas_datareader\google\daily.py:40: UnstableAPIWarning:
    The Google Finance API has not been stable since late 2017. Requests seem
    to fail at random. Failure is especially common when bulk downloading.

    warnings.warn(UNSTABLE_WARNING, UnstableAPIWarning)

    —————————————————————————
    ImmediateDeprecationError Traceback (most recent call last)
    ~\.conda\envs\env_zipline\lib\site-packages\empyrical\utils.py in get_symbol_returns_from_yahoo(symbol, start, end)
    434 try:
    –> 435 px = web.get_data_yahoo(symbol, start=start, end=end)
    436 px[‘date’] = pd.to_datetime(px[‘date’])

    ~\.conda\envs\env_zipline\lib\site-packages\pandas_datareader\data.py in get_data_yahoo(*args, **kwargs)
    62 def get_data_yahoo(*args, **kwargs):
    —> 63 raise ImmediateDeprecationError(DEP_ERROR_MSG.format(‘Yahoo Actions’))
    64 return YahooDailyReader(*args, **kwargs).read()

    ImmediateDeprecationError:
    Yahoo Actions has been immediately deprecated due to large breaks in the API without the
    introduction of a stable replacement. Pull Requests to re-enable these data
    connectors are welcome.

    See https://github.com/pydata/pandas-datareader/issues

    During handling of the above exception, another exception occurred:

    RemoteDataError Traceback (most recent call last)
    in ()
    103 capital_base=10000,
    104 data_frequency = ‘daily’,
    –> 105 bundle=’quandl’
    106 )

    ~\.conda\envs\env_zipline\lib\site-packages\zipline\utils\run_algo.py in run_algorithm(start, end, initialize, capital_base, handle_data, before_trading_start, analyze, data_frequency, data, bundle, bundle_timestamp, trading_calendar, metrics_set, default_extension, extensions, strict_extensions, environ, blotter)
    428 local_namespace=False,
    429 environ=environ,
    –> 430 blotter=blotter,
    431 )

    ~\.conda\envs\env_zipline\lib\site-packages\zipline\utils\run_algo.py in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, data, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, environ, blotter)
    227 ).run(
    228 data,
    –> 229 overwrite_sim_params=False,
    230 )
    231

    ~\.conda\envs\env_zipline\lib\site-packages\zipline\algorithm.py in run(self, data, overwrite_sim_params)
    760 daily_stats = self._create_daily_stats(perfs)
    761
    –> 762 self.analyze(daily_stats)
    763 finally:
    764 self.data_portal = None

    ~\.conda\envs\env_zipline\lib\site-packages\zipline\algorithm.py in analyze(self, perf)
    474
    475 with ZiplineAPI(self):
    –> 476 self._analyze(self, perf)
    477
    478 def __repr__(self):

    in analyze(context, perf)
    88 # Use PyFolio to generate a performance report
    89 returns, positions, transactions = pf.utils.extract_rets_pos_txn_from_zipline(perf)
    —> 90 pf.create_returns_tear_sheet(returns, benchmark_rets=None)
    91
    92

    ~\.conda\envs\env_zipline\lib\site-packages\pyfolio\plotting.py in call_w_context(*args, **kwargs)
    50 if set_context:
    51 with plotting_context(), axes_style():
    —> 52 return func(*args, **kwargs)
    53 else:
    54 return func(*args, **kwargs)

    ~\.conda\envs\env_zipline\lib\site-packages\pyfolio\tears.py in create_returns_tear_sheet(returns, positions, transactions, live_start_date, cone_std, benchmark_rets, bootstrap, return_fig)
    442
    443 if benchmark_rets is None:
    –> 444 benchmark_rets = utils.get_symbol_rets(‘SPY’)
    445
    446 returns = returns[returns.index > benchmark_rets.index[0]]

    ~\.conda\envs\env_zipline\lib\site-packages\pyfolio\utils.py in get_symbol_rets(symbol, start, end)
    598 return SETTINGS[‘returns_func’](symbol,
    599 start=start,
    –> 600 end=end)

    ~\.conda\envs\env_zipline\lib\site-packages\pyfolio\deprecate.py in wrapper(*args, **kwargs)
    41 stacklevel=stacklevel
    42 )
    —> 43 return fn(*args, **kwargs)
    44 return wrapper
    45 return deprecated_dec

    ~\.conda\envs\env_zipline\lib\site-packages\pyfolio\utils.py in default_returns_func(symbol, start, end)
    399 – See full explanation in tears.create_full_tear_sheet (returns).
    400 “””
    –> 401 return empyrical.utils.default_returns_func(symbol, start=None, end=None)
    402
    403

    ~\.conda\envs\env_zipline\lib\site-packages\empyrical\deprecate.py in wrapper(*args, **kwargs)
    41 stacklevel=stacklevel
    42 )
    —> 43 return fn(*args, **kwargs)
    44 return wrapper
    45 return deprecated_dec

    ~\.conda\envs\env_zipline\lib\site-packages\empyrical\utils.py in default_returns_func(symbol, start, end)
    488 symbol=’SPY’,
    489 start=’1/1/1970′,
    –> 490 end=datetime.now())
    491 rets = rets[start:end]
    492 else:

    ~\.conda\envs\env_zipline\lib\site-packages\empyrical\deprecate.py in wrapper(*args, **kwargs)
    41 stacklevel=stacklevel
    42 )
    —> 43 return fn(*args, **kwargs)
    44 return wrapper
    45 return deprecated_dec

    ~\.conda\envs\env_zipline\lib\site-packages\empyrical\utils.py in get_returns_cached(filepath, update_func, latest_dt, **kwargs)
    322
    323 if update_cache:
    –> 324 returns = update_func(**kwargs)
    325 try:
    326 ensure_directory(cache_dir())

    ~\.conda\envs\env_zipline\lib\site-packages\empyrical\deprecate.py in wrapper(*args, **kwargs)
    41 stacklevel=stacklevel
    42 )
    —> 43 return fn(*args, **kwargs)
    44 return wrapper
    45 return deprecated_dec

    ~\.conda\envs\env_zipline\lib\site-packages\empyrical\utils.py in get_symbol_returns_from_yahoo(symbol, start, end)
    441 ‘Yahoo Finance read failed: {}, falling back to Google’.format(e),
    442 UserWarning)
    –> 443 px = web.get_data_google(symbol, start=start, end=end)
    444 rets = px[[‘Close’]].pct_change().dropna()
    445

    ~\.conda\envs\env_zipline\lib\site-packages\pandas_datareader\data.py in get_data_google(*args, **kwargs)
    57
    58 def get_data_google(*args, **kwargs):
    —> 59 return GoogleDailyReader(*args, **kwargs).read()
    60
    61

    ~\.conda\envs\env_zipline\lib\site-packages\pandas_datareader\base.py in read(self)
    204 if isinstance(self.symbols, (compat.string_types, int)):
    205 df = self._read_one_data(self.url,
    –> 206 params=self._get_params(self.symbols))
    207 # Or multiple symbols, (e.g., [‘GOOG’, ‘AAPL’, ‘MSFT’])
    208 elif isinstance(self.symbols, DataFrame):

    ~\.conda\envs\env_zipline\lib\site-packages\pandas_datareader\base.py in _read_one_data(self, url, params)
    82 “”” read one data from specified URL “””
    83 if self._format == ‘string’:
    —> 84 out = self._read_url_as_StringIO(url, params=params)
    85 elif self._format == ‘json’:
    86 out = self._get_response(url, params=params).json()

    ~\.conda\envs\env_zipline\lib\site-packages\pandas_datareader\base.py in _read_url_as_StringIO(self, url, params)
    93 Open url (and retry)
    94 “””
    —> 95 response = self._get_response(url, params=params)
    96 text = self._sanitize_response(response)
    97 out = StringIO()

    ~\.conda\envs\env_zipline\lib\site-packages\pandas_datareader\base.py in _get_response(self, url, params, headers)
    153 msg += ‘\nResponse Text:\n{0}’.format(last_response_text)
    154
    –> 155 raise RemoteDataError(msg)
    156
    157 def _get_crumb(self, *args):

    RemoteDataError: Unable to read URL: https://finance.google.com/finance/historical?output=csv&startdate=Jan+01%2C+1970&enddate=Nov+08%2C+2020&q=SPY
    Response Text:
    b’Sorry… body { font-family: verdana, arial, sans-serif; background-color: #fff; color: #000; }GoogleSorry…We\’re sorry…… but your computer or network may be sending automated queries. To protect our users, we can\’t process your request right now.See Google Help for more information.Google Home

    In [ ]:

    • Hello,
      I noticed that version 0.8.0 was installed instead of 0.9.2 which is the latest version as of now. After updating to the latest version of Pyfolio, the problem is solved.

  122. While Q goes through its changes, many of us readers may just want to get a working installation of zipline going so that we can best following along in the book.

    My first working pass at installing zipline:

    1) conda create -n zip35 python=3.5 anaconda
    2) conda activate zip35
    3) conda install -c Quantopian zipline
    4) pip uninstall gevent

    verify by running “python” (windows) & “import zipline”

  123. Hello everyone,

    hope that anyone can help me to fix this issue. I created the zip35 environment as in chapter 7 stated. I did the replacement work in benchmarks.py and the bypassing in loader.py as well as installing the nb_conda lib and matplotlib. Because my Environment (Python 3.5, Zipline 1.3) i could not install the matplotlib and nb_conda via GUI an had to do it through the Terminal. I have a Windw10 64bit but not the latest version of anaconda navigator (dont know if this is relevant). Also I did the ingesting with the Terminal of my zip35 environment. So my problem is, every time I want to run the code from “First backline testing” this error occurs:

    —————————————————————————
    FileNotFoundError Traceback (most recent call last)
    ~\anaconda3\envs\zip35\lib\site-packages\matplotlib\font_manager.py in
    1352 try:
    -> 1353 fontManager = json_load(_fmcache)
    1354 if (not hasattr(fontManager, ‘_version’) or

    ~\anaconda3\envs\zip35\lib\site-packages\matplotlib\font_manager.py in json_load(filename)
    887 “””
    –> 888 with open(filename, ‘r’) as fh:
    889 return json.load(fh, object_hook=_json_decode)

    FileNotFoundError: [Errno 2] No such file or directory: ‘C:\\Users\\D\\.matplotlib\\fontlist-v300.json’

    During handling of the above exception, another exception occurred:

    TypeError Traceback (most recent call last)
    in
    1 # This ensures that our graphs will be shown properly in the notebook.
    —-> 2 get_ipython().run_line_magic(‘matplotlib’, ‘inline’)
    3
    4 # Import Zipline functions that we need
    5 from zipline import run_algorithm

    ~\anaconda3\envs\zip35\lib\site-packages\IPython\core\interactiveshell.py in run_line_magic(self, magic_name, line, _stack_depth)
    2283 kwargs[‘local_ns’] = sys._getframe(stack_depth).f_locals
    2284 with self.builtin_trap:
    -> 2285 result = fn(*args,**kwargs)
    2286 return result
    2287

    in matplotlib(self, line)

    ~\anaconda3\envs\zip35\lib\site-packages\IPython\core\magic.py in (f, *a, **k)
    185 # but it’s overkill for just that one bit of state.
    186 def magic_deco(arg):
    –> 187 call = lambda f, *a, **k: f(*a, **k)
    188
    189 if callable(arg):

    ~\anaconda3\envs\zip35\lib\site-packages\IPython\core\magics\pylab.py in matplotlib(self, line)
    97 print(“Available matplotlib backends: %s” % backends_list)
    98 else:
    —> 99 gui, backend = self.shell.enable_matplotlib(args.gui)
    100 self._show_matplotlib_backend(args.gui, backend)
    101

    ~\anaconda3\envs\zip35\lib\site-packages\IPython\core\interactiveshell.py in enable_matplotlib(self, gui)
    3351 gui, backend = pt.find_gui_and_backend(self.pylab_gui_select)
    3352
    -> 3353 pt.activate_matplotlib(backend)
    3354 pt.configure_inline_support(self, backend)
    3355

    ~\anaconda3\envs\zip35\lib\site-packages\IPython\core\pylabtools.py in activate_matplotlib(backend)
    311 matplotlib.rcParams[‘backend’] = backend
    312
    –> 313 import matplotlib.pyplot
    314 matplotlib.pyplot.switch_backend(backend)
    315

    ~\anaconda3\envs\zip35\lib\site-packages\matplotlib\pyplot.py in
    30 from cycler import cycler
    31 import matplotlib
    —> 32 import matplotlib.colorbar
    33 import matplotlib.image
    34 from matplotlib import rcsetup, style

    ~\anaconda3\envs\zip35\lib\site-packages\matplotlib\colorbar.py in
    30 import matplotlib.collections as collections
    31 import matplotlib.colors as colors
    —> 32 import matplotlib.contour as contour
    33 import matplotlib.cm as cm
    34 import matplotlib.gridspec as gridspec

    ~\anaconda3\envs\zip35\lib\site-packages\matplotlib\contour.py in
    16 import matplotlib.colors as mcolors
    17 import matplotlib.collections as mcoll
    —> 18 import matplotlib.font_manager as font_manager
    19 import matplotlib.text as text
    20 import matplotlib.cbook as cbook

    ~\anaconda3\envs\zip35\lib\site-packages\matplotlib\font_manager.py in
    1361 raise
    1362 except Exception:
    -> 1363 _rebuild()
    1364 else:
    1365 _rebuild()

    ~\anaconda3\envs\zip35\lib\site-packages\matplotlib\font_manager.py in _rebuild()
    1342 global fontManager
    1343
    -> 1344 fontManager = FontManager()
    1345
    1346 if _fmcache:

    ~\anaconda3\envs\zip35\lib\site-packages\matplotlib\font_manager.py in __init__(self, size, weight)
    976 self.defaultFont = {}
    977
    –> 978 ttffiles = findSystemFonts(paths) + findSystemFonts()
    979 self.defaultFont[‘ttf’] = next(
    980 (fname for fname in ttffiles

    ~\anaconda3\envs\zip35\lib\site-packages\matplotlib\font_manager.py in findSystemFonts(fontpaths, fontext)
    262 fontpaths = [win32FontDirectory()]
    263 # now get all installed fonts directly…
    –> 264 fontfiles.update(win32InstalledFonts(fontext=fontext))
    265 else:
    266 fontpaths = X11FontDirectories

    TypeError: ‘NoneType’ object is not iterable

    I would really appreciate if anyone could help me, thanks in advance!

  124. Hi,

    Has anyone tried to install the norgatedata library: https://pypi.org/project/zipline-norgatedata/
    and seen this message when trying to ingest a bundle:

    File “F:\PyProjects\Python Environments\norgatedata\lib\site-packages\zipline_norgatedata\pipelines.py”, line 3, in
    from zipline.api import sid, symbol
    ImportError: cannot import name ‘sid’

    I tried everything but I don’t know how to solve the problem

  125. Hi Mr. Clenow & community,

    Thank you very much for your books, they’re incredible and I have been learning a lot.

    Also thanks to the community for sharing their experiences on this platform.

    I’m very new to programming and I’ve reached the step of ingesting the quandl bundle in chapter 7 in trading evolved, and I got this error:

    Traceback (most recent call last):
    File “C:\Users\DELL\anaconda3\envs\zipline35\Scripts\zipline-script.py”, line 11, in
    load_entry_point(‘zipline==1.4.1’, ‘console_scripts’, ‘zipline’)()
    File “C:\Users\DELL\anaconda3\envs\zipline35\lib\site-packages\pkg_resources\__init__.py”, line 484, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
    File “C:\Users\DELL\anaconda3\envs\zipline35\lib\site-packages\pkg_resources\__init__.py”, line 2707, in load_entry_point
    return ep.load()
    File “C:\Users\DELL\anaconda3\envs\zipline35\lib\site-packages\pkg_resources\__init__.py”, line 2325, in load
    return self.resolve()
    File “C:\Users\DELL\anaconda3\envs\zipline35\lib\site-packages\pkg_resources\__init__.py”, line 2331, in resolve
    module = __import__(self.module_name, fromlist=[‘__name__’], level=0)
    File “C:\Users\DELL\anaconda3\envs\zipline35\lib\site-packages\zipline\__init__.py”, line 29, in
    from .utils.run_algo import run_algorithm
    File “C:\Users\DELL\anaconda3\envs\zipline35\lib\site-packages\zipline\utils\run_algo.py”, line 20, in
    from zipline.data import bundles
    File “C:\Users\DELL\anaconda3\envs\zipline35\lib\site-packages\zipline\data\bundles\__init__.py”, line 2, in
    from . import quandl # noqa
    File “C:\Users\DELL\anaconda3\envs\zipline35\lib\site-packages\zipline\data\bundles\quandl.py”, line 16, in
    from . import core as bundles
    File “C:\Users\DELL\anaconda3\envs\zipline35\lib\site-packages\zipline\data\bundles\core.py”, line 15, in
    from ..minute_bars import (
    File “C:\Users\DELL\anaconda3\envs\zipline35\lib\site-packages\zipline\data\minute_bars.py”, line 30, in
    import tables
    File “C:\Users\DELL\anaconda3\envs\zipline35\lib\site-packages\tables\__init__.py”, line 93, in
    from .utilsextension import (
    ImportError: DLL load failed: The specified module could not be found.

    I’ve looked through the errors posted and I haven’t seen anything like this one. So any help anyone can provide will be appreciated.

    Kind regards & many thanks,
    Yahya

    • Hi Yahya, Andreas and all,

      I actually experienced the very same issue just today (same error message), so very interested in a possible solution. I also searched other web resources for a possible solution but could not find anything so far.
      Besides this, really enjoying the book – incredible resouce!

      Best,
      Oliver

      • Hi Oliver,

        I asked a programmer friend for help and we found the following:

        File “C:\Users\DELL\anaconda3\envs\zipline35\lib\site-packages\tables\__init__.py”, line 93, in
        from .utilsextension import (
        ImportError: DLL load failed: The specified module could not be found.

        this error can be solved by changing the blosc version (I don’t really understand fully what it is but apparently, it compresses data), we used the below command:

        install blosc=1.14.3 -c anaconda

        According to instructions found in the below page:
        https://github.com/conda-forge/pytables-feedstock/issues/31

        Then we faced another error:

        File “C:\Users\DELL\anaconda3\envs\zip35\lib\site-packages\zipline\utils\run_algo.py”, line 21, in
        from zipline.data.benchmarks import get_benchmark_returns_from_file
        ImportError: cannot import name ‘get_benchmark_returns_from_file’

        And here we found that the name of the function that’s imported from benchmarks.py don’t match with the import command in run_algo.py so we just changed the function name in benchmarks.py to get_benchmark_returns_from_file.

        Hope this helps, and I hope that someone can confirm that what we’ve done is correct.

        Kind Regards,
        Yahya

      • Hi Yahya & Oliver,

        I confirm that

        conda install blosc=1.14.3 -c anaconda

        fixes the


        ImportError: DLL load failed: The specified module could not be found.

        issue.

        Although messages prior to the last error line were a bit different for me.

        I am using the latest Anaconda version and the environment for Zipline is created using Python 3.6. Nothing was customized, simply added pandas and matplotlitb before installing zipline using the following command

        conda install -c quantopian zipline

        Kind regards,
        Denis

  126. Hi Andreas

    I am trying to do the expanded momentum model available on the book website.
    I’m using Python 3.6. (advice on Zipline tutorial).
    I had to change :

    start= pd.Timestamp(1997, 1, 1, 8, 15, 12, 0, pytz.UTC)
    end = pd.Timestamp(2018, 12, 31, 8, 15, 12, 0, pytz.UTC)

    I could run the code until the cell with

    pf.create_returns_tear_sheet(returns, benchmark_rets=None)

    —————————————————————————
    OverflowError Traceback (most recent call last)
    in
    —-> 1 pf.create_returns_tear_sheet(returns, benchmark_rets=None)

    ~\anaconda3\envs\env_zipline36\lib\site-packages\pyfolio\plotting.py in call_w_context(*args, **kwargs)
    47 if set_context:
    48 with context():
    —> 49 return func(*args, **kwargs)
    50 else:
    51 return func(*args, **kwargs)

    ~\anaconda3\envs\env_zipline36\lib\site-packages\pyfolio\tears.py in create_returns_tear_sheet(returns, live_start_date, cone_std, benchmark_rets, bootstrap, return_fig)
    247
    248 if benchmark_rets is None:
    –> 249 benchmark_rets = utils.get_symbol_rets(‘SPY’)
    250 # If the strategy’s history is longer than the benchmark’s, limit
    251 # strategy

    ~\anaconda3\envs\env_zipline36\lib\site-packages\pyfolio\utils.py in get_symbol_rets(symbol, start, end)
    473 return SETTINGS[‘returns_func’](symbol,
    474 start=start,
    –> 475 end=end)
    476
    477

    ~\anaconda3\envs\env_zipline36\lib\site-packages\pyfolio\utils.py in default_returns_func(symbol, start, end)
    256 symbol=’SPY’,
    257 start=’1/1/1970′,
    –> 258 end=datetime.now())
    259 rets = rets[start:end]
    260 else:

    ~\anaconda3\envs\env_zipline36\lib\site-packages\pyfolio\utils.py in get_returns_cached(filepath, update_func, latest_dt, **kwargs)
    169
    170 if update_cache:
    –> 171 returns = update_func(**kwargs)
    172 try:
    173 ensure_directory(cache_dir())

    ~\anaconda3\envs\env_zipline36\lib\site-packages\pyfolio\utils.py in get_symbol_from_yahoo(symbol, start, end)
    212 Returns of symbol in requested period.
    213 “””
    –> 214 px = web.get_data_yahoo(symbol, start=start, end=end)
    215 rets = px[[‘Adj Close’]].pct_change().dropna()
    216 rets.index = rets.index.tz_localize(“UTC”)

    ~\anaconda3\envs\env_zipline36\lib\site-packages\pandas_datareader\data.py in get_data_yahoo(*args, **kwargs)
    80
    81 def get_data_yahoo(*args, **kwargs):
    —> 82 return YahooDailyReader(*args, **kwargs).read()
    83
    84

    ~\anaconda3\envs\env_zipline36\lib\site-packages\pandas_datareader\base.py in read(self)
    249 # If a single symbol, (e.g., ‘GOOG’)
    250 if isinstance(self.symbols, (string_types, int)):
    –> 251 df = self._read_one_data(self.url, params=self._get_params(self.symbols))
    252 # Or multiple symbols, (e.g., [‘GOOG’, ‘AAPL’, ‘MSFT’])
    253 elif isinstance(self.symbols, DataFrame):

    ~\anaconda3\envs\env_zipline36\lib\site-packages\pandas_datareader\yahoo\daily.py in _get_params(self, symbol)
    128 # This needed because yahoo returns data shifted by 4 hours ago.
    129 four_hours_in_seconds = 14400
    –> 130 unix_start = int(time.mktime(self.start.timetuple()))
    131 unix_start += four_hours_in_seconds
    132 day_end = self.end.replace(hour=23, minute=59, second=59)

    OverflowError: mktime argument out of range

    Any help would be appreciated.

    thanks
    M

    • Hi

      It was a benchmark problem.
      I could fix that
      Bench=pd.Series(0,index=returns.index)
      pf.create_returns_tear_sheet(returns, benchmark_rets=Bench)

      Anyway I have a lot of MatplotlibDeprecationWarning…(full screen).

      best regards
      M

  127. Hi Yahya,

    worked for me as well. Thanks a lot for your useful tips – very much appreciated!

  128. Hello.

    Has anyone noticed that placing a order by using the order_target_percent method example:
    order_target_percent(context.stock, stock_weight)
    sometimes consumes more “capital_used” than the one we have, consequently creating “ending_cash” negative?

    Does any one know how to prevent this from happening?

    Best
    FJ

    • in addition to what i’ve mentioned above, in my simulations it’s common to have in the results “gross_leverage” column values up to 1.5 on the days trading occurs.
      I tried to replicate “Backtest Analysis” results from chapter 8 using quandl bundle but unfortunately I couldn’t.
      Same code, same stocks, same dates and same bundle as in the book but i get a much lower equity curve and in the strategy exposure the values oscillate between 0.9x and 1.5. Zipline seems to get the order wrong somehow.

      Has anyone faced this issue?
      Best
      Filipe

  129. Hi Andreas,
    Thank you for your entertaining book. I am trying to get the code TrendModel.py running by ingesting the random_futures_data. When I use

    start = datetime(2001, 1, 1, 8, 15, 12, 0, pytz.UTC)
    end = datetime(2019, 1, 2, 8, 15, 12, 0, pytz.UTC)

    I am getting the following error:

    Traceback (most recent call last):
    File “/Users/jennahaerr/opt/anaconda3/envs/pySystemTrader_py36_v1/lib/python3.6/site-packages/IPython/core/interactiveshell.py”, line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
    File “”, line 1, in
    runfile(‘/Users/jennahaerr/Dropbox (Life Elixir)/Companies/Eonum/gitHub_v1/Finance/pySystemTrader_v2/notebooks/Trading Evolved/Chapter 15 – Futures Trend Following/TrendModel.py’, wdir=’/Users/jennahaerr/Dropbox (Life Elixir)/Companies/Eonum/gitHub_v1/Finance/pySystemTrader_v2/notebooks/Trading Evolved/Chapter 15 – Futures Trend Following’)
    File “/Users/jennahaerr/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/203.5981.165/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py”, line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
    File “/Users/jennahaerr/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/203.5981.165/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py”, line 18, in execfile
    exec(compile(contents+”\n”, file, ‘exec’), glob, loc)
    File “/Users/jennahaerr/Dropbox (Life Elixir)/Companies/Eonum/gitHub_v1/Finance/pySystemTrader_v2/notebooks/Trading Evolved/Chapter 15 – Futures Trend Following/TrendModel.py”, line 328, in
    bundle=’random_futures_data’)
    File “/Users/jennahaerr/opt/anaconda3/envs/pySystemTrader_py36_v1/lib/python3.6/site-packages/zipline/utils/run_algo.py”, line 455, in run_algorithm
    execution_id=execution_id
    File “/Users/jennahaerr/opt/anaconda3/envs/pySystemTrader_py36_v1/lib/python3.6/site-packages/zipline/utils/run_algo.py”, line 238, in _run
    execution_id=execution_id
    File “/Users/jennahaerr/opt/anaconda3/envs/pySystemTrader_py36_v1/lib/python3.6/site-packages/zipline/finance/trading.py”, line 39, in __init__
    assert type(start_session) == pd.Timestamp
    AssertionError

    using:
    start = pd.Timestamp(‘2001-01-01 08:15:12′, tz=’utc’)
    end = pd.Timestamp(‘2019-01-02 08:15:12′, tz=’utc’)

    instead of:
    start = datetime(2001, 1, 1, 8, 15, 12, 0, pytz.UTC)
    end = datetime(2019, 1, 2, 8, 15, 12, 0, pytz.UTC)

    in the TrendModel.py code. However, I am getting the following error:

    Traceback (most recent call last):
    File “/Users/jennahaerr/opt/anaconda3/envs/pySystemTrader_py36_v1/lib/python3.6/site-packages/IPython/core/interactiveshell.py”, line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
    File “”, line 1, in
    runfile(‘/Users/jennahaerr/Dropbox (Life Elixir)/Companies/Eonum/gitHub_v1/Finance/pySystemTrader_v2/notebooks/Trading Evolved/Chapter 15 – Futures Trend Following/TrendModel.py’, wdir=’/Users/jennahaerr/Dropbox (Life Elixir)/Companies/Eonum/gitHub_v1/Finance/pySystemTrader_v2/notebooks/Trading Evolved/Chapter 15 – Futures Trend Following’)
    File “/Users/jennahaerr/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/203.5981.165/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py”, line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
    File “/Users/jennahaerr/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/203.5981.165/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py”, line 18, in execfile
    exec(compile(contents+”\n”, file, ‘exec’), glob, loc)
    File “/Users/jennahaerr/Dropbox (Life Elixir)/Companies/Eonum/gitHub_v1/Finance/pySystemTrader_v2/notebooks/Trading Evolved/Chapter 15 – Futures Trend Following/TrendModel.py”, line 328, in
    bundle=’random_futures_data’)
    File “/Users/jennahaerr/opt/anaconda3/envs/pySystemTrader_py36_v1/lib/python3.6/site-packages/zipline/utils/run_algo.py”, line 455, in run_algorithm
    execution_id=execution_id
    File “/Users/jennahaerr/opt/anaconda3/envs/pySystemTrader_py36_v1/lib/python3.6/site-packages/zipline/utils/run_algo.py”, line 253, in _run
    ‘script’: algotext,
    File “/Users/jennahaerr/opt/anaconda3/envs/pySystemTrader_py36_v1/lib/python3.6/site-packages/zipline/algorithm.py”, line 664, in run
    for perf in self.get_generator():
    File “/Users/jennahaerr/opt/anaconda3/envs/pySystemTrader_py36_v1/lib/python3.6/site-packages/zipline/gens/tradesimulation.py”, line 206, in transform
    for capital_change_packet in every_bar(dt):
    File “/Users/jennahaerr/opt/anaconda3/envs/pySystemTrader_py36_v1/lib/python3.6/site-packages/zipline/gens/tradesimulation.py”, line 133, in every_bar
    handle_data(algo, current_data, dt_to_use)
    File “/Users/jennahaerr/opt/anaconda3/envs/pySystemTrader_py36_v1/lib/python3.6/site-packages/zipline/utils/events.py”, line 216, in handle_data
    dt,
    File “/Users/jennahaerr/opt/anaconda3/envs/pySystemTrader_py36_v1/lib/python3.6/site-packages/zipline/utils/events.py”, line 235, in handle_data
    self.callback(context, data)
    File “/Users/jennahaerr/Dropbox (Life Elixir)/Companies/Eonum/gitHub_v1/Finance/pySystemTrader_v2/notebooks/Trading Evolved/Chapter 15 – Futures Trend Following/TrendModel.py”, line 206, in daily_trade
    bar_count=250,
    File “zipline/_protocol.pyx”, line 121, in zipline._protocol.check_parameters.__call__.assert_keywords_and_call
    File “zipline/_protocol.pyx”, line 709, in zipline._protocol.BarData.history
    File “/Users/jennahaerr/opt/anaconda3/envs/pySystemTrader_py36_v1/lib/python3.6/site-packages/zipline/data/data_portal.py”, line 967, in get_history_window
    field, data_frequency)
    File “/Users/jennahaerr/opt/anaconda3/envs/pySystemTrader_py36_v1/lib/python3.6/site-packages/zipline/data/data_portal.py”, line 806, in _get_history_daily_window
    assets, days_for_window, end_dt, field_to_use, data_frequency
    File “/Users/jennahaerr/opt/anaconda3/envs/pySystemTrader_py36_v1/lib/python3.6/site-packages/zipline/data/data_portal.py”, line 829, in _get_history_daily_window_data
    extra_slot=False
    File “/Users/jennahaerr/opt/anaconda3/envs/pySystemTrader_py36_v1/lib/python3.6/site-packages/zipline/data/data_portal.py”, line 1117, in _get_daily_window_data
    extra_slot)
    File “/Users/jennahaerr/opt/anaconda3/envs/pySystemTrader_py36_v1/lib/python3.6/site-packages/zipline/data/history_loader.py”, line 549, in history
    is_perspective_after)
    File “/Users/jennahaerr/opt/anaconda3/envs/pySystemTrader_py36_v1/lib/python3.6/site-packages/zipline/data/history_loader.py”, line 401, in _ensure_sliding_windows
    end_ix = find_in_sorted_index(cal, end)
    File “/Users/jennahaerr/opt/anaconda3/envs/pySystemTrader_py36_v1/lib/python3.6/site-packages/zipline/utils/pandas_utils.py”, line 141, in find_in_sorted_index
    raise LookupError(“{dt} is not in {dts}”.format(dt=dt, dts=dts))
    LookupError: 2001-01-02 00:00:00+00:00 is not in DatetimeIndex([‘2015-01-02’, ‘2015-01-05’, ‘2015-01-06’, ‘2015-01-07’,
    ‘2015-01-08’, ‘2015-01-09’, ‘2015-01-12’, ‘2015-01-13’,
    ‘2015-01-14’, ‘2015-01-15’,

    ‘2021-12-10’, ‘2021-12-13’, ‘2021-12-14’, ‘2021-12-15’,

    Will you please elaborate?

    • Hello, I have same problem. Have you already solved it? I will glad any help.

      • Same problem to me as well. Appreciate if any help.

      • The random_futures bundle created with the provided data has the dataset starting from mid 2015.

        Change from
        start = datetime(2001, 1, 1, 8, 15, 12, 0, pytz.UTC)
        to
        start = datetime(2016, 1, 1, 8, 15, 12, 0, pytz.UTC)

  130. Hi! I’m wondering if anyone knows a good way to sell a specifik stock after holding it for a certain time? For example, every day check if there is any stock I bought 2 weeks ago and sell those that meets the condition.

  131. Error by running the ‘Momentum Model’ from Chaoter 12.

    Hi Andreas,

    As I run the example code ‘Momentum Model’ from Chapter 12, I got the following errors:

    C:\Users\chenw\anaconda3\envs\env_zipline_36\lib\site-packages\empyrical\stats.py:713: RuntimeWarning: invalid value encountered in true_divide
    out=out,
    C:\Users\chenw\anaconda3\envs\env_zipline_36\lib\site-packages\empyrical\stats.py:799: RuntimeWarning: invalid value encountered in true_divide
    np.divide(average_annual_return, annualized_downside_risk, out=out)
    2003-02-03 – Last Month Result: 0.00%
    —————————————————————————
    KeyError Traceback (most recent call last)
    ~\anaconda3\envs\env_zipline_36\lib\site-packages\zipline\assets\assets.py in _lookup_symbol_strict(self, ownership_map, multi_country, symbol, as_of_date)
    812 try:
    –> 813 owners = ownership_map[company_symbol, share_class_symbol]
    814 assert owners, ’empty owners list for %r’ % symbol

    KeyError: (‘ZBH’, ”)

    I open the ‘sp500.csv’ and find it is not complete. On many days there is no any data such on the 2003-02-03. It is simple empty.

    I use the following codes to check it:

    import pandas as pd

    index_members = pd.read_csv(‘../data/index_members/sp500.csv’, index_col=0, parse_dates=[0])
    index_members.loc[index_members.index == ‘2018-10-25’].values # has data
    index_members.loc[index_members.index == ‘2003-02-03’].values # no data

    Regards

  132. I’m hoping somebody can offer me an easy yes or no answer to an issue I am having. My strategy in zipline closes all positions at EOD, when the code runs

    ”’utils.py in extract_rets_pos_txn_from_zipline(backtest)”’

    it leads to the error code

    ”’AttributeError: ‘DataFrame’ object has no attribute ‘amount””

    It appears this is caused by having no open positions at EOD therefore there is nothing to look at in the positions dataframe, I’m assuming this from

    ”’raise ValueError(“The backtest does not have any positions.”)”’

    I have seen similar issues raised in the past but don’t see any follow-up answers and it seems like this was addresses in later revisions of Pyfolio. I’m running version 0.9.0 which I believe is the most recent. I am hoping somebody will either tell me Pyfolio just can’t be used for these strategies (which I find hard to believe), or they will be able to point me to the simple error I am making. Thank you in advance for you help.

  133. Dear Andreas,
    dear readers,

    I try to replicate the results described in chapter “Your First Zipline Backtest”. I had lots of trouble installing zipline using conda on my Windows machine, however managed to do so. When I run

    # Fire off the backtest
    results = run_algorithm(
    start=start_date,
    end=end_date,
    initialize=initialize,
    analyze=analyze,
    handle_data=handle_data,
    capital_base=10000,
    data_frequency = ‘daily’,
    bundle=’quandl’
    )

    I receive an assertion error. Do you have any idea what the error might be? After all it is exactly the same code from your book, so I am not sure what went wrong? Also, may I ask, what version of matplotlib you used for producing the graphs? As I needed to install it on top of zipline, maybe this caused a conflict with the downgraded conda environment?

    —————————————————————————
    AssertionError Traceback (most recent call last)
    in
    8 capital_base=10000,
    9 data_frequency = ‘daily’,
    —> 10 bundle=’quandl’
    11 )

    ~\miniconda3\envs\env_zipline\lib\site-packages\zipline\utils\run_algo.py in run_algorithm(start, end, initialize, capital_base, handle_data, before_trading_start, analyze, data_frequency, bundle, bundle_timestamp, trading_calendar, metrics_set, benchmark_returns, default_extension, extensions, strict_extensions, environ, blotter)
    407 environ=environ,
    408 blotter=blotter,
    –> 409 benchmark_spec=benchmark_spec,
    410 )
    411

    ~\miniconda3\envs\env_zipline\lib\site-packages\zipline\utils\run_algo.py in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, environ, blotter, benchmark_spec)
    201 trading_calendar=trading_calendar,
    202 capital_base=capital_base,
    –> 203 data_frequency=data_frequency,
    204 ),
    205 metrics_set=metrics_set,

    ~\miniconda3\envs\env_zipline\lib\site-packages\zipline\finance\trading.py in __init__(self, start_session, end_session, trading_calendar, capital_base, emission_rate, data_frequency, arena)
    36 arena=’backtest’):
    37
    —> 38 assert type(start_session) == pd.Timestamp
    39 assert type(end_session) == pd.Timestamp
    40

    AssertionError:

    “””
    Start comment:
    PS: I was only able to install zipline on Windows 10, using the following tweaks from https://github.com/quantopian/zipline/issues/2019. I received DLL errors otherwise:
    1. conda config –set allow_conda_downgrades true
    2. conda install conda=4.6.11
    3. conda create -n env_zipline python=3.6
    4. activate env_zipline
    5. conda install -c Quantopian zipline (at this point, if I import zipline, I’ll encounter the issue mentioned above)
    6. conda install python=3.6.8=h9f7ef89_0
    7. pip install tables==3.5.2 (at this point, error change to ‘ValueError: numpy.ufunc size changed, may ind ted 216 from C header’)
    8. pip install numpy==1.16.1

    “””

  134. Hello guys, I d appreciate any help from you. I try to implement code from p15 FUTURES TREND FOLLOWING and receive the following error:

    Exception ignored in: ‘zipline.assets.continuous_futures.OrderedContracts.contract_before_auto_close’
    AttributeError: ‘NoneType’ object has no attribute ‘next’

    —————————————————————————
    KeyError Traceback (most recent call last)
    in ()
    5 capital_base=starting_portfolio,
    6 data_frequency = ‘daily’,
    —-> 7 bundle=’futures_random’)

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/utils/run_algo.py in run_algorithm(start, end, initialize, capital_base, handle_data, before_trading_start, analyze, data_frequency, data, bundle, bundle_timestamp, trading_calendar, metrics_set, default_extension, extensions, strict_extensions, environ, blotter)
    430 local_namespace=False,
    431 environ=environ,
    –> 432 blotter=blotter,
    433 )

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/utils/run_algo.py in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, data, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, environ, blotter)
    229 ).run(
    230 data,
    –> 231 overwrite_sim_params=False,
    232 )
    233

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/algorithm.py in run(self, data, overwrite_sim_params)
    754 try:
    755 perfs = []
    –> 756 for perf in self.get_generator():
    757 perfs.append(perf)
    758

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/gens/tradesimulation.py in transform(self)
    204 for dt, action in self.clock:
    205 if action == BAR:
    –> 206 for capital_change_packet in every_bar(dt):
    207 yield capital_change_packet
    208 elif action == SESSION_START:

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/gens/tradesimulation.py in every_bar(dt_to_use, current_data, handle_data)
    132 metrics_tracker.process_commission(commission)
    133
    –> 134 handle_data(algo, current_data, dt_to_use)
    135
    136 # grab any new orders from the blotter, then clear the list.

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/utils/events.py in handle_data(self, context, data, dt)
    214 context,
    215 data,
    –> 216 dt,
    217 )
    218

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/utils/events.py in handle_data(self, context, data, dt)
    233 “””
    234 if self.rule.should_trigger(dt):
    –> 235 self.callback(context, data)
    236
    237

    in daily_trade(context, data)
    3 fields = [‘close’, ‘volume’],
    4 frequency = ‘1d’,
    —-> 5 bar_count = 250
    6 )
    7 hist[‘trend’] = hist[‘close’].ewm(span = fast_ma).mean() > hist[‘close’].ewm(span = slow_ma).mean()

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/_protocol.pyx in zipline._protocol.check_parameters.__call__.assert_keywords_and_call (zipline/_protocol.c:3747)()

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/_protocol.pyx in zipline._protocol.BarData.history (zipline/_protocol.c:10190)()

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/data/data_portal.py in get_history_window(self, assets, end_dt, bar_count, frequency, field, data_frequency, ffill)
    965 else:
    966 df = self._get_history_daily_window(assets, end_dt, bar_count,
    –> 967 field, data_frequency)
    968 elif frequency == “1m”:
    969 if field == “price”:

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/data/data_portal.py in _get_history_daily_window(self, assets, end_dt, bar_count, field_to_use, data_frequency)
    804
    805 data = self._get_history_daily_window_data(
    –> 806 assets, days_for_window, end_dt, field_to_use, data_frequency
    807 )
    808 return pd.DataFrame(

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/data/data_portal.py in _get_history_daily_window_data(self, assets, days_for_window, end_dt, field_to_use, data_frequency)
    827 field_to_use,
    828 days_for_window,
    –> 829 extra_slot=False
    830 )
    831 else:

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/data/data_portal.py in _get_daily_window_data(self, assets, field, days_in_window, extra_slot)
    1115 days_in_window,
    1116 field,
    -> 1117 extra_slot)
    1118 if extra_slot:
    1119 return_array[:len(return_array) – 1, :] = data

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/data/history_loader.py in history(self, assets, dts, field, is_perspective_after)
    547 dts,
    548 field,
    –> 549 is_perspective_after)
    550 end_ix = self._calendar.searchsorted(dts[-1])
    551

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/data/history_loader.py in _ensure_sliding_windows(self, assets, dts, field, is_perspective_after)
    429 adj_dts = prefetch_dts
    430 prefetch_len = len(prefetch_dts)
    –> 431 array = self._array(prefetch_dts, needed_assets, field)
    432
    433 if field == ‘sid’:

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/data/history_loader.py in _array(self, dts, assets, field)
    571 dts[0],
    572 dts[-1],
    –> 573 assets,
    574 )[0]
    575

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/data/dispatch_bar_reader.py in load_raw_arrays(self, fields, start_dt, end_dt, sids)
    122 end_dt,
    123 sid_groups[t])
    –> 124 for t in asset_types if sid_groups[t]}
    125
    126 results = []

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/data/dispatch_bar_reader.py in (.0)
    122 end_dt,
    123 sid_groups[t])
    –> 124 for t in asset_types if sid_groups[t]}
    125
    126 results = []

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/data/continuous_future_reader.py in load_raw_arrays(self, columns, start_date, end_date, assets)
    37 start_date,
    38 end_date,
    —> 39 asset.offset
    40 )
    41

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/assets/roll_finder.py in get_rolls(self, root_symbol, start, end, offset)
    89 “””
    90 oc = self.asset_finder.get_ordered_contracts(root_symbol)
    —> 91 front = self._get_active_contract_at_offset(root_symbol, end, 0)
    92 back = oc.contract_at_offset(front, 1, end.value)
    93 if back is not None:

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/assets/roll_finder.py in _get_active_contract_at_offset(self, root_symbol, dt, offset)
    39 session = self.trading_calendar.minute_to_session_label(dt)
    40 front = oc.contract_before_auto_close(session.value)
    —> 41 back = oc.contract_at_offset(front, 1, dt.value)
    42 if back is None:
    43 return front

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/assets/continuous_futures.pyx in zipline.assets.continuous_futures.OrderedContracts.contract_at_offset (zipline/assets/continuous_futures.c:7309)()

    /opt/anaconda3/envs/zip35/lib/python3.5/site-packages/zipline/assets/continuous_futures.pyx in zipline.assets.continuous_futures.OrderedContracts.contract_at_offset (zipline/assets/continuous_futures.c:7061)()

    KeyError: 0

    As far as I can see the issue with rolling futures contracts.

    Some clarification:

    1) I ingested to zipline futures data from random_futures folder.
    2) All my markets from a code are matched with constants.py.
    3) I imported futures data by order of expiration date.

    I will glad any advice.

    • Andreas

      Four readers including me asked the same problems. Would appreciate if you reply with solution. Thanks.

  135. Hi all,

    does anyone know of any internet site for discussion (and for finding help) of this kind of themes? (ie, python programming for trading/investing strategies backtesting)

    thanks a lot in advance
    J

  136. Andreas,

    I’m trying to simulate a stock mean reversion model that buys on limit below previous day’s close price, with daily data every day. However, there’s an issue with zipline’s limit order design: orders are triggered only if the daily “close” is less than or equal to the limit price. This means that if a daily bar’s “open” or “low” are below the limit price, but the close of the day ends up above it (as often happens – e.g. a hammer candle), the orders will not get filled.

    I understand that Quantopian used to operate on minute data. This is why it does not matter if you use time_rules.market_close() or time_rules.market_open() when trading on daily data. Orders are always filled at the last daily bar’s close price. With minute data execution, you can have your limit orders trigger if the next minute’s close is below the limit price.. With daily data – only if the daily close is below the limit price.

    I searched web archive for old Quantopian forum posts where Quantopian techs suggest that zipline code be edited if one wants to change the design of how limit orders are triggered. However, where exactly to edit and how to do it is beyond me.

    Could you, please, offer some insight as to how this problem might be addressed? Certainly, it would be more flexible if zipline offered a choice on how orders are triggered. Currently you cannot even buy/sell on open with daily data, as it is.

    Thanks and best regards,

    Nikolay

  137. Hey all,

    Big fan of the book. Finally getting around to it after a few months of just nibbling on some of the code tests.

    At the beginning of the Backtesting Trading Strategies part on around page 101, and when I run into little errors that I seek solutions for on StackOverflow and Github, it seems that Zipline has completely stopped updating, the zipline documentation on zipline.io is gone, quantopian is abruptdly shutdown, Quandl data is deprecated, etc.

    Does it still make sense to invest the time to learn Zipline, or is a maintained but more complex library like backtrader etc. better?

    • Hi Pat,

      It’s unfortunate that Quantopian folded but I remain a fan of Zipline. It looks like it will be adopted by the open source community now, and it’s still the best Python based backtester (that I’m aware of).

      The most encouraging project at the moment is Zipline Trader. Developed by Shlomi Kushchi and actively maintained, it has a lot of interesting stuff and also a copy of the original docs.

      https://zipline-trader.readthedocs.io/en/latest/index.html

      • Hi Andreas! Thank you for the great book!

        My question about modern situation with Zipline. What do you think about zipline-reloaded – Stefan Jansen project: https://zipline.ml4trading.io/ ?

        What is your recommendations for retail use:
        1) Zipline 1.3 (had tuned by your book)
        2) Zipline-trader
        3) Zipline-reloaded

        Thank you!

      • I think it’s brilliant. That’s why I used Zipline Reloaded in the latest book update. If you’ve got the Kindle version, you should be able to update it to the latest version, released a month or two ago.

  138. Federico Marongiu

    Hi Andreas and thanks for the knowledge shared in this book,
    I would like to ask you if there is any method to replace the .iteritems() function, because it gives me an error and trying to search a solution on the internet it says .iteritems() does not work anymore on Python 3.x.
    Here’s the code and the error:

    def initialize(context):
    dji= [
    “AAPL”,
    “AXP”,
    “BA”,
    “CAT”,
    “CSCO”,
    “CVX”,
    “DIS”,
    “DWDP”,
    “GS”,
    “HD”,
    “IBM”,
    “INTC”,
    “JNJ”,
    “JPM”,
    “KO”,
    “MCD”,
    “MMM”,
    “MRK”,
    “MSFT”,
    “NKE”,
    “PFE”,
    “PG”,
    “TRV”,
    “UNH”,
    “UTX”,
    “V”,
    “VZ”,
    “WBA”,
    “WMT”,
    “XOM”,]

    context.universe=[symbol(s) for s in dji]
    context.history_window = 20
    context.stocks_to_hold = 10
    schedule_function(handle_data, date_rules.month_start(), time_rules.market_close())

    def month_perf(ts):
    perf = (ts[-1]/ts[0])-1
    return perf

    def handle_data(context, data):
    #get history for all stocks
    hist = data.history(context.universe, ‘close’, context.history_window,”1d”)
    perf_table = hist.apply(month_perf)
    buy_list = perf_table[:context.stocks_to_hold]
    the_rest = perf_table[context.stocks_to_hold]
    for stock,perf in buy_list.iteritems():
    stock_weight = 1 / context.stocks_to_hold
    if data.can_trade(stock):
    order_target_percent(stock, stock_weight)
    for stock, perf in the_rest.iteritems():
    if data_can_trade(stock):
    order_target_percent(stock, 0.0)

    def analyze(context, perf):
    returns, positions,transactions= pf.utils.extract_rets_pos_txn_from_zipline(perf)
    pf.create_returns_tear_sheet(returns,benchmark_rets=None)

    start = pd.Timestamp(‘2003-1-1′, tz=’utc’)
    end = pd.Timestamp(‘2017-12-31′, tz=’utc’)
    # Fire off the backtest
    result = run_algorithm(start=start, end=end,
    initialize=initialize,
    analyze=analyze,
    capital_base=10000,
    data_frequency = ‘daily’, bundle=’quandl’ )

    Error:

    File “”, line 55, in handle_data
    for stock, perf in the_rest.iteritems():

    AttributeError: ‘numpy.float64’ object has no attribute ‘iteritems’

    Thank you to whoever replies to this.

  139. Jayant Agarwal

    Hello Sir,

    Recently bought this book and having a great time learning the concepts from this but after reaching to the backtesting chapter and going through the process of creating a separate environment and installing libraries I get stuck in an issue i.e. quandl was not ingesting. I tried different ways written in the books and through sources like github and stackoverflow but could not get anywhere. I tried to run the next code to move forward but found myself stuck in the same issue again.
    It would be highly helpful if you can guide me on this.

    (zip35) C:\Users\jayan>zipline ingest -b quandl
    Traceback (most recent call last):
    File “C:\Users\jayan\anaconda3\envs\zip35\Scripts\zipline-script.py”, line 33, in
    sys.exit(load_entry_point(‘zipline==1.4.1’, ‘console_scripts’, ‘zipline’)())
    File “C:\Users\jayan\anaconda3\envs\zip35\Scripts\zipline-script.py”, line 25, in importlib_load_entry_point
    return next(matches).load()
    File “C:\Users\jayan\anaconda3\envs\zip35\lib\site-packages\importlib_metadata\__init__.py”, line 100, in load
    module = import_module(match.group(‘module’))
    File “C:\Users\jayan\anaconda3\envs\zip35\lib\importlib\__init__.py”, line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
    File “”, line 994, in _gcd_import
    File “”, line 971, in _find_and_load
    File “”, line 941, in _find_and_load_unlocked
    File “”, line 219, in _call_with_frames_removed
    File “”, line 994, in _gcd_import
    File “”, line 971, in _find_and_load
    File “”, line 955, in _find_and_load_unlocked
    File “”, line 665, in _load_unlocked
    File “”, line 678, in exec_module
    File “”, line 219, in _call_with_frames_removed
    File “C:\Users\jayan\anaconda3\envs\zip35\lib\site-packages\zipline\__init__.py”, line 29, in
    from .utils.run_algo import run_algorithm
    File “C:\Users\jayan\anaconda3\envs\zip35\lib\site-packages\zipline\utils\run_algo.py”, line 20, in
    from zipline.data import bundles
    File “C:\Users\jayan\anaconda3\envs\zip35\lib\site-packages\zipline\data\bundles\__init__.py”, line 2, in
    from . import quandl # noqa
    File “C:\Users\jayan\anaconda3\envs\zip35\lib\site-packages\zipline\data\bundles\quandl.py”, line 16, in
    from . import core as bundles
    File “C:\Users\jayan\anaconda3\envs\zip35\lib\site-packages\zipline\data\bundles\core.py”, line 15, in
    from ..minute_bars import (
    File “C:\Users\jayan\anaconda3\envs\zip35\lib\site-packages\zipline\data\minute_bars.py”, line 30, in
    import tables
    File “C:\Users\jayan\anaconda3\envs\zip35\lib\site-packages\tables\__init__.py”, line 99, in
    from .utilsextension import (
    ImportError: DLL load failed: The specified module could not be found.

    Thank you to whoever replies to this.

  140. Dear Andreas,

    Thank you for your great book. I really enjoyed reading it.

    I have problem running trendModel.py. I used the following:

    start = pd.Timestamp(‘2016-01-01′, tz=’utc’)
    end = pd.Timestamp(‘2019-01-02′, tz=’utc’)

    Here is the error I am getting:

    Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/bin/python “/Users/annap/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/203.7148.72/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevconsole.py” –mode=client –port=49859
    import sys; print(‘Python %s on %s’ % (sys.version, sys.platform))
    sys.path.extend([‘/Users/jennahaerr/Dropbox (Life Elixir)/Companies/Eonum/gitHub_v1/Finance/tradingEvolved’])
    Python 3.6.11 | packaged by conda-forge | (default, Aug 5 2020, 20:19:23)
    Type ‘copyright’, ‘credits’ or ‘license’ for more information
    IPython 7.16.1 — An enhanced Interactive Python. Type ‘?’ for help.
    PyDev console: using IPython 7.16.1
    Python 3.6.11 | packaged by conda-forge | (default, Aug 5 2020, 20:19:23)
    [GCC Clang 10.0.1 ] on darwin
    runfile(‘/Users/annap/Dropbox (Life Elixir)/Companies/Eonum/gitHub_v1/Finance/tradingEvolved/notebooks/Trading Evolved/Chapter 15 – Futures Trend Following/TrendModel.py’, wdir=’/Users/jennahaerr/Dropbox (Life Elixir)/Companies/Eonum/gitHub_v1/Finance/tradingEvolved/notebooks/Trading Evolved/Chapter 15 – Futures Trend Following’)
    HTML(value=”)
    Exception ignored in: ‘zipline.assets.continuous_futures.OrderedContracts.contract_before_auto_close’
    AttributeError: ‘NoneType’ object has no attribute ‘next’
    Traceback (most recent call last):
    File “/Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/lib/python3.6/site-packages/IPython/core/interactiveshell.py”, line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
    File “”, line 1, in
    runfile(‘/Users/annap/Dropbox (Life Elixir)/Companies/Eonum/gitHub_v1/Finance/tradingEvolved/notebooks/Trading Evolved/Chapter 15 – Futures Trend Following/TrendModel.py’, wdir=’/Users/jennahaerr/Dropbox (Life Elixir)/Companies/Eonum/gitHub_v1/Finance/tradingEvolved/notebooks/Trading Evolved/Chapter 15 – Futures Trend Following’)
    File “/Users/annap/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/203.7148.72/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py”, line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
    File “/Users/annap/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/203.7148.72/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py”, line 18, in execfile
    exec(compile(contents+”\n”, file, ‘exec’), glob, loc)
    File “/Users/jennahaerr/Dropbox (Life Elixir)/Companies/Eonum/gitHub_v1/Finance/tradingEvolved/notebooks/Trading Evolved/Chapter 15 – Futures Trend Following/TrendModel.py”, line 330, in
    bundle=’random_futures_data’)
    File “/Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/lib/python3.6/site-packages/zipline/utils/run_algo.py”, line 460, in run_algorithm
    execution_id=execution_id
    File “/Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/lib/python3.6/site-packages/zipline/utils/run_algo.py”, line 254, in _run
    ‘script’: algotext,
    File “/Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/lib/python3.6/site-packages/zipline/algorithm.py”, line 664, in run
    for perf in self.get_generator():
    File “/Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/lib/python3.6/site-packages/zipline/gens/tradesimulation.py”, line 205, in transform
    for capital_change_packet in every_bar(dt):
    File “/Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/lib/python3.6/site-packages/zipline/gens/tradesimulation.py”, line 133, in every_bar
    handle_data(algo, current_data, dt_to_use)
    File “/Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/lib/python3.6/site-packages/zipline/utils/events.py”, line 216, in handle_data
    dt,
    File “/Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/lib/python3.6/site-packages/zipline/utils/events.py”, line 235, in handle_data
    self.callback(context, data)
    File “/Users/annap/Dropbox (Life Elixir)/Companies/Eonum/gitHub_v1/Finance/tradingEvolved/notebooks/Trading Evolved/Chapter 15 – Futures Trend Following/TrendModel.py”, line 207, in daily_trade
    bar_count=250,
    File “zipline/_protocol.pyx”, line 121, in zipline._protocol.check_parameters.__call__.assert_keywords_and_call
    File “zipline/_protocol.pyx”, line 709, in zipline._protocol.BarData.history
    File “/Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/lib/python3.6/site-packages/zipline/data/data_portal.py”, line 967, in get_history_window
    field, data_frequency)
    File “/Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/lib/python3.6/site-packages/zipline/data/data_portal.py”, line 806, in _get_history_daily_window
    assets, days_for_window, end_dt, field_to_use, data_frequency
    File “/Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/lib/python3.6/site-packages/zipline/data/data_portal.py”, line 829, in _get_history_daily_window_data
    extra_slot=False
    File “/Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/lib/python3.6/site-packages/zipline/data/data_portal.py”, line 1117, in _get_daily_window_data
    extra_slot)
    File “/Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/lib/python3.6/site-packages/zipline/data/history_loader.py”, line 549, in history
    is_perspective_after)
    File “/Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/lib/python3.6/site-packages/zipline/data/history_loader.py”, line 431, in _ensure_sliding_windows
    array = self._array(prefetch_dts, needed_assets, field)
    File “/Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/lib/python3.6/site-packages/zipline/data/history_loader.py”, line 573, in _array
    assets,
    File “/Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/lib/python3.6/site-packages/zipline/data/dispatch_bar_reader.py”, line 120, in load_raw_arrays
    for t in asset_types if sid_groups[t]}
    File “/Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/lib/python3.6/site-packages/zipline/data/dispatch_bar_reader.py”, line 120, in
    for t in asset_types if sid_groups[t]}
    File “/Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/lib/python3.6/site-packages/zipline/data/continuous_future_reader.py”, line 39, in load_raw_arrays
    asset.offset
    File “/Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/lib/python3.6/site-packages/zipline/assets/roll_finder.py”, line 91, in get_rolls
    front = self._get_active_contract_at_offset(root_symbol, end, 0)
    File “/Users/annap/opt/anaconda3/envs/tradingEvolved_py36_v1/lib/python3.6/site-packages/zipline/assets/roll_finder.py”, line 41, in _get_active_contract_at_offset
    back = oc.contract_at_offset(front, 1, dt.value)
    File “zipline/assets/continuous_futures.pyx”, line 395, in zipline.assets.continuous_futures.OrderedContracts.contract_at_offset
    File “zipline/assets/continuous_futures.pyx”, line 401, in zipline.assets.continuous_futures.OrderedContracts.contract_at_offset
    KeyError: 0

  141. guys i have a weird error in trying to get running zipline, i created virtual environment and installed all the necessary packages and installed successfullly, now here comes the difficult part , which is that when i :

    import zipline

    this works fine…

    but when i run this below :

    from zipline import run_algorithm

    this error pops :

    ImportError: cannot import name ‘run_algorithm’

    what is really going on here ??

    anybody please help me and advance thanks!

  142. Just to say that by using the latest zipline-reloaded package, one can use the book examples using the up to date python and conda versions. A slight adjustment to the book’s code examples is that the datetime to be replace by pandas Timestamp.
    Enjoy the book

  143. Hi Andreas,

    For backtesting stocks strategies for retail trading, which slippage model do you think is more suitable in Zipline?

    Your examples use VolumeShareSlippage, which seems to make sense only if you’re trading very large positions. Otherwise, if you’re a retail trader with a small account, slippage would be negligible and would make the backtests look more favorable than in reality.

    Thankfully, the Quantopian forum has been archived. I’m attaching a post byJessica Stauth, announcing an official change to the default Zipline slippage model from VolumeShareSlippage to FixedBasisPointsSlippage, with a constant slippage of 5 bps applied to every trade.

    Using this model really makes a difference, which could really destroy a huge chunk of the profits in strategies with a high turnover.

    https://quantopian-archive.netlify.app/forum/threads/changes-coming-to-the-default-slippage-model.html

  144. Hi Andreas,
    Quick question. Sorry if this is foolish +/- has been answered elsewhere! On page 68 (kindle) you calculate the pct_change of the ‘SP500’. I was expecting you to calculate the percent change in the actual price. Surely the percent change in the price itself better reflects the movement in your position? Conversely, how does the percent change in the rolling average show your position?
    Best wishes,
    Brett

  145. Sorry, ignore me. I was using my own data and allowed myself to become confused by SMA50 and SPA500.
    Best wishes,
    Brett

  146. Actually, please don’t worry, I got the data from Yahoo finance.

  147. Hello Andreas 🙂
    I tried to recreate the bundle from chapter 23, but when I try to ingest I get the following error at the end. It might be easy to resolve, but I’m pretty new to python and already feel lost in the fog, tried to google it every possible way. I’m using zipline-reloaded. Thanks

    C:\Users\tomas>zipline ingest -b random_stock_data
    [2021-10-29 22:57:27.946828] INFO: zipline.data.bundles.core: Ingesting random_stock_data.
    Loading A…
    Traceback (most recent call last):
    File “c:\users\tomas\appdata\local\programs\python\python38\lib\runpy.py”, line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
    File “c:\users\tomas\appdata\local\programs\python\python38\lib\runpy.py”, line 87, in _run_code
    exec(code, run_globals)
    File “C:\Users\tomas\AppData\Local\Programs\Python\Python38\Scripts\zipline.exe\__main__.py”, line 7, in
    File “c:\users\tomas\appdata\local\programs\python\python38\lib\site-packages\click\core.py”, line 1128, in __call__
    return self.main(*args, **kwargs)
    File “c:\users\tomas\appdata\local\programs\python\python38\lib\site-packages\click\core.py”, line 1053, in main
    rv = self.invoke(ctx)
    File “c:\users\tomas\appdata\local\programs\python\python38\lib\site-packages\click\core.py”, line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
    File “c:\users\tomas\appdata\local\programs\python\python38\lib\site-packages\click\core.py”, line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
    File “c:\users\tomas\appdata\local\programs\python\python38\lib\site-packages\click\core.py”, line 754, in invoke
    return __callback(*args, **kwargs)
    File “c:\users\tomas\appdata\local\programs\python\python38\lib\site-packages\zipline\__main__.py”, line 389, in ingest
    bundles_module.ingest(
    File “c:\users\tomas\appdata\local\programs\python\python38\lib\site-packages\zipline\data\bundles\core.py”, line 445, in ingest
    bundle.ingest(
    File “c:\users\tomas\appdata\local\programs\python\python38\lib\site-packages\zipline\data\bundles\random_stock_data.py”, line 47, in random_stock_data
    daily_bar_writer.write(
    File “c:\users\tomas\appdata\local\programs\python\python38\lib\site-packages\zipline\data\bcolz_daily_bars.py”, line 209, in write
    return self._write_internal(it, assets)
    File “c:\users\tomas\appdata\local\programs\python\python38\lib\site-packages\zipline\data\bcolz_daily_bars.py”, line 269, in _write_internal
    for asset_id, table in iterator:
    File “c:\users\tomas\appdata\local\programs\python\python38\lib\site-packages\zipline\data\bcolz_daily_bars.py”, line 202, in
    ((sid, self.to_ctable(df, invalid_data_behavior)) for sid, df in data),
    File “c:\users\tomas\appdata\local\programs\python\python38\lib\site-packages\zipline\data\bundles\random_stock_data.py”, line 66, in process_stocks
    metadata.loc[sid] = start_date, end_date, ac_date, symbol, “NYSE”
    File “c:\users\tomas\appdata\local\programs\python\python38\lib\site-packages\pandas\core\indexing.py”, line 692, in __setitem__
    iloc._setitem_with_indexer(indexer, value, self.name)
    File “c:\users\tomas\appdata\local\programs\python\python38\lib\site-packages\pandas\core\indexing.py”, line 1629, in _setitem_with_indexer
    self._setitem_with_indexer_missing(indexer, value)
    File “c:\users\tomas\appdata\local\programs\python\python38\lib\site-packages\pandas\core\indexing.py”, line 1918, in _setitem_with_indexer_missing
    raise ValueError(“cannot set a row with mismatched columns”)
    ValueError: cannot set a row with mismatched columns

  148. Dear Mr. Clenow

    first I have to say, that I really love all your books and really learned a lot from them.
    The reason I am writing this email is, that I was wondering if you have an updated version of the csv file including all the constituents of the s&p 500 on any given day, because since the book release there have been many changes and such a list isn’t included in the Norgate Data bundle.
    If you could tell me, how to best get such a csv file, or maybe even have one yourself, I would be really grateful.

    Kind regards
    Luca

    • You won’t need it if you use Norgate. They have a great functionality built-in. Read their docs and you’ll see that their solution is far better than the one I used in the book.

  149. Hi Andreas, I’m a big fan of your books! The cleverness of your momentum score formula (exponentiated least squares slope of log price multiplied by R2 and then annualized) blew my mind.

    Just wanted to point out a couple of minor typos in chapter 18, Trading the curve.
    1. Figure 18-2 on page 302 is labelled “Gasoline in Contango” but it’s actually in backwardation.
    2. page 303 refers to Figure 18-1 and states “the first point in the curve, the closest contract […], the SH9 is currently traded at 907.5” while based on the figure, it’s obviously <890. But perhaps you intended to type "Table" instead of "Figure" because the statement is consistent with Table 18-1 on page 304. (I'm using the "Book Version 1.5 – Mid 2021 Edition")

    Best regards,
    Cyrus

  150. “Book version 1.5 – mid 2021 Edition”
    Guest chapter 22, page 356: at the bottom of the page Carver takes the cumsum of percent returns — this doesn’t make sense — if I’m up 10% yesterday and and down 10% today I am NOT back to where I was at the beginning of yesterday as per Carver’s cumsum would entail (0.1-0.1=0). INSTEAD, I am down 1% (1.10*0.9 – 1 = -0.01)

    so perhaps what Carver meant was something like (returns+1).cumprod() or np.exp(np.log(returns+1).cumsum())-1

    Best,
    Cyrus

  151. Hi Andeas,
    Tiny typo to add to your update list if you want..

    Kindle page 79 “percent chance” to “percent change”?

    Happy New Year and thank you for the excellent book!
    What a gift you’ve given me with this brilliant educational resource.

    Kind regards,
    Kelly

  152. one more to add if you like..

    Kindle book:
    pg. 393
    “We need to ^ one line to this code” ^add

    PS. Ingesting my first bundle. Wish me luck!
    Kelly

  153. Siraphob Pongkritsagorn

    Hi Andreas,

    For preparing equity data locally, I wonder how do you handle those stocks that suddenly got merged/acquired? As I tried to replicate Chapter 12 (Momentum), some of these stocks suddenly experienced a big drop in volume before becoming NaN (I use ffill for these NaN cells) while they still qualified to be selected earlier. Hence, the strategy sometimes could not sell these names due to the 2.5% volume limit, causing my gross_leverage to sometimes go far beyond 1 for an extended period of time.

    Thank you in advance.
    Gem

    • I’d run an automated comparison of local to remote data and flush if any inconsistency. For instance, a daily check if the price a year ago is the same for both sources. If not, assume that the remote source is correct, flush and overwrite local.

      • Siraphob Pongkritsagorn

        Hi Andreas,

        Thank you for a your response.

        However, I’m not sure if my question was clear enough so let me give an example situation:

        * Assuming we have the same criteria as Chapter 12 (Equity Momentum)
        – Stock A qualifies to be bought on month1
        – The strategy buys stock A (along with other qualified stocks)
        – At some point in month 2 stock A got acquired, hence delisted while the strategy is still holding it.

        Now if I left the cells for the days after the acquisition NaN, it’d cause an error. If I ffill the remaining cells using the last traded prices and volumes, the volumes would be super thin that the strategy couldn’t sell it in a day.

        For this situation, I wonder how would you prep your local data for stock A and other stocks that got acquired at some point?

        Thank you for your time.
        Gem

  154. Andreas,

    Thanks for the effort on the book. I am having some issue getting the first zipline backtest up and running due to some errors that I cannot reconcile based on what I have read in your book

    With the unmodified benchmarks.py file i get the following error:

    File ~/opt/anaconda3/envs/env_zipline/lib/python3.8/site-packages/zipline/finance/trading.py:40, in SimulationParameters.__init__(self, start_session, end_session, trading_calendar, capital_base, emission_rate, data_frequency, arena)
    29 def __init__(
    30 self,
    31 start_session,
    (…)
    37 arena=”backtest”,
    38 ):
    —> 40 assert type(start_session) == pd.Timestamp
    41 assert type(end_session) == pd.Timestamp
    43 assert trading_calendar is not None, “Must pass in trading calendar!”

    AssertionError:

    I have modified benchmarks.py as suggested in your book but now when I run the code, I get the following error

    file ~/opt/anaconda3/envs/env_zipline/lib/python3.8/site-packages/trading_calendars/calendar_helpers.py:6, in
    2 import pandas as pd
    4 NANOSECONDS_PER_MINUTE = int(6e10)
    —-> 6 NP_NAT = np.array([pd.NaT], dtype=np.int64)[0]
    9 def next_divider_idx(dividers, minute_val):
    11 divider_idx = np.searchsorted(dividers, minute_val, side=”right”)

    TypeError: int() argument must be a string, a bytes-like object or a number, not ‘NaTType’

    Not really sure how to handle this

  155. Hello, does anyone have list of packages and versions that will work with the code in the book (eg Python version, pandas/numpy, etc)? I am spending more time debugging the errors due to changes in package versions, than using the book. Thanks.

  156. Hello Andreas
    Can you help please. I have been trying to register a custom bundle (without success) based upon pages 416 – 418 of Trading Evolved (version 1). The bundle file (ES_bundle.py) runs without errors when run separately. However, when the extension file is run, no errors occur yet registration of the bundle does not occur. The two files are shown below.

    Zipline-reloaded was downloaded on the 18th September 2022. Python version = 3.8.13. Pandas version = 1.4.3.

    Lines 1 & 2 below represent the extension.py file, to register the bundle. The bundle name is ES_bundle.py and the ingest function name is: database_bundle.

    1)from zipline.data.bundles import register, ES_bundle
    2)register(‘ES_bundle’, ES_bundle.database_bundle, calendar_name=’NYSE’)

    The custom bundle (ES_bundle) is shown below:-
    #This is the Bundle to extract import actual data from a database
    #Date august 2022
    import pandas as pd
    from sqlalchemy import create_engine
    from tqdm import tqdm #This is used for the progress bar

    engine=create_engine(‘sqlite:////home/esmart/Projects/pythonlab/db_pff7.db’)

    #Returns a list of the numbers representing ticker symbols
    def available_stocks():
    symbol_query=”select distinct id from security_price order by id “
    symbols=pd.read_sql_query(symbol_query, engine)
    return symbols.id #Returns a list of numbers which represent the ticker symbols

    #The ingest function needs to have this exact signature
    def database_bundle(environ,
    asset_db_writer,
    minute_bar_writer,
    daily_bar_writer,
    adjustment_writer,
    calendar,
    start_session,
    end_session,
    cache,
    show_progress,
    output_dir):

    #This line may not be necessary
    symbols=available_stocks()
    if not symbols: #Added 22/9/2022
    raise ValueError(“No symbols found in folder.”) #Added 22/9/2022

    #Prepare an empty dataframe for dividends
    divs=pd.DataFrame(columns=[‘sid’,
    ‘amount’
    ‘ex_date’,
    ‘record_date’
    ‘declared_date’,
    ‘pay_date’])

    #Prepare an empty dataframe for splits
    splits=pd.DataFrame(columns=[‘sid’,
    ‘ratio’,
    ‘effective_date’])

    #Prepare an empty dataframe for metadata
    metadata=pd.DataFrame(columns=(‘start_data’,
    ‘end_date’,
    ‘auto_close_date’,
    ‘symbol’,
    ‘exchange’))

    #check valid trading dates, according to the selected exchange calendar
    sessions=calendar.sessions_in_range(start_session, end_session)

    #get data for all stocks and write to zipline
    daily_bar_writer.write(process_stocks(symbols,
    sessions, metadata, divs))

    #write the metadata
    asset_db_writer.write(equities=metadata)

    #write splits and dividends
    adjustment_writer.write(splits=splits,
    dividends=divs)

    “””Generator function to iterate stocks,
    build historical data, metadata and
    dividend data”””

    def process_stocks(symbols, sessions, metadata, divs):
    #loop the stocks, setting a unique security id (sid)
    sid = 0
    for symbol in tqdm(symbols):
    sid += 1

    #make a database query. When necessary add ‘dividend’ to the query.
    query=”””select date, open, high, low, close, volume, adj_close, security_id
    from security_price where id ='{}’ order by date;
    “””.format(symbol)

    #ask the database for the data
    df=pd.read_sql_query(query,engine,index_col=’date’, parse_dates=[‘date’])

    #check first and last date
    start_date=df.index[0]
    end_date=df.index[-1]

    #synch to the official exchange calendar
    df=df.reindex(sessions.tzlocalize(None))[start_date:end_date]

    #forward fill missing data
    df.fillna(method=’ffill’, inplace=True)

    #drop remaining NaN
    df.dropna(inplace=True)

    #the auto_close date is the day after the last trade
    ac_date=end_date + pd.Timedelta(days=1)

    #add a row to the metadata dataframe
    metadata.loc[sid]=start_date, end_date, ac_date, symbol, ‘NYSE’

    #if there is dividend data, add that to the dividend dataframe
    if ‘dividend’ in df.columns:
    #slice off the days with dividends
    tmp=df[df[‘dividend’] != 0.0 ][‘dividend’]
    div=pd.DataFrame(data=tmp.index.tolist(), columns=[‘ex_date’])

    #provide empty columns as we do not have this data for now
    div[‘record_date’]=pd.NaT
    div[‘declared_date’]=pd.NaT
    div[‘pay_date’]=pd.NaT

    #store the dividends and set the security id
    div[‘amount’]=tmp.tolist()
    div[‘sid’]=sid

    #start numbering at where we left off last time
    ind=pd.index(range(divs.shape[0], divs.shape[0] + div.shape[0]))
    div.set_index(ind, inplace=True)

    #Append this stocks dividends to the list of all dividend
    divs=divs.append(div)

    yield sid, df

  157. Hello Andreas
    I posted a question last week about the process to register a custom bundle from your book Trading Evolved. Shortly after I posted the question, it disappeared from view. Therefore, can I confirm that the problem is being investigated.

    Just to be clear, although the extension.py file appears to run without any warnings or errors. When I try to ingest the bundle, there is a message which says: Error: No bundle registered with the name ‘ES_bundle’.

    J.Smith

  158. Richard Lysakowski

    I bought the Kindle edition of your book when it first came out. At that time Zipline support was dropped and the software so buggy to get running that I just shelved the book and used other tools. I did not have the Python skills to fix it at that time. Without Zipline working, much of the code in the book would not work. The Zipline self-support community eventually emerged. I found the Zipline Reloaded has fixed the community support issue and reopened your book today.

    In the messages above you stated “I used Zipline Reloaded in the latest book update. If you’ve got the Kindle version, you should be able to update it to the latest version, released a month or two ago.

    How do I update the Kindle edition of the book? When I reopen it from the Amazon Online Kindle Reader it does not appear to be an updated version. The version on the first page says: Version 1.1

    Can you please provide an up-to-date Github.com link to the updated source code for me to download it? It looks like many bugs were found and changes made to the Zipline-related code. Have those bug fixes been propagated to main github.com branch for you book code?

    Thank you.

    Richard

    • I’m not sure who’s Github rep you’re looking at, but it’s not mine. I didn’t put anything there.

      The easiest way to get started, by far, is to use the Norgate integration: https://pypi.org/project/zipline-norgatedata/ There’s a link near the bottom to code for the book, all adapted for Zipline Reloaded and Norgate data integration. Bug fixes and updates are in there.

      As for updating the book, there should be some menu in your Kindle library on Amazon for that. I wish I could control that part, but it’s all up to Jeff. They tell me that updates should be available to those who already bought it.

Leave a Reply to Ching Ang Cancel reply

Your email address will not be published. Required fields are marked *

*