Following the Trend
Zipline in Docker
This article is contributed by Henrik Nilsson, a clever Swedish guy who read my book and rightly pointed out that I should have mentioned something about how Docker can help simplify the process of setting up and running Zipline. I very much recommend reading and following the instructions below. Having had limited experience with Docker myself, I learned from this as well. Thanks, Henrik! (and sorry for taking so long to post it… it’s been a crazy few months)
Introduction
Docker is a powerful way to share applications and their dependencies between each other. Today’s application usually relies on various dependencies and can be a hassle to install, if even possible.
As of April 2020 the Zipline(1.3.0) that available to download through pypi is released July 18 2018 and depends on running Python 3.5. If you just recently upgraded your operating system you may even find it nearly impossible to get python3.5 running. Second note that, even while still relying on 3.5, Zipline has continued being updated since July 2018. So if you want the latest greatest you probably want to avoid the pypi- package altogether.
Docker makes all this pain go away. It will create a isolated environment that is tailored to the running application, with little-to-none overhead.
Installing Docker
The only requirements that is needed is to have Docker running on your computer.
This shall work regardless of Windows, Mac or Linux. Most containers expect a Mac or Linux environment so they may warn when running inside Windows, but nevertheless it should work.
Follow the steps inside: https://docs.docker.com/get-docker/
Running Zipline with built image
This will be the absolute fastest way by using a built image published to Docker. The steps done before publishing this is described below. In case you are curious why it was not published by Quantopian it still has some parts that needs to be modified for running locally, as described in Trading Evolved.
Docker image is built based on the code: https://github.com/lhjnilsson/zipline
Information about the image is found at docker: https://hub.docker.com/repository/docker/lhjnilsson/zipline
Command below expects you to have downloaded the Trading Evolved.zip and extracted under a folder locally and that you are inside that folder. It will download the image and start a container.
docker run -v pwd:/projects -v ~/.zipline:/root/.zipline -p 8888:8888/tcp --name zipline -it lhjnilsson/ziplineIf this is fine you can proceed to last section “Access Jupyter Interface”!
Manually building Zipline yourself
If you want to build this image yourself instead of Running prebuilt image as explained above you can follow the steps below. There is no benefit in doing one over the other, pure educational.
Getting Zipline from Github
Clone the source from Quantopian Github:
git clone https://github.com/quantopian/zipline.gitAs of writing this guide the latest commit is bc469dbcee337eb223c351fe27429aac538dbf79 if you run into issues, please checkout that commit if you want to have the same start as i did:
git checkout bc469dbcee337eb223c351fe27429aac538dbf79Manual changes before Building
- Range fix of numpy and scipy
The Quantopian hosted Zipline seems to only control one part of the dependeny range. This can be highly dangerous and can make you get unexpected versions that is non- compliant.
At least this was the case for me, i have to update with a new range, code:
- Support Python Datetime as start & end and auto insert NYSE benchmark_returns(Optional)
The older Zipline API(as used in Trading Evolved) uses python datetime- object for start- & end date when running algorithm. New Zipline uses Pandas Timestamp. Code below is a fix to, in case datetime is use, convert it into pandas Timestamp. Second is that instead of using the benchmark from IEXTrading Rest API it now expects a benchmark_returns to be inserted or have a file locally stored with the benchmark(?). In case benchmark_returns is None we use same example as in Trading Evolved to have as benchmark. Code:
Building Zipline
docker build -t zipline .This will take some time(~10min on my computer).
It will pip install all the needed parts and compile the things that are needed to be compiled.
Running Zipline
docker run -v pwd:/projects -v ~/.zipline:/root/.zipline -p 8888:8888/tcp --name zipline -it ziplineOnce Image has been built successfully you are ready to run the container! This example expects that you have downloaded Trading Evolved.zip, extracted the content and are currently inside that folder.
Premier on Docker
Below are example of commands that you probably want to memorize to some extent.
Start a new Container:
docker run -v HOST_FOLDER:/CONTAINER_FOLDER -v -p HOST_PORT:CONTAINER_PORT/tcp --name CONTAINER_NAME -it IMAGEThis will start a new Instance of the image as a running container.
List:
docker psTo list running containers
Execute Command:
docker exec -it CONTAINER_NAME COMMANDGet a Bash- shell inside a running container, from here you can use it like a normal terminal.
Another Example…
docker exec -it -e QUANDL_API_KEY=YOUR_API_KEY zipline zipline ingest -b quandlExecute a Zipline command inside the container, with environment variable set(e.g. Quandl Key!)
In practise this will: Enter Container, Execute Zipline to ingest data from Quandl and Exit.
Stop container:
docker stop CONTAINER_NAMEStop/halt the container(zipline in example).
Start as stopped container/Resume:
docker start CONTAINER_NAMEStart the stopped container.
Remove container:
docker rm CONTAINER_NAMERemove the container fully. Keep in mind that mounted volumes(using -v) are still being stored on your host(!) so they will not be lost.
Access Jupyter Interface
Before you run any backtest, make sure that you have ingested any data, or it will be rather boring.
Change the api key to the quandl key you get from their website.
docker exec -it -e QUANDL_API_KEY=YOUR_API_KEY zipline zipline ingest -b quandlSince we started the container with -v ~/.zipline:/root/.zipline option it is actually stored in our host. So previous ingested data will be present, as well as any updated data will also be stored on your host!
Now, since we passed -p 8888:8888 in our docker- command we can access the notebook from:
You will most likely see a warning about Certificate not Trusted. This is expected. Either accept this or change(in the Docker Image) to start without HTTPS.
Also you should be prompted with a password. The default password is jupyter
Now we should be ready to go! Let’s run some backtests!
Sweet! 🙂
As the contributor of this post i hope you all can solve some of the possible issues with Zipline!
If there is (yet) any concerns about installation or things you wish to have clarified i can try to help in these comments!
Take care! Happy to see myself contributed here
Hi Henrik,
really great work! With docker and your instructions (I am a docker newbie) I had a running zipline installation within a few minutes. Thanks a lot!
However, I noticed that pyfolio is not included in the image which is needed, for instance, for Andreas’ backtest analysis in Chapter 8. Could you give a quick advice or suggestion on how to pip install packages to docker images/containers afterwards?
Thanks again,
Frank
Hello Frank!
From a terminal:
“”
lhjnilsson@nixtop:~$ docker exec -it zipline bash
root@e47232c61cab:/projects# pip install pyfolio
“”
It is based on that you have zipline as the name of your docker- session. (–name zipline from guide)
This will install in locally in your Docker.
It will remain as long as the docker is stored, either as running or paused/stopped.
In case you remove the container- session you would have to redo it.
If you wish to save this modification you can do a “Docker Commit”
Would then be as simple as: “docker commit zipline franks-zipline”
Then it will be stored so the next time you start a docker- session from scratch you run it instead like:
docker run -v
pwd:/projects -v ~/.zipline:/root/.zipline -p 8888:8888/tcp –name zipline -it franks-ziplineHi Henrik,
that was quick and worked like a charm!
Thanks a lot and enjoy your midsummer too!
… something i have noticed that may cause confusion.
The marking that makes pwd be written like it is comes due to “Grave Accent”
So both in guide and comment. please put “grave accent” (`) around pwd to make it run propely.
Have a nice midsummer whoever reads this!
hello, was trying to run this but I get following error message:
docker run -v “pwd”:/projects -v ~/.zipline:/root/.zipline -p 8888:8888/tcp –name zipline -it zipline
Unable to find image ‘zipline:latest’ locally
docker: Error response from daemon: pull access denied for zipline, repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied.
See ‘docker run –help’.
Any idea what this error could be?
Thanks a lot for this. Managed to get it working. I have a question: how can I ingest my custom data as described in the book? I have created the .py file but should I save it within the container and how can I go the registration of the bundle and ingestion?
Thanks a lot in advance and apologies if this is a simple question. I am not very well versed with Docker
i am facing the same issue. any one has any ideas how to register the bundle and ingest in docker? i can’t locate the directories as it is no longer running locally.