MLops for beginners

Day15 — Containerizing our Dev Environment

Building container image for jupyter with all required packages

In most of my previous articles, I was using jupyter notebook for the development. And recently my environment got a bit messy due to multiple versions for the same packages and other stuff.

So I thought of containerizing my development environment and pack it. Thus leveraging all the container benefits for my development environment.

Steps at a glance:

• create a base OS
• Install Python3 on it
• use pip to install all the requirements
• create and work directory and change into that directory
• Launch jupyter on startup

If you have been working with containers lately then the steps should be pretty much clear to you. The idea is to simply create a container that have all my requirements installed and will give me a jupyter notebook to work with.

Let’s see how you can do this for yourself.

Prerequisite:

Docker engine should be installed on your system.

I’ll break the whole process into some doable steps to make things easier to understand and perform.

1. Download a image and install python3 on it.

There are a plenty of docker images present out there on docker hub. You can simply choose anyone as per your requirements. In my previous articles, I used centOS image but installing python3 over that image will make things a bit fat. So instead i used python3 image directly available on docker hub.

list all docker images and grep out python and centos image

python:3.6-slim —  python image
centos:latest — without python installed

So centos:latest will be a lot heavier if we install python on it as compared to directly using python:3.6-slim image. So why not use the slim image. 🤷‍♂️

2. Install requirements.

I have my requirements list ready with me. Either you can use the same or simply make your own requirements list.

absl-py==0.9.0
astunparse==1.6.3
attrs==19.3.0
backcall==0.1.0
bleach==3.1.5
cachetools==4.1.0
certifi==2020.4.5.1
chardet==3.0.4
decorator==4.4.2
defusedxml==0.6.0
entrypoints==0.3
gast==0.3.3
google-auth==1.15.0
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
grpcio==1.29.0
h5py==2.10.0
idna==2.9
importlib-metadata==1.6.0
ipykernel==5.3.0
ipython==7.14.0
ipython-genutils==0.2.0
ipywidgets==7.5.1
jedi==0.17.0
Jinja2==2.11.2
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.1.3
jupyter-console==6.1.0
jupyter-core==4.6.3
Keras==2.3.1
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.2
Markdown==3.2.2
MarkupSafe==1.1.1
mistune==0.8.4
nbconvert==5.6.1
nbformat==5.0.6
notebook==6.0.3
numpy==1.18.4
oauthlib==3.1.0
opt-einsum==3.2.1
packaging==20.4
pandocfilters==1.4.2
parso==0.7.0
pexpect==4.8.0
pickleshare==0.7.5
prometheus-client==0.8.0
prompt-toolkit==3.0.5
protobuf==3.12.2
ptyprocess==0.6.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
Pygments==2.6.1
pyparsing==2.4.7
pyrsistent==0.16.0
python-dateutil==2.8.1
PyYAML==5.3.1
pyzmq==19.0.1
qtconsole==4.7.4
QtPy==1.9.0
requests==2.23.0
requests-oauthlib==1.3.0
rsa==4.0
scipy==1.4.1
Send2Trash==1.5.0
six==1.15.0
tensorboard==2.2.1
tensorboard-plugin-wit==1.6.0.post3
tensorflow==2.2.0
tensorflow-estimator==2.2.0
termcolor==1.1.0
terminado==0.8.3
testpath==0.4.4
tornado==6.0.4
traitlets==4.3.3
urllib3==1.25.9
wcwidth==0.1.9
webencodings==0.5.1
Werkzeug==1.0.1
widgetsnbextension==3.5.1
wrapt==1.12.1
zipp==3.1.0

These are the packages that will download and setup jupyter, tensorflow and keras. You can create your own requirements.txt with your own specific libraries.

3. Now let’s create our Dockerfile that will do everything for us.

FROM python:3.6-slim
# install requirements
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
# workspace
WORKDIR /home/workspace
# Exposing ports
EXPOSE 8888
# Running jupyter notebook
# --NotebookApp.token ='demo' is the password
CMD ["jupyter", "notebook", "--no-browser", "--ip=0.0.0.0", "--allow-root", "--NotebookApp.token='demo'"]

• FROM python:3.6-slim → Use python:3.6-slim image as a base os
• COPY requirements.txt requirements.txt → Copy requierments.txt file inside the container.
• RUN pip install -r requirements.txt→ Install all the requirements from requirements.txt file
• WORKDIR /home/workspace → This is used to change my working directory to /home/workspace.
• EXPOSE 8888 → Expose port 8888 so that we can use jupyter via web browser.
• CMD ["jupyter", "notebook", "--no-browser", "--ip=0.0.0.0", "--allow-root", "--NotebookApp.token='demo'"] → Start jupyter notebook with some additional flags.

4. Now all we need to do is build the docker image and start a container.

Syntax (to build image):

docker build -t <dockerid>/<imagename>:<tag> -f “<filename>” .

What i used :

docker build -t ayedaemon/my_jupyter:v1 -f “dockerfile.jupyter” .

After you have successfully built your image, it’s time to launch a container.

docker container run -d -P -v $PWD/notebooks:/home/workspace --name myjenkins my_jupyter:v1

• -d → start container in detached mode.
• -P → map a random port to the exposed container. Unline -p, which is used to map exposed port to some specific port.
• -v → to add volume to the container.
• --name → to give some name to the container.

On Browser

Here it asks for the password or token. You can either set up a new password or simply login using token. (Token is already specified in Dockerfile)

Now we can use this same container to ship our development environment into any machine that have docker engine installed in it (or even with kubernetes).

But to make this image available from internet, we’ll have to push this to docker hub. Here the role of <dockerID> comes in.

Follow the below steps:

• Go to https://hub.docker.com/ , register yourself, and get a DockerID and your password.
• Set up you docker hub client on your terminal by docker login command.
• use docker push <dockerID>/<containername> command to push this container to docker hub.

Now Sit and Relax.