MLops for beginners

Day15 — Containerizing our Dev Environment

In most of my previous articles, I was using jupyter notebook for the development. And recently my environment got a bit messy due to multiple versions for the same packages and other stuff.

So I thought of containerizing my development environment and pack it. Thus leveraging all the container benefits for my development environment.

Steps at a glance:

  • create a base OS
  • Install Python3 on it
  • use pip to install all the requirements
  • create and work directory and change into that directory
  • Launch jupyter on startup

If you have been working with containers lately then the steps should be pretty much clear to you. The idea is to simply create a container that have all my requirements installed and will give me a jupyter notebook to work with.

Let’s see how you can do this for yourself.

Prerequisite:

Docker engine should be installed on your system.

I’ll break the whole process into some doable steps to make things easier to understand and perform.

1. Download a image and install python3 on it.

There are a plenty of docker images present out there on docker hub. You can simply choose anyone as per your requirements. In my previous articles, I used centOS image but installing python3 over that image will make things a bit fat. So instead i used python3 image directly available on docker hub.

list all docker images and grep out python and centos image
python:3.6-slim —  python imagecentos:latest — without python installed

So centos:latest will be a lot heavier if we install python on it as compared to directly using python:3.6-slim image. So why not use the slim image. 🤷‍♂️

2. Install requirements.

I have my requirements list ready with me. Either you can use the same or simply make your own requirements list.

absl-py==0.9.0
astunparse==1.6.3
attrs==19.3.0
backcall==0.1.0
bleach==3.1.5
cachetools==4.1.0
certifi==2020.4.5.1
chardet==3.0.4
decorator==4.4.2
defusedxml==0.6.0
entrypoints==0.3
gast==0.3.3
google-auth==1.15.0
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
grpcio==1.29.0
h5py==2.10.0
idna==2.9
importlib-metadata==1.6.0
ipykernel==5.3.0
ipython==7.14.0
ipython-genutils==0.2.0
ipywidgets==7.5.1
jedi==0.17.0
Jinja2==2.11.2
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.1.3
jupyter-console==6.1.0
jupyter-core==4.6.3
Keras==2.3.1
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.2
Markdown==3.2.2
MarkupSafe==1.1.1
mistune==0.8.4
nbconvert==5.6.1
nbformat==5.0.6
notebook==6.0.3
numpy==1.18.4
oauthlib==3.1.0
opt-einsum==3.2.1
packaging==20.4
pandocfilters==1.4.2
parso==0.7.0
pexpect==4.8.0
pickleshare==0.7.5
prometheus-client==0.8.0
prompt-toolkit==3.0.5
protobuf==3.12.2
ptyprocess==0.6.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
Pygments==2.6.1
pyparsing==2.4.7
pyrsistent==0.16.0
python-dateutil==2.8.1
PyYAML==5.3.1
pyzmq==19.0.1
qtconsole==4.7.4
QtPy==1.9.0
requests==2.23.0
requests-oauthlib==1.3.0
rsa==4.0
scipy==1.4.1
Send2Trash==1.5.0
six==1.15.0
tensorboard==2.2.1
tensorboard-plugin-wit==1.6.0.post3
tensorflow==2.2.0
tensorflow-estimator==2.2.0
termcolor==1.1.0
terminado==0.8.3
testpath==0.4.4
tornado==6.0.4
traitlets==4.3.3
urllib3==1.25.9
wcwidth==0.1.9
webencodings==0.5.1
Werkzeug==1.0.1
widgetsnbextension==3.5.1
wrapt==1.12.1
zipp==3.1.0

These are the packages that will download and setup jupyter, tensorflow and keras. You can create your own requirements.txt with your own specific libraries.

3. Now let’s create our Dockerfile that will do everything for us.

FROM python:3.6-slim# install requirements
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
# workspace
WORKDIR /home/workspace
# Exposing ports
EXPOSE 8888
# Running jupyter notebook
# --NotebookApp.token ='demo' is the password
CMD ["jupyter", "notebook", "--no-browser", "--ip=0.0.0.0", "--allow-root", "--NotebookApp.token='demo'"]
  • FROM python:3.6-slim → Use python:3.6-slim image as a base os
  • COPY requirements.txt requirements.txt → Copy requierments.txt file inside the container.
  • RUN pip install -r requirements.txt→ Install all the requirements from requirements.txt file
  • WORKDIR /home/workspace → This is used to change my working directory to /home/workspace.
  • EXPOSE 8888 → Expose port 8888 so that we can use jupyter via web browser.
  • CMD ["jupyter", "notebook", "--no-browser", "--ip=0.0.0.0", "--allow-root", "--NotebookApp.token='demo'"] → Start jupyter notebook with some additional flags.

4. Now all we need to do is build the docker image and start a container.

Syntax (to build image):

docker build -t <dockerid>/<imagename>:<tag> -f “<filename>” .

What i used :

docker build -t ayedaemon/my_jupyter:v1 -f “dockerfile.jupyter” .

After you have successfully built your image, it’s time to launch a container.

docker container run -d -P -v $PWD/notebooks:/home/workspace --name myjenkins my_jupyter:v1
  • -d → start container in detached mode.
  • -P → map a random port to the exposed container. Unline -p, which is used to map exposed port to some specific port.
  • -v → to add volume to the container.
  • --name → to give some name to the container.
On Browser

Here it asks for the password or token. You can either set up a new password or simply login using token. (Token is already specified in Dockerfile)

Now we can use this same container to ship our development environment into any machine that have docker engine installed in it (or even with kubernetes).

But to make this image available from internet, we’ll have to push this to docker hub. Here the role of <dockerID> comes in.

Follow the below steps:

  • Go to https://hub.docker.com/ , register yourself, and get a DockerID and your password.
  • Set up you docker hub client on your terminal by docker login command.
  • use docker push <dockerID>/<containername> command to push this container to docker hub.

Now Sit and Relax.

--

--

--

Connecting the dots and rest is magic. https://ayedaemon.github.io/

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Git Basics on CMD and Source Tree for Code Push & Pull Code

Yarn Hadoop Architecture

Why such a hype for microservices?

C# Code Snippet — InvokeRepeating in Godot

The Path to Develop Professional Programming Skills

Securing your middleware with JWT

GDG Turkey Summit (EN)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Rishabh Umrao

Rishabh Umrao

Connecting the dots and rest is magic. https://ayedaemon.github.io/

More from Medium

Getting Hands-on with Kubernetes using Simple Python Application (Part I)

Set up configs like memory limits in Docker for Windows and WSL2

Running Scalable Lightweight Session aware Python Flask application on Kubernetes

Deploying Flask App on Kubernetes

Setting up Python Interpreter and running Python Code on Docker Container