Notes on Dockers

Notes on Dockers

Notes, Daily commands, high level concepts related to the daily use of dockers for software development purpose.

Concepts of Dockers

How dockers system works ?, Its all refer to a file system which differentiate it from virtual machines. In my initial finding it uses the base concept from Cgroups and Linux Containers.

What is talks actually On High Level:

It talks about the layers of files, like It first picks up the base file system which it says like base image of containers(e.g. specific linux version) and then on top it java installation (JDK stuff), and then on top of it specific webservice deployment. So now this web service running only in an isolated environment and not conflicting with any other. its like a stack of files.

To understand the insight of dockers, need to understand the concepts of cgroups and Linux Containers (LXC):

cgroups (abbreviated from control groups) is a Linux kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes.

Refer : https://en.wikipedia.org/wiki/Cgroups
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01

https://en.wikipedia.org/wiki/LXC

LXC (Linux Containers) is an operating-system-level virtualization method for running multiple isolated Linux systems (containers) on a control host using a single Linux kernel.

Refer : https://www.docker.com/

Refer : Linux Containers : https://linuxcontainers.org/

 

docker rmi XXX (delete images)

docker -version 

eSumit@Sumits-MacBook-Pro:~/Documents/Public Github/GitPush/CloudManager$ docker version
Client:
 Version: 17.09.1-ce
 API version: 1.32
 Go version: go1.8.3
 Git commit: 19e2cf6
 Built: Thu Dec 7 22:22:25 2017
 OS/Arch: darwin/amd64

Server:
 Version: 17.09.1-ce
 API version: 1.32 (minimum version 1.12)
 Go version: go1.8.3
 Git commit: 19e2cf6
 Built: Thu Dec 7 22:28:28 2017
 OS/Arch: linux/amd64
 Experimental: true
eSumit@Sumits-MacBook-Pro:~/Documents/Public Github/GitPush/CloudManager$

docker info

eSumit@Sumits-MacBook-Pro:~/Documents/Public Github/GitPush/CloudManager$ docker info
Containers: 5
 Running: 0
 Paused: 0
 Stopped: 5
Images: 10
Server Version: 17.09.1-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 63
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
 Profile: default
Kernel Version: 4.9.49-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.952GiB
Name: moby
ID: F6DU:FNP2:QGDU:SCJY:INGB:II7A:HN6N:ANLE:2VGE:AJN3:74UF:66M4
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 18
 Goroutines: 29
 System Time: 2018-02-05T08:28:35.381917475Z
 EventsListeners: 1
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

eSumit@Sumits-MacBook-Pro:~/Documents/Public Github/GitPush/CloudManager$

 

5 Important Commands on Dockers ( from Medium)

  1. docker logs <container_id>
    Hopefully you’ve already tried this, but if not, start here. This’ll give you the full STDOUT and STDERR from the command that was run initially in your container.
  2. docker stats <container_id>
    If you just need to keep an eye on the metrics of your container to work out what’s gone wrong, docker stats can help: it’ll give you a live stream of resource usage, so you can see just how much memory you’ve leaked so far.
  3. docker cp <container_id>:/path/to/useful/file /local-path
    Often just getting hold of more log files is enough to sort you out. If you already know what you want, docker cp has your back: copy any file from any container back out onto your local machine, so you can examine it in depth (especially useful analysing heap dumps).
  4. docker exec -it <container_id> /bin/bash
    Next up, if you can run the container (if it’s crashed, you can restart it with docker start <container_id>), shell in directly and start digging around for further details by hand.
  5. docker commit <container_id> my-broken-container &&
    docker run -it my-broken-container /bin/bash
    Can’t start your container at all? If you’ve got a initial command or entrypoint that immediately crashes, Docker will immediately shut it back down for you. This can make your container unstartable, so you can’t shell in any more, which really gets in the way.
    Fortunately, there’s a workaround: save the current state of the shut-down container as a new image, and start that with a different command to avoid your existing failures.
    Have a failing entrypoint instead? There’s an entrypoint override command-line flag too.

View story at Medium.com

How to check if docker installed correctly 

Read these four steps :

To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the “hello-world” image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.

 

 

eSumit@Sumits-MacBook-Pro:~/Documents/Public Github/GitPush/CloudManager$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
ca4f61b1923c: Pull complete 
Digest: sha256:66ef312bbac49c39a89aa9bcc3cb4f3c9e7de3788c944158df3ee0176d32b751
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 (amd64)
 3. The Docker daemon created a new container from that image which runs the
 executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
 to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://cloud.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/

eSumit@Sumits-MacBook-Pro:~/Documents/Public Github/GitPush/CloudManager$

Difference between Docker-Machine and Dockers

Docker-Machine to create a virtual machine with the help of Virtual-Box (Driver) stuff if creating machine on local or other driver can be used e.g. creating on aws, So on local with Docker-Machine we can create, start and stop virtual host e.g. it runs inside virtual box.

Docker start, run individual docker components

Important :

docker run to run a new container

docker ps to see running and stopped containers

docker images to see info about images

docker run Run a command in a new container

docker start Start one or more stopped containers

eSumit@Sumits-MacBook-Pro:~$ docker-machine --help
Usage: docker-machine [OPTIONS] COMMAND [arg...]

Create and manage machines running Docker.

Version: 0.13.0, build 9ba6da9

Author:
 Docker Machine Contributors - <https://github.com/docker/machine>

Options:
 --debug, -D Enable debug mode
 --storage-path, -s "/Users/eSumit/.docker/machine" Configures storage path [$MACHINE_STORAGE_PATH]
 --tls-ca-cert CA to verify remotes against [$MACHINE_TLS_CA_CERT]
 --tls-ca-key Private key to generate certificates [$MACHINE_TLS_CA_KEY]
 --tls-client-cert Client cert to use for TLS [$MACHINE_TLS_CLIENT_CERT]
 --tls-client-key Private key used in client TLS auth [$MACHINE_TLS_CLIENT_KEY]
 --github-api-token Token to use for requests to the Github API [$MACHINE_GITHUB_API_TOKEN]
 --native-ssh Use the native (Go-based) SSH implementation. [$MACHINE_NATIVE_SSH]
 --bugsnag-api-token BugSnag API token for crash reporting [$MACHINE_BUGSNAG_API_TOKEN]
 --help, -h show help
 --version, -v print the version
 
Commands:
 active Print which machine is active
 config Print the connection config for machine
 create Create a machine
 env Display the commands to set up the environment for the Docker client
 inspect Inspect information about a machine
 ip Get the IP address of a machine
 kill Kill a machine
 ls List machines
 provision Re-provision existing machines
 regenerate-certs Regenerate TLS Certificates for a machine
 restart Restart a machine
 rm Remove a machine
 ssh Log into or run a command on a machine with SSH.
 scp Copy files between machines
 mount Mount or unmount a directory from a machine with SSHFS.
 start Start a machine
 status Get the status of a machine
 stop Stop a machine
 upgrade Upgrade a machine to the latest version of Docker
 url Get the URL of a machine
 version Show the Docker Machine version or a machine docker version
 help Shows a list of commands or help for one command
 
Run 'docker-machine COMMAND --help' for more information on a command.
eSumit@Sumits-MacBook-Pro:~$

Docker –Help :

eSumit@Sumits-MacBook-Pro:~$ docker --help

Usage: docker COMMAND

A self-sufficient runtime for containers

Options:
 --config string Location of client config files (default
 "/Users/eSumit/.docker")
 -D, --debug Enable debug mode
 --help Print usage
 -H, --host list Daemon socket(s) to connect to
 -l, --log-level string Set the logging level
 ("debug"|"info"|"warn"|"error"|"fatal")
 (default "info")
 --tls Use TLS; implied by --tlsverify
 --tlscacert string Trust certs signed only by this CA (default
 "/Users/eSumit/.docker/ca.pem")
 --tlscert string Path to TLS certificate file (default
 "/Users/eSumit/.docker/cert.pem")
 --tlskey string Path to TLS key file (default
 "/Users/eSumit/.docker/key.pem")
 --tlsverify Use TLS and verify the remote
 -v, --version Print version information and quit

Management Commands:
 checkpoint Manage checkpoints
 config Manage Docker configs
 container Manage containers
 image Manage images
 network Manage networks
 node Manage Swarm nodes
 plugin Manage plugins
 secret Manage Docker secrets
 service Manage services
 stack Manage Docker stacks
 swarm Manage Swarm
 system Manage Docker
 volume Manage volumes

Commands:
 attach Attach local standard input, output, and error streams to a running container
 build Build an image from a Dockerfile
 commit Create a new image from a container's changes
 cp Copy files/folders between a container and the local filesystem
 create Create a new container
 deploy Deploy a new stack or update an existing stack
 diff Inspect changes to files or directories on a container's filesystem
 events Get real time events from the server
 exec Run a command in a running container
 export Export a container's filesystem as a tar archive
 history Show the history of an image
 images List images
 import Import the contents from a tarball to create a filesystem image
 info Display system-wide information
 inspect Return low-level information on Docker objects
 kill Kill one or more running containers
 load Load an image from a tar archive or STDIN
 login Log in to a Docker registry
 logout Log out from a Docker registry
 logs Fetch the logs of a container
 pause Pause all processes within one or more containers
 port List port mappings or a specific mapping for the container
 ps List containers
 pull Pull an image or a repository from a registry
 push Push an image or a repository to a registry
 rename Rename a container
 restart Restart one or more containers
 rm Remove one or more containers
 rmi Remove one or more images
 run Run a command in a new container
 save Save one or more images to a tar archive (streamed to STDOUT by default)
 search Search the Docker Hub for images
 start Start one or more stopped containers
 stats Display a live stream of container(s) resource usage statistics
 stop Stop one or more running containers
 tag Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
 top Display the running processes of a container
 unpause Unpause all processes within one or more containers
 update Update configuration of one or more containers
 version Show the Docker version information
 wait Block until one or more containers stop, then print their exit codes

A Cluster = A Swarm

Containerising software applications

CI/CD WorkFlows

A Docker Image contains – FileSystem ( with application code and dependencies), Metadata ( influences how a container runs), A command (Gets executed as a process on invocation)

Two Ways to create docker images :

  1. Registry (Base Image), Local System (Docker File), Local system (App Files) – Build all together – and produce Image-A — Dockerfile Instructions
  2. Local Repo (Image-A_ Run inside the Container (FileSystem) If anything changes in the file system creates an Image-B (Local Repo) — Committing a Container

#Delete every Docker containers

image – a container is the runtime instance of a image , Its like container get created from an image if used docker run image commands

# Must be run first because images are attached to containers

docker rm -f $(docker ps -a -q)

#Delete every Docker image

docker rmi -f $(docker images -q)

References :

https://nickjanetakis.com/blog/get-to-know-dockers-ecosystem

Docker-Machine

Exercise : Create Default and Env

esumits-MacBook-Pro:napp esumit$ docker-machine create default

Creating CA: /Users/esumit/.docker/machine/certs/ca.pem

Creating client certificate: /Users/esumit/.docker/machine/certs/cert.pem

Running pre-create checks…

(default) Image cache directory does not exist, creating it at /Users/esumit/.docker/machine/cache…

(default) No default Boot2Docker ISO found locally, downloading the latest release…

(default) Latest release for github.com/boot2docker/boot2docker is v18.04.0-ce

(default) Downloading /Users/esumit/.docker/machine/cache/boot2docker.iso from https://github.com/boot2docker/boot2docker/releases/download/v18.04.0-ce/boot2docker.iso&#8230;

(default) 0%….10%….20%….30%….40%….50%….60%….70%….80%….90%….100%

Creating machine…

(default) Copying /Users/esumit/.docker/machine/cache/boot2docker.iso to /Users/esumit/.docker/machine/machines/default/boot2docker.iso…

(default) Creating VirtualBox VM…

(default) Creating SSH key…

(default) Starting the VM…

(default) Check network to re-create if needed…

(default) Found a new host-only adapter: “vboxnet1”

(default) Waiting for an IP…

Waiting for machine to be running, this may take a few minutes…

Detecting operating system of created instance…

Waiting for SSH to be available…

Detecting the provisioner…

Provisioning with boot2docker…

Copying certs to the local machine directory…

Copying certs to the remote machine…

Setting Docker configuration on the remote daemon…

Checking connection to Docker…

Docker is up and running!

To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env default

esumits-MacBook-Pro:napp esumit$ docker-machine env default

export DOCKER_TLS_VERIFY=”1″

export DOCKER_HOST=”tcp://192.168.99.100:2376″

export DOCKER_CERT_PATH=”/Users/esumit/.docker/machine/machines/default”

export DOCKER_MACHINE_NAME=”default”

# Run this command to configure your shell:

# eval $(docker-machine env default)

esumits-MacBook-Pro:napp esumit$ eval $(docker-machine env default)

esumits-MacBook-Pro:napp esumit$ docker ls

docker: ‘ls’ is not a docker command.

See ‘docker –help’

esumits-MacBook-Pro:napp esumit$ docker ls

docker: ‘ls’ is not a docker command.

See ‘docker –help’

esumits-MacBook-Pro:napp esumit$ docker ps

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

esumits-MacBook-Pro:napp esumit$ docker-machine ls

NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER        ERRORS

default   *        virtualbox   Running   tcp://192.168.99.100:2376           v18.04.0-ce   

esumits-MacBook-Pro:napp esumit$ docker images

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

esumits-MacBook-Pro:napp esumit$ docker pull nginx

Using default tag: latest

latest: Pulling from library/nginx

2a72cbf407d6: Pull complete

04b2d3302d48: Pull complete

e7f619103861: Pull complete

Digest: sha256:18156dcd747677b03968621b2729d46021ce83a5bc15118e5bcced925fb4ebb9

Status: Downloaded newer image for nginx:latest

esumits-MacBook-Pro:napp esumit$ docker pull nginx:1.11

1.11: Pulling from library/nginx

6d827a3ef358: Pull complete

f8f2e0556751: Pull complete

5c9972dca3fd: Pull complete

451b9524cb06: Pull complete

Digest: sha256:e6693c20186f837fc393390135d8a598a96a833917917789d63766cab6c59582

Status: Downloaded newer image for nginx:1.11

esumits-MacBook-Pro:napp esumit$ docker images

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

nginx               latest              b175e7467d66        11 days ago         109MB

nginx               1.11                5766334bdaa0        12 months ago       183MB

 

Docker Build

Exercise :  Build a Docker Image and Push it to Docker-Hub

Docker File : Referred here : https://github.com/xiaopeng163/docker-k8s-lab

FROM python:2.7
MAINTAINER Peng Xiao "xiaoquwl@gmail.com"
COPY . /app
WORKDIR /app
RUN pip install -r requirements.txt
EXPOSE 5000
CMD [ "python", "app.py" ]

esumits-MacBook-Pro:flask-hello-world esumit$ ls

Dockerfile app.py requirements.txt

esumits-MacBook-Pro:flask-hello-world esumit$ docker build -t esumit/flask-hello-world:1.0 .

Sending build context to Docker daemon  4.096kB

Step 1/7 : FROM python:2.7

2.7: Pulling from library/python

f2b6b4884fc8: Pull complete

4fb899b4df21: Pull complete

74eaa8be7221: Pull complete

2d6e98fe4040: Pull complete

414666f7554d: Pull complete

bb0bcc8d7f6a: Pull complete

ace2d3087f57: Pull complete

da74659b9184: Pull complete

Digest: sha256:f34253be2e6444e8deb93e6972b36265c0c2f7153a6bd2e994ed27aba7af84a3

Status: Downloaded newer image for python:2.7

—> 446084dab7ab

Step 2/7 : MAINTAINER Peng Xiao “xiaoquwl@gmail.com”

—> Running in b39874a591cd

Removing intermediate container b39874a591cd

—> e17fd6f14bc9

Step 3/7 : COPY . /app

—> e808b6791da8

Step 4/7 : WORKDIR /app

Removing intermediate container 65233833b377

—> b8257ab0187d

Step 5/7 : RUN pip install -r requirements.txt

—> Running in 987ad39840fe

Collecting Flask (from -r requirements.txt (line 1))

  Downloading https://files.pythonhosted.org/packages/77/32/e3597cb19ffffe724ad4bf0beca4153419918e7fa4ba6a34b04ee4da3371/Flask-0.12.2-py2.py3-none-any.whl (83kB)

Collecting itsdangerous>=0.21 (from Flask->-r requirements.txt (line 1))

  Downloading https://files.pythonhosted.org/packages/dc/b4/a60bcdba945c00f6d608d8975131ab3f25b22f2bcfe1dab221165194b2d4/itsdangerous-0.24.tar.gz (46kB)

Collecting Jinja2>=2.4 (from Flask->-r requirements.txt (line 1))

  Downloading https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl (126kB)

Collecting Werkzeug>=0.7 (from Flask->-r requirements.txt (line 1))

  Downloading https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl (322kB)

Collecting click>=2.0 (from Flask->-r requirements.txt (line 1))

  Downloading https://files.pythonhosted.org/packages/34/c1/8806f99713ddb993c5366c362b2f908f18269f8d792aff1abfd700775a77/click-6.7-py2.py3-none-any.whl (71kB)

Collecting MarkupSafe>=0.23 (from Jinja2>=2.4->Flask->-r requirements.txt (line 1))

  Downloading https://files.pythonhosted.org/packages/4d/de/32d741db316d8fdb7680822dd37001ef7a448255de9699ab4bfcbdf4172b/MarkupSafe-1.0.tar.gz

Building wheels for collected packages: itsdangerous, MarkupSafe

  Running setup.py bdist_wheel for itsdangerous: started

  Running setup.py bdist_wheel for itsdangerous: finished with status ‘done’

  Stored in directory: /root/.cache/pip/wheels/2c/4a/61/5599631c1554768c6290b08c02c72d7317910374ca602ff1e5

  Running setup.py bdist_wheel for MarkupSafe: started

  Running setup.py bdist_wheel for MarkupSafe: finished with status ‘done’

  Stored in directory: /root/.cache/pip/wheels/33/56/20/ebe49a5c612fffe1c5a632146b16596f9e64676768661e4e46

Successfully built itsdangerous MarkupSafe

Installing collected packages: itsdangerous, MarkupSafe, Jinja2, Werkzeug, click, Flask

Successfully installed Flask-0.12.2 Jinja2-2.10 MarkupSafe-1.0 Werkzeug-0.14.1 click-6.7 itsdangerous-0.24

Removing intermediate container 987ad39840fe

—> 0dd2fd8ee3c7

Step 6/7 : EXPOSE 5000

—> Running in ef19979316fa

Removing intermediate container ef19979316fa

—> b5fd11bd3624

Step 7/7 : CMD [ “python”, “app.py” ]

—> Running in 465f96161d4d

Removing intermediate container 465f96161d4d

—> a5606dfe34d4

Successfully built a5606dfe34d4

Successfully tagged esumit/flask-hello-world:1.0

Push to Docker Hub

esumits-MacBook-Pro:flask-hello-world esumit$ docker login

Login with your Docker ID to push and pull images from Docker Hub. If you don’t have a Docker ID, head over to https://hub.docker.com to create one.

Username: xxxxxx

Password:

Login Succeeded

esumits-MacBook-Pro:flask-hello-world esumit$ docker push esumit/flask-hello-world

The push refers to repository [docker.io/esumit/flask-hello-world]

0c8a536dda2c: Pushed

ada46425680f: Pushed

6bdccf632521: Mounted from library/python

62d47657687c: Mounted from library/python

4e32c2de91a6: Mounted from library/python

6e1b48dc2ccc: Mounted from library/python

ff57bdb79ac8: Mounted from library/python

6e5e20cbf4a7: Mounted from library/python

86985c679800: Mounted from library/python

8fad67424c4e: Mounted from library/python

1.0: digest: sha256:d97fb4f98f2a46cd4ae067f6c6cbdb31a6cb2a7eb2499a701c5364e368f4b56b size: 2429

Docker Push to eSumit Repo
Docker Push to eSumit Repo

 Command to Delete all exited containers

docker rm $(docker ps -aq –filter “status=exited”)

Docker Networking 

Each Docker have his own network interface , routing table, firewall rules

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s