Notes, Daily commands, high level concepts related to the daily use of dockers for software development purpose.
Concepts of Dockers
How dockers system works ?, Its all refer to a file system which differentiate it from virtual machines. In my initial finding it uses the base concept from Cgroups and Linux Containers.
What is talks actually On High Level:
It talks about the layers of files, like It first picks up the base file system which it says like base image of containers(e.g. specific linux version) and then on top it java installation (JDK stuff), and then on top of it specific webservice deployment. So now this web service running only in an isolated environment and not conflicting with any other. its like a stack of files.
To understand the insight of dockers, need to understand the concepts of cgroups and Linux Containers (LXC):
cgroups (abbreviated from control groups) is a Linux kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes.
Refer : https://en.wikipedia.org/wiki/Cgroups
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01
https://en.wikipedia.org/wiki/LXC
LXC (Linux Containers) is an operating-system-level virtualization method for running multiple isolated Linux systems (containers) on a control host using a single Linux kernel.
Refer : https://www.docker.com/
Refer : Linux Containers : https://linuxcontainers.org/
docker rmi XXX (delete images)
docker -version
eSumit@Sumits-MacBook-Pro:~/Documents/Public Github/GitPush/CloudManager$ docker version Client: Version: 17.09.1-ce API version: 1.32 Go version: go1.8.3 Git commit: 19e2cf6 Built: Thu Dec 7 22:22:25 2017 OS/Arch: darwin/amd64 Server: Version: 17.09.1-ce API version: 1.32 (minimum version 1.12) Go version: go1.8.3 Git commit: 19e2cf6 Built: Thu Dec 7 22:28:28 2017 OS/Arch: linux/amd64 Experimental: true eSumit@Sumits-MacBook-Pro:~/Documents/Public Github/GitPush/CloudManager$
docker info
eSumit@Sumits-MacBook-Pro:~/Documents/Public Github/GitPush/CloudManager$ docker info Containers: 5 Running: 0 Paused: 0 Stopped: 5 Images: 10 Server Version: 17.09.1-ce Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 63 Dirperm1 Supported: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0 runc version: 3f2f8b84a77f73d38244dd690525642a72156c64 init version: 949e6fa Security Options: seccomp Profile: default Kernel Version: 4.9.49-moby Operating System: Alpine Linux v3.5 OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 1.952GiB Name: moby ID: F6DU:FNP2:QGDU:SCJY:INGB:II7A:HN6N:ANLE:2VGE:AJN3:74UF:66M4 Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): true File Descriptors: 18 Goroutines: 29 System Time: 2018-02-05T08:28:35.381917475Z EventsListeners: 1 No Proxy: *.local, 169.254/16 Registry: https://index.docker.io/v1/ Experimental: true Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false eSumit@Sumits-MacBook-Pro:~/Documents/Public Github/GitPush/CloudManager$
5 Important Commands on Dockers ( from Medium)
- docker logs <container_id>
Hopefully you’ve already tried this, but if not, start here. This’ll give you the full STDOUT and STDERR from the command that was run initially in your container. - docker stats <container_id>
If you just need to keep an eye on the metrics of your container to work out what’s gone wrong, docker stats can help: it’ll give you a live stream of resource usage, so you can see just how much memory you’ve leaked so far. - docker cp <container_id>:/path/to/useful/file /local-path
Often just getting hold of more log files is enough to sort you out. If you already know what you want, docker cp has your back: copy any file from any container back out onto your local machine, so you can examine it in depth (especially useful analysing heap dumps). - docker exec -it <container_id> /bin/bash
Next up, if you can run the container (if it’s crashed, you can restart it with docker start <container_id>), shell in directly and start digging around for further details by hand. - docker commit <container_id> my-broken-container &&
docker run -it my-broken-container /bin/bash
Can’t start your container at all? If you’ve got a initial command or entrypoint that immediately crashes, Docker will immediately shut it back down for you. This can make your container unstartable, so you can’t shell in any more, which really gets in the way.
Fortunately, there’s a workaround: save the current state of the shut-down container as a new image, and start that with a different command to avoid your existing failures.
Have a failing entrypoint instead? There’s an entrypoint override command-line flag too.
How to check if docker installed correctly
Read these four steps :
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the “hello-world” image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
eSumit@Sumits-MacBook-Pro:~/Documents/Public Github/GitPush/CloudManager$ docker run hello-world Unable to find image 'hello-world:latest' locally latest: Pulling from library/hello-world ca4f61b1923c: Pull complete Digest: sha256:66ef312bbac49c39a89aa9bcc3cb4f3c9e7de3788c944158df3ee0176d32b751 Status: Downloaded newer image for hello-world:latest Hello from Docker! This message shows that your installation appears to be working correctly. To generate this message, Docker took the following steps: 1. The Docker client contacted the Docker daemon. 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. (amd64) 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading. 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal. To try something more ambitious, you can run an Ubuntu container with: $ docker run -it ubuntu bash Share images, automate workflows, and more with a free Docker ID: https://cloud.docker.com/ For more examples and ideas, visit: https://docs.docker.com/engine/userguide/ eSumit@Sumits-MacBook-Pro:~/Documents/Public Github/GitPush/CloudManager$
Difference between Docker-Machine and Dockers
Docker-Machine to create a virtual machine with the help of Virtual-Box (Driver) stuff if creating machine on local or other driver can be used e.g. creating on aws, So on local with Docker-Machine we can create, start and stop virtual host e.g. it runs inside virtual box.
Docker start, run individual docker components
Important :
docker run to run a new container
docker ps to see running and stopped containers
docker images to see info about images
docker run Run a command in a new container
docker start Start one or more stopped containers
eSumit@Sumits-MacBook-Pro:~$ docker-machine --help
Usage: docker-machine [OPTIONS] COMMAND [arg...]
Create and manage machines running Docker.
Version: 0.13.0, build 9ba6da9
Author:
Docker Machine Contributors - <https://github.com/docker/machine>
Options:
--debug, -D Enable debug mode
--storage-path, -s "/Users/eSumit/.docker/machine" Configures storage path [$MACHINE_STORAGE_PATH]
--tls-ca-cert CA to verify remotes against [$MACHINE_TLS_CA_CERT]
--tls-ca-key Private key to generate certificates [$MACHINE_TLS_CA_KEY]
--tls-client-cert Client cert to use for TLS [$MACHINE_TLS_CLIENT_CERT]
--tls-client-key Private key used in client TLS auth [$MACHINE_TLS_CLIENT_KEY]
--github-api-token Token to use for requests to the Github API [$MACHINE_GITHUB_API_TOKEN]
--native-ssh Use the native (Go-based) SSH implementation. [$MACHINE_NATIVE_SSH]
--bugsnag-api-token BugSnag API token for crash reporting [$MACHINE_BUGSNAG_API_TOKEN]
--help, -h show help
--version, -v print the version
Commands:
active Print which machine is active
config Print the connection config for machine
create Create a machine
env Display the commands to set up the environment for the Docker client
inspect Inspect information about a machine
ip Get the IP address of a machine
kill Kill a machine
ls List machines
provision Re-provision existing machines
regenerate-certs Regenerate TLS Certificates for a machine
restart Restart a machine
rm Remove a machine
ssh Log into or run a command on a machine with SSH.
scp Copy files between machines
mount Mount or unmount a directory from a machine with SSHFS.
start Start a machine
status Get the status of a machine
stop Stop a machine
upgrade Upgrade a machine to the latest version of Docker
url Get the URL of a machine
version Show the Docker Machine version or a machine docker version
help Shows a list of commands or help for one command
Run 'docker-machine COMMAND --help' for more information on a command.
eSumit@Sumits-MacBook-Pro:~$
Docker –Help :
eSumit@Sumits-MacBook-Pro:~$ docker --help
Usage: docker COMMAND
A self-sufficient runtime for containers
Options:
--config string Location of client config files (default
"/Users/eSumit/.docker")
-D, --debug Enable debug mode
--help Print usage
-H, --host list Daemon socket(s) to connect to
-l, --log-level string Set the logging level
("debug"|"info"|"warn"|"error"|"fatal")
(default "info")
--tls Use TLS; implied by --tlsverify
--tlscacert string Trust certs signed only by this CA (default
"/Users/eSumit/.docker/ca.pem")
--tlscert string Path to TLS certificate file (default
"/Users/eSumit/.docker/cert.pem")
--tlskey string Path to TLS key file (default
"/Users/eSumit/.docker/key.pem")
--tlsverify Use TLS and verify the remote
-v, --version Print version information and quit
Management Commands:
checkpoint Manage checkpoints
config Manage Docker configs
container Manage containers
image Manage images
network Manage networks
node Manage Swarm nodes
plugin Manage plugins
secret Manage Docker secrets
service Manage services
stack Manage Docker stacks
swarm Manage Swarm
system Manage Docker
volume Manage volumes
Commands:
attach Attach local standard input, output, and error streams to a running container
build Build an image from a Dockerfile
commit Create a new image from a container's changes
cp Copy files/folders between a container and the local filesystem
create Create a new container
deploy Deploy a new stack or update an existing stack
diff Inspect changes to files or directories on a container's filesystem
events Get real time events from the server
exec Run a command in a running container
export Export a container's filesystem as a tar archive
history Show the history of an image
images List images
import Import the contents from a tarball to create a filesystem image
info Display system-wide information
inspect Return low-level information on Docker objects
kill Kill one or more running containers
load Load an image from a tar archive or STDIN
login Log in to a Docker registry
logout Log out from a Docker registry
logs Fetch the logs of a container
pause Pause all processes within one or more containers
port List port mappings or a specific mapping for the container
ps List containers
pull Pull an image or a repository from a registry
push Push an image or a repository to a registry
rename Rename a container
restart Restart one or more containers
rm Remove one or more containers
rmi Remove one or more images
run Run a command in a new container
save Save one or more images to a tar archive (streamed to STDOUT by default)
search Search the Docker Hub for images
start Start one or more stopped containers
stats Display a live stream of container(s) resource usage statistics
stop Stop one or more running containers
tag Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
top Display the running processes of a container
unpause Unpause all processes within one or more containers
update Update configuration of one or more containers
version Show the Docker version information
wait Block until one or more containers stop, then print their exit codes
A Cluster = A Swarm
Containerising software applications
CI/CD WorkFlows
A Docker Image contains – FileSystem ( with application code and dependencies), Metadata ( influences how a container runs), A command (Gets executed as a process on invocation)
Two Ways to create docker images :
- Registry (Base Image), Local System (Docker File), Local system (App Files) – Build all together – and produce Image-A — Dockerfile Instructions
- Local Repo (Image-A_ Run inside the Container (FileSystem) If anything changes in the file system creates an Image-B (Local Repo) — Committing a Container
#Delete every Docker containers
image – a container is the runtime instance of a image , Its like container get created from an image if used docker run image commands
# Must be run first because images are attached to containers
docker rm -f $(docker ps -a -q)
#Delete every Docker image
docker rmi -f $(docker images -q)
References :
https://nickjanetakis.com/blog/get-to-know-dockers-ecosystem
Few Docker’s commands
To show only running containers use the given command:
docker ps
To show all containers use the given command:
docker ps -a
To show the latest created container (includes all states) use the given command:
docker ps -l
To show n last created containers (includes all states) use the given command:
docker ps -n=-1
To display total file sizes use the given command:
docker ps -s
The content presented above is from docker.com.
In the new version of Docker, commands are updated, and some management commands are added:
docker container ls
Is used to list all the running containers.
docker container ls -a
And then, if you want to clean them all,
docker rm $(docker ps -aq)
Is used to list all the containers created irrespective of its state.
Docker-Machine
Exercise : Create Default and Env
esumits-MacBook-Pro:napp esumit$ docker-machine create default
Creating CA: /Users/esumit/.docker/machine/certs/ca.pem
Creating client certificate: /Users/esumit/.docker/machine/certs/cert.pem
Running pre-create checks…
(default) Image cache directory does not exist, creating it at /Users/esumit/.docker/machine/cache…
(default) No default Boot2Docker ISO found locally, downloading the latest release…
(default) Latest release for github.com/boot2docker/boot2docker is v18.04.0-ce
(default) Downloading /Users/esumit/.docker/machine/cache/boot2docker.iso from https://github.com/boot2docker/boot2docker/releases/download/v18.04.0-ce/boot2docker.iso…
(default) 0%….10%….20%….30%….40%….50%….60%….70%….80%….90%….100%
Creating machine…
(default) Copying /Users/esumit/.docker/machine/cache/boot2docker.iso to /Users/esumit/.docker/machine/machines/default/boot2docker.iso…
(default) Creating VirtualBox VM…
(default) Creating SSH key…
(default) Starting the VM…
(default) Check network to re-create if needed…
(default) Found a new host-only adapter: “vboxnet1”
(default) Waiting for an IP…
Waiting for machine to be running, this may take a few minutes…
Detecting operating system of created instance…
Waiting for SSH to be available…
Detecting the provisioner…
Provisioning with boot2docker…
Copying certs to the local machine directory…
Copying certs to the remote machine…
Setting Docker configuration on the remote daemon…
Checking connection to Docker…
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env default
esumits-MacBook-Pro:napp esumit$ docker-machine env default
export DOCKER_TLS_VERIFY=”1″
export DOCKER_HOST=”tcp://192.168.99.100:2376″
export DOCKER_CERT_PATH=”/Users/esumit/.docker/machine/machines/default”
export DOCKER_MACHINE_NAME=”default”
# Run this command to configure your shell:
# eval $(docker-machine env default)
esumits-MacBook-Pro:napp esumit$ eval $(docker-machine env default)
esumits-MacBook-Pro:napp esumit$ docker ls
docker: ‘ls’ is not a docker command.
See ‘docker –help’
esumits-MacBook-Pro:napp esumit$ docker ls
docker: ‘ls’ is not a docker command.
See ‘docker –help’
esumits-MacBook-Pro:napp esumit$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
esumits-MacBook-Pro:napp esumit$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
default * virtualbox Running tcp://192.168.99.100:2376 v18.04.0-ce
esumits-MacBook-Pro:napp esumit$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
esumits-MacBook-Pro:napp esumit$ docker pull nginx
Using default tag: latest
latest: Pulling from library/nginx
2a72cbf407d6: Pull complete
04b2d3302d48: Pull complete
e7f619103861: Pull complete
Digest: sha256:18156dcd747677b03968621b2729d46021ce83a5bc15118e5bcced925fb4ebb9
Status: Downloaded newer image for nginx:latest
esumits-MacBook-Pro:napp esumit$ docker pull nginx:1.11
1.11: Pulling from library/nginx
6d827a3ef358: Pull complete
f8f2e0556751: Pull complete
5c9972dca3fd: Pull complete
451b9524cb06: Pull complete
Digest: sha256:e6693c20186f837fc393390135d8a598a96a833917917789d63766cab6c59582
Status: Downloaded newer image for nginx:1.11
esumits-MacBook-Pro:napp esumit$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx latest b175e7467d66 11 days ago 109MB
nginx 1.11 5766334bdaa0 12 months ago 183MB
Docker Build
Exercise : Build a Docker Image and Push it to Docker-Hub
Docker File : Referred here : https://github.com/xiaopeng163/docker-k8s-lab
FROM python:2.7 MAINTAINER Peng Xiao "xiaoquwl@gmail.com" COPY . /app WORKDIR /app RUN pip install -r requirements.txt EXPOSE 5000 CMD [ "python", "app.py" ]
esumits-MacBook-Pro:flask-hello-world esumit$ ls
Dockerfile app.py requirements.txt
esumits-MacBook-Pro:flask-hello-world esumit$ docker build -t esumit/flask-hello-world:1.0 .
Sending build context to Docker daemon 4.096kB
Step 1/7 : FROM python:2.7
2.7: Pulling from library/python
f2b6b4884fc8: Pull complete
4fb899b4df21: Pull complete
74eaa8be7221: Pull complete
2d6e98fe4040: Pull complete
414666f7554d: Pull complete
bb0bcc8d7f6a: Pull complete
ace2d3087f57: Pull complete
da74659b9184: Pull complete
Digest: sha256:f34253be2e6444e8deb93e6972b36265c0c2f7153a6bd2e994ed27aba7af84a3
Status: Downloaded newer image for python:2.7
—> 446084dab7ab
Step 2/7 : MAINTAINER Peng Xiao “xiaoquwl@gmail.com”
—> Running in b39874a591cd
Removing intermediate container b39874a591cd
—> e17fd6f14bc9
Step 3/7 : COPY . /app
—> e808b6791da8
Step 4/7 : WORKDIR /app
Removing intermediate container 65233833b377
—> b8257ab0187d
Step 5/7 : RUN pip install -r requirements.txt
—> Running in 987ad39840fe
Collecting Flask (from -r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/77/32/e3597cb19ffffe724ad4bf0beca4153419918e7fa4ba6a34b04ee4da3371/Flask-0.12.2-py2.py3-none-any.whl (83kB)
Collecting itsdangerous>=0.21 (from Flask->-r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/dc/b4/a60bcdba945c00f6d608d8975131ab3f25b22f2bcfe1dab221165194b2d4/itsdangerous-0.24.tar.gz (46kB)
Collecting Jinja2>=2.4 (from Flask->-r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl (126kB)
Collecting Werkzeug>=0.7 (from Flask->-r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl (322kB)
Collecting click>=2.0 (from Flask->-r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/34/c1/8806f99713ddb993c5366c362b2f908f18269f8d792aff1abfd700775a77/click-6.7-py2.py3-none-any.whl (71kB)
Collecting MarkupSafe>=0.23 (from Jinja2>=2.4->Flask->-r requirements.txt (line 1))
Building wheels for collected packages: itsdangerous, MarkupSafe
Running setup.py bdist_wheel for itsdangerous: started
Running setup.py bdist_wheel for itsdangerous: finished with status ‘done’
Stored in directory: /root/.cache/pip/wheels/2c/4a/61/5599631c1554768c6290b08c02c72d7317910374ca602ff1e5
Running setup.py bdist_wheel for MarkupSafe: started
Running setup.py bdist_wheel for MarkupSafe: finished with status ‘done’
Stored in directory: /root/.cache/pip/wheels/33/56/20/ebe49a5c612fffe1c5a632146b16596f9e64676768661e4e46
Successfully built itsdangerous MarkupSafe
Installing collected packages: itsdangerous, MarkupSafe, Jinja2, Werkzeug, click, Flask
Successfully installed Flask-0.12.2 Jinja2-2.10 MarkupSafe-1.0 Werkzeug-0.14.1 click-6.7 itsdangerous-0.24
Removing intermediate container 987ad39840fe
—> 0dd2fd8ee3c7
Step 6/7 : EXPOSE 5000
—> Running in ef19979316fa
Removing intermediate container ef19979316fa
—> b5fd11bd3624
Step 7/7 : CMD [ “python”, “app.py” ]
—> Running in 465f96161d4d
Removing intermediate container 465f96161d4d
—> a5606dfe34d4
Successfully built a5606dfe34d4
Successfully tagged esumit/flask-hello-world:1.0
Push to Docker Hub
esumits-MacBook-Pro:flask-hello-world esumit$ docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don’t have a Docker ID, head over to https://hub.docker.com to create one.
Username: xxxxxx
Password:
Login Succeeded
esumits-MacBook-Pro:flask-hello-world esumit$ docker push esumit/flask-hello-world
The push refers to repository [docker.io/esumit/flask-hello-world]
0c8a536dda2c: Pushed
ada46425680f: Pushed
6bdccf632521: Mounted from library/python
62d47657687c: Mounted from library/python
4e32c2de91a6: Mounted from library/python
6e1b48dc2ccc: Mounted from library/python
ff57bdb79ac8: Mounted from library/python
6e5e20cbf4a7: Mounted from library/python
86985c679800: Mounted from library/python
8fad67424c4e: Mounted from library/python
1.0: digest: sha256:d97fb4f98f2a46cd4ae067f6c6cbdb31a6cb2a7eb2499a701c5364e368f4b56b size: 2429
Command to Delete all exited containers
docker rm $(docker ps -aq –filter “status=exited”)
Docker Networking
Each Docker have his own network interface , routing table, firewall rules
Docker Image vs Docker Container
I couldn’t understand the concept of image and layer in spite of reading all the questions here and then eventually stumbled upon this excellent documentation from Docker (duh!).
The example there is really the key to understand the whole concept. It is a lengthy post, so I am summarising the key points that need to be really grasped to get clarity.
Image: A Docker image is built up from a series of read-only layers
Layer: Each layer represents an instruction in the image’s Dockerfile.
Example: The below Dockerfile contains four commands, each of which creates a layer.
FROM ubuntu:15.04
COPY . /app
RUN make /app
CMD python /app/app.py
Importantly, each layer is only a set of differences from the layer before it.
Container. When you create a new container, you add a new writable layer on top of the underlying layers. This layer is often called the “container layer”. All changes made to the running container, such as writing new files, modifying existing files, and deleting files, are written to this thin writable container layer.
Hence, the major difference between a container and an image is the top writable layer. All writes to the container that add new or modify existing data are stored in this writable layer. When the container is deleted, the writable layer is also deleted. The underlying image remains unchanged.
Understanding images cnd Containers from a size-on-disk perspective
To view the approximate size of a running container, you can use the docker ps -s command. You get size and virtual size as two of the outputs:
Size: the amount of data (on disk) that is used for the writable layer of each container
Virtual Size: the amount of data used for the read-only image data used by the container. Multiple containers may share some or all read-only image data. Hence these are not additive. I.e. you can’t add all the virtual sizes to calculate how much size on disk is used by the image
Another important concept is the copy-on-write strategy
If a file or directory exists in a lower layer within the image, and another layer (including the writable layer) needs read access to it, it just uses the existing file. The first time another layer needs to modify the file (when building the image or running the container), the file is copied into that layer and modified.
————————
Dockerfile > (Build) > Image > (Run) > Container.
Dockerfile: contains a set of docker instructions that provisions your operating system the way you like, and installs/configure all your software’s.
Image: compiled Dockerfile. Saves you time from rebuilding the Dockerfile every time you need to run a container. And it’s a way to hide your provision code.
Container: the virtual operating system itself, you can ssh into it and run any commands you wish, as if it’s a real environment. You can run 1000+ containers from the same Image.
