DISCLAIMER:
- This will be an overly simpliflied explanation
- Refer to documentation and references for a full understanding
- This is basically a guide i would write for the struggling poly kid i was 4 mth ago.
Containers allows for OS-level virtualisation.
Basically its a Virtual Machine that virtualises processes and not the whole computer, which makes it much more efficient.
Containers were initially made for Linux, therefore most containers will be linux based.
Main benefits:
- Run an application with libraries, dependacies all pre-loaded.
- Ensure application consistency. IE, if it runs on my system A, it should run on system B. (with some caveats)
Virtual Machines:
- Emulate entire system
- Very resource intensive
Containers:
- "Emulate" processes (Are native processes that emulate other OS)
- Very little resource overhead
A method to ensure whatever runs on system A, will run on system B.
Achieved by having system "run" an OS as a native process. Be able to interact with the OS as if its a different computer.
You take an OS, tell docker how to build + modify it, untill you get a usuable program.
The main benefit of containers is isolation,
allowing us to package an application with all of its dependencies into a standardized unit.
In essence, we can put our setup into the container (such as installing dependacies), and it will work 'anywhere'.
The most popular containerisation software is Docker.
However for High Performance Computing(HPC) in scientific context, Singularity is a popular option.
- dockerfile -> List of instructions on how to build an image
- Image -> Image built using dockerfile
- Container -> Image that is running
- Docker Daemon -> Docker "program/runtime". Must be installed + running to dun docker.
- DockerHub -> Online Repository for docker images. Think github but for docker.
- .dockerignore -> Docker version of .gitignore, choose what dockerfile should ignore.
- Define your
dockerfile, define how you want your image to be built. - Build your image, usually takes awhile.
- Obtain completed docker image.
- Run the docker image locally.
- Push docker image to DockerHub.
- Pull docker image on machine 2.
- Run the docker image on machine 2.
Windows Subsystem for Linux, basically aims to have windows be able to run Linux "natively".
As of WSL2, they had decided on using a lightweight VM.
I would reccomend you use this to run containers, good choice if you do not have a spare laptop with linux to use.
Resources:
People have made many guides, heres some of them, find other favourites
- Learning docker by making web app
- Docker page official guide
- Pull and run the basic docker container hello-world
When in doubt, ask the reference:
How Image Layers work:
- After each dockerfile command, a new read-only layer is created
- Final top layer is a modifiable layer
- Idea: different images can share lower level layers.
Best Practices:
- Reduce number of dockerfile commands, put more commands into same dockerfile command.
- If unsure, Test commands using
docker exec / run, write down all commands used. - Then, implement commands in dockerfile.
List of all common docker commands: Reference
docker run:
- Run a container
$ docker run [OPTIONS] IMAGE[:TAG|@DIGEST] [COMMAND] [ARG...]
Reference
docker stop:
- Stop a running container
docker stop [OPTIONS] CONTAINER
Reference
docker exec:
- runs a new command in a running container
docker exec [OPTIONS] CONTAINER COMMAND [ARG...]
Reference
docker build:
- Build an image from dockerfile
docker build [OPTIONS] PATH | URL | -
Reference
docker pull:
- Pull an image or a repository from a registry
docker pull [OPTIONS] NAME[:TAG|@DIGEST]
Reference
docker push:
- Push an image or a repository to a registry
docker push [OPTIONS] NAME[:TAG]
Reference
Container and Image management
docker ps:
- List containers
docker ps [OPTIONS]
Reference
docker images:
- List images
docker images [OPTIONS] [REPOSITORY[:TAG]]
Reference
docker image:
- Manage images
docker image COMMAND
Reference
docker image rm:
- Remove one or more images
docker image rm [OPTIONS] IMAGE [IMAGE...]
Reference
Save and Load image from .tar file
docker save:
- Save one or more images to a tar archive (streamed to STDOUT by default)
docker save [OPTIONS] IMAGE [IMAGE...]
Reference
docker load:
- Load an image from a tar archive or STDIN
docker load [OPTIONS]
Reference
Advanced
docker buildx:
- Build with BuildKit, X-perimental builder. Typically used for cross compilation
- Complex, but good to use for x-compilation. Double check containers are proper architecture.
docker buildx COMMAND
Reference
docker commit:
- Create a new image from a container’s changes, aka save a running container as a new image.
- Not Reccomended, its reccomended that changes to containers be done in a dockerfile for consistency.
docker commit [OPTIONS] CONTAINER [REPOSITORY[:TAG]]
Reference
Instruction Set Architecture:
- Ensures CPU with the same architecture will be able to run the same software. Wiki
How the magic works:
- Programs are compiled to only run on 1 Specific OS and 1 Specific Architecture.
- Docker daemon abstracts away the Operating System -> OS-level virtualisation.
- Therefore, as long architecture is the same, docker image can run.
As long architecture is the same, Machine will be able to run the docker image.
If not, have to use a VM or emulation technique.
Proper References for Deeper Reading:
Use dockerhub to upload containers for future use and easy access.
Official Guide: Docker Hub guide
Docker has an official naming scheme, you use this push and pull docker images. \
<docker_ID>/<repo_name>:<desired_tag>
docker pull <docker_ID>/<repo_name>:<desired_tag>docker pull hello-world
Pull an official repo, only referencing repo name:
docker pull python
Pull a normal repo, specify docker_ID and repo name:
docker pull continuumio/anaconda3[Auto pulls :latest]
Pull a specific tag, specify tag:
docker pull continuumio/anaconda3:latest
Pull a specific image, specify image hash:
docker pull continuumio/anaconda3:2020.11@sha256:0b2047cdc438807b87d53272c3d5b10c8238fe65a2fedf9bd72de0b7ba360cb1[Always pull specific image]
Build your image:
docker build -t <your_docker_ID>/<your_repo_name>:<desired_tag> .docker build -t 16fb/deepheadpose:ZX .
Ensure your docker image follows DockerHub format, if not retag your image.
Push to DockerHub:
docker push <your_docker_ID>/<your_repo_name>:<desired_tag>docker push 16fb/deepheadpose:ZX
Follow this StackOverflow Guide.
Very usefull and helpful when you C: drive is out of space.
Experimental builder tool, can be used to cross compile and put multiple images into same tag on dockerhub.
Has other uses as well.
Some Guides to get started:
Windows 10:
As of writing(13/1/2021), stable version of WSL2 with docker does not support GPU Usage(GPU Passthrough).
There is a preview / insider build for windows that does allow GPU usage.
Hopefully by the time you read this, the preview build will have been pushed to live.
Ubuntu/Mac:
Possible, refer to dockerOnUbuntu.md and dockerWithGPUOnUbuntu.md for more details.
DockerHub -> Done
Building + Running Docker containers -> Simple one done
docker and WSL2
typical docker commands: -> done
image management commands: -> done
how docker takes up space on your computer in windows -> referenced guide
docker layers -> done
.dockerignore -> mentioned somewhere
docker buildx -> done
diff arch -> Mentioned
GPU support in containers. -> done