Docker compose has nice support for GPUs, K8s has moved their cluster-wide GPU scheduler from experimental to stable status. Docker swarm has yet to support the device option used in docker compose so the mechanisms for supporting GPUs on swarm are a bit more open-ended.
- NVIDIA container runtime for docker. The runtime is no longer required to run GPU support with the docker cli or compose; however, it appears necessary so that one can set
Default Runtime: nvidiafor swarm mode. - docker compose GPU support
- Good GitHub Gist Reference for an overview on Swarm with GPUs. It is a bit dated, but has good links and conversation.
- Miscellaneous Options for docker configuration. Go down to "Node Generic Resources" for an explanation of how this is intended to support NVIDIA GPUs. The main idea is one has to change the
/etc/docker/daemon.jsonfile to advertise thenode-generic-resources(NVIDIA GPUs) on each node. GPUs have to be added by hand the thedaemon.jsonfile, swarm does not detect and advertise them automatically. - How to create a service with generic resources. This shows how to create stacks/services requesting the generic resources advertised in the
/etc/docker/daemon.jsonfile. - Quick blog overview confirming these basic approaches.
- Really good overview on Generic Resources in swarm.
Both solutions need to follow these steps first:
- Install
nvidia-container-runtime. Follow the steps here. Takes <5 minutes. - Update
/etc/docker/daemon.jsonto usenvidiaas the default runtime.
{
"default-runtime": "nvidia",
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
}
}- Restart the docker daemon on each node
sudo service docker restart. Confirm the default runtime isnvidiawithdocker info.
You're done. When you deploy a service to a node, it will by default see all the GPUs on that node. Generally this means you are deploying global services (one per node) or assigning services to specific nodes so that there aren't accidental collisions between services accessing the same GPU resources simultaneously.
If you want to expose only certain GPUs to a given service (e.g., multiple services on one node with each having access only to its own GPU(s)) use the NVIDIA_VISIBLE_DEVICES environment variable for each service. To do this dynamically so that each service gets access to its own GPU using docker service templates looks like this:
services:
my-service-node-001:
image: blah blah
environment:
- NVIDIA_VISIBLE_DEVICES={{.Task.Slot}}
deploy:
replicas: 15
placement:
constraints:
- node.hostname==some-node-001
Because {{.Task.Slot}} starts counting at 1, you may want to include a global service in the template to make use of GPU 0.
Advertise NVIDA GPUs using Node Generic Resources. This is the most general purpose approach and will enable services to simply declare the required GPU resources and swarm will schedule them accordingly.
The /etc/docker/daemon.json file on each node needs to be updated to advertise its GPU resources. You can find the UUID for each GPU by running nvidia-smi -a | grep UUID. You only need to include GPU plus the first 8 digits of the UUID, it seems, i.e., GPU-ba74caf3 for the UUID. The following needs to be added to the daemon.json file already declaring nvidia as the default runtime.
{
"node-generic-resources": [
"NVIDIA-GPU=GPU-ba74caf3",
"NVIDIA-GPU=GPU-dl23cdb4"
]
}Enable GPU resource advertising by uncommenting the swarm-resource = "DOCKER_RESOURCE_GPU" line (line 2) in /etc/nvidia-container-runtime/config.toml.
The docker daemon must be restarted after updating these files by running sudo service docker restart on each node. Services can now request GPUs using the generic-resource flag.
docker service create \
--name cuda \
--generic-resource "NVIDIA-GPU=2" \
--generic-resource "SSD=1" \
nvidia/cudaThe names for node-generic-resources in /etc/docker/daemon.json could be anything you want. So if you want to declare NVIDIA-H100 and NVIDIA-4090 you could and then request specific GPU types with --generic-resource "NVIDIA-H100".
To request GPU resources in a docker-compose.yaml file for the stack use the following under the deploy key.
services:
my-gpu-service:
...
deploy:
resources:
reservations:
generic_resources:
- discrete_resource_spec:
kind: "NVIDIA-GPU"
value: 2
Hello sir:
My system had 2 GPUs. I add node-generic-resources" to /etc/docker/daemon.json
{
"runtimes": {
"nvidia": {
"path": "/usr/bin/nvidia-container-runtime",
"runtimeArgs": []
}
},
"default-runtime": "nvidia",
"node-generic-resources": [
"NVIDIA-GPU=GPU-7af7a99b",
"NVIDIA-GPU=GPU-e934c858"
]
}
also uncomment swarm-resource = "DOCKER_RESOURCE_GPU"
When I run docker service create --replicas 1 --name swarm-test --generic-resource "NVIDIA-GPU=1" swarmimage nvidia-smi
=>The container always use all GPU devices.
But if I run docker run -it --gpus 1 swarmimage nvidia-smi
=>The container only use GPU #0 device.
Would you please help this case?