Skip to content

Instantly share code, notes, and snippets.

@vfarcic

vfarcic/03-networking.sh

Created January 21, 2017 23:27
Show Gist options
  • Save vfarcic/fd7d7e04e1133fc3c90084c4c1a919fe to your computer and use it in GitHub Desktop.
Save vfarcic/fd7d7e04e1133fc3c90084c4c1a919fe to your computer and use it in GitHub Desktop.
Download ZIP
Raw
03-networking.sh
for i in 1 2 3; do
docker-machine create -d virtualbox node-$i
done
eval $(docker-machine env node-1)
docker swarm init \
--advertise-addr $(docker-machine ip node-1)
TOKEN=$(docker swarm join-token -q worker)
for i in 2 3; do
eval $(docker-machine env node-$i)
docker swarm join \
--token $TOKEN \
--advertise-addr $(docker-machine ip node-$i) \
$(docker-machine ip node-1):2377
done
eval $(docker-machine env node-1)
docker node ls
docker service create --name go-demo-db \
mongo:3.2.10
docker service inspect --pretty go-demo-db
docker service rm go-demo-db
docker network create --driver overlay go-demo
docker service create --name go-demo-db \
--network go-demo \
mongo:3.2.10
docker service inspect --pretty go-demo-db
docker service create --name util \
--network go-demo --mode global \
alpine sleep 1000000000
docker service ps util
ID=$(docker ps -q --filter label=com.docker.swarm.service.name=util)
docker exec -it $ID apk add --update drill
docker exec -it $ID drill go-demo-db
docker network create --driver overlay proxy
docker network ls -f "driver=overlay"
docker service create --name go-demo \
-e DB=go-demo-db \
--network go-demo \
--network proxy \
vfarcic/go-demo:1.0
docker service create --name proxy \
-p 80:80 \
-p 443:443 \
-p 8080:8080 \
--network proxy \
-e MODE=swarm \
vfarcic/docker-flow-proxy
docker service ps proxy
curl "$(docker-machine ip node-1):8080/v1/docker-flow-proxy/reconfigure?serviceName=go-demo&servicePath=/demo&port=8080"
curl -i "$(docker-machine ip node-1)/demo/hello"
curl -i "$(docker-machine ip node-3)/demo/hello"
NODE=$(docker service ps proxy | tail -n +2 | awk '{print $4}')
eval $(docker-machine env $NODE)
ID=$(docker ps -q \
--filter label=com.docker.swarm.service.name=proxy)
docker exec -it \
$ID cat /cfg/haproxy.cfg
eval $(docker-machine env node-1)
docker service scale go-demo=5
ID=$(docker ps -q --filter label=com.docker.swarm.service.name=util)
docker exec -it $ID apk add --update drill
docker exec -it $ID drill go-demo
docker-machine rm -f node-1 node-2 node-3
@scgm11
Copy link

scgm11 commented Apr 18, 2017

maybe the creation of the docker cluster should be something like:

for i in 1 2 3; do
  docker-machine create  --virtualbox-hostonly-cidr "10.10.10.1/24" -d virtualbox node-$i
done

because if you have the same subnetwork 99 on you system this would fail to create:

Error creating machine: Error in driver during machine creation: Error setting up host only network on machine start: host-only cidr conflicts with the network address of a host interface

the network could be anything you want.

btw excellent book, I´ve already did 2.0 and Im doing 2.1 (bought them both on leanpub)

@damianoneill
Copy link

@vfarcic when running on my host, the script blocks at the following section;

mh1cr1iblyxzreata8ppzbj7i
overall progress: 3 out of 3 tasks
cbqb2hbw9c5b: running   [==================================================>]
xwotfszpzrss: running   [==================================================>]
s6ht1fuerqz2: running   [==================================================>]
verify: Service converged
ID                  NAME                             IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ohl2h4fx0oju        util.xwotfszpzrss9e0mdkvy0avj3   alpine:latest       node-2              Running             Running 6 seconds ago
wc3ttsf6529t        util.s6ht1fuerqz26zk4n4ldtawu5   alpine:latest       node-1              Running             Running 11 seconds ago
0cl1ze9n4gkr        util.cbqb2hbw9c5brus71y2fzj77m   alpine:latest       node-3              Running             Running 10 seconds ago
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
(1/2) Installing ldns (1.7.1-r1)
(2/2) Installing drill (1.7.1-r1)
Executing busybox-1.31.1-r8.trigger
OK: 6 MiB in 16 packages
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 11236
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; go-demo-db.	IN	A

;; ANSWER SECTION:
go-demo-db.	600	IN	A	10.0.1.2

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 0 msec
;; SERVER: 127.0.0.11
;; WHEN: Tue Jan  7 10:14:33 2020
;; MSG SIZE  rcvd: 54
bvcjv2pv90gtfxn7u1b5uubj9
NETWORK ID          NAME                DRIVER              SCOPE
pn6lnd2bha6d        go-demo             overlay             swarm
dmtipuxk9xhu        ingress             overlay             swarm
bvcjv2pv90gt        proxy               overlay             swarm
wx6untueojdzbj7ef6y2ub1ih
overall progress: 0 out of 1 tasks
1/1: starting  [============================================>      ]

It repeated iterates through the states; starting, preparing, ready, starting, ...

But never progresses past this point.

The full output is

10:06 $ ./03-networking.sh
Running pre-create checks...
Creating machine...
(node-1) Copying /Users/doneill/.docker/machine/cache/boot2docker.iso to /Users/doneill/.docker/machine/machines/node-1/boot2docker.iso...
(node-1) Creating VirtualBox VM...
(node-1) Creating SSH key...
(node-1) Starting the VM...
(node-1) Check network to re-create if needed...
(node-1) Waiting for an IP...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env node-1
Running pre-create checks...
Creating machine...
(node-2) Copying /Users/doneill/.docker/machine/cache/boot2docker.iso to /Users/doneill/.docker/machine/machines/node-2/boot2docker.iso...
(node-2) Creating VirtualBox VM...
(node-2) Creating SSH key...
(node-2) Starting the VM...
(node-2) Check network to re-create if needed...
(node-2) Waiting for an IP...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env node-2
Running pre-create checks...
Creating machine...
(node-3) Copying /Users/doneill/.docker/machine/cache/boot2docker.iso to /Users/doneill/.docker/machine/machines/node-3/boot2docker.iso...
(node-3) Creating VirtualBox VM...
(node-3) Creating SSH key...
(node-3) Starting the VM...
(node-3) Check network to re-create if needed...
(node-3) Waiting for an IP...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env node-3
Swarm initialized: current node (s6ht1fuerqz26zk4n4ldtawu5) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-12svu2fxj3rgd98gyyuff74doh3vk971cqg1taydypel394uxy-ds5myprae9sfyscll62hds0py 192.168.99.110:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

This node joined a swarm as a worker.
This node joined a swarm as a worker.
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
s6ht1fuerqz26zk4n4ldtawu5 *   node-1              Ready               Active              Leader              19.03.5
xwotfszpzrss9e0mdkvy0avj3     node-2              Ready               Active                                  19.03.5
cbqb2hbw9c5brus71y2fzj77m     node-3              Ready               Active                                  19.03.5
jsxggr6155zy4npsty7oscn07
overall progress: 1 out of 1 tasks
1/1: running   [==================================================>]
verify: Service converged

ID:		jsxggr6155zy4npsty7oscn07
Name:		go-demo-db
Service Mode:	Replicated
 Replicas:	1
Placement:
UpdateConfig:
 Parallelism:	1
 On failure:	pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Update order:      stop-first
RollbackConfig:
 Parallelism:	1
 On failure:	pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Rollback order:    stop-first
ContainerSpec:
 Image:		mongo:3.2.10@sha256:532a19da83ee0e4e2a2ec6bc4212fc4af26357c040675d5c2629a4e4c4563cef
 Init:		false
Resources:
Endpoint Mode:	vip

go-demo-db
pn6lnd2bha6dinfqvyc4ityfp
qe2y0uwa9pxsmn5gi7xk2xavo
overall progress: 1 out of 1 tasks
1/1: running   [==================================================>]
verify: Service converged

ID:		qe2y0uwa9pxsmn5gi7xk2xavo
Name:		go-demo-db
Service Mode:	Replicated
 Replicas:	1
Placement:
UpdateConfig:
 Parallelism:	1
 On failure:	pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Update order:      stop-first
RollbackConfig:
 Parallelism:	1
 On failure:	pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Rollback order:    stop-first
ContainerSpec:
 Image:		mongo:3.2.10@sha256:532a19da83ee0e4e2a2ec6bc4212fc4af26357c040675d5c2629a4e4c4563cef
 Init:		false
Resources:
Networks: go-demo
Endpoint Mode:	vip

mh1cr1iblyxzreata8ppzbj7i
overall progress: 3 out of 3 tasks
cbqb2hbw9c5b: running   [==================================================>]
xwotfszpzrss: running   [==================================================>]
s6ht1fuerqz2: running   [==================================================>]
verify: Service converged
ID                  NAME                             IMAGE               NODE                DESIRED STATE       CURRENT STATE            ERROR               PORTS
ohl2h4fx0oju        util.xwotfszpzrss9e0mdkvy0avj3   alpine:latest       node-2              Running             Running 6 seconds ago
wc3ttsf6529t        util.s6ht1fuerqz26zk4n4ldtawu5   alpine:latest       node-1              Running             Running 11 seconds ago
0cl1ze9n4gkr        util.cbqb2hbw9c5brus71y2fzj77m   alpine:latest       node-3              Running             Running 10 seconds ago
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.11/community/x86_64/APKINDEX.tar.gz
(1/2) Installing ldns (1.7.1-r1)
(2/2) Installing drill (1.7.1-r1)
Executing busybox-1.31.1-r8.trigger
OK: 6 MiB in 16 packages
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 11236
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; go-demo-db.	IN	A

;; ANSWER SECTION:
go-demo-db.	600	IN	A	10.0.1.2

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 0 msec
;; SERVER: 127.0.0.11
;; WHEN: Tue Jan  7 10:14:33 2020
;; MSG SIZE  rcvd: 54
bvcjv2pv90gtfxn7u1b5uubj9
NETWORK ID          NAME                DRIVER              SCOPE
pn6lnd2bha6d        go-demo             overlay             swarm
dmtipuxk9xhu        ingress             overlay             swarm
bvcjv2pv90gt        proxy               overlay             swarm
wx6untueojdzbj7ef6y2ub1ih
overall progress: 0 out of 1 tasks
1/1: ready     [======================================>            ]

CTRL-C (twice) at this point results in

overall progress: 0 out of 1 tasks
1/1: starting  [============================================>      ]
^COperation continuing in background.
Use `docker service ps wx6untueojdzbj7ef6y2ub1ih` to check progress.
cosxwh7e6r2z696mgyf5wkomu
overall progress: 0 out of 1 tasks
1/1: starting  [============================================>      ]
^COperation continuing in background.
Use `docker service ps cosxwh7e6r2z696mgyf5wkomu` to check progress.
ID                  NAME                IMAGE                              NODE                DESIRED STATE       CURRENT STATE             ERROR               PORTS
u43pnqufaqir        proxy.1             vfarcic/docker-flow-proxy:latest   node-2              Running             Starting 10 seconds ago
curl: (7) Failed to connect to 192.168.99.110 port 8080: Connection refused
curl: (7) Failed to connect to 192.168.99.110 port 80: Connection refused
curl: (7) Failed to connect to 192.168.99.112 port 80: Connection refused
global
    pidfile /var/run/haproxy.pid

10:20 $ sw_vers
ProductName: Mac OS X
ProductVersion: 10.14.6
BuildVersion: 18G1012
✔ ~
10:20 $ docker --version
Docker version 19.03.5, build 633a0ea
✔ ~
10:21 $ docker-machine --version
docker-machine version 0.16.2, build bd45ab13

Virtualbox Version 6.0.10 r132072 (Qt5.6.3)

I also cleared the machine directory cache and downloaded the latest boot2docker version to confirm I'm on the latest.

(node-1) Latest release for github.com/boot2docker/boot2docker is v19.03.5
(node-1) Downloading /Users/doneill/.docker/machine/cache/boot2docker.iso from https://github.com/boot2docker/boot2docker/releases/download/v19.03.5/boot2docker.iso...

Any suggestions on what the problem might be?

Thanks,
Damian.

@vfarcic
Copy link
Author

vfarcic commented Jan 7, 2020

Which command hangs? Is it...

docker service create --name go-demo \
  -e DB=go-demo-db \
  --network go-demo \
  --network proxy \
  vfarcic/go-demo:1.0

?

@damianoneill
Copy link

Hi Victor, yes this is where it stalls.

bash -x 03-networking.sh 

...

+ docker network create --driver overlay proxy
nrxfg3ylsofyfqcrndkt52xax
+ docker network ls -f driver=overlay
NETWORK ID          NAME                DRIVER              SCOPE
ulkj0k2890ny        go-demo             overlay             swarm
m7s0kdu89tis        ingress             overlay             swarm
nrxfg3ylsofy        proxy               overlay             swarm
+ docker service create --name go-demo -e DB=go-demo-db --network go-demo --network proxy vfarcic/go-demo:1.0
3unhhv3w2x50k5otnkiw46b4h
overall progress: 0 out of 1 tasks
1/1: starting  [============================================>      ]

@vfarcic
Copy link
Author

vfarcic commented Jan 7, 2020

Can you inspect the service in a separate terminal? Something like docker service inspect go-demo --pretty. While there, output the services as well (docker service ls).

@vfarcic
Copy link
Author

vfarcic commented Jan 7, 2020

I just run the same commands and it worked fine. Do you have some kind of a firewall/VPN?

@damianoneill
Copy link

Requested outputs below.

Note, I updated the script to exit before creating the service and ran the service command manually to ensure it wasnt a timing issue.

As before I had to CTRL-C the command as it kept circling around the states.

✔ ~/projects/github.com/vfarcic/go-demo [master|✚ 1…3]
12:03 $ docker service inspect go-demo --pretty

ID:		vcqyztrqikym758h42ig3ah6b
Name:		go-demo
Service Mode:	Replicated
 Replicas:	1
Placement:
UpdateConfig:
 Parallelism:	1
 On failure:	pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Update order:      stop-first
RollbackConfig:
 Parallelism:	1
 On failure:	pause
 Monitoring Period: 5s
 Max failure ratio: 0
 Rollback order:    stop-first
ContainerSpec:
 Image:		vfarcic/go-demo:1.0@sha256:a578350216c5939ee518c9f57d040cf8a54ebc5d1a85292bfdf505394b040e5b
 Env:		DB=go-demo-db
 Init:		false
Resources:
Networks: proxy go-demo
Endpoint Mode:	vip
12:03 $ docker service ls
ID                  NAME                MODE                REPLICAS            IMAGE                 PORTS
vcqyztrqikym        go-demo             replicated          0/1                 vfarcic/go-demo:1.0
dxhiwhcq7ijb        go-demo-db          replicated          1/1                 mongo:3.2.10
mxzo3gm6q4jz        util                global              3/3                 alpine:latest

Not running on a VPN, have also connected to different external networks to ensure no uplink issues.

I am running McAfee endpoint security for Mac, but the Firewall function is off.

@vfarcic
Copy link
Author

vfarcic commented Jan 7, 2020

I'll go through it tomorrow.

My Swarm skills became a bit rusty. It's been 3 years since I'm not using it. From my perspective, Swarm is dead, and that's very unfortunate. Docker (company) abandoned it and (almost) everyone switched to Kubernetes. Now I believe it is a dead technology :(

@vfarcic
Copy link
Author

vfarcic commented Jan 7, 2020

Are you going through this code using The DevOps 2.1 Toolkit book?

@damianoneill
Copy link

Hi Victor, yes I'm using the book. I'm in Chpt 3 Setting up a cluster

The text from the book is

All the commands from this chapter are available in the 03-networking.sh (https://gist.github.com/vfarcic/fd7d7e04e1133fc3c90084c4c1a919fe) Gist.

@vfarcic
Copy link
Author

vfarcic commented Jan 7, 2020

I strongly recommend not wasting time with Swarm. Send me an email to [email protected] and I'll send you a free copy of "The DevOps 2.3 Toolkit: Kubernetes".

@damianoneill
Copy link

Little further.

16:35 $ docker ps -a
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS                            PORTS               NAMES
fd247afc63a6        vfarcic/go-demo:1.0   "go-demo"                9 minutes ago       Up 6 seconds (health: starting)   8080/tcp            go-demo.1.9d434uglfzh53vmkqeqw0zlvn
e522aed5538d        vfarcic/go-demo:1.0   "go-demo"                9 minutes ago       Exited (2) 12 seconds ago                             go-demo.1.lv69xtzp42tlniuai834xn9b5
c605db4e16de        vfarcic/go-demo:1.0   "go-demo"                9 minutes ago       Exited (2) 27 seconds ago                             go-demo.1.41mc1zsqjc82d0k51idasr3dy
3c67bcccd9dd        vfarcic/go-demo:1.0   "go-demo"                9 minutes ago       Exited (2) 43 seconds ago                             go-demo.1.03i2yoirlxpyximk0k6dyninu
6eaf090b1929        vfarcic/go-demo:1.0   "go-demo"                10 minutes ago      Exited (2) 59 seconds ago                             go-demo.1.i3poq3wx4yeabg0wnytua8jxm
690c6b18a849        alpine:latest         "sleep 1000000000"       29 minutes ago      Up 20 minutes                                         util.pa11gdrab3db7tch6enx98mfn.nc1s0hd05maftw89jlu0ncmlp
091ba9e4f713        mongo:3.2.10          "/entrypoint.sh mong…"   29 minutes ago      Up 20 minutes                     27017/tcp           go-demo-db.1.s3hemcmhy9m5otm74z8yz42tg
docker logs e522aed5538d
panic: no reachable servers

goroutine 1 [running]:
panic(0x7b6480, 0xc82000b7f0)
	/usr/local/go/src/runtime/panic.go:481 +0x3e6
main.setupDb()
	/usr/src/myapp/main.go:43 +0xc5
main.main()
	/usr/src/myapp/main.go:30 +0x14
✔ ~

The gist is pulling the image from hub vfarcic/go-demo:1.0 so can't see the code in go mainline.

Current code does not have setupDb at line 43 in the main.go.

Might be good to update the gist to reflect the latest image? i.e.

docker service create --name go-demo \
   -e DB=go-demo-db \
   --network go-demo \
   --network proxy \
   vfarcic/go-demo:latest

Doing this shows the issue is with the panic when connecting to mongo, so looks like the os.Getenv("DB") is not being returned correctly or the host is not reachable or the port is not exposed.

@damianoneill
Copy link

damianoneill commented Jan 7, 2020 •
edited
Loading

Might be a port issue.

docker network create --attachable --driver overlay go-demo

I needed to make the network attachable to run below.

$ docker run -e DB=go-demo-db --network go-demo -it vfarcic/go-demo:latest /bin/sh
/ # env
HOSTNAME=47a4294f8e92
SHLVL=1
HOME=/root
DB=go-demo-db
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
/ # ping go-demo-db
PING go-demo-db (10.0.1.2): 56 data bytes
64 bytes from 10.0.1.2: seq=0 ttl=64 time=0.082 ms
64 bytes from 10.0.1.2: seq=1 ttl=64 time=0.084 ms
^C
--- go-demo-db ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.082/0.083/0.084 ms
/ # telnet go-demo-db 27017

Should the create service for mongo expose the default port?

docker service create --name go-demo-db \
  --network go-demo \
  mongo:3.2.10

@damianoneill
Copy link

Not a port issue, confirmed port exposed

nmap -p 27017 go-demo-db

Starting Nmap 7.12 ( https://nmap.org ) at 2020-01-07 17:25 UTC
Nmap scan report for go-demo-db (10.0.1.2)
Host is up (-0.20s latency).
PORT      STATE    SERVICE
27017/tcp filtered mongod
MAC Address: 02:42:0A:00:01:04 (Unknown)

Nmap done: 1 IP address (1 host up) scanned in 0.46 seconds

@damianoneill
Copy link

Some progress

The problem is with the proxy network, removing this from the service create command allows the service to be created.

$ docker service create --name go-demo    -e DB=go-demo-db    --network go-demo     vfarcic/go-demo:latest
qfqpgs962otjbulsfof0wzvnt
overall progress: 1 out of 1 tasks
1/1: running   [==================================================>]
verify: Service converged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment