Skip to content

Instantly share code, notes, and snippets.

@harshal-shah

harshal-shah/kops_issue_syslog.log

Last active July 5, 2018 16:08
Show Gist options
  • Save harshal-shah/04b3038aea786e5e9d7a7d524091de96 to your computer and use it in GitHub Desktop.
Save harshal-shah/04b3038aea786e5e9d7a7d524091de96 to your computer and use it in GitHub Desktop.
Download ZIP
syslog captured during k8s master outage
Raw
kops_issue_syslog.log
Jul 5 13:22:56 ip-172-24-102-238 docker[8538]: I0705 13:22:56.252986 8583 aws_volume.go:72] AWS API Request: ec2metadata/GetMetadata
Jul 5 13:22:56 ip-172-24-102-238 docker[8538]: I0705 13:22:56.253994 8583 aws_volume.go:72] AWS API Request: ec2metadata/GetMetadata
Jul 5 13:22:56 ip-172-24-102-238 docker[8538]: I0705 13:22:56.254718 8583 aws_volume.go:72] AWS API Request: ec2metadata/GetMetadata
Jul 5 13:22:56 ip-172-24-102-238 docker[8538]: I0705 13:22:56.257659 8583 aws_volume.go:72] AWS API Request: ec2/DescribeInstances
Jul 5 13:22:56 ip-172-24-102-238 docker[8538]: I0705 13:22:56.486326 8583 main.go:187] cluster-id: production.mydomain.io
Jul 5 13:22:56 ip-172-24-102-238 docker[8538]: I0705 13:22:56.486677 8583 dnscontroller.go:101] starting DNS controller
Jul 5 13:22:56 ip-172-24-102-238 docker[8538]: I0705 13:22:56.486760 8583 dnscache.go:75] querying all DNS zones (no cached results)
Jul 5 13:22:56 ip-172-24-102-238 docker[8538]: I0705 13:22:56.486917 8583 route53.go:53] AWS request: route53 ListHostedZones
Jul 5 13:22:56 ip-172-24-102-238 docker[8538]: I0705 13:22:56.487405 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:22:56 ip-172-24-102-238 docker[8538]: W0705 13:22:56.540663 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:22:56 ip-172-24-102-238 docker[8538]: W0705 13:22:56.540686 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:22:56 ip-172-24-102-238 docker[8538]: W0705 13:22:56.540694 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:22:56 ip-172-24-102-238 docker[8538]: W0705 13:22:56.540700 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:22:56 ip-172-24-102-238 docker[8538]: W0705 13:22:56.540711 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:22:56 ip-172-24-102-238 docker[8538]: W0705 13:22:56.540722 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:22:56 ip-172-24-102-238 docker[8538]: W0705 13:22:56.540745 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:22:56 ip-172-24-102-238 docker[8538]: W0705 13:22:56.540758 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:22:56 ip-172-24-102-238 docker[8538]: I0705 13:22:56.540804 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0f1b533a98788c1ab"
Jul 5 13:22:56 ip-172-24-102-238 docker[8538]: I0705 13:22:56.540979 8583 aws_volume.go:72] AWS API Request: ec2/AttachVolume
Jul 5 13:22:57 ip-172-24-102-238 docker[8538]: I0705 13:22:57.443298 8583 aws_volume.go:72] AWS API Request: ec2/AttachVolume
Jul 5 13:22:59 ip-172-24-102-238 docker[8538]: I0705 13:22:59.347494 8583 aws_volume.go:72] AWS API Request: ec2/AttachVolume
Jul 5 13:23:02 ip-172-24-102-238 docker[8538]: I0705 13:23:02.437746 8583 aws_volume.go:72] AWS API Request: ec2/AttachVolume
Jul 5 13:23:02 ip-172-24-102-238 docker[8538]: W0705 13:23:02.456467 8583 volume_mounter.go:260] Error attaching volume "vol-0f1b533a98788c1ab": Error attaching EBS volume "vol-0f1b533a98788c1ab": RequestLimitExceeded: Request limit exceeded.
Jul 5 13:23:02 ip-172-24-102-238 docker[8538]: I0705 13:23:02.456492 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:23:02 ip-172-24-102-238 docker[8538]: I0705 13:23:02.456684 8583 aws_volume.go:72] AWS API Request: ec2/AttachVolume
Jul 5 13:23:03 ip-172-24-102-238 docker[8538]: I0705 13:23:03.244476 8583 aws_volume.go:72] AWS API Request: ec2/AttachVolume
Jul 5 13:23:04 ip-172-24-102-238 docker[8538]: I0705 13:23:04.886824 8583 aws_volume.go:72] AWS API Request: ec2/AttachVolume
Jul 5 13:23:07 ip-172-24-102-238 docker[8538]: I0705 13:23:07.244990 8583 aws_volume.go:72] AWS API Request: ec2/AttachVolume
Jul 5 13:23:07 ip-172-24-102-238 docker[8538]: W0705 13:23:07.262345 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": Error attaching EBS volume "vol-0d86190f17f570d98": RequestLimitExceeded: Request limit exceeded.
Jul 5 13:23:07 ip-172-24-102-238 docker[8538]: I0705 13:23:07.262370 8583 volume_mounter.go:274] Currently attached volumes: []
Jul 5 13:23:07 ip-172-24-102-238 docker[8538]: I0705 13:23:07.262380 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:23:07 ip-172-24-102-238 docker[8538]: I0705 13:23:07.267563 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:23:07 ip-172-24-102-238 docker[8538]: I0705 13:23:07.267585 8583 kube_boot.go:188] kubelet systemd service not running. Starting
Jul 5 13:23:07 ip-172-24-102-238 docker[8538]: I0705 13:23:07.317376 8583 kube_boot.go:193] 'systemctl start kubelet' output:
Jul 5 13:23:07 ip-172-24-102-238 docker[8538]: W0705 13:23:07.321621 8583 rbac.go:52] Error configuring RBAC: error creating service accounts: Post https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:23:07 ip-172-24-102-238 docker[8538]: W0705 13:23:07.321640 8583 rbac.go:52] Error configuring RBAC: error creating cluster role bindings: unable to create RBAC clusterrolebinding: Post https://127.0.0.1/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:23:07 ip-172-24-102-238 docker[8538]: W0705 13:23:07.321651 8583 kube_boot.go:152] error initializing rbac: error creating service accounts: Post https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:23:07 ip-172-24-102-238 docker[8538]: I0705 13:23:07.321662 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:23:07 ip-172-24-102-238 docker[8538]: I0705 13:23:07.321730 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:23:07 ip-172-24-102-238 docker[8538]: I0705 13:23:07.343529 8583 channels.go:48] error running channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes:
Jul 5 13:23:07 ip-172-24-102-238 docker[8538]: I0705 13:23:07.343563 8583 channels.go:49] Error: error querying kubernetes version: Get https://127.0.0.1/version: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:23:07 ip-172-24-102-238 docker[8538]: I0705 13:23:07.343585 8583 channels.go:34] apply channel output was: Error: error querying kubernetes version: Get https://127.0.0.1/version: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:23:07 ip-172-24-102-238 docker[8538]: W0705 13:23:07.343601 8583 kube_boot.go:157] error applying channel "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml": error running channels: exit status 1
Jul 5 13:24:07 ip-172-24-102-238 docker[8538]: I0705 13:24:07.344024 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:24:07 ip-172-24-102-238 docker[8538]: W0705 13:24:07.412942 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:24:07 ip-172-24-102-238 docker[8538]: W0705 13:24:07.412969 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:24:07 ip-172-24-102-238 docker[8538]: W0705 13:24:07.412981 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:24:07 ip-172-24-102-238 docker[8538]: W0705 13:24:07.412994 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:24:07 ip-172-24-102-238 docker[8538]: W0705 13:24:07.413007 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:24:07 ip-172-24-102-238 docker[8538]: W0705 13:24:07.413021 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:24:07 ip-172-24-102-238 docker[8538]: W0705 13:24:07.413035 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:24:07 ip-172-24-102-238 docker[8538]: W0705 13:24:07.413049 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:24:07 ip-172-24-102-238 docker[8538]: I0705 13:24:07.413067 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0f1b533a98788c1ab"
Jul 5 13:24:07 ip-172-24-102-238 docker[8538]: I0705 13:24:07.413214 8583 aws_volume.go:72] AWS API Request: ec2/AttachVolume
Jul 5 13:24:07 ip-172-24-102-238 docker[8538]: I0705 13:24:07.956442 8583 aws_volume.go:72] AWS API Request: ec2/AttachVolume
Jul 5 13:24:09 ip-172-24-102-238 docker[8538]: I0705 13:24:09.778963 8583 aws_volume.go:72] AWS API Request: ec2/AttachVolume
Jul 5 13:24:12 ip-172-24-102-238 docker[8538]: I0705 13:24:12.498647 8583 aws_volume.go:72] AWS API Request: ec2/AttachVolume
Jul 5 13:24:12 ip-172-24-102-238 docker[8538]: W0705 13:24:12.518755 8583 volume_mounter.go:260] Error attaching volume "vol-0f1b533a98788c1ab": Error attaching EBS volume "vol-0f1b533a98788c1ab": RequestLimitExceeded: Request limit exceeded.
Jul 5 13:24:12 ip-172-24-102-238 docker[8538]: I0705 13:24:12.518781 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:24:12 ip-172-24-102-238 docker[8538]: I0705 13:24:12.518977 8583 aws_volume.go:72] AWS API Request: ec2/AttachVolume
Jul 5 13:24:13 ip-172-24-102-238 docker[8538]: I0705 13:24:13.497675 8583 aws_volume.go:72] AWS API Request: ec2/AttachVolume
Jul 5 13:24:14 ip-172-24-102-238 docker[8538]: I0705 13:24:14.717397 8583 aws_volume.go:72] AWS API Request: ec2/AttachVolume
Jul 5 13:24:17 ip-172-24-102-238 docker[8538]: I0705 13:24:17.325471 8583 aws_volume.go:72] AWS API Request: ec2/AttachVolume
Jul 5 13:24:17 ip-172-24-102-238 docker[8538]: W0705 13:24:17.351221 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": Error attaching EBS volume "vol-0d86190f17f570d98": RequestLimitExceeded: Request limit exceeded.
Jul 5 13:24:17 ip-172-24-102-238 docker[8538]: I0705 13:24:17.351241 8583 volume_mounter.go:274] Currently attached volumes: []
Jul 5 13:24:17 ip-172-24-102-238 docker[8538]: I0705 13:24:17.351251 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:24:17 ip-172-24-102-238 docker[8538]: I0705 13:24:17.380820 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:24:17 ip-172-24-102-238 docker[8538]: I0705 13:24:17.380871 8583 kube_boot.go:184] kubelet systemd service already running
Jul 5 13:24:17 ip-172-24-102-238 docker[8538]: W0705 13:24:17.381469 8583 rbac.go:52] Error configuring RBAC: error creating service accounts: Post https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:24:17 ip-172-24-102-238 docker[8538]: W0705 13:24:17.381490 8583 rbac.go:52] Error configuring RBAC: error creating cluster role bindings: unable to create RBAC clusterrolebinding: Post https://127.0.0.1/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:24:17 ip-172-24-102-238 docker[8538]: W0705 13:24:17.381500 8583 kube_boot.go:152] error initializing rbac: error creating service accounts: Post https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:24:17 ip-172-24-102-238 docker[8538]: I0705 13:24:17.381512 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:24:17 ip-172-24-102-238 docker[8538]: I0705 13:24:17.381554 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:24:17 ip-172-24-102-238 docker[8538]: I0705 13:24:17.403108 8583 channels.go:48] error running channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes:
Jul 5 13:24:17 ip-172-24-102-238 docker[8538]: I0705 13:24:17.403151 8583 channels.go:49] Error: error querying kubernetes version: Get https://127.0.0.1/version: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:24:17 ip-172-24-102-238 docker[8538]: I0705 13:24:17.403175 8583 channels.go:34] apply channel output was: Error: error querying kubernetes version: Get https://127.0.0.1/version: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:24:17 ip-172-24-102-238 docker[8538]: W0705 13:24:17.403188 8583 kube_boot.go:157] error applying channel "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml": error running channels: exit status 1
Jul 5 13:25:17 ip-172-24-102-238 docker[8538]: I0705 13:25:17.403612 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:25:17 ip-172-24-102-238 docker[8538]: W0705 13:25:17.440432 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:25:17 ip-172-24-102-238 docker[8538]: W0705 13:25:17.440455 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:25:17 ip-172-24-102-238 docker[8538]: W0705 13:25:17.440465 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:25:17 ip-172-24-102-238 docker[8538]: W0705 13:25:17.440472 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:25:17 ip-172-24-102-238 docker[8538]: W0705 13:25:17.440480 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:25:17 ip-172-24-102-238 docker[8538]: W0705 13:25:17.440495 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:25:17 ip-172-24-102-238 docker[8538]: W0705 13:25:17.440509 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:25:17 ip-172-24-102-238 docker[8538]: W0705 13:25:17.440539 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:25:17 ip-172-24-102-238 docker[8538]: I0705 13:25:17.440556 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0f1b533a98788c1ab"
Jul 5 13:25:17 ip-172-24-102-238 docker[8538]: I0705 13:25:17.440734 8583 aws_volume.go:72] AWS API Request: ec2/AttachVolume
Jul 5 13:25:18 ip-172-24-102-238 docker[8538]: I0705 13:25:18.102262 8583 aws_volume.go:72] AWS API Request: ec2/AttachVolume
Jul 5 13:25:19 ip-172-24-102-238 docker[8538]: I0705 13:25:19.889991 8583 aws_volume.go:72] AWS API Request: ec2/AttachVolume
Jul 5 13:25:20 ip-172-24-102-238 docker[8538]: I0705 13:25:20.129385 8583 aws_volume.go:398] AttachVolume request returned {
Jul 5 13:25:20 ip-172-24-102-238 docker[8538]: I0705 13:25:20.129560 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:25:20 ip-172-24-102-238 docker[8538]: W0705 13:25:20.207991 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:25:20 ip-172-24-102-238 docker[8538]: W0705 13:25:20.208016 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:25:20 ip-172-24-102-238 docker[8538]: W0705 13:25:20.208023 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:25:20 ip-172-24-102-238 docker[8538]: W0705 13:25:20.208030 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:25:20 ip-172-24-102-238 docker[8538]: I0705 13:25:20.208044 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:25:20 ip-172-24-102-238 docker[8538]: W0705 13:25:20.208054 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": All devices in use
Jul 5 13:25:20 ip-172-24-102-238 docker[8538]: I0705 13:25:20.208073 8583 volume_mounter.go:274] Currently attached volumes: [{"ID":"vol-0f1b533a98788c1ab","LocalDevice":"/dev/xvdz","AttachedTo":"","Mountpoint":"","Status":"available","Info":{"Description":"vol-0f1b533a98788c1ab","EtcdClusters":[{"clusterKey":"main","nodeName":"c","nodeNames":["a","b","c"]}]}}]
Jul 5 13:25:20 ip-172-24-102-238 docker[8538]: I0705 13:25:20.208188 8583 volume_mounter.go:58] Master volume "vol-0f1b533a98788c1ab" is attached at "/dev/xvdz"
Jul 5 13:25:20 ip-172-24-102-238 docker[8538]: I0705 13:25:20.208664 8583 volume_mounter.go:72] Doing safe-format-and-mount of /dev/xvdz to /mnt/master-vol-0f1b533a98788c1ab
Jul 5 13:25:20 ip-172-24-102-238 docker[8538]: I0705 13:25:20.208716 8583 aws_volume.go:320] nvme path not found "/rootfs/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0f1b533a98788c1ab"
Jul 5 13:25:20 ip-172-24-102-238 docker[8538]: I0705 13:25:20.208736 8583 volume_mounter.go:107] Waiting for volume "vol-0f1b533a98788c1ab" to be attached
Jul 5 13:25:21 ip-172-24-102-238 docker[8538]: I0705 13:25:21.208959 8583 aws_volume.go:320] nvme path not found "/rootfs/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0f1b533a98788c1ab"
Jul 5 13:25:21 ip-172-24-102-238 docker[8538]: I0705 13:25:21.208994 8583 volume_mounter.go:107] Waiting for volume "vol-0f1b533a98788c1ab" to be attached
Jul 5 13:25:22 ip-172-24-102-238 docker[8538]: I0705 13:25:22.209240 8583 volume_mounter.go:110] Found volume "vol-0f1b533a98788c1ab" mounted at device "/dev/xvdz"
Jul 5 13:25:22 ip-172-24-102-238 docker[8538]: I0705 13:25:22.209829 8583 volume_mounter.go:147] Creating mount directory "/rootfs/mnt/master-vol-0f1b533a98788c1ab"
Jul 5 13:25:22 ip-172-24-102-238 docker[8538]: I0705 13:25:22.210722 8583 volume_mounter.go:152] Mounting device "/dev/xvdz" on "/mnt/master-vol-0f1b533a98788c1ab"
Jul 5 13:25:22 ip-172-24-102-238 docker[8538]: I0705 13:25:22.210748 8583 mount_linux.go:496] Checking for issues with fsck on disk: /dev/xvdz
Jul 5 13:25:22 ip-172-24-102-238 docker[8538]: I0705 13:25:22.239950 8583 mount_linux.go:515] Attempting to mount disk: /dev/xvdz /mnt/master-vol-0f1b533a98788c1ab
Jul 5 13:25:22 ip-172-24-102-238 docker[8538]: I0705 13:25:22.303264 8583 mount_linux.go:211] Detected OS with systemd
Jul 5 13:25:22 ip-172-24-102-238 docker[8538]: I0705 13:25:22.303865 8583 volume_mounter.go:195] mounting inside container: /rootfs/dev/xvdz -> /rootfs/mnt/master-vol-0f1b533a98788c1ab
Jul 5 13:25:22 ip-172-24-102-238 docker[8538]: I0705 13:25:22.303892 8583 mount_linux.go:143] Mounting cmd (systemd-run) with arguments ([--description=Kubernetes transient mount for /rootfs/mnt/master-vol-0f1b533a98788c1ab --scope -- mount /rootfs/dev/xvdz /rootfs/mnt/master-vol-0f1b533a98788c1ab])
Jul 5 13:25:22 ip-172-24-102-238 docker[8538]: I0705 13:25:22.334711 8583 volume_mounter.go:80] mounted master volume "vol-0f1b533a98788c1ab" on /mnt/master-vol-0f1b533a98788c1ab
Jul 5 13:25:22 ip-172-24-102-238 docker[8538]: I0705 13:25:22.334757 8583 kube_boot.go:119] Found etcd cluster spec on volume "vol-0f1b533a98788c1ab": {"clusterKey":"main","nodeName":"c","nodeNames":["a","b","c"]}
Jul 5 13:25:22 ip-172-24-102-238 docker[8538]: I0705 13:25:22.334815 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:25:22 ip-172-24-102-238 docker[8538]: I0705 13:25:22.335207 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Jul 5 13:25:22 ip-172-24-102-238 docker[8538]: I0705 13:25:22.335261 8583 dnscontroller.go:610] Update desired state: protokube/etcd-c.internal.production.mydomain.io: [{A etcd-c.internal.production.mydomain.io 172.24.102.238 false}]
Jul 5 13:25:22 ip-172-24-102-238 docker[8538]: I0705 13:25:22.344852 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:25:22 ip-172-24-102-238 docker[8538]: I0705 13:25:22.344905 8583 kube_boot.go:184] kubelet systemd service already running
Jul 5 13:25:22 ip-172-24-102-238 docker[8538]: I0705 13:25:22.401122 8583 etcd_cluster.go:292] Updated etcd manifest: /etc/kubernetes/manifests/etcd.manifest
Jul 5 13:25:26 ip-172-24-102-238 docker[8538]: I0705 13:25:26.793198 8583 dnscontroller.go:256] Using default TTL of 1m0s
Jul 5 13:25:26 ip-172-24-102-238 docker[8538]: I0705 13:25:26.793252 8583 dnscontroller.go:258] updating records for {A etcd-c.internal.production.mydomain.io}: [] -> [172.24.102.238]
Jul 5 13:25:26 ip-172-24-102-238 docker[8538]: I0705 13:25:26.793326 8583 dnscontroller.go:421] Querying all dnsprovider records for zone "live-k8s.mydomain.io."
Jul 5 13:25:26 ip-172-24-102-238 docker[8538]: I0705 13:25:26.793528 8583 route53.go:53] AWS request: route53 ListResourceRecordSets
Jul 5 13:25:27 ip-172-24-102-238 docker[8538]: I0705 13:25:27.154467 8583 dnscontroller.go:570] Adding DNS changes to batch {A etcd-c.internal.production.mydomain.io} [172.24.102.238]
Jul 5 13:25:27 ip-172-24-102-238 docker[8538]: I0705 13:25:27.154515 8583 dnscontroller.go:301] applying DNS changeset for zone live-k8s.mydomain.io.::ZY9SNN4V5GYR0
Jul 5 13:25:27 ip-172-24-102-238 docker[8538]: I0705 13:25:27.154896 8583 route53.go:53] AWS request: route53 ChangeResourceRecordSets
Jul 5 13:25:35 ip-172-24-102-238 docker[8538]: I0705 13:25:35.338911 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:25:35 ip-172-24-102-238 docker[8538]: I0705 13:25:35.338970 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:25:35 ip-172-24-102-238 docker[8538]: I0705 13:25:35.583170 8583 channels.go:34] apply channel output was: I0705 13:25:35.375450 11181 addons.go:38] Loading addons channel from "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:26:22 ip-172-24-102-238 docker[8538]: I0705 13:26:22.401450 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Jul 5 13:26:35 ip-172-24-102-238 docker[8538]: I0705 13:26:35.583650 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:26:35 ip-172-24-102-238 docker[8538]: W0705 13:26:35.667831 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:26:35 ip-172-24-102-238 docker[8538]: W0705 13:26:35.667857 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:26:35 ip-172-24-102-238 docker[8538]: W0705 13:26:35.667864 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:26:35 ip-172-24-102-238 docker[8538]: W0705 13:26:35.667871 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:26:35 ip-172-24-102-238 docker[8538]: W0705 13:26:35.667878 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:26:35 ip-172-24-102-238 docker[8538]: W0705 13:26:35.667885 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:26:35 ip-172-24-102-238 docker[8538]: W0705 13:26:35.667905 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:26:35 ip-172-24-102-238 docker[8538]: W0705 13:26:35.667918 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:26:35 ip-172-24-102-238 docker[8538]: I0705 13:26:35.667951 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:26:35 ip-172-24-102-238 docker[8538]: W0705 13:26:35.667968 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": All devices in use
Jul 5 13:26:35 ip-172-24-102-238 docker[8538]: I0705 13:26:35.667986 8583 volume_mounter.go:274] Currently attached volumes: [{"ID":"vol-0f1b533a98788c1ab","LocalDevice":"/dev/xvdz","AttachedTo":"i-020821d707dfc72a5","Mountpoint":"","Status":"in-use","Info":{"Description":"vol-0f1b533a98788c1ab","EtcdClusters":[{"clusterKey":"main","nodeName":"c","nodeNames":["a","b","c"]}]}}]
Jul 5 13:26:35 ip-172-24-102-238 docker[8538]: I0705 13:26:35.668027 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:26:35 ip-172-24-102-238 docker[8538]: I0705 13:26:35.693619 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:26:35 ip-172-24-102-238 docker[8538]: I0705 13:26:35.693653 8583 kube_boot.go:184] kubelet systemd service already running
Jul 5 13:26:35 ip-172-24-102-238 docker[8538]: I0705 13:26:35.738823 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:26:35 ip-172-24-102-238 docker[8538]: I0705 13:26:35.738891 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:26:35 ip-172-24-102-238 docker[8538]: I0705 13:26:35.983175 8583 channels.go:34] apply channel output was: I0705 13:26:35.770086 13702 addons.go:38] Loading addons channel from "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:27:22 ip-172-24-102-238 docker[8538]: I0705 13:27:22.404144 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Jul 5 13:27:35 ip-172-24-102-238 docker[8538]: I0705 13:27:35.983597 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:27:36 ip-172-24-102-238 docker[8538]: W0705 13:27:36.058188 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:27:36 ip-172-24-102-238 docker[8538]: W0705 13:27:36.058215 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:27:36 ip-172-24-102-238 docker[8538]: W0705 13:27:36.058223 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:27:36 ip-172-24-102-238 docker[8538]: W0705 13:27:36.058232 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:27:36 ip-172-24-102-238 docker[8538]: W0705 13:27:36.058244 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:27:36 ip-172-24-102-238 docker[8538]: W0705 13:27:36.058269 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:27:36 ip-172-24-102-238 docker[8538]: W0705 13:27:36.058280 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:27:36 ip-172-24-102-238 docker[8538]: W0705 13:27:36.058286 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:27:36 ip-172-24-102-238 docker[8538]: I0705 13:27:36.058299 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:27:36 ip-172-24-102-238 docker[8538]: W0705 13:27:36.058308 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": All devices in use
Jul 5 13:27:36 ip-172-24-102-238 docker[8538]: I0705 13:27:36.058327 8583 volume_mounter.go:274] Currently attached volumes: [{"ID":"vol-0f1b533a98788c1ab","LocalDevice":"/dev/xvdz","AttachedTo":"i-020821d707dfc72a5","Mountpoint":"","Status":"in-use","Info":{"Description":"vol-0f1b533a98788c1ab","EtcdClusters":[{"clusterKey":"main","nodeName":"c","nodeNames":["a","b","c"]}]}}]
Jul 5 13:27:36 ip-172-24-102-238 docker[8538]: I0705 13:27:36.058371 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:27:36 ip-172-24-102-238 docker[8538]: I0705 13:27:36.084102 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:27:36 ip-172-24-102-238 docker[8538]: I0705 13:27:36.084142 8583 kube_boot.go:184] kubelet systemd service already running
Jul 5 13:27:36 ip-172-24-102-238 docker[8538]: I0705 13:27:36.124598 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:27:36 ip-172-24-102-238 docker[8538]: I0705 13:27:36.124647 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:27:36 ip-172-24-102-238 docker[8538]: I0705 13:27:36.358815 8583 channels.go:34] apply channel output was: I0705 13:27:36.151637 14713 addons.go:38] Loading addons channel from "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:28:22 ip-172-24-102-238 docker[8538]: I0705 13:28:22.405985 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Jul 5 13:28:36 ip-172-24-102-238 docker[8538]: I0705 13:28:36.359364 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:28:36 ip-172-24-102-238 docker[8538]: W0705 13:28:36.447158 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:28:36 ip-172-24-102-238 docker[8538]: W0705 13:28:36.447185 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:28:36 ip-172-24-102-238 docker[8538]: W0705 13:28:36.447192 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:28:36 ip-172-24-102-238 docker[8538]: W0705 13:28:36.447204 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:28:36 ip-172-24-102-238 docker[8538]: W0705 13:28:36.447218 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:28:36 ip-172-24-102-238 docker[8538]: W0705 13:28:36.447237 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:28:36 ip-172-24-102-238 docker[8538]: W0705 13:28:36.447247 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:28:36 ip-172-24-102-238 docker[8538]: W0705 13:28:36.447263 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:28:36 ip-172-24-102-238 docker[8538]: I0705 13:28:36.447284 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:28:36 ip-172-24-102-238 docker[8538]: W0705 13:28:36.447313 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": All devices in use
Jul 5 13:28:36 ip-172-24-102-238 docker[8538]: I0705 13:28:36.447344 8583 volume_mounter.go:274] Currently attached volumes: [{"ID":"vol-0f1b533a98788c1ab","LocalDevice":"/dev/xvdz","AttachedTo":"i-020821d707dfc72a5","Mountpoint":"","Status":"in-use","Info":{"Description":"vol-0f1b533a98788c1ab","EtcdClusters":[{"clusterKey":"main","nodeName":"c","nodeNames":["a","b","c"]}]}}]
Jul 5 13:28:36 ip-172-24-102-238 docker[8538]: I0705 13:28:36.447388 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:28:36 ip-172-24-102-238 docker[8538]: I0705 13:28:36.479689 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:28:36 ip-172-24-102-238 docker[8538]: I0705 13:28:36.479723 8583 kube_boot.go:184] kubelet systemd service already running
Jul 5 13:28:36 ip-172-24-102-238 docker[8538]: I0705 13:28:36.538483 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:28:36 ip-172-24-102-238 docker[8538]: I0705 13:28:36.538546 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:28:36 ip-172-24-102-238 docker[8538]: I0705 13:28:36.760254 8583 channels.go:34] apply channel output was: I0705 13:28:36.565340 15542 addons.go:38] Loading addons channel from "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:29:22 ip-172-24-102-238 docker[8538]: I0705 13:29:22.408752 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Jul 5 13:29:36 ip-172-24-102-238 docker[8538]: I0705 13:29:36.760747 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:29:36 ip-172-24-102-238 docker[8538]: W0705 13:29:36.879306 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:29:36 ip-172-24-102-238 docker[8538]: W0705 13:29:36.879331 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:29:36 ip-172-24-102-238 docker[8538]: W0705 13:29:36.879338 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:29:36 ip-172-24-102-238 docker[8538]: W0705 13:29:36.879345 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:29:36 ip-172-24-102-238 docker[8538]: W0705 13:29:36.879354 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:29:36 ip-172-24-102-238 docker[8538]: W0705 13:29:36.879375 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:29:36 ip-172-24-102-238 docker[8538]: W0705 13:29:36.879388 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:29:36 ip-172-24-102-238 docker[8538]: W0705 13:29:36.879398 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:29:36 ip-172-24-102-238 docker[8538]: I0705 13:29:36.879415 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:29:36 ip-172-24-102-238 docker[8538]: W0705 13:29:36.879429 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": All devices in use
Jul 5 13:29:36 ip-172-24-102-238 docker[8538]: I0705 13:29:36.879458 8583 volume_mounter.go:274] Currently attached volumes: [{"ID":"vol-0f1b533a98788c1ab","LocalDevice":"/dev/xvdz","AttachedTo":"i-020821d707dfc72a5","Mountpoint":"","Status":"in-use","Info":{"Description":"vol-0f1b533a98788c1ab","EtcdClusters":[{"clusterKey":"main","nodeName":"c","nodeNames":["a","b","c"]}]}}]
Jul 5 13:29:36 ip-172-24-102-238 docker[8538]: I0705 13:29:36.879503 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:29:36 ip-172-24-102-238 docker[8538]: I0705 13:29:36.904264 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:29:36 ip-172-24-102-238 docker[8538]: I0705 13:29:36.904323 8583 kube_boot.go:184] kubelet systemd service already running
Jul 5 13:29:36 ip-172-24-102-238 docker[8538]: I0705 13:29:36.941428 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:29:36 ip-172-24-102-238 docker[8538]: I0705 13:29:36.941480 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:29:37 ip-172-24-102-238 docker[8538]: I0705 13:29:37.204558 8583 channels.go:34] apply channel output was: I0705 13:29:36.972161 16509 addons.go:38] Loading addons channel from "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:30:22 ip-172-24-102-238 docker[8538]: I0705 13:30:22.410771 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Jul 5 13:30:37 ip-172-24-102-238 docker[8538]: I0705 13:30:37.205097 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:30:37 ip-172-24-102-238 docker[8538]: W0705 13:30:37.478061 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:30:37 ip-172-24-102-238 docker[8538]: W0705 13:30:37.478090 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:30:37 ip-172-24-102-238 docker[8538]: W0705 13:30:37.478097 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:30:37 ip-172-24-102-238 docker[8538]: W0705 13:30:37.478104 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:30:37 ip-172-24-102-238 docker[8538]: W0705 13:30:37.478112 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:30:37 ip-172-24-102-238 docker[8538]: W0705 13:30:37.478123 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:30:37 ip-172-24-102-238 docker[8538]: W0705 13:30:37.478136 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:30:37 ip-172-24-102-238 docker[8538]: W0705 13:30:37.478152 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:30:37 ip-172-24-102-238 docker[8538]: I0705 13:30:37.478172 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:30:37 ip-172-24-102-238 docker[8538]: W0705 13:30:37.478186 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": All devices in use
Jul 5 13:30:37 ip-172-24-102-238 docker[8538]: I0705 13:30:37.478202 8583 volume_mounter.go:274] Currently attached volumes: [{"ID":"vol-0f1b533a98788c1ab","LocalDevice":"/dev/xvdz","AttachedTo":"i-020821d707dfc72a5","Mountpoint":"","Status":"in-use","Info":{"Description":"vol-0f1b533a98788c1ab","EtcdClusters":[{"clusterKey":"main","nodeName":"c","nodeNames":["a","b","c"]}]}}]
Jul 5 13:30:37 ip-172-24-102-238 docker[8538]: I0705 13:30:37.478241 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:30:37 ip-172-24-102-238 docker[8538]: I0705 13:30:37.485920 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:30:37 ip-172-24-102-238 docker[8538]: I0705 13:30:37.485945 8583 kube_boot.go:184] kubelet systemd service already running
Jul 5 13:30:37 ip-172-24-102-238 docker[8538]: I0705 13:30:37.545971 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:30:37 ip-172-24-102-238 docker[8538]: I0705 13:30:37.546024 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:30:37 ip-172-24-102-238 docker[8538]: I0705 13:30:37.991186 8583 channels.go:34] apply channel output was: I0705 13:30:37.574684 18154 addons.go:38] Loading addons channel from "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:31:22 ip-172-24-102-238 docker[8538]: I0705 13:31:22.414091 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Jul 5 13:31:37 ip-172-24-102-238 docker[8538]: I0705 13:31:37.991666 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:31:38 ip-172-24-102-238 docker[8538]: W0705 13:31:38.115423 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:31:38 ip-172-24-102-238 docker[8538]: W0705 13:31:38.115448 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:31:38 ip-172-24-102-238 docker[8538]: W0705 13:31:38.115456 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:31:38 ip-172-24-102-238 docker[8538]: W0705 13:31:38.115463 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:31:38 ip-172-24-102-238 docker[8538]: W0705 13:31:38.115470 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:31:38 ip-172-24-102-238 docker[8538]: W0705 13:31:38.115477 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:31:38 ip-172-24-102-238 docker[8538]: W0705 13:31:38.115494 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:31:38 ip-172-24-102-238 docker[8538]: W0705 13:31:38.115507 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:31:38 ip-172-24-102-238 docker[8538]: I0705 13:31:38.115545 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:31:38 ip-172-24-102-238 docker[8538]: W0705 13:31:38.115561 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": All devices in use
Jul 5 13:31:38 ip-172-24-102-238 docker[8538]: I0705 13:31:38.115579 8583 volume_mounter.go:274] Currently attached volumes: [{"ID":"vol-0f1b533a98788c1ab","LocalDevice":"/dev/xvdz","AttachedTo":"i-020821d707dfc72a5","Mountpoint":"","Status":"in-use","Info":{"Description":"vol-0f1b533a98788c1ab","EtcdClusters":[{"clusterKey":"main","nodeName":"c","nodeNames":["a","b","c"]}]}}]
Jul 5 13:31:38 ip-172-24-102-238 docker[8538]: I0705 13:31:38.115622 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:31:38 ip-172-24-102-238 docker[8538]: I0705 13:31:38.132964 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:31:38 ip-172-24-102-238 docker[8538]: I0705 13:31:38.133004 8583 kube_boot.go:184] kubelet systemd service already running
Jul 5 13:31:38 ip-172-24-102-238 docker[8538]: I0705 13:31:38.172854 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:31:38 ip-172-24-102-238 docker[8538]: I0705 13:31:38.172918 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:31:38 ip-172-24-102-238 docker[8538]: I0705 13:31:38.405179 8583 channels.go:34] apply channel output was: I0705 13:31:38.198466 19571 addons.go:38] Loading addons channel from "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:32:22 ip-172-24-102-238 docker[8538]: I0705 13:32:22.415741 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Jul 5 13:32:38 ip-172-24-102-238 docker[8538]: I0705 13:32:38.405664 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:32:38 ip-172-24-102-238 docker[8538]: W0705 13:32:38.491756 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:32:38 ip-172-24-102-238 docker[8538]: W0705 13:32:38.491784 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:32:38 ip-172-24-102-238 docker[8538]: W0705 13:32:38.491792 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:32:38 ip-172-24-102-238 docker[8538]: W0705 13:32:38.491799 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:32:38 ip-172-24-102-238 docker[8538]: W0705 13:32:38.491812 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:32:38 ip-172-24-102-238 docker[8538]: W0705 13:32:38.491833 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:32:38 ip-172-24-102-238 docker[8538]: W0705 13:32:38.491866 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:32:38 ip-172-24-102-238 docker[8538]: W0705 13:32:38.491880 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:32:38 ip-172-24-102-238 docker[8538]: I0705 13:32:38.491903 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:32:38 ip-172-24-102-238 docker[8538]: W0705 13:32:38.491918 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": All devices in use
Jul 5 13:32:38 ip-172-24-102-238 docker[8538]: I0705 13:32:38.491927 8583 volume_mounter.go:274] Currently attached volumes: [{"ID":"vol-0f1b533a98788c1ab","LocalDevice":"/dev/xvdz","AttachedTo":"i-020821d707dfc72a5","Mountpoint":"","Status":"in-use","Info":{"Description":"vol-0f1b533a98788c1ab","EtcdClusters":[{"clusterKey":"main","nodeName":"c","nodeNames":["a","b","c"]}]}}]
Jul 5 13:32:38 ip-172-24-102-238 docker[8538]: I0705 13:32:38.491957 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:32:38 ip-172-24-102-238 docker[8538]: I0705 13:32:38.528553 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:32:38 ip-172-24-102-238 docker[8538]: I0705 13:32:38.528588 8583 kube_boot.go:184] kubelet systemd service already running
Jul 5 13:32:38 ip-172-24-102-238 docker[8538]: I0705 13:32:38.567323 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:32:38 ip-172-24-102-238 docker[8538]: I0705 13:32:38.567390 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:32:38 ip-172-24-102-238 docker[8538]: I0705 13:32:38.789002 8583 channels.go:34] apply channel output was: I0705 13:32:38.595391 20578 addons.go:38] Loading addons channel from "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:33:22 ip-172-24-102-238 docker[8538]: I0705 13:33:22.418162 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Jul 5 13:33:38 ip-172-24-102-238 docker[8538]: I0705 13:33:38.789410 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:33:39 ip-172-24-102-238 docker[8538]: W0705 13:33:39.031599 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:33:39 ip-172-24-102-238 docker[8538]: W0705 13:33:39.031626 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:33:39 ip-172-24-102-238 docker[8538]: W0705 13:33:39.031635 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:33:39 ip-172-24-102-238 docker[8538]: W0705 13:33:39.031642 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:33:39 ip-172-24-102-238 docker[8538]: W0705 13:33:39.031651 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:33:39 ip-172-24-102-238 docker[8538]: W0705 13:33:39.031658 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:33:39 ip-172-24-102-238 docker[8538]: W0705 13:33:39.031678 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:33:39 ip-172-24-102-238 docker[8538]: W0705 13:33:39.031690 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:33:39 ip-172-24-102-238 docker[8538]: I0705 13:33:39.031735 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:33:39 ip-172-24-102-238 docker[8538]: W0705 13:33:39.031762 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": All devices in use
Jul 5 13:33:39 ip-172-24-102-238 docker[8538]: I0705 13:33:39.031777 8583 volume_mounter.go:274] Currently attached volumes: [{"ID":"vol-0f1b533a98788c1ab","LocalDevice":"/dev/xvdz","AttachedTo":"i-020821d707dfc72a5","Mountpoint":"","Status":"in-use","Info":{"Description":"vol-0f1b533a98788c1ab","EtcdClusters":[{"clusterKey":"main","nodeName":"c","nodeNames":["a","b","c"]}]}}]
Jul 5 13:33:39 ip-172-24-102-238 docker[8538]: I0705 13:33:39.031874 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:33:39 ip-172-24-102-238 docker[8538]: I0705 13:33:39.065385 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:33:39 ip-172-24-102-238 docker[8538]: I0705 13:33:39.065432 8583 kube_boot.go:184] kubelet systemd service already running
Jul 5 13:33:39 ip-172-24-102-238 docker[8538]: I0705 13:33:39.137598 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:33:39 ip-172-24-102-238 docker[8538]: I0705 13:33:39.137658 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:33:39 ip-172-24-102-238 docker[8538]: I0705 13:33:39.395621 8583 channels.go:34] apply channel output was: I0705 13:33:39.166174 21934 addons.go:38] Loading addons channel from "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:34:22 ip-172-24-102-238 docker[8538]: I0705 13:34:22.424367 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Jul 5 13:34:39 ip-172-24-102-238 docker[8538]: I0705 13:34:39.396135 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:34:39 ip-172-24-102-238 docker[8538]: W0705 13:34:39.469144 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:34:39 ip-172-24-102-238 docker[8538]: W0705 13:34:39.469174 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:34:39 ip-172-24-102-238 docker[8538]: W0705 13:34:39.469186 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:34:39 ip-172-24-102-238 docker[8538]: W0705 13:34:39.469197 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:34:39 ip-172-24-102-238 docker[8538]: W0705 13:34:39.469211 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:34:39 ip-172-24-102-238 docker[8538]: W0705 13:34:39.469232 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:34:39 ip-172-24-102-238 docker[8538]: W0705 13:34:39.469247 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:34:39 ip-172-24-102-238 docker[8538]: W0705 13:34:39.469262 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:34:39 ip-172-24-102-238 docker[8538]: I0705 13:34:39.469281 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:34:39 ip-172-24-102-238 docker[8538]: W0705 13:34:39.469297 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": All devices in use
Jul 5 13:34:39 ip-172-24-102-238 docker[8538]: I0705 13:34:39.469312 8583 volume_mounter.go:274] Currently attached volumes: [{"ID":"vol-0f1b533a98788c1ab","LocalDevice":"/dev/xvdz","AttachedTo":"i-020821d707dfc72a5","Mountpoint":"","Status":"in-use","Info":{"Description":"vol-0f1b533a98788c1ab","EtcdClusters":[{"clusterKey":"main","nodeName":"c","nodeNames":["a","b","c"]}]}}]
Jul 5 13:34:39 ip-172-24-102-238 docker[8538]: I0705 13:34:39.469351 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:34:39 ip-172-24-102-238 docker[8538]: I0705 13:34:39.475973 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:34:39 ip-172-24-102-238 docker[8538]: I0705 13:34:39.476008 8583 kube_boot.go:184] kubelet systemd service already running
Jul 5 13:34:53 ip-172-24-102-238 docker[8538]: W0705 13:34:53.486864 8583 rbac.go:52] Error configuring RBAC: error creating service accounts: client: etcd member http://127.0.0.1:4001 has no leader
Jul 5 13:34:53 ip-172-24-102-238 docker[8538]: W0705 13:34:53.486894 8583 rbac.go:52] Error configuring RBAC: error creating cluster role bindings: unable to create RBAC clusterrolebinding: client: etcd member http://127.0.0.1:4001 has no leader
Jul 5 13:34:53 ip-172-24-102-238 docker[8538]: W0705 13:34:53.486907 8583 kube_boot.go:152] error initializing rbac: error creating service accounts: client: etcd member http://127.0.0.1:4001 has no leader
Jul 5 13:34:53 ip-172-24-102-238 docker[8538]: I0705 13:34:53.486919 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:34:53 ip-172-24-102-238 docker[8538]: I0705 13:34:53.486973 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:34:53 ip-172-24-102-238 docker[8538]: I0705 13:34:53.707201 8583 channels.go:34] apply channel output was: I0705 13:34:53.519646 23650 addons.go:38] Loading addons channel from "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:35:22 ip-172-24-102-238 docker[8538]: I0705 13:35:22.426680 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Jul 5 13:35:53 ip-172-24-102-238 docker[8538]: I0705 13:35:53.707634 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:35:53 ip-172-24-102-238 docker[8538]: W0705 13:35:53.912234 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:35:53 ip-172-24-102-238 docker[8538]: W0705 13:35:53.912261 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:35:53 ip-172-24-102-238 docker[8538]: W0705 13:35:53.912273 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:35:53 ip-172-24-102-238 docker[8538]: W0705 13:35:53.912321 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:35:53 ip-172-24-102-238 docker[8538]: W0705 13:35:53.912338 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:35:53 ip-172-24-102-238 docker[8538]: W0705 13:35:53.912356 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:35:53 ip-172-24-102-238 docker[8538]: W0705 13:35:53.912371 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:35:53 ip-172-24-102-238 docker[8538]: W0705 13:35:53.912386 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:35:53 ip-172-24-102-238 docker[8538]: I0705 13:35:53.912405 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:35:53 ip-172-24-102-238 docker[8538]: W0705 13:35:53.912415 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": All devices in use
Jul 5 13:35:53 ip-172-24-102-238 docker[8538]: I0705 13:35:53.912424 8583 volume_mounter.go:274] Currently attached volumes: [{"ID":"vol-0f1b533a98788c1ab","LocalDevice":"/dev/xvdz","AttachedTo":"i-020821d707dfc72a5","Mountpoint":"","Status":"in-use","Info":{"Description":"vol-0f1b533a98788c1ab","EtcdClusters":[{"clusterKey":"main","nodeName":"c","nodeNames":["a","b","c"]}]}}]
Jul 5 13:35:53 ip-172-24-102-238 docker[8538]: I0705 13:35:53.912457 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:35:53 ip-172-24-102-238 docker[8538]: I0705 13:35:53.941247 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:35:53 ip-172-24-102-238 docker[8538]: I0705 13:35:53.941288 8583 kube_boot.go:184] kubelet systemd service already running
Jul 5 13:36:04 ip-172-24-102-238 docker[8538]: W0705 13:36:04.078103 8583 rbac.go:52] Error configuring RBAC: error creating service accounts: client: etcd member http://127.0.0.1:4001 has no leader
Jul 5 13:36:04 ip-172-24-102-238 docker[8538]: W0705 13:36:04.078128 8583 rbac.go:52] Error configuring RBAC: error creating cluster role bindings: unable to create RBAC clusterrolebinding: EOF
Jul 5 13:36:04 ip-172-24-102-238 docker[8538]: W0705 13:36:04.078139 8583 kube_boot.go:152] error initializing rbac: error creating service accounts: client: etcd member http://127.0.0.1:4001 has no leader
Jul 5 13:36:04 ip-172-24-102-238 docker[8538]: I0705 13:36:04.078151 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:36:04 ip-172-24-102-238 docker[8538]: I0705 13:36:04.078203 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:36:04 ip-172-24-102-238 docker[8538]: I0705 13:36:04.434288 8583 channels.go:48] error running channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes:
Jul 5 13:36:04 ip-172-24-102-238 docker[8538]: I0705 13:36:04.434330 8583 channels.go:49] I0705 13:36:04.118939 24121 addons.go:38] Loading addons channel from "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:36:04 ip-172-24-102-238 docker[8538]: I0705 13:36:04.434356 8583 channels.go:34] apply channel output was: I0705 13:36:04.118939 24121 addons.go:38] Loading addons channel from "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:36:04 ip-172-24-102-238 docker[8538]: W0705 13:36:04.434374 8583 kube_boot.go:157] error applying channel "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml": error running channels: exit status 1
Jul 5 13:36:22 ip-172-24-102-238 docker[8538]: I0705 13:36:22.428816 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: I0705 13:37:04.434843 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: W0705 13:37:04.477396 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: W0705 13:37:04.477426 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: W0705 13:37:04.477436 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: W0705 13:37:04.477443 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: W0705 13:37:04.477451 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: W0705 13:37:04.477458 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: W0705 13:37:04.477468 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: W0705 13:37:04.477488 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: I0705 13:37:04.477510 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: W0705 13:37:04.477531 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": All devices in use
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: I0705 13:37:04.477541 8583 volume_mounter.go:274] Currently attached volumes: [{"ID":"vol-0f1b533a98788c1ab","LocalDevice":"/dev/xvdz","AttachedTo":"i-020821d707dfc72a5","Mountpoint":"","Status":"in-use","Info":{"Description":"vol-0f1b533a98788c1ab","EtcdClusters":[{"clusterKey":"main","nodeName":"c","nodeNames":["a","b","c"]}]}}]
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: I0705 13:37:04.477586 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: I0705 13:37:04.484625 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: I0705 13:37:04.484650 8583 kube_boot.go:184] kubelet systemd service already running
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: W0705 13:37:04.491303 8583 rbac.go:52] Error configuring RBAC: error creating service accounts: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:4001: getsockopt: connection refused
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: W0705 13:37:04.491326 8583 rbac.go:52] Error configuring RBAC: error creating cluster role bindings: unable to create RBAC clusterrolebinding: dial tcp 127.0.0.1:4001: getsockopt: connection refused
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: W0705 13:37:04.491336 8583 kube_boot.go:152] error initializing rbac: error creating service accounts: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:4001: getsockopt: connection refused
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: I0705 13:37:04.491346 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: I0705 13:37:04.491385 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: I0705 13:37:04.710171 8583 channels.go:48] error running channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes:
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: I0705 13:37:04.710267 8583 channels.go:49] I0705 13:37:04.523690 24483 addons.go:38] Loading addons channel from "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: I0705 13:37:04.710308 8583 channels.go:34] apply channel output was: I0705 13:37:04.523690 24483 addons.go:38] Loading addons channel from "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:37:04 ip-172-24-102-238 docker[8538]: W0705 13:37:04.710324 8583 kube_boot.go:157] error applying channel "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml": error running channels: exit status 1
Jul 5 13:37:22 ip-172-24-102-238 docker[8538]: I0705 13:37:22.431305 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Jul 5 13:38:04 ip-172-24-102-238 docker[8538]: I0705 13:38:04.710792 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:38:04 ip-172-24-102-238 docker[8538]: W0705 13:38:04.771937 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:38:04 ip-172-24-102-238 docker[8538]: W0705 13:38:04.772002 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:38:04 ip-172-24-102-238 docker[8538]: W0705 13:38:04.772020 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:38:04 ip-172-24-102-238 docker[8538]: W0705 13:38:04.772031 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:38:04 ip-172-24-102-238 docker[8538]: W0705 13:38:04.772046 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:38:04 ip-172-24-102-238 docker[8538]: W0705 13:38:04.772058 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:38:04 ip-172-24-102-238 docker[8538]: W0705 13:38:04.772073 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:38:04 ip-172-24-102-238 docker[8538]: W0705 13:38:04.772085 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:38:04 ip-172-24-102-238 docker[8538]: I0705 13:38:04.772106 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:38:04 ip-172-24-102-238 docker[8538]: W0705 13:38:04.772120 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": All devices in use
Jul 5 13:38:04 ip-172-24-102-238 docker[8538]: I0705 13:38:04.772137 8583 volume_mounter.go:274] Currently attached volumes: [{"ID":"vol-0f1b533a98788c1ab","LocalDevice":"/dev/xvdz","AttachedTo":"i-020821d707dfc72a5","Mountpoint":"","Status":"in-use","Info":{"Description":"vol-0f1b533a98788c1ab","EtcdClusters":[{"clusterKey":"main","nodeName":"c","nodeNames":["a","b","c"]}]}}]
Jul 5 13:38:04 ip-172-24-102-238 docker[8538]: I0705 13:38:04.772191 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:38:04 ip-172-24-102-238 docker[8538]: I0705 13:38:04.811883 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:38:04 ip-172-24-102-238 docker[8538]: I0705 13:38:04.811973 8583 kube_boot.go:184] kubelet systemd service already running
Jul 5 13:38:22 ip-172-24-102-238 docker[8538]: I0705 13:38:22.433319 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Jul 5 13:38:23 ip-172-24-102-238 docker[8538]: W0705 13:38:23.202396 8583 rbac.go:52] Error configuring RBAC: error creating service accounts: client: etcd member http://127.0.0.1:4001 has no leader
Jul 5 13:38:23 ip-172-24-102-238 docker[8538]: W0705 13:38:23.202418 8583 rbac.go:52] Error configuring RBAC: error creating cluster role bindings: unable to create RBAC clusterrolebinding: client: etcd member http://127.0.0.1:4001 has no leader
Jul 5 13:38:23 ip-172-24-102-238 docker[8538]: W0705 13:38:23.202427 8583 kube_boot.go:152] error initializing rbac: error creating service accounts: client: etcd member http://127.0.0.1:4001 has no leader
Jul 5 13:38:23 ip-172-24-102-238 docker[8538]: I0705 13:38:23.202435 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:38:23 ip-172-24-102-238 docker[8538]: I0705 13:38:23.202475 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:38:23 ip-172-24-102-238 docker[8538]: I0705 13:38:23.468117 8583 channels.go:34] apply channel output was: I0705 13:38:23.230184 25023 addons.go:38] Loading addons channel from "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:39:22 ip-172-24-102-238 docker[8538]: I0705 13:39:22.436196 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Jul 5 13:39:23 ip-172-24-102-238 docker[8538]: I0705 13:39:23.468613 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:39:23 ip-172-24-102-238 docker[8538]: W0705 13:39:23.550707 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:39:23 ip-172-24-102-238 docker[8538]: W0705 13:39:23.550739 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:39:23 ip-172-24-102-238 docker[8538]: W0705 13:39:23.550752 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:39:23 ip-172-24-102-238 docker[8538]: W0705 13:39:23.550763 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:39:23 ip-172-24-102-238 docker[8538]: W0705 13:39:23.550776 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:39:23 ip-172-24-102-238 docker[8538]: W0705 13:39:23.550790 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:39:23 ip-172-24-102-238 docker[8538]: W0705 13:39:23.550804 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:39:23 ip-172-24-102-238 docker[8538]: W0705 13:39:23.550815 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:39:23 ip-172-24-102-238 docker[8538]: I0705 13:39:23.550833 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:39:23 ip-172-24-102-238 docker[8538]: W0705 13:39:23.550849 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": All devices in use
Jul 5 13:39:23 ip-172-24-102-238 docker[8538]: I0705 13:39:23.550861 8583 volume_mounter.go:274] Currently attached volumes: [{"ID":"vol-0f1b533a98788c1ab","LocalDevice":"/dev/xvdz","AttachedTo":"i-020821d707dfc72a5","Mountpoint":"","Status":"in-use","Info":{"Description":"vol-0f1b533a98788c1ab","EtcdClusters":[{"clusterKey":"main","nodeName":"c","nodeNames":["a","b","c"]}]}}]
Jul 5 13:39:23 ip-172-24-102-238 docker[8538]: I0705 13:39:23.550900 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:39:23 ip-172-24-102-238 docker[8538]: I0705 13:39:23.559903 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:39:23 ip-172-24-102-238 docker[8538]: I0705 13:39:23.559928 8583 kube_boot.go:184] kubelet systemd service already running
Jul 5 13:39:27 ip-172-24-102-238 docker[8538]: W0705 13:39:27.841527 8583 rbac.go:52] Error configuring RBAC: error creating service accounts: Post https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts: EOF
Jul 5 13:39:27 ip-172-24-102-238 docker[8538]: W0705 13:39:27.841554 8583 rbac.go:52] Error configuring RBAC: error creating cluster role bindings: unable to create RBAC clusterrolebinding: Post https://127.0.0.1/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings: read tcp 127.0.0.1:34082->127.0.0.1:443: read: connection reset by peer
Jul 5 13:39:27 ip-172-24-102-238 docker[8538]: W0705 13:39:27.841565 8583 kube_boot.go:152] error initializing rbac: error creating service accounts: Post https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts: EOF
Jul 5 13:39:27 ip-172-24-102-238 docker[8538]: I0705 13:39:27.841646 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:39:27 ip-172-24-102-238 docker[8538]: I0705 13:39:27.841757 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:39:27 ip-172-24-102-238 docker[8538]: I0705 13:39:27.861067 8583 channels.go:48] error running channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes:
Jul 5 13:39:27 ip-172-24-102-238 docker[8538]: I0705 13:39:27.861098 8583 channels.go:49] Error: error querying kubernetes version: Get https://127.0.0.1/version: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:39:27 ip-172-24-102-238 docker[8538]: I0705 13:39:27.861119 8583 channels.go:34] apply channel output was: Error: error querying kubernetes version: Get https://127.0.0.1/version: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:39:27 ip-172-24-102-238 docker[8538]: W0705 13:39:27.861138 8583 kube_boot.go:157] error applying channel "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml": error running channels: exit status 1
Jul 5 13:40:22 ip-172-24-102-238 docker[8538]: I0705 13:40:22.438033 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: I0705 13:40:27.861561 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: W0705 13:40:27.903535 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: W0705 13:40:27.903559 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: W0705 13:40:27.903567 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: W0705 13:40:27.903577 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: W0705 13:40:27.903590 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: W0705 13:40:27.903602 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: W0705 13:40:27.903615 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: W0705 13:40:27.903637 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: I0705 13:40:27.903656 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: W0705 13:40:27.903681 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": All devices in use
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: I0705 13:40:27.903695 8583 volume_mounter.go:274] Currently attached volumes: [{"ID":"vol-0f1b533a98788c1ab","LocalDevice":"/dev/xvdz","AttachedTo":"i-020821d707dfc72a5","Mountpoint":"","Status":"in-use","Info":{"Description":"vol-0f1b533a98788c1ab","EtcdClusters":[{"clusterKey":"main","nodeName":"c","nodeNames":["a","b","c"]}]}}]
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: I0705 13:40:27.903741 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: I0705 13:40:27.932308 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: I0705 13:40:27.932341 8583 kube_boot.go:184] kubelet systemd service already running
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: W0705 13:40:27.932946 8583 rbac.go:52] Error configuring RBAC: error creating service accounts: Post https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: W0705 13:40:27.932964 8583 rbac.go:52] Error configuring RBAC: error creating cluster role bindings: unable to create RBAC clusterrolebinding: Post https://127.0.0.1/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: W0705 13:40:27.932972 8583 kube_boot.go:152] error initializing rbac: error creating service accounts: Post https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: I0705 13:40:27.932979 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: I0705 13:40:27.933010 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: I0705 13:40:27.952900 8583 channels.go:48] error running channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes:
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: I0705 13:40:27.952943 8583 channels.go:49] Error: error querying kubernetes version: Get https://127.0.0.1/version: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: I0705 13:40:27.952966 8583 channels.go:34] apply channel output was: Error: error querying kubernetes version: Get https://127.0.0.1/version: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:40:27 ip-172-24-102-238 docker[8538]: W0705 13:40:27.952983 8583 kube_boot.go:157] error applying channel "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml": error running channels: exit status 1
Jul 5 13:41:22 ip-172-24-102-238 docker[8538]: I0705 13:41:22.440329 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Jul 5 13:41:27 ip-172-24-102-238 docker[8538]: I0705 13:41:27.953383 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: W0705 13:41:28.000564 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: W0705 13:41:28.000592 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: W0705 13:41:28.000610 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: W0705 13:41:28.000624 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: W0705 13:41:28.000638 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: W0705 13:41:28.000650 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: W0705 13:41:28.000663 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: W0705 13:41:28.000671 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: I0705 13:41:28.000683 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: W0705 13:41:28.000692 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": All devices in use
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: I0705 13:41:28.000701 8583 volume_mounter.go:274] Currently attached volumes: [{"ID":"vol-0f1b533a98788c1ab","LocalDevice":"/dev/xvdz","AttachedTo":"i-020821d707dfc72a5","Mountpoint":"","Status":"in-use","Info":{"Description":"vol-0f1b533a98788c1ab","EtcdClusters":[{"clusterKey":"main","nodeName":"c","nodeNames":["a","b","c"]}]}}]
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: I0705 13:41:28.000739 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: I0705 13:41:28.007739 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: I0705 13:41:28.007777 8583 kube_boot.go:184] kubelet systemd service already running
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: W0705 13:41:28.008300 8583 rbac.go:52] Error configuring RBAC: error creating service accounts: Post https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: W0705 13:41:28.008325 8583 rbac.go:52] Error configuring RBAC: error creating cluster role bindings: unable to create RBAC clusterrolebinding: Post https://127.0.0.1/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: W0705 13:41:28.008336 8583 kube_boot.go:152] error initializing rbac: error creating service accounts: Post https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: I0705 13:41:28.008362 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: I0705 13:41:28.008402 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: I0705 13:41:28.030531 8583 channels.go:48] error running channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes:
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: I0705 13:41:28.030573 8583 channels.go:49] Error: error querying kubernetes version: Get https://127.0.0.1/version: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: I0705 13:41:28.030594 8583 channels.go:34] apply channel output was: Error: error querying kubernetes version: Get https://127.0.0.1/version: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:41:28 ip-172-24-102-238 docker[8538]: W0705 13:41:28.030611 8583 kube_boot.go:157] error applying channel "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml": error running channels: exit status 1
Jul 5 13:42:22 ip-172-24-102-238 docker[8538]: I0705 13:42:22.442297 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Jul 5 13:42:28 ip-172-24-102-238 docker[8538]: I0705 13:42:28.031058 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:42:28 ip-172-24-102-238 docker[8538]: W0705 13:42:28.087698 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:42:28 ip-172-24-102-238 docker[8538]: W0705 13:42:28.087726 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:42:28 ip-172-24-102-238 docker[8538]: W0705 13:42:28.087739 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:42:28 ip-172-24-102-238 docker[8538]: W0705 13:42:28.087752 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:42:28 ip-172-24-102-238 docker[8538]: W0705 13:42:28.087765 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:42:28 ip-172-24-102-238 docker[8538]: W0705 13:42:28.087776 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:42:28 ip-172-24-102-238 docker[8538]: W0705 13:42:28.087802 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:42:28 ip-172-24-102-238 docker[8538]: W0705 13:42:28.087823 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:42:28 ip-172-24-102-238 docker[8538]: I0705 13:42:28.087849 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:42:28 ip-172-24-102-238 docker[8538]: W0705 13:42:28.087873 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": All devices in use
Jul 5 13:42:28 ip-172-24-102-238 docker[8538]: I0705 13:42:28.087884 8583 volume_mounter.go:274] Currently attached volumes: [{"ID":"vol-0f1b533a98788c1ab","LocalDevice":"/dev/xvdz","AttachedTo":"i-020821d707dfc72a5","Mountpoint":"","Status":"in-use","Info":{"Description":"vol-0f1b533a98788c1ab","EtcdClusters":[{"clusterKey":"main","nodeName":"c","nodeNames":["a","b","c"]}]}}]
Jul 5 13:42:28 ip-172-24-102-238 docker[8538]: I0705 13:42:28.087925 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:42:28 ip-172-24-102-238 docker[8538]: I0705 13:42:28.104723 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:42:28 ip-172-24-102-238 docker[8538]: I0705 13:42:28.104764 8583 kube_boot.go:184] kubelet systemd service already running
Jul 5 13:42:28 ip-172-24-102-238 docker[8538]: I0705 13:42:28.120401 8583 request.go:711] Got a Retry-After %!s(int=1) response for attempt 1 to https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts
Jul 5 13:42:29 ip-172-24-102-238 docker[8538]: I0705 13:42:29.121980 8583 request.go:711] Got a Retry-After %!s(int=1) response for attempt 2 to https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts
Jul 5 13:42:30 ip-172-24-102-238 docker[8538]: I0705 13:42:30.123533 8583 request.go:711] Got a Retry-After %!s(int=1) response for attempt 3 to https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts
Jul 5 13:42:31 ip-172-24-102-238 docker[8538]: I0705 13:42:31.124961 8583 request.go:711] Got a Retry-After %!s(int=1) response for attempt 4 to https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts
Jul 5 13:42:32 ip-172-24-102-238 docker[8538]: I0705 13:42:32.126347 8583 request.go:711] Got a Retry-After %!s(int=1) response for attempt 5 to https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts
Jul 5 13:42:33 ip-172-24-102-238 docker[8538]: I0705 13:42:33.129054 8583 request.go:711] Got a Retry-After %!s(int=1) response for attempt 6 to https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts
Jul 5 13:42:34 ip-172-24-102-238 docker[8538]: I0705 13:42:34.130267 8583 request.go:711] Got a Retry-After %!s(int=1) response for attempt 7 to https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts
Jul 5 13:42:35 ip-172-24-102-238 docker[8538]: I0705 13:42:35.131649 8583 request.go:711] Got a Retry-After %!s(int=1) response for attempt 8 to https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts
Jul 5 13:42:36 ip-172-24-102-238 docker[8538]: I0705 13:42:36.133235 8583 request.go:711] Got a Retry-After %!s(int=1) response for attempt 9 to https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts
Jul 5 13:42:37 ip-172-24-102-238 docker[8538]: I0705 13:42:37.135953 8583 request.go:711] Got a Retry-After %!s(int=1) response for attempt 1 to https://127.0.0.1/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings
Jul 5 13:42:38 ip-172-24-102-238 docker[8538]: I0705 13:42:38.137417 8583 request.go:711] Got a Retry-After %!s(int=1) response for attempt 2 to https://127.0.0.1/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings
Jul 5 13:42:39 ip-172-24-102-238 docker[8538]: I0705 13:42:39.138810 8583 request.go:711] Got a Retry-After %!s(int=1) response for attempt 3 to https://127.0.0.1/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings
Jul 5 13:42:40 ip-172-24-102-238 docker[8538]: I0705 13:42:40.140239 8583 request.go:711] Got a Retry-After %!s(int=1) response for attempt 4 to https://127.0.0.1/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings
Jul 5 13:42:41 ip-172-24-102-238 docker[8538]: I0705 13:42:41.141637 8583 request.go:711] Got a Retry-After %!s(int=1) response for attempt 5 to https://127.0.0.1/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings
Jul 5 13:42:42 ip-172-24-102-238 docker[8538]: I0705 13:42:42.143039 8583 request.go:711] Got a Retry-After %!s(int=1) response for attempt 6 to https://127.0.0.1/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings
Jul 5 13:42:43 ip-172-24-102-238 docker[8538]: W0705 13:42:43.143576 8583 rbac.go:52] Error configuring RBAC: error creating service accounts: Unable to refresh the Webhook configuration: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:4001: getsockopt: connection refused
Jul 5 13:42:43 ip-172-24-102-238 docker[8538]: W0705 13:42:43.143598 8583 rbac.go:52] Error configuring RBAC: error creating cluster role bindings: unable to create RBAC clusterrolebinding: Post https://127.0.0.1/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:42:43 ip-172-24-102-238 docker[8538]: W0705 13:42:43.143609 8583 kube_boot.go:152] error initializing rbac: error creating service accounts: Unable to refresh the Webhook configuration: client: etcd cluster is unavailable or misconfigured; error #0: dial tcp 127.0.0.1:4001: getsockopt: connection refused
Jul 5 13:42:43 ip-172-24-102-238 docker[8538]: I0705 13:42:43.143617 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:42:43 ip-172-24-102-238 docker[8538]: I0705 13:42:43.143666 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:42:43 ip-172-24-102-238 docker[8538]: I0705 13:42:43.161183 8583 channels.go:48] error running channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes:
Jul 5 13:42:43 ip-172-24-102-238 docker[8538]: I0705 13:42:43.161206 8583 channels.go:49] Error: error querying kubernetes version: Get https://127.0.0.1/version: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:42:43 ip-172-24-102-238 docker[8538]: I0705 13:42:43.161220 8583 channels.go:34] apply channel output was: Error: error querying kubernetes version: Get https://127.0.0.1/version: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:42:43 ip-172-24-102-238 docker[8538]: W0705 13:42:43.161229 8583 kube_boot.go:157] error applying channel "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml": error running channels: exit status 1
Jul 5 13:43:22 ip-172-24-102-238 docker[8538]: I0705 13:43:22.444803 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: I0705 13:43:43.161648 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: W0705 13:43:43.204387 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: W0705 13:43:43.204409 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: W0705 13:43:43.204416 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: W0705 13:43:43.204423 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: W0705 13:43:43.204432 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: W0705 13:43:43.204447 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: W0705 13:43:43.204461 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: W0705 13:43:43.204487 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: I0705 13:43:43.204507 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: W0705 13:43:43.204538 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": All devices in use
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: I0705 13:43:43.204557 8583 volume_mounter.go:274] Currently attached volumes: [{"ID":"vol-0f1b533a98788c1ab","LocalDevice":"/dev/xvdz","AttachedTo":"i-020821d707dfc72a5","Mountpoint":"","Status":"in-use","Info":{"Description":"vol-0f1b533a98788c1ab","EtcdClusters":[{"clusterKey":"main","nodeName":"c","nodeNames":["a","b","c"]}]}}]
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: I0705 13:43:43.204600 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: I0705 13:43:43.210313 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: I0705 13:43:43.210341 8583 kube_boot.go:184] kubelet systemd service already running
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: W0705 13:43:43.210988 8583 rbac.go:52] Error configuring RBAC: error creating service accounts: Post https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: W0705 13:43:43.211005 8583 rbac.go:52] Error configuring RBAC: error creating cluster role bindings: unable to create RBAC clusterrolebinding: Post https://127.0.0.1/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: W0705 13:43:43.211016 8583 kube_boot.go:152] error initializing rbac: error creating service accounts: Post https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: I0705 13:43:43.211029 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: I0705 13:43:43.211072 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: I0705 13:43:43.230547 8583 channels.go:48] error running channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes:
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: I0705 13:43:43.230671 8583 channels.go:49] Error: error querying kubernetes version: Get https://127.0.0.1/version: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: I0705 13:43:43.230747 8583 channels.go:34] apply channel output was: Error: error querying kubernetes version: Get https://127.0.0.1/version: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:43:43 ip-172-24-102-238 docker[8538]: W0705 13:43:43.230799 8583 kube_boot.go:157] error applying channel "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml": error running channels: exit status 1
Jul 5 13:44:22 ip-172-24-102-238 docker[8538]: I0705 13:44:22.446730 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: I0705 13:44:43.231338 8583 aws_volume.go:72] AWS API Request: ec2/DescribeVolumes
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: W0705 13:44:43.289448 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Environment=Live
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: W0705 13:44:43.289475 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Group=Kubernetes
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: W0705 13:44:43.289488 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Squad=delta-force
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: W0705 13:44:43.289500 8583 aws_volume.go:232] unknown tag on volume "vol-0f1b533a98788c1ab": Tribe=Sioux
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: W0705 13:44:43.289514 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Squad=delta-force
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: W0705 13:44:43.289525 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Group=Kubernetes
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: W0705 13:44:43.289547 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Environment=Live
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: W0705 13:44:43.289576 8583 aws_volume.go:232] unknown tag on volume "vol-0d86190f17f570d98": Tribe=Sioux
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: I0705 13:44:43.289595 8583 volume_mounter.go:255] Trying to mount master volume: "vol-0d86190f17f570d98"
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: W0705 13:44:43.289615 8583 volume_mounter.go:260] Error attaching volume "vol-0d86190f17f570d98": All devices in use
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: I0705 13:44:43.289632 8583 volume_mounter.go:274] Currently attached volumes: [{"ID":"vol-0f1b533a98788c1ab","LocalDevice":"/dev/xvdz","AttachedTo":"i-020821d707dfc72a5","Mountpoint":"","Status":"in-use","Info":{"Description":"vol-0f1b533a98788c1ab","EtcdClusters":[{"clusterKey":"main","nodeName":"c","nodeNames":["a","b","c"]}]}}]
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: I0705 13:44:43.289678 8583 kube_boot.go:169] ensuring that kubelet systemd service is running
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: I0705 13:44:43.316124 8583 kube_boot.go:182] 'systemctl status kubelet' output:
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: I0705 13:44:43.316164 8583 kube_boot.go:184] kubelet systemd service already running
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: W0705 13:44:43.316880 8583 rbac.go:52] Error configuring RBAC: error creating service accounts: Post https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: W0705 13:44:43.316902 8583 rbac.go:52] Error configuring RBAC: error creating cluster role bindings: unable to create RBAC clusterrolebinding: Post https://127.0.0.1/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: W0705 13:44:43.316913 8583 kube_boot.go:152] error initializing rbac: error creating service accounts: Post https://127.0.0.1/api/v1/namespaces/kube-system/serviceaccounts: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: I0705 13:44:43.316926 8583 channels.go:31] checking channel: "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml"
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: I0705 13:44:43.316974 8583 channels.go:45] Running command: channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: I0705 13:44:43.339584 8583 channels.go:48] error running channels apply channel s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml --v=4 --yes:
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: I0705 13:44:43.339616 8583 channels.go:49] Error: error querying kubernetes version: Get https://127.0.0.1/version: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: I0705 13:44:43.339638 8583 channels.go:34] apply channel output was: Error: error querying kubernetes version: Get https://127.0.0.1/version: dial tcp 127.0.0.1:443: getsockopt: connection refused
Jul 5 13:44:43 ip-172-24-102-238 docker[8538]: W0705 13:44:43.339654 8583 kube_boot.go:157] error applying channel "s3://kops-s3-bucket/production.mydomain.io/addons/bootstrap-channel.yaml": error running channels: exit status 1
Jul 5 13:45:22 ip-172-24-102-238 docker[8538]: I0705 13:45:22.449147 8583 kube_dns.go:36] Creating DNS record: etcd-c.internal.production.mydomain.io => [172.24.102.238]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment