使用kubernetes创建容器一直处于ContainerCreating状态的原因查找与解决

最近刚刚入手研究kubernetes,运行容器的时候,发现一直处于ContainerCreating状态,悲了个催,刚入手就遇到了点麻烦,下面来讲讲如何查找问题及解决的

运行容器命令:

[root@master-149 ~]# kubectl run my-alpine --image=alpine --replicas=2 ping www.baidu.com

查看pods状态

[root@master-149 ~]# kubectl get pods
NAME                         READY     STATUS              RESTARTS   AGE
my-alpine-2150523991-knzcx   0/1       ContainerCreating   0          6m
my-alpine-2150523991-lmvv5   0/1       ContainerCreating   0          6m

一直处于ContainerCreating状态,开始查找原因
执行如下命令:

[root@master-149 ~]# kubectl describe pod my-alpine
Name:		my-alpine-2150523991-knzcx
Namespace:	default
Node:		node-150/192.168.10.150
Start Time:	Sat, 19 Nov 2016 18:20:52 +0800
Labels:		pod-template-hash=2150523991,run=my-alpine
Status:		Pending
IP:		
Controllers:	ReplicaSet/my-alpine-2150523991
Containers:
  my-alpine:
    Container ID:	
    Image:		alpine
    Image ID:		
    Port:		
    Args:
      ping
      www.baidu.com
    QoS Tier:
      cpu:		BestEffort
      memory:		BestEffort
    State:		Waiting
      Reason:		ContainerCreating
    Ready:		False
    Restart Count:	0
    Environment Variables:
Conditions:
  Type		Status
  Ready 	False 
No volumes.
Events:
  FirstSeen	LastSeen	Count	From			SubobjectPath	Type		Reason		Message
  ---------	--------	-----	----			-------------	--------	------		-------
  7m		7m		1	{default-scheduler }			Normal		Scheduled	Successfully assigned my-alpine-2150523991-knzcx to node-150
  6m		6m		1	{kubelet node-150}			Warning		FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "POD" with ErrImagePull: "image pull failed for gcr.io/google_containers/pause:2.0, this may be because there are no credentials on this request.  details: (unable to ping registry endpoint https://gcr.io/v0/\nv2 ping attempt failed with error: Get https://gcr.io/v2/: dial tcp 64.233.189.82:443: getsockopt: connection refused\n v1 ping attempt failed with error: Get https://gcr.io/v1/_ping: dial tcp 64.233.189.82:443: getsockopt: connection refused)"

  4m	47s	3	{kubelet node-150}		Warning	FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "POD" with ErrImagePull: "image pull failed for gcr.io/google_containers/pause:2.0, this may be because there are no credentials on this request.  details: (unable to ping registry endpoint https://gcr.io/v0/\nv2 ping attempt failed with error: Get https://gcr.io/v2/: dial tcp 74.125.204.82:443: getsockopt: connection refused\n v1 ping attempt failed with error: Get https://gcr.io/v1/_ping: dial tcp 74.125.204.82:443: getsockopt: connection refused)"

  4m	8s	6	{kubelet node-150}		Warning	FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "POD" with ImagePullBackOff: "Back-off pulling image \"gcr.io/google_containers/pause:2.0\""


Name:		my-alpine-2150523991-lmvv5
Namespace:	default
Node:		node-150/192.168.10.150
Start Time:	Sat, 19 Nov 2016 18:20:52 +0800
Labels:		pod-template-hash=2150523991,run=my-alpine
Status:		Pending
IP:		
Controllers:	ReplicaSet/my-alpine-2150523991
Containers:
  my-alpine:
    Container ID:	
    Image:		alpine
    Image ID:		
    Port:		
    Args:
      ping
      www.baidu.com
    QoS Tier:
      cpu:		BestEffort
      memory:		BestEffort
    State:		Waiting
      Reason:		ContainerCreating
    Ready:		False
    Restart Count:	0
    Environment Variables:
Conditions:
  Type		Status
  Ready 	False 
No volumes.
Events:
  FirstSeen	LastSeen	Count	From			SubobjectPath	Type		Reason		Message
  ---------	--------	-----	----			-------------	--------	------		-------
  7m		7m		1	{default-scheduler }			Normal		Scheduled	Successfully assigned my-alpine-2150523991-lmvv5 to node-150
  5m		1m		3	{kubelet node-150}			Warning		FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "POD" with ErrImagePull: "image pull failed for gcr.io/google_containers/pause:2.0, this may be because there are no credentials on this request.  details: (unable to ping registry endpoint https://gcr.io/v0/\nv2 ping attempt failed with error: Get https://gcr.io/v2/: dial tcp 74.125.204.82:443: getsockopt: connection refused\n v1 ping attempt failed with error: Get https://gcr.io/v1/_ping: dial tcp 74.125.204.82:443: getsockopt: connection refused)"

  3m	1m	4	{kubelet node-150}		Warning	FailedSync	Error syncing pod, skipping: failed to "StartContainer" for "POD" with ImagePullBackOff: "Back-off pulling image \"gcr.io/google_containers/pause:2.0\""

其中:

Error syncing pod, skipping: failed to "StartContainer" for "POD" with ErrImagePull: "image pull failed for gcr.io/google_containers/pause:2.0, this may be because there are no credentials on this request.  details: (unable to ping registry endpoint https://gcr.io/v0/\nv2 ping attempt failed with error: Get https://gcr.io/v2/: dial tcp 74.125.204.82:443: getsockopt: connection refused\n v1 ping attempt failed with error: Get https://gcr.io/v1/_ping: dial tcp 74.125.204.82:443: getsockopt: connection refused)

不难看出,无法访问到gcr.io
有如下解决办法:

  1. 翻墙
  2. 修改hosts文件(这里我用的是“61.91.161.217  gcr.io”,但是可能会失效)
  3. 从其他源下载容器“pause:2.0”,然后打tag为“gcr.io/google_containers/pause:2.0”
问题描述: mysql-3173324660 创建一直处于 ContainerCreating状态 [root@user1-group1-295 ceph]# kubectl get pods NAME READY STATUS RESTARTS AGE mysql-3173324660-jd8jw 0/1 ContainerCreating 0 1m mysql-sonar-1457409014-hsbwg 1/1 Running 4 117d kubectl describe po/mysql-3173324660-jd8jw 报挂卷失败: ![图片说明](https://img-ask.csdn.net/upload/201709/04/1504506802_346149.png) Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 2m 2m 1 default-scheduler Normal Scheduled Successfully assigned mysql-3173324660-jd8jw to user1-group1-296 34s 34s 1 kubelet, user1-group1-296 Warning FailedMount Unable to mount volumes for pod "mysql-3173324660-jd8jw_default(cc915019-9139-11e7-b200-fa163e9b5d77)": timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-3173324660-jd8jw". list of unattached/unmounted volumes=[mysql-persistent-storage] 34s 34s 1 kubelet, user1-group1-296 Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-3173324660-jd8jw". list of unattached/unmounted volumes=[mysql-persistent-storage] 2m 24s 9 kubelet, user1-group1-296 Warning FailedMount MountVolume.SetUp failed for volume "kubernetes.io/rbd/cc915019-9139-11e7-b200-fa163e9b5d77-mysql-pv" (spec.Name: "mysql-pv") pod "cc915019-9139-11e7-b200-fa163e9b5d77" (UID: "cc915019-9139-11e7-b200-fa163e9b5d77") with: rbd: failed to mount rbd volume /dev/rbd4 [xfs] to /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/rbd-image-imagemysql1, error executable file not found in $PATH 按照楼下朋友指点需要查看的几个点,rdb正常,pv和pvc都是bound: [root@user1-group1-295 ceph]# rbd list caasmysqlimage ceph-image ceph-image-new foo image-mysql-wb1 imagemysql1 imagewb imagewb1 imagewbb1 imagewbb2 jdk-image mysql-image mysql-image-1 [root@user1-group1-295 ceph]# kubectl get pv NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE mysql-pv 2Gi RWO Recycle Bound default/mysql-pv-claim 12m [root@user1-group1-295 ceph]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE mysql-pv-claim Bound mysql-pv 2Gi RWO 12m [root@user1-group1-295 ceph]# ======================================================= Sep 04 06:25:03 左右创建的pod: 创建pod失败时候的日志如下: [root@user1-group1-295 ceph]# journalctl -u kubelet -f -- Logs begin at Tue 2017-07-11 06:42:53 UTC. -- Sep 04 06:23:21 user1-group1-295.novalocal kubelet[15976]: I0904 06:23:21.226456 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/0af2b564-3b6c-11e7-81c5-fa163e9b5d77-default-token-r5gl6" (spec.Name: "default-token-r5gl6") pod "0af2b564-3b6c-11e7-81c5-fa163e9b5d77" (UID: "0af2b564-3b6c-11e7-81c5-fa163e9b5d77"). Sep 04 06:23:22 user1-group1-295.novalocal kubelet[15976]: I0904 06:23:22.229211 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fe04c9ea-56f9-11e7-b76e-fa163e9b5d77-traefik-ingress-controller-token-xmzl4" (spec.Name: "traefik-ingress-controller-token-xmzl4") pod "fe04c9ea-56f9-11e7-b76e-fa163e9b5d77" (UID: "fe04c9ea-56f9-11e7-b76e-fa163e9b5d77"). Sep 04 06:23:23 user1-group1-295.novalocal kubelet[15976]: I0904 06:23:23.736718 16005 server.go:778] GET /metrics: (107.402434ms) 200 [[Go-http-client/1.1] 172.17.0.162:48938] Sep 04 06:23:33 user1-group1-295.novalocal kubelet[15976]: I0904 06:23:33.740537 16005 server.go:778] GET /metrics: (111.112116ms) 200 [[Go-http-client/1.1] 172.17.0.162:48960] Sep 04 06:23:34 user1-group1-295.novalocal kubelet[15976]: I0904 06:23:34.171001 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/432189b6-57fe-11e7-b76e-fa163e9b5d77-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "432189b6-57fe-11e7-b76e-fa163e9b5d77" (UID: "432189b6-57fe-11e7-b76e-fa163e9b5d77"). Sep 04 06:23:43 user1-group1-295.novalocal kubelet[15976]: I0904 06:23:43.736004 16005 server.go:778] GET /metrics: (106.587002ms) 200 [[Go-http-client/1.1] 172.17.0.162:48982] Sep 04 06:23:44 user1-group1-295.novalocal kubelet[15976]: I0904 06:23:44.201917 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77" (UID: "2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77"). Sep 04 06:23:50 user1-group1-295.novalocal kubelet[15976]: I0904 06:23:50.219064 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fcfb6601-360d-11e7-a88c-fa163ed46388-default-token-9prvk" (spec.Name: "default-token-9prvk") pod "fcfb6601-360d-11e7-a88c-fa163ed46388" (UID: "fcfb6601-360d-11e7-a88c-fa163ed46388"). Sep 04 06:23:53 user1-group1-295.novalocal kubelet[15976]: I0904 06:23:53.227001 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/5abc1b0a-8617-11e7-b200-fa163e9b5d77-default-token-7x6qd" (spec.Name: "default-token-7x6qd") pod "5abc1b0a-8617-11e7-b200-fa163e9b5d77" (UID: "5abc1b0a-8617-11e7-b200-fa163e9b5d77"). Sep 04 06:23:53 user1-group1-295.novalocal kubelet[15976]: I0904 06:23:53.743604 16005 server.go:778] GET /metrics: (114.703271ms) 200 [[Go-http-client/1.1] 172.17.0.162:49004] Sep 04 06:24:03 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:03.732337 16005 server.go:778] GET /metrics: (103.103667ms) 200 [[Go-http-client/1.1] 172.17.0.162:49026] Sep 04 06:24:05 user1-group1-295.novalocal kubelet[15976]: W0904 06:24:05.500408 16005 container_gc.go:305] Failed to remove container "/k8s_kube-controller-manager.b0c85041_kube-controller-manager-user1-group1-295_kube-system_24921f1c706c18a694bb4a7942186800_c66f920e": Error response from daemon: {"message":"Driver overlay failed to remove root filesystem d992e1cb757ed839b70a0f896bf30d78c35a26cdab41ef7db30a59ad44bada5c: remove /var/lib/docker/overlay/ea36794bdd3e2509cd8d77444133e0f64a94d33f9e019d3826a9c4db10015453/merged: device or resource busy"} Sep 04 06:24:06 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:06.262225 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fccdbfd7-360d-11e7-a88c-fa163ed46388-default-token-9prvk" (spec.Name: "default-token-9prvk") pod "fccdbfd7-360d-11e7-a88c-fa163ed46388" (UID: "fccdbfd7-360d-11e7-a88c-fa163ed46388"). Sep 04 06:24:11 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:11.175838 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/05c2105e-4b6a-11e7-ad4d-fa163ed46388-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "05c2105e-4b6a-11e7-ad4d-fa163ed46388" (UID: "05c2105e-4b6a-11e7-ad4d-fa163ed46388"). Sep 04 06:24:13 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:13.740341 16005 server.go:778] GET /metrics: (110.717336ms) 200 [[Go-http-client/1.1] 172.17.0.162:49048] Sep 04 06:24:15 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:15.989430 16005 container_manager_linux.go:439] Discovered runtime cgroups name: /system.slice/docker.service Sep 04 06:24:23 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:23.740110 16005 server.go:778] GET /metrics: (110.727464ms) 200 [[Go-http-client/1.1] 172.17.0.162:49070] Sep 04 06:24:33 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:33.736772 16005 server.go:778] GET /metrics: (107.362082ms) 200 [[Go-http-client/1.1] 172.17.0.162:49092] Sep 04 06:24:34 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:34.247795 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/f1b4decb-33af-11e7-a587-fa163ed46388-default-token-r5gl6" (spec.Name: "default-token-r5gl6") pod "f1b4decb-33af-11e7-a587-fa163ed46388" (UID: "f1b4decb-33af-11e7-a587-fa163ed46388"). Sep 04 06:24:42 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:42.169852 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fe04c9ea-56f9-11e7-b76e-fa163e9b5d77-traefik-ingress-controller-token-xmzl4" (spec.Name: "traefik-ingress-controller-token-xmzl4") pod "fe04c9ea-56f9-11e7-b76e-fa163e9b5d77" (UID: "fe04c9ea-56f9-11e7-b76e-fa163e9b5d77"). Sep 04 06:24:43 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:43.739418 16005 server.go:778] GET /metrics: (110.330416ms) 200 [[Go-http-client/1.1] 172.17.0.162:49114] Sep 04 06:24:48 user1-group1-295.novalocal kubelet[15976]: E0904 06:24:48.725052 16005 fsHandler.go:121] failed to collect filesystem stats - rootDiskErr: du command failed on /var/lib/docker/overlay/32c3a6427d80d828ac8692f5f308b021fc8a2a28be6c131525eff5ef909c41dc with output stdout: 2730096 /var/lib/docker/overlay/32c3a6427d80d828ac8692f5f308b021fc8a2a28be6c131525eff5ef909c41dc Sep 04 06:24:48 user1-group1-295.novalocal kubelet[15976]: , stderr: du: cannot access '/var/lib/docker/overlay/32c3a6427d80d828ac8692f5f308b021fc8a2a28be6c131525eff5ef909c41dc/merged/proc/30247/task/30247/fd/4': No such file or directory Sep 04 06:24:48 user1-group1-295.novalocal kubelet[15976]: du: cannot access '/var/lib/docker/overlay/32c3a6427d80d828ac8692f5f308b021fc8a2a28be6c131525eff5ef909c41dc/merged/proc/30247/task/30247/fdinfo/4': No such file or directory Sep 04 06:24:48 user1-group1-295.novalocal kubelet[15976]: du: cannot access '/var/lib/docker/overlay/32c3a6427d80d828ac8692f5f308b021fc8a2a28be6c131525eff5ef909c41dc/merged/proc/30247/fd/4': No such file or directory Sep 04 06:24:48 user1-group1-295.novalocal kubelet[15976]: du: cannot access '/var/lib/docker/overlay/32c3a6427d80d828ac8692f5f308b021fc8a2a28be6c131525eff5ef909c41dc/merged/proc/30247/fdinfo/4': No such file or directory Sep 04 06:24:48 user1-group1-295.novalocal kubelet[15976]: - exit status 1, rootInodeErr: <nil>, extraDiskErr: <nil> Sep 04 06:24:48 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:48.725072 16005 fsHandler.go:131] du and find on following dirs took 1.057182281s: [/var/lib/docker/overlay/32c3a6427d80d828ac8692f5f308b021fc8a2a28be6c131525eff5ef909c41dc /var/lib/docker/containers/7d9aa1bfde24865c84c5a7e8577965741ec339379213af49f400deef4592803a] Sep 04 06:24:50 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:50.194610 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/0af2b564-3b6c-11e7-81c5-fa163e9b5d77-default-token-r5gl6" (spec.Name: "default-token-r5gl6") pod "0af2b564-3b6c-11e7-81c5-fa163e9b5d77" (UID: "0af2b564-3b6c-11e7-81c5-fa163e9b5d77"). Sep 04 06:24:53 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:53.741418 16005 server.go:778] GET /metrics: (112.212086ms) 200 [[Go-http-client/1.1] 172.17.0.162:49136] Sep 04 06:24:59 user1-group1-295.novalocal kubelet[15976]: I0904 06:24:59.218706 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/5abc1b0a-8617-11e7-b200-fa163e9b5d77-default-token-7x6qd" (spec.Name: "default-token-7x6qd") pod "5abc1b0a-8617-11e7-b200-fa163e9b5d77" (UID: "5abc1b0a-8617-11e7-b200-fa163e9b5d77"). Sep 04 06:25:02 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:02.226260 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/432189b6-57fe-11e7-b76e-fa163e9b5d77-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "432189b6-57fe-11e7-b76e-fa163e9b5d77" (UID: "432189b6-57fe-11e7-b76e-fa163e9b5d77"). Sep 04 06:25:03 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:03.744718 16005 server.go:778] GET /metrics: (115.63146ms) 200 [[Go-http-client/1.1] 172.17.0.162:49158] Sep 04 06:25:05 user1-group1-295.novalocal kubelet[15976]: W0904 06:25:05.623008 16005 container_gc.go:305] Failed to remove container "/k8s_kube-controller-manager.b0c85041_kube-controller-manager-user1-group1-295_kube-system_24921f1c706c18a694bb4a7942186800_c66f920e": Error response from daemon: {"message":"Driver overlay failed to remove root filesystem d992e1cb757ed839b70a0f896bf30d78c35a26cdab41ef7db30a59ad44bada5c: remove /var/lib/docker/overlay/ea36794bdd3e2509cd8d77444133e0f64a94d33f9e019d3826a9c4db10015453/merged: device or resource busy"} Sep 04 06:25:06 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:06.243201 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77" (UID: "2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77"). Sep 04 06:25:12 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:12.260590 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fcfb6601-360d-11e7-a88c-fa163ed46388-default-token-9prvk" (spec.Name: "default-token-9prvk") pod "fcfb6601-360d-11e7-a88c-fa163ed46388" (UID: "fcfb6601-360d-11e7-a88c-fa163ed46388"). Sep 04 06:25:13 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:13.737764 16005 server.go:778] GET /metrics: (108.38973ms) 200 [[Go-http-client/1.1] 172.17.0.162:49180] Sep 04 06:25:23 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:23.727904 16005 server.go:778] GET /metrics: (98.765996ms) 200 [[Go-http-client/1.1] 172.17.0.162:49202] Sep 04 06:25:24 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:24.194371 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/05c2105e-4b6a-11e7-ad4d-fa163ed46388-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "05c2105e-4b6a-11e7-ad4d-fa163ed46388" (UID: "05c2105e-4b6a-11e7-ad4d-fa163ed46388"). Sep 04 06:25:33 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:33.738025 16005 server.go:778] GET /metrics: (109.050426ms) 200 [[Go-http-client/1.1] 172.17.0.162:49224] Sep 04 06:25:36 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:36.226919 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fccdbfd7-360d-11e7-a88c-fa163ed46388-default-token-9prvk" (spec.Name: "default-token-9prvk") pod "fccdbfd7-360d-11e7-a88c-fa163ed46388" (UID: "fccdbfd7-360d-11e7-a88c-fa163ed46388"). Sep 04 06:25:43 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:43.737571 16005 server.go:778] GET /metrics: (108.474848ms) 200 [[Go-http-client/1.1] 172.17.0.162:49246] Sep 04 06:25:45 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:45.903049 16005 image_gc_manager.go:270] [imageGCManager]: Disk usage on "/dev/vda1" (/var/lib/docker/overlay) is at 94% which is over the high threshold (90%). Trying to free 7021084672 bytes Sep 04 06:25:45 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:45.916512 16005 image_gc_manager.go:335] [imageGCManager]: Removing image "sha256:573ddcad1ff5243330f172efbf5d44a1c2c35abcae3c2839673817dff41764a6" to free 217108251 bytes Sep 04 06:25:45 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:45.918875 16005 image_gc_manager.go:335] [imageGCManager]: Removing image "sha256:54e0498c3dadcfc3036086232145c96d334295a318c12cf8af025bf59ccd7d4e" to free 987088941 bytes Sep 04 06:25:45 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:45.922480 16005 image_gc_manager.go:335] [imageGCManager]: Removing image "sha256:c1951085bcc973dcf859ca1842d049c2e394092c06a0dc8d5aa47f4b8b55de80" to free 42634444 bytes Sep 04 06:25:45 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:45.924666 16005 image_gc_manager.go:335] [imageGCManager]: Removing image "sha256:958a7ae9e56979be256796dabd5845c704f784cd422734184999cf91f24c2547" to free 109368581 bytes Sep 04 06:25:45 user1-group1-295.novalocal kubelet[15976]: E0904 06:25:45.927024 16005 kubelet.go:1165] Image garbage collection failed: wanted to free 7021084672, but freed 0 space with errors in image deletion: [Error response from daemon: {"message":"conflict: unable to delete 573ddcad1ff5 (cannot be forced) - image is being used by running container a0abb2be0dfd"}, Error response from daemon: {"message":"conflict: unable to delete 54e0498c3dad (must be forced) - image is being used by stopped container 675afc7fd4b2"}, Error response from daemon: {"message":"conflict: unable to delete c1951085bcc9 (cannot be forced) - image is being used by running container 0ce4eb29420d"}, Error response from daemon: {"message":"conflict: unable to delete 958a7ae9e569 (must be forced) - image is referenced in multiple repositories"}] Sep 04 06:25:47 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:47.257883 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/f1b4decb-33af-11e7-a587-fa163ed46388-default-token-r5gl6" (spec.Name: "default-token-r5gl6") pod "f1b4decb-33af-11e7-a587-fa163ed46388" (UID: "f1b4decb-33af-11e7-a587-fa163ed46388"). Sep 04 06:25:53 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:53.733853 16005 server.go:778] GET /metrics: (104.78036ms) 200 [[Go-http-client/1.1] 172.17.0.162:49268] Sep 04 06:25:57 user1-group1-295.novalocal kubelet[15976]: I0904 06:25:57.186726 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/0af2b564-3b6c-11e7-81c5-fa163e9b5d77-default-token-r5gl6" (spec.Name: "default-token-r5gl6") pod "0af2b564-3b6c-11e7-81c5-fa163e9b5d77" (UID: "0af2b564-3b6c-11e7-81c5-fa163e9b5d77"). Sep 04 06:26:03 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:03.737476 16005 server.go:778] GET /metrics: (108.293704ms) 200 [[Go-http-client/1.1] 172.17.0.162:49290] Sep 04 06:26:05 user1-group1-295.novalocal kubelet[15976]: W0904 06:26:05.656712 16005 container_gc.go:305] Failed to remove container "/k8s_kube-controller-manager.b0c85041_kube-controller-manager-user1-group1-295_kube-system_24921f1c706c18a694bb4a7942186800_c66f920e": Error response from daemon: {"message":"Driver overlay failed to remove root filesystem d992e1cb757ed839b70a0f896bf30d78c35a26cdab41ef7db30a59ad44bada5c: remove /var/lib/docker/overlay/ea36794bdd3e2509cd8d77444133e0f64a94d33f9e019d3826a9c4db10015453/merged: device or resource busy"} Sep 04 06:26:06 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:06.213274 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fe04c9ea-56f9-11e7-b76e-fa163e9b5d77-traefik-ingress-controller-token-xmzl4" (spec.Name: "traefik-ingress-controller-token-xmzl4") pod "fe04c9ea-56f9-11e7-b76e-fa163e9b5d77" (UID: "fe04c9ea-56f9-11e7-b76e-fa163e9b5d77"). Sep 04 06:26:13 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:13.741089 16005 server.go:778] GET /metrics: (111.980692ms) 200 [[Go-http-client/1.1] 172.17.0.162:49312] Sep 04 06:26:20 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:20.255745 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fcfb6601-360d-11e7-a88c-fa163ed46388-default-token-9prvk" (spec.Name: "default-token-9prvk") pod "fcfb6601-360d-11e7-a88c-fa163ed46388" (UID: "fcfb6601-360d-11e7-a88c-fa163ed46388"). Sep 04 06:26:23 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:23.734496 16005 server.go:778] GET /metrics: (102.819982ms) 200 [[Go-http-client/1.1] 172.17.0.162:49334] Sep 04 06:26:24 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:24.274689 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/432189b6-57fe-11e7-b76e-fa163e9b5d77-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "432189b6-57fe-11e7-b76e-fa163e9b5d77" (UID: "432189b6-57fe-11e7-b76e-fa163e9b5d77"). Sep 04 06:26:28 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:28.186208 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/5abc1b0a-8617-11e7-b200-fa163e9b5d77-default-token-7x6qd" (spec.Name: "default-token-7x6qd") pod "5abc1b0a-8617-11e7-b200-fa163e9b5d77" (UID: "5abc1b0a-8617-11e7-b200-fa163e9b5d77"). Sep 04 06:26:33 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:33.200294 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77" (UID: "2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77"). Sep 04 06:26:33 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:33.736186 16005 server.go:778] GET /metrics: (106.916768ms) 200 [[Go-http-client/1.1] 172.17.0.162:49356] Sep 04 06:26:42 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:42.224907 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/05c2105e-4b6a-11e7-ad4d-fa163ed46388-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "05c2105e-4b6a-11e7-ad4d-fa163ed46388" (UID: "05c2105e-4b6a-11e7-ad4d-fa163ed46388"). Sep 04 06:26:43 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:43.738326 16005 server.go:778] GET /metrics: (109.178882ms) 200 [[Go-http-client/1.1] 172.17.0.162:49378] Sep 04 06:26:46 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:46.235835 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fccdbfd7-360d-11e7-a88c-fa163ed46388-default-token-9prvk" (spec.Name: "default-token-9prvk") pod "fccdbfd7-360d-11e7-a88c-fa163ed46388" (UID: "fccdbfd7-360d-11e7-a88c-fa163ed46388"). Sep 04 06:26:53 user1-group1-295.novalocal kubelet[15976]: I0904 06:26:53.734508 16005 server.go:778] GET /metrics: (105.402837ms) 200 [[Go-http-client/1.1] 172.17.0.162:49400] Sep 04 06:27:03 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:03.741988 16005 server.go:778] GET /metrics: (112.954759ms) 200 [[Go-http-client/1.1] 172.17.0.162:49422] Sep 04 06:27:05 user1-group1-295.novalocal kubelet[15976]: W0904 06:27:05.784294 16005 container_gc.go:305] Failed to remove container "/k8s_kube-controller-manager.b0c85041_kube-controller-manager-user1-group1-295_kube-system_24921f1c706c18a694bb4a7942186800_c66f920e": Error response from daemon: {"message":"Driver overlay failed to remove root filesystem d992e1cb757ed839b70a0f896bf30d78c35a26cdab41ef7db30a59ad44bada5c: remove /var/lib/docker/overlay/ea36794bdd3e2509cd8d77444133e0f64a94d33f9e019d3826a9c4db10015453/merged: device or resource busy"} Sep 04 06:27:09 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:09.205529 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fe04c9ea-56f9-11e7-b76e-fa163e9b5d77-traefik-ingress-controller-token-xmzl4" (spec.Name: "traefik-ingress-controller-token-xmzl4") pod "fe04c9ea-56f9-11e7-b76e-fa163e9b5d77" (UID: "fe04c9ea-56f9-11e7-b76e-fa163e9b5d77"). Sep 04 06:27:13 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:13.739225 16005 server.go:778] GET /metrics: (109.968574ms) 200 [[Go-http-client/1.1] 172.17.0.162:49444] Sep 04 06:27:17 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:17.227946 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/0af2b564-3b6c-11e7-81c5-fa163e9b5d77-default-token-r5gl6" (spec.Name: "default-token-r5gl6") pod "0af2b564-3b6c-11e7-81c5-fa163e9b5d77" (UID: "0af2b564-3b6c-11e7-81c5-fa163e9b5d77"). Sep 04 06:27:17 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:17.228615 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/f1b4decb-33af-11e7-a587-fa163ed46388-default-token-r5gl6" (spec.Name: "default-token-r5gl6") pod "f1b4decb-33af-11e7-a587-fa163ed46388" (UID: "f1b4decb-33af-11e7-a587-fa163ed46388"). Sep 04 06:27:23 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:23.729005 16005 server.go:778] GET /metrics: (99.830606ms) 200 [[Go-http-client/1.1] 172.17.0.162:49466] Sep 04 06:27:25 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:25.252704 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/432189b6-57fe-11e7-b76e-fa163e9b5d77-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "432189b6-57fe-11e7-b76e-fa163e9b5d77" (UID: "432189b6-57fe-11e7-b76e-fa163e9b5d77"). Sep 04 06:27:27 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:27.258375 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/fcfb6601-360d-11e7-a88c-fa163ed46388-default-token-9prvk" (spec.Name: "default-token-9prvk") pod "fcfb6601-360d-11e7-a88c-fa163ed46388" (UID: "fcfb6601-360d-11e7-a88c-fa163ed46388"). Sep 04 06:27:33 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:33.737406 16005 server.go:778] GET /metrics: (107.868122ms) 200 [[Go-http-client/1.1] 172.17.0.162:49488] Sep 04 06:27:40 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:40.196971 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/5abc1b0a-8617-11e7-b200-fa163e9b5d77-default-token-7x6qd" (spec.Name: "default-token-7x6qd") pod "5abc1b0a-8617-11e7-b200-fa163e9b5d77" (UID: "5abc1b0a-8617-11e7-b200-fa163e9b5d77"). Sep 04 06:27:40 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:40.197045 16005 operation_generator.go:597] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77-default-token-t9rq2" (spec.Name: "default-token-t9rq2") pod "2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77" (UID: "2f7c9bd1-5afe-11e7-b76e-fa163e9b5d77"). Sep 04 06:27:43 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:43.737770 16005 server.go:778] GET /metrics: (108.444824ms) 200 [[Go-http-client/1.1] 172.17.0.162:49510] Sep 04 06:27:53 user1-group1-295.novalocal kubelet[15976]: I0904 06:27:53.728799 16005 server.go:778] GET /metrics: (99.267527ms) 200 [[Go-http-client/1.1] 172.17.0.162:49532]
©️2020 CSDN 皮肤主题: 大白 设计师:CSDN官方博客 返回首页