I searched around a bit a while back and found stuff about using loop device, which we are not, and using fstrim to free space. I ended up just rebuilding my kubernetes working nodes to clear the space back up last time and switching from XFS to EXT4 after reading some things about potential issues with XFS releasing space with thin provisioning. We appear to be creeping back up to the same point now, though, and I am wondering what is going on.
My suspicion is the device mapper is being used to create thin volumes and then they are being orphaned, but I can't see them using the dmsetup command. This problem seems to be unique to the kubernetes worker nodes, I guess it could be something with Rancher and Kubernetes and how it is interacting with Docker?
docker_host:~ # cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.0 (Maipo)
docker_host:~ # docker info
Containers: 101
Running: 83
Paused: 0
Stopped: 18
Images: 56
Server Version: 17.03.2-ce
Storage Driver: devicemapper
Pool Name: k8s_vol1-docker_thinpool
Pool Blocksize: 131.1 kB
Base Device Size: 10.74 GB
Backing Filesystem: ext4
Data file:
Metadata file:
Data Space Used: 48.32 GB
Data Space Total: 214.7 GB
Data Space Available: 166.4 GB
Metadata Space Used: 862.3 MB
Metadata Space Total: 2.252 GB
Metadata Space Available: 1.39 GB
Thin Pool Minimum Free Space: 21.47 GB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-514.10.2.el7.x86_64
Operating System: Red Hat Enterprise Linux Server 7.0 (Maipo)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 62.76 GiB
Name: docker_host
ID: *******
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
###.###.###.###:####
127.0.0.0/8
Live Restore Enabled: false
docker_host:~ # lvs
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
docker_containers k8s_vol1 -wi-ao---- 50.00g
docker_thinpool k8s_vol1 twi-aotz-- 200.00g 22.50 38.28
kubelet k8s_vol1 -wi-ao---- 10.00g
optlv k8s_vol1 -wi-ao---- 10.00g
rook_ceph k8s_vol1 -wi-ao---- 200.00g
root os -wi-ao---- 10.00g
swap os -wi-ao---- 2.00g
tmp os -wi-ao---- 2.00g
var os -wi-ao---- 5.00g
docker_host:~ # pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 os lvm2 a-- 29.47g 10.47g
/dev/sdb k8s_vol1 lvm2 a-- 1000.00g 527.80g
docker_host:~ # vgs
VG #PV #LV #SN Attr VSize VFree
k8s_vol1 1 5 0 wz--n- 1000.00g 527.80g
os 1 4 0 wz--n- 29.47g 10.47g
docker_host:~ # dmsetup ls
docker-253:3-101498017-72bcd76347e26fed9b0332842d42dab557262ef56d6abefb6faff3247efa1cf8 (253:79)
docker-253:3-101498017-cb9540890830b416bbc03da01f68a446604b55682b3e5e82740eeb37a8e8fb24 (253:12)
docker-253:3-101498017-692db6edc24c878f9caa45c89c930120e6c0b3897f62e468aa1ba56482743a8f (253:95)
docker-253:3-101498017-c31f527f16b1d4126a0b18c22b54ec1121d83f3f3a13a38f218a8956b700202e (253:74)
docker-253:3-101498017-2ca06d5355e51a41695b3a407c141d540ce3eaebfaf79d55c39e7bf5b14ec0fd (253:88)
docker-253:3-101498017-1e91e539f9a112909e7439bca13629864a2d7f1797c3cc42944dc01a6ed07001 (253:39)
docker-253:3-101498017-b72394e1b06d75059d7696e41cb92e2904dfb68c66b4d44bb7e5710c9d078734 (253:27)
docker-253:3-101498017-3bb55da27654a22e832141e1ef8f943ac22b3f06f96315033a99529b43a1fd7e (253:90)
docker-253:3-101498017-a2e755e978eeda8788df96e95eb811e40f1e747aa08b9dbdd3bd9affcbf4765f (253:14)
os-tmp (253:7)
docker-253:3-101498017-58aae6188dca08a773f7b0dd3c051ca315006054ae87a6864488750419183451 (253:59)
docker-253:3-101498017-7d23c8c042d3eca16b215cd90a244d931fa8c1f7c0d9f6354c8aba1442d185a9 (253:62)
docker-253:3-101498017-081c71e18b31c9bb8e92bba85e97768be3c9bbfc0d5fb9d0bdd0f9d5299278ef (253:76)
docker-253:3-101498017-57fe21d56db1e80144e67f29eea5d52eeb2140248f13ecdda24430f33a76b75f (253:68)
docker-253:3-101498017-9b4be0f839de894812e1918c7d46eecaff84f816d58ac17818b4b9069088081b (253:82)
docker-253:3-101498017-66c8b4df11fff26acb065b57fa1ead99cdbb91ca53af794b7d5f9fa93dc421ca (253:80)
docker-253:3-101498017-e8658773d4478b95145832dd4ce68ff8acb90e0128cf1a4cd1592b3d10b56c14 (253:40)
docker-253:3-101498017-83d87da21d2cc6717b3b9bf64f85552af572363cb75eaf6aa8a5443dc1815ede (253:16)
docker-253:3-101498017-d6f8df43759458ef8c66ee84a618e98d6b1e3f3664e69134136df5fa41b0f980 (253:46)
docker-253:3-101498017-861e4d8fd60e438a93589bf114df45fde0dde59e805287b477aef8fd30335ad0 (253:73)
docker-253:3-101498017-15a7b53ae901d70b709733315f70b3e7b77d48177bf719799436877b04661533 (253:72)
docker-253:3-101498017-82d44c0769935c8e64d28806b525f4daf2c9011fbb8566b10c7c357dee88b295 (253:70)
docker-253:3-101498017-cde50281f6d9a07d8d227d97ab1621c00dea5467dfca97f7cea8e668d6cd4195 (253:45)
docker-253:3-101498017-dc85b9e7171b469ba9f32e8a1f91671a03e2933606aa1b53edeadc3de13d4972 (253:61)
docker-253:3-101498017-7688a3352e7b0caa35b55a83c8f62fd1edb5b527bbce25bfa6374763bcc789f2 (253:60)
docker-253:3-101498017-0334a53c8f4afd225a961ca5e001cd0daed7d549027963e6b4af3601773315e6 (253:35)
docker-253:3-101498017-9a5f122b66aa832856a6cf9714517ccd2d5862fb29e71ce72af4c365f27a1520 (253:19)
docker-253:3-101498017-4a4e61a6cb0c02b3a918af44b98b554252d45cea5651acba3ec2f4e2d7d05cf0 (253:83)
docker-253:3-101498017-2d847821dc876c1cf9339e006f34562fdae9f4f5dae0219476045eada99c1b01 (253:15)
docker-253:3-101498017-c7f082591d74f0a1b2ded0e9c57ce5a76c3a92a2f119e63018b9d44aa5c5fb94 (253:65)
docker-253:3-101498017-d053d6aca55c7498de05048e17876f4081725e7c58a425950f465b6ec44702df (253:29)
os-var (253:8)
docker-253:3-101498017-e0d6285982adca5d02f380a8d34793cfee94da06543e751e6e65211a055291b4 (253:85)
docker-253:3-101498017-280657ed58b3c2b6ad2985cf810f78f0a5a1224e194357912176c0140de26701 (253:23)
k8s_vol1-optlv (253:2)
docker-253:3-101498017-94bf39e488c4de0ceb0547a49cbecc496ab7660113dafe3b0fd78bd7cb2aee51 (253:43)
docker-253:3-101498017-b16d5d13faec6ace9dee33ee736a720b265273d07514575f3e8c1815e3351035 (253:38)
docker-253:3-101498017-b9b8e8346860c24c9629241dd586e330c1948432a54f8d5f2efd114329be2dfd (253:57)
docker-253:3-101498017-a96d4b4b9aac6459b36ca93bd71092f60fc72a204fc13bd83c3efdd4b46e6f52 (253:17)
k8s_vol1-docker_containers (253:3)
docker-253:3-101498017-b8baaa0a194a40ffd4b165a1a381b84706cb6a60fd46d80306efec146e431f95 (253:77)
docker-253:3-101498017-c055e9a480a884f00333dd6a2c335da3409aefcc807ee52e6bf8f78362095bc9 (253:26)
k8s_vol1-rook_ceph (253:10)
docker-253:3-101498017-9ce68f7c91a3ed26264668f3de3ae558975a92b9f8d6233a82c7bff784d21ccb (253:49)
docker-253:3-101498017-c6d44daa361bc0439bba8346b193b04c4c8252e600ed722de7b7aa8cdc0b28d2 (253:63)
docker-253:3-101498017-5ce0e9df887903b61491b9ca51b49ebdd1cfa790ae010606d6098049683a86f3 (253:71)
docker-253:3-101498017-01e0ff2289f9d72c93d6cf94187d7682bec99e3a02e4cdc4916f6d1d4cbdf5dc (253:41)
docker-253:3-101498017-8a552cf69c2d25746795a0e5a3e9535c0c926246c865052252be88bd23980490 (253:92)
docker-253:3-101498017-9c74c5623eb36c4ffa7ced7ceb838ba227cd5b434a9ae06e3131a9c90c7a238c (253:22)
docker-253:3-101498017-19f400317e661c19ddd8ee6a6d7f25eeb48e54c67ed8c31667babbff70486899 (253:67)
docker-253:3-101498017-c7e0ad7332c64312fb79d4a3aebead2c842d5606f90caef786ba0ca6fd61d669 (253:54)
docker-253:3-101498017-aea189e064aa1820ce56a3155d5c949dca68008d22582c533b357654329d8a31 (253:66)
docker-253:3-101498017-26273229d473b48898bc0426694ade5010f36ddab8948529fbbdc7f5acc5841d (253:36)
docker-253:3-101498017-1cee6b102ce6671c0a44e1350c562208e1af3ea3e2b7d58b6967dffca32265d2 (253:37)
docker-253:3-101498017-f56c67d4d326547eb35a6c8e9170f74d23d137371312e53c78fef27abe6aba61 (253:24)
docker-253:3-101498017-9d3379bcb8edbb38e0db85fb8a54e46db953c2fa6fc5ebca18e2a4c3f6d47e84 (253:47)
docker-253:3-101498017-456b66ba8079c962fb9c6239d33440984d86a02ea6bfdc6718597a70307b529f (253:32)
k8s_vol1-docker_thinpool_tdata (253:6)
docker-253:3-101498017-5aec78cb52a7e922839edcb7b6307a5f2d8d1847fc66061fd0c3a84cc2330e0e (253:48)
os-swap (253:0)
docker-253:3-101498017-f12784005d2ba8707dfcbb8424921ad2721698079253a3121289dd39689de282 (253:87)
docker-253:3-101498017-c0663ce2cafe4a458da70bbc96b5068c8f62989a4e53f289874df2f724057e81 (253:69)
docker-253:3-101498017-7cf9e87a2f613f0c7d7ca8c827986091e8b8c8132c7f1326a6f3d30b385d1d17 (253:33)
os-root (253:1)
docker-253:3-101498017-54df327f48ba863ffd70b1c249aa7da4d913fa317af99af971a63439e8efe139 (253:13)
docker-253:3-101498017-458803691c98537e1c92371a1fa956cf968f791f2bf51bff7678595c5160ba4f (253:64)
docker-253:3-101498017-d07e3bfe5c48623aceb7625c9e2d66ddfe523a06c303801ab4c29f6c33c68cdd (253:50)
docker-253:3-101498017-d9e37c4a49eba4d82e22dd07f6317d3d5714037f19e2aa3f15ab86c0162c8ed2 (253:21)
k8s_vol1-docker_thinpool_tmeta (253:5)
k8s_vol1-docker_thinpool (253:9)
docker-253:3-101498017-3c8c62d635216ac3ff6421ccb8d4dc835f399a08ec6b660b3c8edbd5503e2903 (253:42)
k8s_vol1-kubelet (253:4)
docker-253:3-101498017-2d05d9275ecf718697c59aba19f67a3347fe4395dc37d89dd323d2889652d7ed (253:58)
docker-253:3-101498017-cb5813d7f1064e9b26fec5d91e1665205927769841df552bf1b72e65320d22ff (253:55)
docker-253:3-101498017-f697572e0551ec13f1c09d5e7a6e163b861ee08251a7e4d4e42fd4c589e418b2 (253:78)
docker-253:3-101498017-a61b78adab88a851dc75bb73daa289b8127d608e8f934b75f7689b2bac3924de (253:31)
docker-253:3-101498017-e36d383f3d2aa4e02c57374bbac54c9b998a22d2ac2cfc4bbbe2092b914cc05f (253:11)
docker-253:3-101498017-87f74f834c7eb9aed8b1d781638a375107237e9e0d32ad5ad1e22297a1ac6c4c (253:25)
docker-253:3-101498017-2fdea3314ebf45424145f8c67458de28ffb8e512919c2d4429a14be9a37e64f5 (253:18)
docker-253:3-101498017-cbf7504c9b084a6942c28c4944669b0e7d549cad9fb9b79b8d12e1a87c030a35 (253:75)
docker-253:3-101498017-e88e9db7e1fd3706918f7269ee38f62b7a9a5b12318991dbd19299cf2630d736 (253:89)
docker-253:3-101498017-a09c944b5dadb6aff47b31099cc0884af9996d0526403d9bbc8c86823051aaff (253:51)
docker-253:3-101498017-3c7a5d24808362d90d1b22074235943378b4488122733453e369e4509e5ee2a2 (253:20)
docker-253:3-101498017-4e1f157bba6ead05baa86d72fb8f7e90705090db89196cb84d1dab916c369a47 (253:84)
docker-253:3-101498017-1f7cb658dbbb2ab9635eb2dd6c29219fb84a659d60bb5fef6fdee7977b658cd7 (253:30)
docker-253:3-101498017-7b5a238eebb203422ab5daa6a8706950337124e2ce0bcc622b613231ed03f224 (253:34)
docker-253:3-101498017-8e564cde63bdb70ea017b2cef69b68cfa10e22ddea0451229fd1bb7ab67d05fa (253:28)
docker-253:3-101498017-7d0a3e631d18fc56e7d80b3132786c46cd2cf4efc24ea6ff367e17e208474da2 (253:81)
docker-253:3-101498017-3c897c44a0b7287dc91afcdf4e913dec6084e8816b7a0c2b1dd3bc832cf116f7 (253:94)
docker-253:3-101498017-7e8da0104cc1b0434c6d95cfdbb68a89f2c7a9ec8502467613bf1e61052de6f5 (253:56)
docker-253:3-101498017-9b3b2b78d0ed36d8c382520cb983f179777e936212e90b87edcec88a8adf91b9 (253:52)
docker-253:3-101498017-661cacf156942f9ffe9018140ceca192df12d33628b9d483a9a0439fe2e2402d (253:86)
docker-253:3-101498017-547a6a68db282fabcbf4377151b4d3df076a4b8497012d36e217187fcd6eaf57 (253:53)
docker-253:3-101498017-0e536306352fea1c007f71e8e87703ab79eefcd1ea8330b7b8e9f4ce6fc5d0ce (253:44)
docker_host:~ # df | grep dm- | wc -l
83
docker_host:~ # dmsetup ls | wc -l
94
docker_host:~ # dmsetup ls | grep docker | wc -l
87
docker_host:~ # df | grep dm-
/dev/dm-12 10190100 1129120 8520308 12% /var/lib/docker/devicemapper/mnt/cb9540890830b416bbc03da01f68a446604b55682b3e5e82740eeb37a8e8fb24
/dev/dm-13 10190100 1129448 8519980 12% /var/lib/docker/devicemapper/mnt/54df327f48ba863ffd70b1c249aa7da4d913fa317af99af971a63439e8efe139
/dev/dm-14 10190100 165092 9484336 2% /var/lib/docker/devicemapper/mnt/a2e755e978eeda8788df96e95eb811e40f1e747aa08b9dbdd3bd9affcbf4765f
/dev/dm-15 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/2d847821dc876c1cf9339e006f34562fdae9f4f5dae0219476045eada99c1b01
/dev/dm-17 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/a96d4b4b9aac6459b36ca93bd71092f60fc72a204fc13bd83c3efdd4b46e6f52
/dev/dm-18 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/2fdea3314ebf45424145f8c67458de28ffb8e512919c2d4429a14be9a37e64f5
/dev/dm-19 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/9a5f122b66aa832856a6cf9714517ccd2d5862fb29e71ce72af4c365f27a1520
/dev/dm-20 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/3c7a5d24808362d90d1b22074235943378b4488122733453e369e4509e5ee2a2
/dev/dm-21 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/d9e37c4a49eba4d82e22dd07f6317d3d5714037f19e2aa3f15ab86c0162c8ed2
/dev/dm-24 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/f56c67d4d326547eb35a6c8e9170f74d23d137371312e53c78fef27abe6aba61
/dev/dm-22 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/9c74c5623eb36c4ffa7ced7ceb838ba227cd5b434a9ae06e3131a9c90c7a238c
/dev/dm-25 10190100 319768 9329660 4% /var/lib/docker/devicemapper/mnt/87f74f834c7eb9aed8b1d781638a375107237e9e0d32ad5ad1e22297a1ac6c4c
/dev/dm-23 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/280657ed58b3c2b6ad2985cf810f78f0a5a1224e194357912176c0140de26701
/dev/dm-27 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/b72394e1b06d75059d7696e41cb92e2904dfb68c66b4d44bb7e5710c9d078734
/dev/dm-29 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/d053d6aca55c7498de05048e17876f4081725e7c58a425950f465b6ec44702df
/dev/dm-28 10190100 143968 9505460 2% /var/lib/docker/devicemapper/mnt/8e564cde63bdb70ea017b2cef69b68cfa10e22ddea0451229fd1bb7ab67d05fa
/dev/dm-26 10190100 281632 9367796 3% /var/lib/docker/devicemapper/mnt/c055e9a480a884f00333dd6a2c335da3409aefcc807ee52e6bf8f78362095bc9
/dev/dm-11 10190100 632460 9016968 7% /var/lib/docker/devicemapper/mnt/e36d383f3d2aa4e02c57374bbac54c9b998a22d2ac2cfc4bbbe2092b914cc05f
/dev/dm-16 10190100 64968 9584460 1% /var/lib/docker/devicemapper/mnt/83d87da21d2cc6717b3b9bf64f85552af572363cb75eaf6aa8a5443dc1815ede
/dev/dm-30 10190100 61484 9587944 1% /var/lib/docker/devicemapper/mnt/1f7cb658dbbb2ab9635eb2dd6c29219fb84a659d60bb5fef6fdee7977b658cd7
/dev/dm-31 10190100 546804 9102624 6% /var/lib/docker/devicemapper/mnt/a61b78adab88a851dc75bb73daa289b8127d608e8f934b75f7689b2bac3924de
/dev/dm-33 10190100 319768 9329660 4% /var/lib/docker/devicemapper/mnt/7cf9e87a2f613f0c7d7ca8c827986091e8b8c8132c7f1326a6f3d30b385d1d17
/dev/dm-34 10190100 104316 9545112 2% /var/lib/docker/devicemapper/mnt/7b5a238eebb203422ab5daa6a8706950337124e2ce0bcc622b613231ed03f224
/dev/dm-35 10190100 82192 9567236 1% /var/lib/docker/devicemapper/mnt/0334a53c8f4afd225a961ca5e001cd0daed7d549027963e6b4af3601773315e6
/dev/dm-38 10190100 536900 9112528 6% /var/lib/docker/devicemapper/mnt/b16d5d13faec6ace9dee33ee736a720b265273d07514575f3e8c1815e3351035
/dev/dm-37 10190100 536900 9112528 6% /var/lib/docker/devicemapper/mnt/1cee6b102ce6671c0a44e1350c562208e1af3ea3e2b7d58b6967dffca32265d2
/dev/dm-32 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/456b66ba8079c962fb9c6239d33440984d86a02ea6bfdc6718597a70307b529f
/dev/dm-36 10190100 632460 9016968 7% /var/lib/docker/devicemapper/mnt/26273229d473b48898bc0426694ade5010f36ddab8948529fbbdc7f5acc5841d
/dev/dm-41 10190100 536904 9112524 6% /var/lib/docker/devicemapper/mnt/01e0ff2289f9d72c93d6cf94187d7682bec99e3a02e4cdc4916f6d1d4cbdf5dc
/dev/dm-40 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/e8658773d4478b95145832dd4ce68ff8acb90e0128cf1a4cd1592b3d10b56c14
/dev/dm-42 10190100 536908 9112520 6% /var/lib/docker/devicemapper/mnt/3c8c62d635216ac3ff6421ccb8d4dc835f399a08ec6b660b3c8edbd5503e2903
/dev/dm-39 10190100 89256 9560172 1% /var/lib/docker/devicemapper/mnt/1e91e539f9a112909e7439bca13629864a2d7f1797c3cc42944dc01a6ed07001
/dev/dm-44 10190100 89256 9560172 1% /var/lib/docker/devicemapper/mnt/0e536306352fea1c007f71e8e87703ab79eefcd1ea8330b7b8e9f4ce6fc5d0ce
/dev/dm-43 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/94bf39e488c4de0ceb0547a49cbecc496ab7660113dafe3b0fd78bd7cb2aee51
/dev/dm-46 10190100 141136 9508292 2% /var/lib/docker/devicemapper/mnt/d6f8df43759458ef8c66ee84a618e98d6b1e3f3664e69134136df5fa41b0f980
/dev/dm-55 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/cb5813d7f1064e9b26fec5d91e1665205927769841df552bf1b72e65320d22ff
/dev/dm-56 10190100 85836 9563592 1% /var/lib/docker/devicemapper/mnt/7e8da0104cc1b0434c6d95cfdbb68a89f2c7a9ec8502467613bf1e61052de6f5
/dev/dm-57 10190100 76872 9572556 1% /var/lib/docker/devicemapper/mnt/b9b8e8346860c24c9629241dd586e330c1948432a54f8d5f2efd114329be2dfd
/dev/dm-54 10190100 78100 9571328 1% /var/lib/docker/devicemapper/mnt/c7e0ad7332c64312fb79d4a3aebead2c842d5606f90caef786ba0ca6fd61d669
/dev/dm-53 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/547a6a68db282fabcbf4377151b4d3df076a4b8497012d36e217187fcd6eaf57
/dev/dm-52 10190100 287272 9362156 3% /var/lib/docker/devicemapper/mnt/9b3b2b78d0ed36d8c382520cb983f179777e936212e90b87edcec88a8adf91b9
/dev/dm-50 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/d07e3bfe5c48623aceb7625c9e2d66ddfe523a06c303801ab4c29f6c33c68cdd
/dev/dm-51 10190100 1019208 8630220 11% /var/lib/docker/devicemapper/mnt/a09c944b5dadb6aff47b31099cc0884af9996d0526403d9bbc8c86823051aaff
/dev/dm-58 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/2d05d9275ecf718697c59aba19f67a3347fe4395dc37d89dd323d2889652d7ed
/dev/dm-59 10190100 177592 9471836 2% /var/lib/docker/devicemapper/mnt/58aae6188dca08a773f7b0dd3c051ca315006054ae87a6864488750419183451
/dev/dm-60 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/7688a3352e7b0caa35b55a83c8f62fd1edb5b527bbce25bfa6374763bcc789f2
/dev/dm-61 10190100 73788 9575640 1% /var/lib/docker/devicemapper/mnt/dc85b9e7171b469ba9f32e8a1f91671a03e2933606aa1b53edeadc3de13d4972
/dev/dm-63 10190100 141160 9508268 2% /var/lib/docker/devicemapper/mnt/c6d44daa361bc0439bba8346b193b04c4c8252e600ed722de7b7aa8cdc0b28d2
/dev/dm-68 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/57fe21d56db1e80144e67f29eea5d52eeb2140248f13ecdda24430f33a76b75f
/dev/dm-69 10190100 598648 9050780 7% /var/lib/docker/devicemapper/mnt/c0663ce2cafe4a458da70bbc96b5068c8f62989a4e53f289874df2f724057e81
/dev/dm-45 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/cde50281f6d9a07d8d227d97ab1621c00dea5467dfca97f7cea8e668d6cd4195
/dev/dm-48 10190100 205664 9443764 3% /var/lib/docker/devicemapper/mnt/5aec78cb52a7e922839edcb7b6307a5f2d8d1847fc66061fd0c3a84cc2330e0e
/dev/dm-67 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/19f400317e661c19ddd8ee6a6d7f25eeb48e54c67ed8c31667babbff70486899
/dev/dm-74 10190100 601884 9047544 7% /var/lib/docker/devicemapper/mnt/c31f527f16b1d4126a0b18c22b54ec1121d83f3f3a13a38f218a8956b700202e
/dev/dm-64 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/458803691c98537e1c92371a1fa956cf968f791f2bf51bff7678595c5160ba4f
/dev/dm-65 10190100 954536 8694892 10% /var/lib/docker/devicemapper/mnt/c7f082591d74f0a1b2ded0e9c57ce5a76c3a92a2f119e63018b9d44aa5c5fb94
/dev/dm-75 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/cbf7504c9b084a6942c28c4944669b0e7d549cad9fb9b79b8d12e1a87c030a35
/dev/dm-76 10190100 439296 9210132 5% /var/lib/docker/devicemapper/mnt/081c71e18b31c9bb8e92bba85e97768be3c9bbfc0d5fb9d0bdd0f9d5299278ef
/dev/dm-66 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/aea189e064aa1820ce56a3155d5c949dca68008d22582c533b357654329d8a31
/dev/dm-62 10190100 386920 9262508 5% /var/lib/docker/devicemapper/mnt/7d23c8c042d3eca16b215cd90a244d931fa8c1f7c0d9f6354c8aba1442d185a9
/dev/dm-47 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/9d3379bcb8edbb38e0db85fb8a54e46db953c2fa6fc5ebca18e2a4c3f6d47e84
/dev/dm-49 10190100 681736 8967692 8% /var/lib/docker/devicemapper/mnt/9ce68f7c91a3ed26264668f3de3ae558975a92b9f8d6233a82c7bff784d21ccb
/dev/dm-94 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/3c897c44a0b7287dc91afcdf4e913dec6084e8816b7a0c2b1dd3bc832cf116f7
/dev/dm-95 10190100 112708 9536720 2% /var/lib/docker/devicemapper/mnt/692db6edc24c878f9caa45c89c930120e6c0b3897f62e468aa1ba56482743a8f
/dev/dm-70 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/82d44c0769935c8e64d28806b525f4daf2c9011fbb8566b10c7c357dee88b295
/dev/dm-72 10190100 430968 9218460 5% /var/lib/docker/devicemapper/mnt/15a7b53ae901d70b709733315f70b3e7b77d48177bf719799436877b04661533
/dev/dm-79 10190100 319764 9329664 4% /var/lib/docker/devicemapper/mnt/72bcd76347e26fed9b0332842d42dab557262ef56d6abefb6faff3247efa1cf8
/dev/dm-78 10190100 319764 9329664 4% /var/lib/docker/devicemapper/mnt/f697572e0551ec13f1c09d5e7a6e163b861ee08251a7e4d4e42fd4c589e418b2
/dev/dm-77 10190100 319764 9329664 4% /var/lib/docker/devicemapper/mnt/b8baaa0a194a40ffd4b165a1a381b84706cb6a60fd46d80306efec146e431f95
/dev/dm-73 10190100 319764 9329664 4% /var/lib/docker/devicemapper/mnt/861e4d8fd60e438a93589bf114df45fde0dde59e805287b477aef8fd30335ad0
/dev/dm-71 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/5ce0e9df887903b61491b9ca51b49ebdd1cfa790ae010606d6098049683a86f3
/dev/dm-80 10190100 879836 8769592 10% /var/lib/docker/devicemapper/mnt/66c8b4df11fff26acb065b57fa1ead99cdbb91ca53af794b7d5f9fa93dc421ca
/dev/dm-86 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/661cacf156942f9ffe9018140ceca192df12d33628b9d483a9a0439fe2e2402d
/dev/dm-89 10190100 285304 9364124 3% /var/lib/docker/devicemapper/mnt/e88e9db7e1fd3706918f7269ee38f62b7a9a5b12318991dbd19299cf2630d736
/dev/dm-84 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/4e1f157bba6ead05baa86d72fb8f7e90705090db89196cb84d1dab916c369a47
/dev/dm-85 10190100 1019240 8630188 11% /var/lib/docker/devicemapper/mnt/e0d6285982adca5d02f380a8d34793cfee94da06543e751e6e65211a055291b4
/dev/dm-81 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/7d0a3e631d18fc56e7d80b3132786c46cd2cf4efc24ea6ff367e17e208474da2
/dev/dm-82 10190100 523004 9126424 6% /var/lib/docker/devicemapper/mnt/9b4be0f839de894812e1918c7d46eecaff84f816d58ac17818b4b9069088081b
/dev/dm-83 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/4a4e61a6cb0c02b3a918af44b98b554252d45cea5651acba3ec2f4e2d7d05cf0
/dev/dm-87 10190100 1329112 8320316 14% /var/lib/docker/devicemapper/mnt/f12784005d2ba8707dfcbb8424921ad2721698079253a3121289dd39689de282
/dev/dm-88 10190100 37648 9611780 1% /var/lib/docker/devicemapper/mnt/2ca06d5355e51a41695b3a407c141d540ce3eaebfaf79d55c39e7bf5b14ec0fd
/dev/dm-90 10190100 417500 9231928 5% /var/lib/docker/devicemapper/mnt/3bb55da27654a22e832141e1ef8f943ac22b3f06f96315033a99529b43a1fd7e
/dev/dm-92 10190100 319764 9329664 4% /var/lib/docker/devicemapper/mnt/8a552cf69c2d25746795a0e5a3e9535c0c926246c865052252be88bd23980490
docker_host:~ # total=0; for s in `df | grep dm- | awk '{print $3}'`; do total=$(expr $total + $s); done; echo $(expr $total / 1024)
21255
docker_host:~ # docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 43 41 11.58 GB 450.3 MB (3%)
Containers 101 83 1.665 GB 83 B (0%)
Local Volumes 7 7 124.9 MB 0 B (0%)
docker_host:~ # cat /etc/docker/
daemon.json key.json
docker_host:~ # cat /etc/docker/daemon.json
{
"storage-driver": "devicemapper",
"storage-opts": [
"dm.thinpooldev=/dev/mapper/k8s_vol1-docker_thinpool",
"dm.use_deferred_removal=true",
"dm.use_deferred_deletion=true",
"dm.fs=ext4"
],
"insecure-registries": ["###.###.###.###:####"]
}
