I have a raidz3 zpool on Ubuntu 18.04 with the Ubuntu built-in zfs features, there was a system crash recently, I guess it's caused by an unavailable(failed) disk, and after rebooting, the pool was missing:
$ sudo zpool status
no pools available
Tried to list the pool, but it says: The pool cannot be imported due to damaged devices or data.
$ sudo zpool import
pool: content-pool
id: 3621552755300412622
state: UNAVAIL
status: One or more devices were being resilvered.
action: The pool cannot be imported due to damaged devices or data.
config:
.
.
.
I've confirmed that all the disks are detected by the system properly, can be located in /dev/disk/by/-id/ / /dev/
Tried to import it using following trying, also tried with -F/-f/-m/-o readonly=on parameter:
sudo zpool import content-poolsudo zpool import -d /dev/disk/by-id/ content-pool
All returned:
cannot import content-pool': one or more devices is currently unavailable
Also tried to use zdb -l to dump vdev labels from the partitions of alive hard drive, seems the label can be retrieved successfully, the output are almost the same, except the guid: part, and there are 36 children(0-35) just like to original working raidz3 pool:
------------------------------------
LABEL 0
------------------------------------
version: 5000
name: 'content-pool'
state: 0
txg: 8348544
pool_guid: 3621552755300412622
errata: 0
hostname: 'content-ftp'
top_guid: 11498459744868090541
guid: 7798174214455629273
vdev_children: 2
vdev_tree:
type: 'raidz'
id: 0
guid: 11498459744868090541
nparity: 3
metaslab_array: 166
metaslab_shift: 40
ashift: 12
asize: 216041772810240
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 7798174214455629273
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D168PVC8-part1'
whole_disk: 1
DTL: 215
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 9655586118508030176
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D363TCL9-part1'
whole_disk: 1
DTL: 214
create_txg: 4
children[2]:
type: 'disk'
id: 2
guid: 1036444324318734332
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D363TDVX-part1'
whole_disk: 1
DTL: 213
create_txg: 4
children[3]:
type: 'disk'
Is there any chance that I can force the import and try to fix the data? Hopefully raidz3 can restore the damaged data? Thank you all!!!
Update
Somehow after disable autoimport by setting zfs_autoimport_disable=0 in /etc/modprobe.d/zfs.conf, zpool status works
$ sudo zpool status
pool: content-pool
state: UNAVAIL
status: One or more devices are faulted in response to persistent errors. There are insufficient replicas for the pool to
continue functioning.
action: Destroy and re-create the pool from a backup source. Manually marking the device
repaired using 'zpool clear' may allow some data to be recovered.
scan: none requested
config:
NAME STATE READ WRITE CKSUM
content-pool UNAVAIL 0 0 0 insufficient replicas
raidz3-0 DEGRADED 0 0 0
ata-WDC_WD60EFRX-68L0BN1_WD-WX11D168PVC8 ONLINE 0 0 0
ata-WDC_WD60EFRX-68L0BN1_WD-WX11D363TCL9 ONLINE 0 0 0
ata-WDC_WD60EFRX-68L0BN1_WD-WX11D363TDVX ONLINE 0 0 0
ata-WDC_WD60EFRX-68L0BN1_WD-WX11D36JR303 ONLINE 0 0 0
ata-WDC_WD60EFRX-68L0BN1_WD-WX11D36JRJD1 ONLINE 0 0 0
ata-WDC_WD60EFRX-68L0BN1_WD-WX11DB56Z35P ONLINE 0 0 0
ata-WDC_WD60EFRX-68L0BN1_WD-WX11DB56Z4JD ONLINE 0 0 0
ata-WDC_WD60EFRX-68L0BN1_WD-WX11DB56ZEX2 ONLINE 0 0 0
ata-WDC_WD60EFRX-68L0BN1_WD-WX21D36PPEKZ ONLINE 0 0 0
ata-WDC_WD60EFRX-68L0BN1_WD-WXB1HB4HV3ZZ ONLINE 0 0 0
ata-WDC_WD60EFRX-68L0BN1_WD-WXB1HB4LMMMM ONLINE 0 0 0
ata-WDC_WD60EFRX-68L0BN1_WD-WXC1HB4HMLCD ONLINE 0 0 0
ata-WDC_WD60EFRX-68L0BN1_WD-WXN1H84CCEVF ONLINE 0 0 0
ata-WDC_WD60EFRX-68MYMN1_WD-WX11D557Y3SR ONLINE 0 0 0
ata-WDC_WD60EFRX-68MYMN1_WD-WX11D557YAVR ONLINE 0 0 0
ata-WDC_WD60EFRX-68MYMN1_WD-WX11D557YCC2 ONLINE 0 0 0
ata-WDC_WD60EFRX-68MYMN1_WD-WX11D55PX3KS ONLINE 0 0 0
ata-WDC_WD60EFRX-68MYMN1_WD-WX11D55PX3YX ONLINE 0 0 0
ata-WDC_WD60EFRX-68MYMN1_WD-WX11D55PXTSC ONLINE 0 0 0
ata-WDC_WD60EFRX-68MYMN1_WD-WX11DC4497YV ONLINE 0 0 0
ata-WDC_WD60EFRX-68MYMN1_WD-WX31D2504K74 ONLINE 0 0 0
ata-WDC_WD60EFRX-68MYMN1_WD-WX31D65A2C9E ONLINE 0 0 0
ata-WDC_WD60EFRX-68MYMN1_WD-WX41D259T63X ONLINE 0 0 0
ata-WDC_WD60EFRX-68MYMN1_WD-WX41D259TE9L ONLINE 0 0 0
ata-WDC_WD60EFRX-68MYMN1_WD-WX41D259TJU9 ONLINE 0 0 0
ata-WDC_WD60EFRX-68MYMN1_WD-WX41D259TVTR ONLINE 0 0 0
1078152416620325459 UNAVAIL 0 0 0 was /dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX61D6574E35-part1
ata-WDC_WD60EFRX-68MYMN1_WD-WX61D6574KZ5 ONLINE 0 0 0
ata-WDC_WD60EFRX-68MYMN1_WD-WX61D6574SXZ ONLINE 0 0 0
ata-WDC_WD60EFRX-68MYMN1_WD-WX61D6574V07 ONLINE 0 0 0
ata-WDC_WD60EFRX-68MYMN1_WD-WX71D65JEZ2C ONLINE 0 0 0
ata-WDC_WD60EFRX-68MYMN1_WD-WX81D6550UR4 ONLINE 0 0 0
ata-WDC_WD60EFRX-68L0BN1_WD-WXB1HB4HSKRY ONLINE 0 0 0
ata-WDC_WD60EFRX-68MYMN1_WD-WX81D65D9LD6 ONLINE 0 0 0
ata-WDC_WD60EFRX-68MYMN1_WD-WX91D65DCE04 ONLINE 0 0 0
ata-WDC_WD60EFRX-68MYMN1_WD-WX91D65DCKR5 ONLINE 0 0 0
ata-WDC_WD60EFRX-68L0BN1_WD-WX11DB56ZTEP FAULTED 0 0 0 external device fault
Now the problem has become that how to recover from a faulted pool, I can't do almost any operation to the pool:
$ sudo zpool clear content-pool ata-WDC_WD60EFRX-68L0BN1_WD-WX11DB56ZTEP
cannot clear errors for ata-WDC_WD60EFRX-68L0BN1_WD-WX11DB56ZTEP: one or more devices is currently unavailable
$ sudo zpool offline content-pool ata-WDC_WD60EFRX-68L0BN1_WD-WX11DB56ZTEP
cannot open 'content-pool': pool is unavailable
$ sudo zpool scrub content-pool
cannot scrub 'content-pool': pool is currently unavailable
Even I tried to replace the dead drive or clear the pool status:
$ sudo zpool replace content-pool 1078152416620325459 ata-WDC_WD60EFRX-68L0BN1_WD-WX11D168PDHT
cannot open 'content-pool': pool is unavailable
$ sudo zpool clear content-pool
cannot clear errors for content-pool: one or more devices is currently unavailable
Tried to retrieve some details from sudo zpool events -vf but no result from above operations
update: back to the state that pool can't be imported again
current zpool import:
(the faulted ata-WDC_WD60EFRX-68L0BN1_WD-WX11DB56ZTEP was added by accident before, there is no data in it)
$ sudo zpool import
pool: content-pool
id: 3621552755300412622
state: UNAVAIL
status: One or more devices are faulted.
action: The pool cannot be imported due to damaged devices or data.
config:
content-pool UNAVAIL insufficient replicas
raidz3-0 DEGRADED
ata-WDC_WD60EFRX-68L0BN1_WD-WX11D168PVC8 ONLINE
ata-WDC_WD60EFRX-68L0BN1_WD-WX11D363TCL9 ONLINE
ata-WDC_WD60EFRX-68L0BN1_WD-WX11D363TDVX ONLINE
ata-WDC_WD60EFRX-68L0BN1_WD-WX11D36JR303 ONLINE
ata-WDC_WD60EFRX-68L0BN1_WD-WX11D36JRJD1 ONLINE
ata-WDC_WD60EFRX-68L0BN1_WD-WX11DB56Z35P ONLINE
ata-WDC_WD60EFRX-68L0BN1_WD-WX11DB56Z4JD ONLINE
ata-WDC_WD60EFRX-68L0BN1_WD-WX11DB56ZEX2 ONLINE
ata-WDC_WD60EFRX-68L0BN1_WD-WX21D36PPEKZ ONLINE
ata-WDC_WD60EFRX-68L0BN1_WD-WXB1HB4HV3ZZ ONLINE
ata-WDC_WD60EFRX-68L0BN1_WD-WXB1HB4LMMMM ONLINE
ata-WDC_WD60EFRX-68L0BN1_WD-WXC1HB4HMLCD ONLINE
ata-WDC_WD60EFRX-68L0BN1_WD-WXN1H84CCEVF ONLINE
ata-WDC_WD60EFRX-68MYMN1_WD-WX11D557Y3SR ONLINE
ata-WDC_WD60EFRX-68MYMN1_WD-WX11D557YAVR ONLINE
ata-WDC_WD60EFRX-68MYMN1_WD-WX11D557YCC2 ONLINE
ata-WDC_WD60EFRX-68MYMN1_WD-WX11D55PX3KS ONLINE
ata-WDC_WD60EFRX-68MYMN1_WD-WX11D55PX3YX ONLINE
ata-WDC_WD60EFRX-68MYMN1_WD-WX11D55PXTSC ONLINE
ata-WDC_WD60EFRX-68MYMN1_WD-WX11DC4497YV ONLINE
ata-WDC_WD60EFRX-68MYMN1_WD-WX31D2504K74 ONLINE
ata-WDC_WD60EFRX-68MYMN1_WD-WX31D65A2C9E ONLINE
ata-WDC_WD60EFRX-68MYMN1_WD-WX41D259T63X ONLINE
ata-WDC_WD60EFRX-68MYMN1_WD-WX41D259TE9L ONLINE
ata-WDC_WD60EFRX-68MYMN1_WD-WX41D259TJU9 ONLINE
ata-WDC_WD60EFRX-68MYMN1_WD-WX41D259TVTR ONLINE
1078152416620325459 UNAVAIL
ata-WDC_WD60EFRX-68MYMN1_WD-WX61D6574KZ5 ONLINE
ata-WDC_WD60EFRX-68MYMN1_WD-WX61D6574SXZ ONLINE
ata-WDC_WD60EFRX-68MYMN1_WD-WX61D6574V07 ONLINE
ata-WDC_WD60EFRX-68MYMN1_WD-WX71D65JEZ2C ONLINE
ata-WDC_WD60EFRX-68MYMN1_WD-WX81D6550UR4 ONLINE
ata-WDC_WD60EFRX-68L0BN1_WD-WXB1HB4HSKRY ONLINE
ata-WDC_WD60EFRX-68MYMN1_WD-WX81D65D9LD6 ONLINE
ata-WDC_WD60EFRX-68MYMN1_WD-WX91D65DCE04 ONLINE
ata-WDC_WD60EFRX-68MYMN1_WD-WX91D65DCKR5 ONLINE
ata-WDC_WD60EFRX-68L0BN1_WD-WX11DB56ZTEP FAULTED corrupted data
One of the zdb -l:
(All the disk in the same pool has almost the same zdb -l result, except the guid: part is different)
$ sudo zdb -l /dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX11D55PX3KS-part1
------------------------------------
LABEL 0
------------------------------------
version: 5000
name: 'content-pool'
state: 0
txg: 8348544
pool_guid: 3621552755300412622
errata: 0
hostname: 'content-ftp'
top_guid: 11498459744868090541
guid: 11818476807007794000
vdev_children: 2
vdev_tree:
type: 'raidz'
id: 0
guid: 11498459744868090541
nparity: 3
metaslab_array: 166
metaslab_shift: 40
ashift: 12
asize: 216041772810240
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 7798174214455629273
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D168PVC8-part1'
whole_disk: 1
DTL: 215
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 9655586118508030176
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D363TCL9-part1'
whole_disk: 1
DTL: 214
create_txg: 4
children[2]:
type: 'disk'
id: 2
guid: 1036444324318734332
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D363TDVX-part1'
whole_disk: 1
DTL: 213
create_txg: 4
children[3]:
type: 'disk'
id: 3
guid: 1671251092857451329
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D36JR303-part1'
whole_disk: 1
DTL: 212
create_txg: 4
children[4]:
type: 'disk'
id: 4
guid: 4180777695415546678
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11D36JRJD1-part1'
whole_disk: 1
DTL: 211
create_txg: 4
children[5]:
type: 'disk'
id: 5
guid: 9587968578374614856
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11DB56Z35P-part1'
whole_disk: 1
DTL: 210
create_txg: 4
children[6]:
type: 'disk'
id: 6
guid: 2726944175730713275
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11DB56Z4JD-part1'
whole_disk: 1
DTL: 209
create_txg: 4
children[7]:
type: 'disk'
id: 7
guid: 15058102474771879213
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX11DB56ZEX2-part1'
whole_disk: 1
DTL: 208
create_txg: 4
children[8]:
type: 'disk'
id: 8
guid: 2176583035980080357
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WX21D36PPEKZ-part1'
whole_disk: 1
DTL: 207
create_txg: 4
children[9]:
type: 'disk'
id: 9
guid: 12213605498789728384
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WXB1HB4HV3ZZ-part1'
whole_disk: 1
DTL: 206
create_txg: 4
children[10]:
type: 'disk'
id: 10
guid: 16692863623985590265
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WXB1HB4LMMMM-part1'
whole_disk: 1
DTL: 205
create_txg: 4
children[11]:
type: 'disk'
id: 11
guid: 4131222994089368621
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WXC1HB4HMLCD-part1'
whole_disk: 1
DTL: 204
create_txg: 4
children[12]:
type: 'disk'
id: 12
guid: 10405780603842776111
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WXN1H84CCEVF-part1'
whole_disk: 1
DTL: 203
create_txg: 4
children[13]:
type: 'disk'
id: 13
guid: 4802054319391370846
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX11D557Y3SR-part1'
whole_disk: 1
DTL: 202
create_txg: 4
children[14]:
type: 'disk'
id: 14
guid: 14336049427456271508
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX11D557YAVR-part1'
whole_disk: 1
DTL: 201
create_txg: 4
children[15]:
type: 'disk'
id: 15
guid: 5171261257149665977
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX11D557YCC2-part1'
whole_disk: 1
DTL: 200
create_txg: 4
children[16]:
type: 'disk'
id: 16
guid: 11818476807007794000
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX11D55PX3KS-part1'
whole_disk: 1
DTL: 199
create_txg: 4
children[17]:
type: 'disk'
id: 17
guid: 12514554540177747308
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX11D55PX3YX-part1'
whole_disk: 1
DTL: 198
create_txg: 4
children[18]:
type: 'disk'
id: 18
guid: 411055325634056854
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX11D55PXTSC-part1'
whole_disk: 1
DTL: 197
create_txg: 4
children[19]:
type: 'disk'
id: 19
guid: 9168663998575981435
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX11DC4497YV-part1'
whole_disk: 1
DTL: 196
create_txg: 4
children[20]:
type: 'disk'
id: 20
guid: 4385217325944977250
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX31D2504K74-part1'
whole_disk: 1
DTL: 195
create_txg: 4
children[21]:
type: 'disk'
id: 21
guid: 12169669977314753282
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX31D65A2C9E-part1'
whole_disk: 1
DTL: 194
create_txg: 4
children[22]:
type: 'disk'
id: 22
guid: 2739496474415237206
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX41D259T63X-part1'
whole_disk: 1
DTL: 193
create_txg: 4
children[23]:
type: 'disk'
id: 23
guid: 15921933850909040039
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX41D259TE9L-part1'
whole_disk: 1
DTL: 192
create_txg: 4
children[24]:
type: 'disk'
id: 24
guid: 13900183369170026888
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX41D259TJU9-part1'
whole_disk: 1
DTL: 191
create_txg: 4
children[25]:
type: 'disk'
id: 25
guid: 10729475082755539900
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX41D259TVTR-part1'
whole_disk: 1
DTL: 190
create_txg: 4
children[26]:
type: 'disk'
id: 26
guid: 1078152416620325459
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX61D6574E35-part1'
whole_disk: 1
not_present: 1
DTL: 189
create_txg: 4
children[27]:
type: 'disk'
id: 27
guid: 7169212150010223263
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX61D6574KZ5-part1'
whole_disk: 1
DTL: 188
create_txg: 4
children[28]:
type: 'disk'
id: 28
guid: 10688002672848252538
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX61D6574SXZ-part1'
whole_disk: 1
DTL: 187
create_txg: 4
children[29]:
type: 'disk'
id: 29
guid: 14475182699961610412
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX61D6574V07-part1'
whole_disk: 1
DTL: 186
create_txg: 4
children[30]:
type: 'disk'
id: 30
guid: 1251108739797802502
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX71D65JEZ2C-part1'
whole_disk: 1
DTL: 185
create_txg: 4
children[31]:
type: 'disk'
id: 31
guid: 16567657162171866083
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX81D6550UR4-part1'
whole_disk: 1
DTL: 184
create_txg: 4
children[32]:
type: 'disk'
id: 32
guid: 11324965671804235780
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68L0BN1_WD-WXB1HB4HSKRY-part1'
whole_disk: 1
DTL: 1921
create_txg: 4
children[33]:
type: 'disk'
id: 33
guid: 5940180339143158814
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX81D65D9LD6-part1'
whole_disk: 1
DTL: 183
create_txg: 4
children[34]:
type: 'disk'
id: 34
guid: 437257513201166423
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX91D65DCE04-part1'
whole_disk: 1
DTL: 182
create_txg: 4
children[35]:
type: 'disk'
id: 35
guid: 13641210868471885912
path: '/dev/disk/by-id/ata-WDC_WD60EFRX-68MYMN1_WD-WX91D65DCKR5-part1'
whole_disk: 1
DTL: 178
create_txg: 4
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
labels = 0 1 2 3
