##### OK first I'll try listing all detected zfs pools. ZfsMediaPool is raidz2 and all SATA 12tb disks. It has been rock solid for years. **This is via the 100m optical cable** ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: FAULTED status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. The pool may be active on another system, but can be imported using the '-f' flag. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E config: ZfsMediaPool FAULTED corrupted data raidz2-0 DEGRADED PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 FAULTED corrupted data PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 UNAVAIL corrupted data PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 FAULTED corrupted data ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: FAULTED status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. The pool may be active on another system, but can be imported using the '-f' flag. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E config: ZfsMediaPool FAULTED corrupted data raidz2-0 DEGRADED PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 FAULTED corrupted data PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 FAULTED corrupted data #WTF now it's ok? ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: ZfsMediaPool ONLINE raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: ZfsMediaPool ONLINE raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: ZfsMediaPool ONLINE raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE # OK, looks stable, lets try mounting/importing it ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ ZfsMediaPool cannot import 'ZfsMediaPool': I/O error Recovery is possible, but will result in some data loss. Returning the pool to its state as of Fri Apr 14 11:38:29 2023 should correct the problem. Approximately 6 seconds of data must be discarded, irreversibly. After rewind, at least one persistent user-data error will remain. Recovery can be attempted by executing 'zpool import -F ZfsMediaPool'. A scrub of the pool is strongly recommended after recovery. ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: FAULTED status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. The pool may be active on another system, but can be imported using the '-f' flag. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E config: ZfsMediaPool FAULTED corrupted data raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 UNAVAIL corrupted data logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE #WTF it's crapped out at import. ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: FAULTED status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. The pool may be active on another system, but can be imported using the '-f' flag. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E config: ZfsMediaPool FAULTED corrupted data raidz2-0 DEGRADED PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 FAULTED corrupted data PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 UNAVAIL corrupted data PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 FAULTED corrupted data ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: FAULTED status: The pool metadata is corrupted. action: The pool cannot be imported due to damaged devices or data. The pool may be active on another system, but can be imported using the '-f' flag. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-72 config: ZfsMediaPool FAULTED corrupted data raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: ZfsMediaPool ONLINE raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: ZfsMediaPool ONLINE raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: FAULTED status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. The pool may be active on another system, but can be imported using the '-f' flag. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E config: ZfsMediaPool FAULTED corrupted data raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 UNAVAIL corrupted data PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: ZfsMediaPool ONLINE raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: ZfsMediaPool ONLINE raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: ZfsMediaPool ONLINE raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: ZfsMediaPool ONLINE raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: ZfsMediaPool ONLINE raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: FAULTED status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. The pool may be active on another system, but can be imported using the '-f' flag. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E config: ZfsMediaPool FAULTED corrupted data raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 UNAVAIL corrupted data PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: FAULTED status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. The pool may be active on another system, but can be imported using the '-f' flag. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E config: ZfsMediaPool FAULTED corrupted data raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 UNAVAIL corrupted data PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 UNAVAIL corrupted data PCI0@0-PEG1@1,1-PEGP@0-@7:0 UNAVAIL corrupted data PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: FAULTED status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. The pool may be active on another system, but can be imported using the '-f' flag. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E config: ZfsMediaPool FAULTED corrupted data raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 UNAVAIL corrupted data PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 UNAVAIL corrupted data PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: FAULTED status: The pool metadata is corrupted. action: The pool cannot be imported due to damaged devices or data. The pool may be active on another system, but can be imported using the '-f' flag. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-72 config: ZfsMediaPool FAULTED corrupted data raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: ZfsMediaPool ONLINE raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: FAULTED status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. The pool may be active on another system, but can be imported using the '-f' flag. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E config: ZfsMediaPool FAULTED corrupted data raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 UNAVAIL corrupted data PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 UNAVAIL corrupted data PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: FAULTED status: The pool metadata is corrupted. action: The pool cannot be imported due to damaged devices or data. The pool may be active on another system, but can be imported using the '-f' flag. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-72 config: ZfsMediaPool FAULTED corrupted data raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: ZfsMediaPool ONLINE raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: ONLINE status: One or more devices contains corrupted data. action: The pool can be imported using its name or numeric identifier. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J config: ZfsMediaPool ONLINE raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 UNAVAIL corrupted data PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: FAULTED status: The pool metadata is corrupted. action: The pool cannot be imported due to damaged devices or data. The pool may be active on another system, but can be imported using the '-f' flag. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-72 config: ZfsMediaPool FAULTED corrupted data raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: ONLINE status: One or more devices contains corrupted data. action: The pool can be imported using its name or numeric identifier. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J config: ZfsMediaPool ONLINE raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 UNAVAIL corrupted data PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: ZfsMediaPool ONLINE raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: ZfsMediaPool ONLINE raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: ZfsMediaPool ONLINE raidz2-0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@e:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 ONLINE ComicBookGuy:root root# zpool import -d /var/run/disk/by-path/ pool: ZfsMediaPool id: 626277063290310723 state: FAULTED status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. The pool may be active on another system, but can be imported using the '-f' flag. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E config: ZfsMediaPool FAULTED corrupted data raidz2-0 DEGRADED PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@9:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@10:0 FAULTED corrupted data PCI0@0-PEG1@1,1-PEGP@0-@e:0 UNAVAIL corrupted data PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@f:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@d:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE cache PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:4 spares PCI0@0-PEG1@1,1-PEGP@0-@c:0 logs PCI0@0-RP09@1D-PXSX@0-IONVMeController-IONVMeBlockStorageDevice@1-@1:3 FAULTED corrupted data #so this is maybe something with these SATA disks, I have more 4tb SATA disks. #lets experiment here. #Ok lets take 8 4g SATA disks and make a new pool #This is by the 100m cable ############################################################################### Channel 1: ATTO ExpressSAS H1288 ############################################################################### Enclosure 0: Areca Te ARC-403801.37.69 0137 Enclosure Status: Ok Slot Count: 16 ID Slot Vendor Product / Rev Group Name Capacity Serial Number ---- ---- -------- ---------------- ---- ---------------- ---------- 17 0 ATA ST4000NM0033-9ZM GA6E *Available 3.64 TB SN:S1Z2BPT1 18 1 ATA ST4000NM0033-9ZM GA6E *Available 3.64 TB SN:Z1Z2TPM1 19 2 ATA ST4000NM0033-9ZM SN07 *Available 3.64 TB SN:S1Z0KARA 20 3 ATA ST4000NM0033-9ZM SN07 *Available 3.64 TB SN:Z1ZAK6Z8 21 4 ATA ST4000NM0033-9ZM GA6E *Available 3.64 TB SN:Z1Z2XD5M 22 5 ATA ST4000NM0033-9ZM GA6E *Available 3.64 TB SN:Z1Z5WW7T 23 6 ATA ST4000NM0033-9ZM GA6E *Available 3.64 TB SN:Z1Z2WN24 24 7 ATA ST4000NM0033-9ZM SN07 *Available 3.64 TB SN:Z1ZAK739 #ls /var/run/disk/by-path/PCI0\@0-PEG1\@1\,1-PEGP\@0-\@1*:0 -al #zpool create raid2 raidz2 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@11:0 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@12:0 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@13:0 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@14:0 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@15:0 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@16:0 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@17:0 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@18:0 #Now it's online and created. I do a scrub and it goes down hard. ComicBookGuy:~ root# zpool status -v pool: raid2 state: SUSPENDED status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC scan: scrub in progress since Fri Apr 14 12:27:29 2023 1.37M scanned at 3.66K/s, 0B issued at 0B/s, 3.76M total 0B repaired, 0.00% done, no estimated completion time config: NAME STATE READ WRITE CKSUM raid2 UNAVAIL 0 0 0 insufficient replicas raidz2-0 UNAVAIL 0 0 0 insufficient replicas PCI0@0-PEG1@1,1-PEGP@0-@11:0 FAULTED 18 0 0 too many errors PCI0@0-PEG1@1,1-PEGP@0-@12:0 FAULTED 15 0 0 too many errors PCI0@0-PEG1@1,1-PEGP@0-@13:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@14:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@15:0 FAULTED 12 0 0 too many errors PCI0@0-PEG1@1,1-PEGP@0-@16:0 FAULTED 9 0 0 too many errors PCI0@0-PEG1@1,1-PEGP@0-@17:0 FAULTED 6 0 0 too many errors PCI0@0-PEG1@1,1-PEGP@0-@18:0 ONLINE 0 0 0 errors: List of errors unavailable: pool I/O is currently suspended #I'm switching back to the 3m copper cable #On copper it works fine: ComicBookGuy:root root# zpool status pool: raid2 state: ONLINE scan: scrub repaired 0B in 00:00:01 with 0 errors on Fri Apr 14 12:44:04 2023 config: NAME STATE READ WRITE CKSUM raid2 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@0:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@1:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE 0 0 0 errors: No known data errors ComicBookGuy:root root# dd if=/dev/zero of=/Volumes/raid2/50g-zero bs=1m count=51200 51200+0 records in 51200+0 records out 53687091200 bytes transferred in 106.644582 secs (503420710 bytes/sec) ComicBookGuy:root root# dd if=/dev/zero of=/Volumes/raid2/100g-zero bs=1m count=102400 102400+0 records in 102400+0 records out 107374182400 bytes transferred in 209.963624 secs (511394214 bytes/sec) #Performance looks good. #Maybe it's the number of disks? #Switch back to optical 100m and make a 4 disk raid2 Enclosure 0: Areca Te ARC-403801.37.69 0137 Enclosure Status: Ok Slot Count: 16 ID Slot Vendor Product / Rev Group Name Capacity Serial Number ---- ---- -------- ---------------- ---- ---------------- ---------- 4 4 ATA ST4000NM0033-9ZM GA6E *Available 3.64 TB SN:Z1Z2XD5M 5 5 ATA ST4000NM0033-9ZM GA6E *Available 3.64 TB SN:Z1Z5WW7T 6 6 ATA ST4000NM0033-9ZM GA6E *Available 3.64 TB SN:Z1Z2WN24 7 7 ATA ST4000NM0033-9ZM SN07 *Available 3.64 TB SN:Z1ZAK739 ComicBookGuy:root root# zpool create raid2-4 raidz2 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@4:0 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@6:0 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@5:0 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@7:0 ComicBookGuy:root root# zpool status pool: raid2-4 state: ONLINE config: NAME STATE READ WRITE CKSUM raid2-4 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE 0 0 0 errors: No known data errors ComicBookGuy:root root# dd if=/dev/zero of=/Volumes/raid2-4/50g-zero bs=1m count=51200 51200+0 records in 51200+0 records out 53687091200 bytes transferred in 158.096684 secs (339583917 bytes/sec) #I do a scrub and it fails on the optical cable. ComicBookGuy:root root# zpool status pool: raid2-4 state: SUSPENDED status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC scan: scrub in progress since Fri Apr 14 13:09:10 2023 100G scanned at 4.36G/s, 41.3G issued at 1.80G/s, 100G total 0B repaired, 41.21% done, 00:00:32 to go config: NAME STATE READ WRITE CKSUM raid2-4 ONLINE 0 0 0 raidz2-0 ONLINE 933 30 0 PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE 186 273 0 PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE 148 290 0 PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE 186 148 0 PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE 4 153 0 errors: 482 data errors, use '-v' for a list #Now we try a stripe, as myabe it's the raidz having issues. #Ok, 4 disk stripe now on 100m fiber zpool create stripe4 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@0:0 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@2:0 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@1:0 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@3:0 ComicBookGuy:root root# zpool status pool: stripe4 state: ONLINE config: NAME STATE READ WRITE CKSUM stripe4 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@0:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@1:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE 0 0 0 errors: No known data errors #DD'd the 50g file over, it was slower, like 70mbit/s ComicBookGuy:root root# zpool scrub stripe4 ComicBookGuy:root root# zpool status pool: stripe4 state: SUSPENDED status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC scan: scrub in progress since Fri Apr 14 13:56:17 2023 5.35G scanned at 914M/s, 6.54G issued at 1.09G/s, 5.35G total 0B repaired, 122.06% done, no estimated completion time config: NAME STATE READ WRITE CKSUM stripe4 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@0:0 ONLINE 72 0 2 PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE 52 0 0 PCI0@0-PEG1@1,1-PEGP@0-@1:0 ONLINE 32 0 0 PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE 66 0 0 errors: 40728 data errors, use '-v' for a list #same thing, it can't read and gives tons of errors. #Lets try a 4 disk mirror รง4 mirror /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@5:0 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@6:0 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@7:0 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@8:0 zpool status m4 pool: m4 state: ONLINE config: NAME STATE READ WRITE CKSUM m4 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE 0 0 0 errors: No known data errors ComicBookGuy:root root# dd if=/dev/zero of=/Volumes/m4/50g-zero bs=1m count=51200 51200+0 records in 51200+0 records out 53687091200 bytes transferred in 283.789402 secs (189179338 bytes/sec) #we do a scrub ComicBookGuy:root root# zpool status m4 pool: m4 state: SUSPENDED status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC scan: scrub in progress since Fri Apr 14 14:08:45 2023 50.0G scanned at 2.78G/s, 21.3G issued at 1.18G/s, 50.0G total 0B repaired, 42.50% done, 00:00:24 to go config: NAME STATE READ WRITE CKSUM m4 ONLINE 0 0 0 mirror-0 ONLINE 169K 0 0 PCI0@0-PEG1@1,1-PEGP@0-@5:0 ONLINE 128 260 0 PCI0@0-PEG1@1,1-PEGP@0-@6:0 ONLINE 122 232 0 PCI0@0-PEG1@1,1-PEGP@0-@7:0 ONLINE 128 0 0 PCI0@0-PEG1@1,1-PEGP@0-@8:0 ONLINE 116 113 0 #reboot and lets make a 2 disk mirror on the 100m cable. ComicBookGuy:root root# zpool create m2 mirror /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@1:0 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@2:0 ComicBookGuy:root root# zpool status pool: m2 state: ONLINE config: NAME STATE READ WRITE CKSUM m2 ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@1:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE 0 0 0 errors: No known data errors ComicBookGuy:root root# dd if=/dev/zero of=/Volumes/m2/50g-zero bs=1m count=51200 51200+0 records in 51200+0 records out 53687091200 bytes transferred in 293.570962 secs (182876027 bytes/sec) ComicBookGuy:root root# zpool status pool: m2 state: SUSPENDED status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC scan: scrub in progress since Fri Apr 14 14:28:50 2023 50.0G scanned at 7.14G/s, 16.4G issued at 2.34G/s, 50.0G total 0B repaired, 32.77% done, 00:00:14 to go config: NAME STATE READ WRITE CKSUM m2 ONLINE 0 0 0 mirror-0 ONLINE 129K 0 0 PCI0@0-PEG1@1,1-PEGP@0-@1:0 ONLINE 64 0 0 PCI0@0-PEG1@1,1-PEGP@0-@2:0 ONLINE 64 61 0 errors: 132087 data errors, use '-v' for a list # ok lets mirror the two 12g sas disks I have zpool create sas-m mirror /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@4:0 /var/run/disk/by-path/PCI0@0-PEG1@1,1-PEGP@0-@3:0 ComicBookGuy:root root# zpool status sas-m pool: sas-m state: ONLINE config: NAME STATE READ WRITE CKSUM sas-m ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE 0 0 0 ComicBookGuy:root root# dd if=/dev/zero of=/Volumes/sas-m/100g-zero bs=1m count=102400 102400+0 records in 102400+0 records out 107374182400 bytes transferred in 526.444670 secs (203961002 bytes/sec) ComicBookGuy:root root# zpool scrub sas-m ComicBookGuy:root root# zpool status sas-m pool: sas-m state: ONLINE scan: scrub in progress since Fri Apr 14 16:29:37 2023 115G scanned at 5.77G/s, 4.06G issued at 208M/s, 115G total 0B repaired, 3.52% done, 00:09:08 to go config: NAME STATE READ WRITE CKSUM sas-m ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE 0 0 0 errors: No known data errors ComicBookGuy:root root# zpool status sas-m pool: sas-m state: ONLINE scan: scrub in progress since Fri Apr 14 16:29:37 2023 115G scanned at 333M/s, 79.4G issued at 229M/s, 115G total 0B repaired, 68.78% done, 00:02:41 to go config: NAME STATE READ WRITE CKSUM sas-m ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE 0 0 0 errors: No known data errors ComicBookGuy:root root# zpool status sas-m pool: sas-m state: ONLINE scan: scrub repaired 0B in 00:08:35 with 0 errors on Fri Apr 14 16:38:12 2023 config: NAME STATE READ WRITE CKSUM sas-m ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@4:0 ONLINE 0 0 0 PCI0@0-PEG1@1,1-PEGP@0-@3:0 ONLINE 0 0 0 errors: No known data errors it scrubs and finishes on SAS just fine. The only variable here is SATA disks. I've tried HGST, SEGATE, and WD SATA disks, all have the same issues in the ARECA controller. I've tried them in both enclosures as well, with the same issues across both enclosures. If SATA disks are connected directly to the ATTO controller, they work fine. If SATA disks are in the ARECA enclosure, and it's connected via copper cables, they work fine. If SATA disks are in the ARECA enclosure and connected via the 10m cable they work, but will not be auto detected unless 'edfb off' is active on the enclosure Is it a latency issue on the 100m cable? It works fine with SAS disks, so IDK. The expander controller should handle it.