Disk Layout

Boot disk = 128g mirror ZFS

ZFS storage

5 vdev's mirrored of the SAS disks 
2 mirror NVME 256g partition, but dedicated on the whole disk.  This can grow.
optional log and l2arc on the boot NVME's

NVME name spaces

The NVME come setup as 1.88T disks:

tnvmcap   : 1,880,375,648,256
unvmcap   : 375,648,256

I suspect this is a 2.0 TiB (2,199,023,255,552b) provisioned down in the controller or about 85%. moving this to a 1.56TB disk will under provision this and make it perform better in the event we use it as log or write intensive.