Storage Server
My documents on building a storage server and proxmox host.
Parts
- X10DSC+ Motherboard
- AOM-S3108M-H8L RAID/HBA
- AOC-MTG-I4S quad SFP+ NIC
- 8x32 GB RAM (256g)
- 24x Seagate ST16000NM002G Exos X16 16TB 12Gb/s SAS
- 375 GB Optane SSD for l2arc/ZIL SSDPEL1K375GA01
- 3 4tb m2 SSD for Special Devices
Disk Layout
ZFS zraid2
Zpool 3 - 6 disk raidz2 1 - 5 disk raidz2 Special - 3 m2 flash Zil - 16 GiB - optane l2arc - 333 GiB - optane Note optane is shared between Zil and l2arc
Space on this should be 15 disks of 16 tb, with redundancy or 240tb. This will leave one disk open as a hot spare.
The sata disks will be linux boot disks in a standard linux raid (maybe zfs mirror?)
SAS controller
The built in controller supports HBA mode per supermicro.
https://docs.broadcom.com/doc/pub-005110
Updating Disks
Firmware
Current firmware on the disks is E004, but disks have E002 on them
for i in `seq 2 25` ; do SeaChest_Firmware --downloadFW ./EvansExosX16SAS-STD-512E-E004.LOD -d /dev/sg$i; done #SeaChest_Firmware -s ========================================================================================== SeaChest_Firmware - Seagate drive utilities - NVMe Enabled Copyright (c) 2014-2021 Seagate Technology LLC and/or its Affiliates, All Rights Reserved SeaChest_Firmware Version: 3.0.0-2_2_1 X86_64 Build Date: Apr 27 2021 Today: Tue Oct 10 22:14:38 2023 User: root ========================================================================================== nvme_ioctl_id: Inappropriate ioctl for device Vendor Handle Model Number Serial Number FwRev LSI /dev/sg0 SAS3x28 0705 LSI /dev/sg1 SAS3x28 0705 SEAGATE /dev/sg10 ST16000NM002G ZL20AJ3P E004 SEAGATE /dev/sg11 ST16000NM002G ZL231860 E004 SEAGATE /dev/sg12 ST16000NM002G ZL21T6Q1 E004 SEAGATE /dev/sg13 ST16000NM002G ZL231PV0 E004 SEAGATE /dev/sg14 ST16000NM002G ZL21TDHH E004 SEAGATE /dev/sg15 ST16000NM002G ZL21Y85R E004 SEAGATE /dev/sg16 ST16000NM002G ZL21S9FQ E004 SEAGATE /dev/sg17 ST16000NM002G ZL21T38M E004 SEAGATE /dev/sg18 ST16000NM002G ZL22C3MJ E004 SEAGATE /dev/sg19 ST16000NM002G ZL21S9J3 E004 SEAGATE /dev/sg2 ST16000NM002G ZL21S9WK E004 SEAGATE /dev/sg20 ST16000NM002G ZL21S9AW E004 SEAGATE /dev/sg21 ST16000NM002G ZL21TGY1 E004 SEAGATE /dev/sg22 ST16000NM002G ZL20CRL7 E004 SEAGATE /dev/sg23 ST16000NM002G ZL21RP9E E004 SEAGATE /dev/sg24 ST16000NM002G ZL21RNZW E004 SEAGATE /dev/sg25 ST16000NM002G ZL21JYXF E004 ATA /dev/sg26 Samsung SSD 870 EVO 1TB S75BNL0W812633P SVT03B6Q ATA /dev/sg27 Samsung SSD 870 EVO 1TB S75BNS0W642820L SVT03B6Q SEAGATE /dev/sg3 ST16000NM002G ZL21V48W E004 SEAGATE /dev/sg4 ST16000NM002G ZL21T7XK E004 SEAGATE /dev/sg5 ST16000NM002G ZL21T8HS E004 SEAGATE /dev/sg6 ST16000NM002G ZL21SBMS E004 SEAGATE /dev/sg7 ST16000NM002G ZL21SRYP E004 SEAGATE /dev/sg8 ST16000NM002G ZL21LVPQ E004 SEAGATE /dev/sg9 ST16000NM002G ZL21TB40 E004 NVMe /dev/nvme0n1 Samsung SSD 990 PRO 4TB S7KGNJ0W912464T 0B2QJXG7
Low Level Format
It's necessary to low level format these disks as we need to turn off Protection Type 2, test the sectors of the drive and make it a 4096 sector size.
# SeaChest_Format --protectionType 0 --formatUnit 4096 --confirm this-will-erase-data --poll -d /dev/sg2
