3

i know that my setup isn't supported officially by this kind of hardware, but still i am really astonished how bad it behaves. i am running a ML350 Gen9 with 128GB of RAM, dual Xeon E5-2630 V3 on which i am using the P440ar to run 8 SSDs Crucial MX500 4TB and a P840 with 14 HDDs Seagate Barracuda SATA. Both hardware raid card have the latest firmware, so as the SAS expander cards and the rest of the server overall (used the latest iso firmware image available from HP). I configured the 8 SSDs in Raid10, tried several stip size and disable the smart path, tried with or without physical cache activated, i really have bad performance when using fio :

read 4K : 80MB/s write 4K : 65MB/s write 1M : 750MB/s read 1M : 3140MB/s

with 8 drive and 2 lines for the link to the RAID card, i assume that i should have 24GB bandwidth, so the read looks to perform well, as i run the fio command using a 10GB file size, so more than the 2GB RAM the controller have. However, the write and 4K performance doesn't look good for a RAID 10 of 8 drives to me.

I plan to install Proxmox to run VMs and containers, various purpose. I need to do additional tests of the Raid 5 volume, but it was even performing less good than that from what i recall. What am i missing ? I understand that these are consumer drives, but still with such amount of drives it should perform better, shouldn't it ? I am not an expert so if you want additional information of course, let me know.

Edit : I did the tests in RAID5 for the HDD, here is the results :

Read 4K : 40,5MB/s Write 4K : 7MB/s Read 1M : 420MB/s Write 1M : 330MB/s

So same terrible performance especially with 4K that we saw with the RAID10 above. As suggested by a member below, i tried to switch my Proxmox to ZFS after placed both controllers in HBA mode. Here is some benchmarks since i now switched to ZFS with the SSDs

Read 4K : 823MB/s Write 4K : 537MB/s Read 1M : 1792MB/s Write 1M : 1892MB/s

Each of the tests run for some minutes, making sure that not only the RAM usage talks and i take the average bandwidth from fio. I also did some test in a VM of OpenMediaVault, to cross-check these numbers, i can say that minus few percentage drop due to the VM environment, the numbers follow the same trend. I need still to test the RaidZ 5 variant, will share the results.

mauro23
  • 31
  • 2
  • 2
    "not supported" = "will not be answered on ServerFault as off topic" – Nikita Kipriyanov Aug 05 '23 at 05:34
  • I was running a DL380e Gen8 server before, with a P822 and 2 MSA60 connected to it with a RAID 6 of 24x 2TB SAS HDD, but this consumes way too much power and i have to find an alternative. As this is to run at home, i tried to minimize the budget as even if i have a lot of services running and backups from some customers abroad, i don't want to overkill my budget with enterprise class 4TB storage drives. That's the reason why. Even if it is an "unsupported" way of using that server, i am really surprised that it performs so slow. – mauro23 Aug 05 '23 at 14:32
  • I would avoid unsupported configurations. You may try using the Starwind virtual appliance as it does support hardware RAID controllers. Ask their engineer if they can help you. https://www.starwindsoftware.com/vhci-appliance – Stuka Aug 08 '23 at 11:48

1 Answers1

4

Turn your HP P440ar RAID card into HBA by following the sequence below.

  1. Get into HP SmartArray configuration utility by pressing “F10” during the boot process
  2. Delete all previously created RAID sets by using "Clear configuration" option
  3. Select "Enable HBA Mode"
  4. Restart

You do ZFS with Proxmox then. Your consumer-grade SSDs don’t have PLP (power loss protection) so make sure you run some beefy UPS with Linux integration, the one UPS signals Linux it’s running on batt power so Linux could shut down VMs and itself gracefully (scripting is trivial, there’s a ton of them, they’re UPS specific though, Google for “PowerChute” term as an example).

Either way, by running everything in a 100% unsupported mode you’re walking on thin ice. Good luck!

BaronSamedi1958
  • 220
  • 1
  • 4
  • 1
    I will try to compare a ZFS deployment vs a hardware raid one with the same drives and the same configuration. To be honest, i had an existing deployment on an old server with a weekly report of my LUN using SSACLI, it would be a pitty to have to redo that with ZFS but if i can get a decent performance and safety of the RAID, why not. – mauro23 Aug 05 '23 at 14:34
  • So did some tests yesterday, in ZFS for both SSD array and HDD, i used RaidZ 10 and RaidZ 2, so i know that it doesn't compare 100% with the Raid5 i configured first, but i wanted originally to be in Raid6 but the terrible performance made me switch to Raid5 to check and it wasn't that much better. Overall, i noticed a huge performance improvement, but sometimes it still behaves strangely and i am wondering why. I got better, but still rather slow (in my opinion) performance with ZFS default setting. Then i turned some settings off like 'compression', 'atime' and 'sync'. I saw improvement. – mauro23 Aug 07 '23 at 04:43