Three of my Huawei 2288H V5 nodes started throwing storage I/O errors after I upgraded the cluster to Proxmox VE 9. The RAID controller — a Broadcom/LSI SAS3508 in OEM Huawei trim — was loading fine, VMs were starting, but under any sustained disk load I was seeing megaraid_sas: FW reported fatal error in dmesg and controllers dropping offline. Huawei support eventually confirmed what I suspected: firmware 5.140.00-3319 is EOL, last validated on Debian 10, and there is no supported upgrade path for this OEM variant.
The upstream megaraid_sas driver in kernel 6.x tightened error handling and command timeout behavior. On older firmware that predates some of those ABI changes, the driver and firmware end up in a disagreement that looks like a hard fault. You’re not getting a kernel bug — you’re getting a firmware that can no longer keep up with the driver’s expectations.
The firmware EOL situation is worse than it sounds
Broadcom’s own release of the SAS3508 firmware has been updated well past 5.140. The problem is OEM variants. Huawei, Dell, HPE — they all ship locked firmware images that only flash via their own tools against their own hardware IDs. You cannot take a Broadcom reference firmware and flash it onto a Huawei OEM card. The controller will refuse it. And Huawei has quietly stopped publishing updated firmware for the 2288H V5 generation.
Running storcli64 /c0 show confirms the situation:
storcli64 /c0 show | grep -E "FW Package|Controller Status|ROC temperature"
FW Package Build = 5.140.00-3319
Controller Status = Degraded
ROC temperature(Degree Celsius) = 62
The Degraded controller status here is the driver-level report — not necessarily a physical failure. It means the driver has lost confidence in the firmware’s ability to handle its command queue.
megaraid_sas module parameters can buy you stability
The in-kernel megaraid_sas driver exposes several tunable parameters. Two of them matter here: msix_vectors and throttlequeuedepth.
By default on modern kernels, megaraid_sas tries to negotiate a large number of MSI-X vectors and a deep command queue. Older firmware has lower internal limits. When the driver pushes more concurrent commands than the firmware can track, it trips the firmware’s internal watchdog and you get the fatal error event.
Drop both values in a modprobe config:
cat /etc/modprobe.d/megaraid_sas.conf
options megaraid_sas msix_vectors=1 throttlequeuedepth=16
Then rebuild initramfs and reboot:
update-initramfs -u -k all
reboot
After the reboot, verify the parameters took effect:
cat /sys/module/megaraid_sas/parameters/msix_vectors
cat /sys/module/megaraid_sas/parameters/throttlequeuedepth
Both should reflect your set values. Check dmesg for the driver init lines:
dmesg | grep megaraid
[ 4.123456] megaraid_sas: Loading driver, version 07.727.03.00-rc1
[ 4.234567] megaraid_sas 0000:02:00.0: firmware version: 5.140.00-3319
[ 4.345678] megaraid_sas 0000:02:00.0: MSI-X vectors requested: 1
[ 4.456789] megaraid_sas 0000:02:00.0: queue depth: 16
On my three nodes, this eliminated the fatal error events under normal VM workloads. It reduces theoretical throughput — you’re capping parallelism — but on a node where the alternative is a controller going offline mid-workload, it’s the correct trade.
Baseline the array health before trusting it
Before you stop worrying, confirm the array itself is clean. The firmware instability can mask real background media errors that got silently swallowed.
storcli64 /c0/eall/sall show | grep -E "DG|State|Med|Pred"
Look for anything under Med (media errors) or Pred (predictive failure flag). A non-zero media error count on a drive that’s been running through these crash cycles is worth taking seriously.
Full array status at a glance:
storcli64 /c0/vall show
--------------------------------------------------------------
DG/VD TYPE State Access Consist Cache Cac sCC Size Name
--------------------------------------------------------------
0/0 RAID5 Optl RW Yes RWTD - ON 10.913 TB
--------------------------------------------------------------
Optl (Optimal) with Consist = Yes means the array is clean and consistent. If you see Pdgd (Partially Degraded) or Dgrd (Degraded), stop here and rebuild before doing anything else.
Kernel cmdline fallback for early boot crashes
If the node is crashing before it even gets to userspace — before update-initramfs gets a chance to apply the modprobe config — you need to set the parameters via the kernel command line instead.
Edit /etc/default/grub:
GRUB_CMDLINE_LINUX_DEFAULT="quiet megaraid_sas.msix_vectors=1 megaraid_sas.throttlequeuedepth=16"
update-grub
reboot
This is also the approach to use on nodes where the initramfs itself is on the RAID volume — you don’t want to be in a position where the driver panics before it can load the config from disk.
How the driver-firmware handshake breaks down
┌─────────────────────────────────────────────────────────────┐
│ Kernel 6.x megaraid_sas driver (07.727.x) │
│ │
│ Default: msix_vectors=16, queue_depth=256 │
│ │ │
│ ▼ │
│ Sends 256 concurrent IOPs + MSI-X negotiation │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────┐ │
│ │ SAS3508 OEM FW 5.140.00-3319 (EOL) │ │
│ │ Internal queue limit: ~32 commands │ │
│ │ MSI-X vectors available: 1-2 │ │
│ │ Result: FIRMWARE WATCHDOG TRIPS → FATAL EVENT │ │
│ └─────────────────────────────────────────────────┘ │
│ │
│ With: msix_vectors=1, queue_depth=16 │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────┐ │
│ │ SAS3508 OEM FW 5.140.00-3319 │ │
│ │ Queue fits within firmware limits │ │
│ │ Result: STABLE │ │
│ └─────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
Long-term: plan around this hardware
The throttlequeuedepth=16 fix is a mitigation, not a solution. You are running production VMs on a controller with EOL firmware that the vendor has no intention of updating. The stability window you’ve bought is narrow — a future kernel update could change driver behavior in a way that breaks even the throttled config.
If these nodes are in your critical path, start the hardware replacement conversation now. If you’re on a budget cycle, at minimum mirror your most important VMs to nodes with supported controllers and set up Proxmox HA groups to keep those VMs off the SAS3508 nodes under normal scheduling:
# Pin a VM away from the affected nodes using HA groups
ha-manager add vm:100 --group no-sas3508-nodes --state started
Where no-sas3508-nodes is an HA group configured in the Proxmox datacenter that excludes the affected nodes.
pvesh set /cluster/ha/groups/no-sas3508-nodes \
--nodes "pve04,pve05,pve06" \
--nofailback 0 \
--restricted 1
That keeps your critical VMs off the degraded hardware while you work the replacement timeline.