Add rate limiting
A host CPU can write POST codes much faster than the BMC can handle
them, considering all the D-Bus/IPC work required. Ideally `dbus-broker`
would apply backpressure when it gets full of unhandled signals, but its
quota mechanism uses a simple per-user accounting that doesn't
differentiate between all the connections from OpenBMC daemons running
as root. So there is no way to configure it to prevent just `snoopd`
from sending too many messages - instead it will disconnect arbitrary
services leading to mass chaos.
So without a D-Bus policy mechanism to prevent excess memory usage,
there are 2 different failure cases during a POST code storm:
1. `snoopd` continues to send messages faster than `post-code-manager`
can process them, leading to `dbus-broker` consuming all the system
memory.
2. `snoopd` fills up the D-Bus socket buffer. Once sd-bus fails to send
a message across the socket, it starts queuing messages internally
leading to `snoopd` consuming all the system memory. This only
happens because we get stuck in the `snoopd` read loop during a POST
code storm, and we don't process other events that would allow the
write queue to drain.
As a workaround, introduce configurable rate limiting to `snoopd`. A new
meson option 'rate-limit' sets the corresponding '--rate-limit'
command-line parameter. These options take an integer value representing
the maximum number of POST codes to process per second. The default
meson option value is 1000, and the value of 0 will disable rate limiting.
Tested: Ran the POST code stress on host for 30 minutes:
```
[root@sut ~]# stress-ng --ioport 2
```
Watched BMC process memory usage and CPU usage in `top`, verified that
`post-code-manager`, `dbus-broker`, and `snoopd` each used less than 10%
CPU and 2% memory on AST2600 with 512 MiB of DRAM.
Change-Id: If03a01e0cd62366d188109bb4dff52958346e1db
Signed-off-by: Jonathan Doman <jonathan.doman@intel.com>
4 files changed