tree f4740ee21401b39ee67d475e6325f453345c6d06
parent 65d1753ca99624e9f6e8479b66a82764a75d4b88
author Jonathan Doman <jonathan.doman@intel.com> 1682534739 -0700
committer Jonathan Doman <jonathan.doman@intel.com> 1682715380 -0700

Add rate limiting

A host CPU can write POST codes much faster than the BMC can handle
them, considering all the D-Bus/IPC work required. Ideally `dbus-broker`
would apply backpressure when it gets full of unhandled signals, but its
quota mechanism uses a simple per-user accounting that doesn't
differentiate between all the connections from OpenBMC daemons running
as root. So there is no way to configure it to prevent just `snoopd`
from sending too many messages - instead it will disconnect arbitrary
services leading to mass chaos.

So without a D-Bus policy mechanism to prevent excess memory usage,
there are 2 different failure cases during a POST code storm:
1. `snoopd` continues to send messages faster than `post-code-manager`
   can process them, leading to `dbus-broker` consuming all the system
   memory.
2. `snoopd` fills up the D-Bus socket buffer. Once sd-bus fails to send
   a message across the socket, it starts queuing messages internally
   leading to `snoopd` consuming all the system memory. This only
   happens because we get stuck in the `snoopd` read loop during a POST
   code storm, and we don't process other events that would allow the
   write queue to drain.

As a workaround, introduce configurable rate limiting to `snoopd`. A new
meson option 'rate-limit' sets the corresponding '--rate-limit'
command-line parameter. These options take an integer value representing
the maximum number of POST codes to process per second. The default
meson option value is 1000, and the value of 0 will disable rate limiting.

Tested: Ran the POST code stress on host for 30 minutes:
```
[root@sut ~]# stress-ng --ioport 2
```

Watched BMC process memory usage and CPU usage in `top`, verified that
`post-code-manager`, `dbus-broker`, and `snoopd` each used less than 10%
CPU and 2% memory on AST2600 with 512 MiB of DRAM.

Change-Id: If03a01e0cd62366d188109bb4dff52958346e1db
Signed-off-by: Jonathan Doman <jonathan.doman@intel.com>