IPMI Architecture

IPMI is a hard problem to solve. New bits bolted onto legacy stuff makes for quite the Frankensteinian monster. If we break it apart again and look at each piece, maybe we can sleep with fewer nightmares.

High-Level Overview

IPMI is all about commands and responses. Channels provide a mechanism for transporting the data, each with a slightly different protocol and transport layer, but ultimately, the highest level data is a raw IPMI command consisting of a NetFn/LUN, Command, and optional data. Each response is likewise a Completion Code and optional data. So the first step is to break apart channels and the IPMI queue.

___________          ___________________
| KCS/BT  |          |                 |
| Channel | <------> |                 |
----------/          | IPMI Daemon     |
-----------          |   (ipmid)       |
| RMCP+   |          |                 |
| Channel | <------> |                 |
----------/          ------------------/

The IPMI messages that get passed to and from the IPMI daemon (ipmid) are basically the equivalent of ipmitool's "raw" commands with a little more information about the message.

Message Data:
  byte:userID - IPMI user ID of caller (0 for session-less channels)
  enum:privilege - ADMIN, USER, OPERATOR, CALLBACK; will be less than or
    equal to the privilege of the user and less than or equal to the max
    privilege of this channel
  enum:channel - what channel the request came in on (LAN0, LAN1, KCS/BT,
    IPMB0, etc.), also used to route the response back to the caller.
  integer:msgID - identifier for the message to match with the response
  byte:LUN - LUN from netFn/LUN pair (0-3, as per the IPMI spec)
  byte:netFn - netFn from netFn/LUN pair (as per the IPMI spec)
  byte:cmd - IPMI command ID (as per the IPMI spec)
  array<byte>:data - optional command data (as per the IPMI spec)
Response Data:
  enum:channel - what channel the request came in on
  integer:msgID - what request this response matches
  byte:CC - IPMI completion code
  array<byte>:data - optional response data

The next part is to provide a higher-level, strongly-typed, modern C++ mechanism for registering handlers. Each handler will specify exactly what arguments are needed for the command and what types will be returned in the response. This way, the ipmid queue can unpack requests and pack responses in a safe manner.

To be able to operate in a manner like the current IPMI provider system works, the registration mechanism will need to be able to either be mostly header-only or otherwise runtime linkable so that external provider libraries can be used to add IPMI commands.

Details and Implementation

For session-less channels (like BT, KCS, and IPMB), the only privilege check will be to see that the requested privilege is less than or equal to the channel's maximum privilege. If the channel has a session and authenticates users, the privilege must be less than or equal to the channel's maximum privilege and the user's maximum privilege.

Ipmid takes the LUN/netFN/Cmd tuple and looks up the corresponding handler function. If the requested privilege is less than or equal to the required privilege for the given registered command, the request may proceed. If any of these checks fail, ipmid returns with Insufficient Privilege.

At this point, the IPMI command is run through the filter hooks. The default hook is ACCEPT, where the command just passes onto the execution phase. But OEMs and providers can register hooks that would ultimately block IPMI commands from executing, much like the IPMI 2.0 Spec's Firmware Firewall. The hook would be passed in the context of the IPMI call and the raw content of the call and has the opportunity to return any valid IPMI completion code. Any non-zero completion code would prevent the command from executing and would be returned to the caller.

The next phase is parameter unpacking and validation. This is done by compiler-generated code with variadic templates at handler registration time. The registration function is a templated function that allows any type of handler to be passed in so that the types of the handler can be extracted and unpacked.

This can be done with something along these lines:

class ipmiQueue {
  template <typename MessageHandler, typename... InputArgs>
  auto register_ipmi_handler(
      const std::vector<enum ipmiChannel>& channelList,
      uint8_t lun, uint8_t netFn, uint8_t cmd,
      MessageHandler handler) {
    ...
  }
  template <typename MessageHandler, typename... InputArgs>
  auto register_ipmi_handler_async(
      const std::vector<enum ipmiChannel>& channelList,
      uint8_t lun, uint8_t netFn, uint8_t cmd,
      MessageHandler handler) {
    ...
  }
  template <typename MessageHandler, typename... ReplyArgs>
  auto async_reply(ipmi::handlerContext&, ReplyArgs) {
    ...
  }
};
 ...
 namespace ipmi {
   constexpr uint8_t appNetFn = 6;
   class handlerContext {
     enum ipmiChannel channel;
     uint32_t msgId;
     uint8_t userId;
     enum ipmiPriv privilege;
   };
   namespace app {
     constexpr uint8_t setUserAccessCmd = 0x43;
   }
 }

std::tuple<ipmi::compCode> setUserAccess(
    ipmi::handlerContext& context,
    uint1_t changeBit, // one bit integer type
    uint1_t callbackRestricted,
    uint1_t linkAuth,
    uint1_t ipmiEnable,
    uint4_t channelNumber,
    uint2_t reserved1,
    uint6_t userId,
    uint4_t reserved2,
    uint4_t privLimit,
    std::optional<uint4_t> reserved3,
    std::optional<uint4_t> userSessionLimit) {
  ...
  return std::tuple<ipmi::compCodeNormal>;
}

ipmi::register_ipmi_handler(ipmi::lun0, ipmi::appNetFn,
                            ipmi::app::setUserAccessCmd,
                            setUserAccess);
void getUserAccess(
    ipmi::handlerContext& context,
    uint4_t reserved1,
    uint4_t channelNumber,
    uint2_t reserved2,
    uint6_t userId) {
  ...
    auto reply = std::make_shared<std::tuple<ipmi::compCode,
      uint2_t, uint6_t, uint2_t, uint6_t, uint2_t, uint6_t, uint1_t,
      uint1_t, uint4_t>>();
    async_call([&]() {
      std::get<0>(*reply) = ipmi::compCodeNormal;
      std::get<2>(*reply) = ...;
      ipmi::async_reply(context, *reply);
    });
 ...
 }

ipmi::register_ipmi_handler_async(ipmi::lun0, ipmi::appNetFn,
                            ipmi::app::setUserAccessCmd,
                            setUserAccess);

Ideally, we would have support for asynchronous handling of IPMI calls. This means that the queue could have multiple in-flight calls that are waiting on another D-Bus call to return. With asynchronous calls, this will not block the rest of the queue's operation, allowing for maximum throughput and minimum delay. Synchronous calls would emit warnings if they hold up the queue for too long. If it is possible to do, it would be nice to abstract the D-Bus interface call interface so that it could put off returning the result to a function and handle other stuff while waiting. Then if the IPMI method only had D-Bus calls, it could be written in a synchronous method but still allow the rest of the queue to behave as if it was written as an asynchronous callback.

Passing the reply tuple in as a shared pointer allows for multiple levels of nested lambdas so that the owner is never destroyed and the lifetime of the object is preserved. This is helpful if multiple D-Bus calls need to be made to gather information. Alternately, it might only need to be generated in the last stage when something of value is generated. Either way, the tuple is ultimately passed into the templated async_reply function that packs the parameters back into a vector to send back to the requester. The context is guaranteed to be valid until either a synchronous call returns or async_reply is called.

Using templates, it is possible to extract the return type and argument types of the handlers and use that to unpack (and validate) the arguments from the incoming request and then pack the result back into a vector of bytes to send back to the caller. The deserializer will keep track of the number of bits it has unpacked and then compare it with the total number of bits that the method is requesting. In the example, we are assuming that the non-full-byte integers are packed bits in the message in most-significant-bit first order (same order the specification describes them in). Optional arguments can be used easily with C++17's std::optional (or using boost::optional for earlier C++). Actually calling the handler with the extracted tuple of arguments is easy with C++17's std::apply (or can be written by hand if necessary). The moral of the story here is that we should use C++17 since it is available with Yocto 2.4.

For multi-byte parameters, endianness matters, so we should define some types that can denote that: be_int32_t be_uint32_t, le_int32_t, le_uint32_t. Alternately, we could only specify big-endian variants because most of the IPMI spec uses little-endian representations.

To start with, we can implement the templated registration scheme, but still allow for a legacy registration method so that all the currently implemented IPMI handlers can still work until they have been rewritten to use the new mechanism. When all the current commands have been rewritten, we can remove the legacy interface. All commands registering with the legacy interface will get logged with a message saying that interface is deprecated.

Things that would be nice to have are as follows:

  • nested types for arguments: e.g., std::array<std::tuple<uint1_t, uint1_t, uint2_t, uint4_t>, 4>
  • a nested callback mechanism (the one that comes to mind is set/get lan parameters) where the handler is really ultimately split into subhandlers with different trailing parameters by examining the first few bytes. In this case, you read a few common bytes and then need to re-interpret the large trailing buffer. If we can provide the message parser to the handlers, then they can re-parse the big buffer using compiler-generated code rather than re-writing their own sub-parser.
  • C++17 (as noted above for std::apply and std::optional (and possibly other shiny goodness)