Author: Ivan Mikhaylov, i.mikhaylov@yadro.com
Other contributors: Alexander Amelkin, a.amelkin@yadro.com Alexander Filippov, a.filippov@yadro.com
Created: 2019-07-23
End users of OpenBMC may take actions that change the system state and/or configuration. Such actions may be taken using any of the numerous interfaces provided by OpenBMC. That includes RedFish, IPMI, ssh or serial console shell, and other interfaces, including the future ones.
Consequences of those actions may sometimes be harmful and an investigation may be conducted in order to find out the person responsible for the unwelcome changes. Currently, most changes leave no trace in OpenBMC logs, which hampers the aforementioned investigation.
It is required to develop a mechanism that would allow for tracking such user activity, logging it, and taking certain actions if necessary.
YADRO had an internal solution for the problem. It was only applicable to an outdated version of OpenBMC and needed a redesign. There was also a parallel effort by IBM that can be found here: REST and Redfish Traffic Logging
This design assumes that an end user is never given a direct access to the system shell. The shell allows for direct manipulation of user database (add/remove users, change passwords) and system configuration (network scripts, etc.), and it doesn't seem feasible to track such user actions taken within the shell. This design assumes that all user interaction with OpenBMC is limited to controlled interfaces served by other Phosphor OpenBMC components interacting via D-Bus.
The main idea is to catch D-Bus requests sent by user interfaces, then handle the request according to the configuration. In future, support for flexible policies may be implemented that would allow for better flexibility in handling and tracking.
The phosphor-audit service represents a service that provides user activity tracking and corresponding action taking in response of user actions.
The key benefit of using phosphor-audit is that all action handling will be kept inside this project instead of spreading it across multiple dedicated interface services with a risk of missing a handler for some action in one of them and bloating the codebase.
The component diagram below shows the example of service overview.
+----------------+ audit event +-----------------+ | IPMI NET +-----------+ | action | +----------------+ | | +-------------+ | | | | logging | | +----------------+ | | +-------------+ | | IPMI HOST +-----------+ +--------------+ | | +----------------+ | | audit | | +-------------+ | +----->+ service +----->| | command | | +----------------+ | | | | +-------------+ | | RedFish/REST +-----------+ +--------------+ | | +----------------+ | | +-------------+ | | | | notify | | +----------------+ | | +-------------+ | | any service +-----------+ | | +----------------+ | +-------------+ | | | ... | | | +-------------+ | +-----------------+
The audit event from diagram generated by an application to track user activity. The application sends 'signal' to audit service via D-Bus. What is happening next in audit service's handler depends on user requirements and needs. It is possible to just store logs, run arbitrary command or notify someone in handler or we can do all of the above and all of this can be optional.
Audit event call
Audit event call performs preprocessing of incoming data at application side before sending it to the audit service, if the request is filtered out, it will be dropped at this moment and will no longer be processed. After the filter check, the audit event call sends the data through D-Bus to the audit service which makes a decision regarding next steps. Also, it caches list of possible commands (blacklist or whitelist) and status of its service (disabled or enabled). If the service in undefined state, the call checks if service alive or not.
audit_event(type, rc, request, user, host, data)
- type - type of event source : IPMI, REST, PAM, etc.
- rc - return code of the handler event (status, rc, etc.)
- request - a generalized identifier of the event, e.g. ipmi command (cmd/netfn/lun), web path, or anything else that can describe the event.
- user - the user account on behalf of which the event was processed. depends on context, NA/None in case of user inaccessibility.
- source - identifier of the host that the event has originated from. This can be literally "host" for events originating from the local host (via locally connected IPMI), or an IP address or a hostname of a remote host.
- data - any supplementary data that can help better identify the event (e.g., some first bytes of the IPMI command data).
Service itself can control flow of events with configuration on its side.
Pseudocode for example:
audit_event(NET_IPMI, "access denied"(rc=-1), "ipmi cmd", "qwerty223",
"192.168.0.1", <some additional data if needed>)
audit_event(REST, "login successful"(rc=200), "rest login",
"qwerty223", "192.168.0.1", NULL)
audit_event(HOST_IPMI, "shutting down the host"(rc=0), "host poweroff",
NULL, NULL, NULL)
audit_event(blob_data) Blob can be described as structure:
struct blob_audit
{
uint8_t type;
int32_t rc;
uint32_t request_id;
char *user;
sockaddr_in6 *addr;
struct iovec *data;
}
When the call reaches the server destination via D-Bus, the server already knows that the call should be processed via predefined list of actions which are set in the server configuration.
Step by step execution of call:
How the checks will be processed at client's layer:
The configuration structure can be described as tree with set of options, as example of structure:
[IPMI]
[Enabled]
[Whitelist]
[Cmd 0x01] ["reset request"]
[Cmd 0x02] ["hello world"]
[Cmd 0x03] ["goodbye cruel world"]
[Actions]
[Notify type1] [Recipient]
[Notify type2] [Recipient]
[Notify type3] [Recipient]
[Logging type] [Options]
[Exec] [ExternalCommand]
[REST]
[Disabled]
[Blacklist]
[Path1] [Options]
[Path2] [Options]
[Actions]
[Notify type2] [Recipient]
[Logging type] [Options]
Options can be updated via D-Bus properties. The audit service listens changes on configuration file and emit 'PropertiesChanged' signal with changed details.
Possible list of requests which have to be filtered and processed. 'Whitelist' filters possible requests which can be processed. 'Blacklist' blocks only exact requests.
Each audit processing type can be disabled or enabled at runtime via config file or D-Bus property.
The end recipient notification system with different transports.
phosphor-logging, journald or anything else suitable for.
Running a command as consequenced action.
An example of possible flow:
+----------------+ | NET IPMI | | REQUEST | +----------------+ | +--------------------------------------------------------------------------+ | +-------v--------+ IPMI | | | NET IPMI | | | +----------------+ | | | | | +-------v--------+ +---------------------------+ | | | rc = handle() +------->| audit_event<NET_IPMI>() | | | +----------------+ +---------------------------+ | | | | | | | | | | +-------v--------+ | | | | Processing | | | | | further | | | | +----------------+ | | +--------------------------------------------------------------------------+ | | +--------------------------------------------------------------------------+ | +-----------------------------+ | | | Audit Service | | | | | | | | | | | +-----v------+ | | NO | Is logging | YES | | +------+ enabled +--------------------+ | | | | for type? | | | | | +------------+ +-------v-----+ | | | NO | Is request | YES | | | +--------+ type +--------+ | | | | | filtered? | | | | | | +-------------+ | | | | | | | | | +-------v-------+ | | | | | Notify | | | | | | Administrator | | | | | +---------------+ | | | | | | | | | +-------v-------+ | | | | | Log Event | | | | | +---------------+ | | | | | | | | | +-------v-------+ | | | | | User | | | | | | actions | | | | | +---------------+ | | | | | | | | | +-------v-------+ | | | +-------------->| End |<----------------------+ | | +---------------+ | | | +--------------------------------------------------------------------------+
The unified model for reporting accidents to the end user, where the transport can be:
Sending a note to directed recipient which set in configuration via sendmail or anything else.
SNMP
Sending a notification via SNMP trap messages to directed recipient which set in configuration.
Instant messengers
Sending a notification to directed recipient which set in configuration via jabber/sametime/gtalk/etc.
D-Bus
Notify the other service which set in configuration via 'method_call' or 'signal'.
Notifications will be skipped in case if there is no any of above configuration rules is set inside configuration. It is possible to pick up rules at runtime.
Processing user requests in each dedicated interface service and logging them separately for each of the interfaces. Scattered handling looks like an error-prone and rigid approach.
Improves system manageability and security.
Impacts when phosphor-audit is not enabled:
Impacts when phosphor-audit is enabled: All of the above, plus:
dbus-send as command-line tool for generating audit events.
Scenarios: