Author: Andrew Geissler (geissonator)
Other contributors:
Created: June 23rd, 2020
BMCs can reboot for a lot of different reasons. User request, firmware update, and a variety of different error scenarios. When the BMC is rebooted while the host is up and running, there needs to be a process by which the two synchronize with each other and the BMC gets itself into a state that matches with the host.
A good portion of this is explained in the phosphor-state-manager README.
This design doc is written to formalize the design and add some more details on dealing with both IPMI and PLDM communication to the host as well as desired behavior when unable to talk with the host.
The high level flow is that OpenBMC software first checks for pgood, and if set, it then checks the host to see if it is up and alive. If the power is on and the host is running, then files are created in the filesystem to indicate this:
It should be noted that although full support is not in place for multi-chassis and multi-host systems, the framework is there to build on. op-reset-chassis-running@.service is a templated service, checking pgood in its instances power domain. It creates a file in the filesystem, /run/openbmc/chassis@%i-on, to indicate power is on for that instance. Similar implementation is done for the host via phosphor-reset-host-check@.service and the file /run/openbmc/host@%i-on.
If chassis power is on and the host is up, then obmc-chassis-poweron@.target and obmc-host-start@.target are started.
The /run/ files are used by OpenBMC services to determine if they need to run or not when the chassis and host targets are started. For example, if the chassis is already powered on and the host is running, there is no need to actually turn power on, or start the host. The behavior wanted is that these services "start" but do nothing. That is commonly done within a systemd service file via the following:
ConditionPathExists=!/run/openbmc/chassis@%i-onConditionPathExists=!/run/openbmc/host@%i-onThis allows the targets to start and for the BMC to get in synch with the state of the chassis and host without any special software checks.
Different systems have different requirements on what the behavior should be when the chassis power is on, but the host is unreachable. This design needs to allow this customization. For example, some systems would prefer to just leave the system in whatever state it is in and let the user correct things. Some systems want to recover automatically (i.e. reboot the host) for the user. Some systems have a hybrid approach where depending on where the host was in its boot, the BMC may leave it or recover.
xyz.openbmc_project.Condition.HostFirmware, which will have a CurrentFirmwareCondition property which other applications can read to determine if the host is running.CurrentFirmwareCondition property should not return "Running" unless the Hypervisor firmware is running.High Level Flow:
xyz.openbmc_project.Condition.HostFirmware PDI interfaceCurrentFirmwareCondition property of all interface. If any call returns that a host is running then create file and start host target.IPMI and PLDM software will be started as applicable. A combination of systemd services and applications within phosphor-state-manager will coordinate the checking of pgood, and if set, request the IPMI and PLDM applications to discover if the host is running. Based on the response from these queries the software in phosphor-state-manager will take the appropriate action of creating the /run files and starting the chassis and host targets or entering into recovery of the host.
The systemd targets responsible for this and any common services will be hosted within phosphor-state-manager. Any system or company specific services can be installed in the common targets:
A separate service and application will be created within phosphor-state-manager to execute the following logic in situations where chassis power is on but the host has failed to respond to any of the different mechanisms to communicate with it:
IBM systems will utilize a processor CFAM register. The specific register is Mailbox scratch register 12.
If the chassis power is on but the BMC is unable to communicate with the host via IPMI or PLDM, then the BMC will read this processor CFAM register.
The Host code will write 0xA5000001 to this register to indicate when it has reached a state in which it can boot an operating system without needing the BMC. If the BMC sees this value written in the CFAM register, then it will leave the host as-is, even if it is unable to communicate with the host over IPMI or PLDM. It will log an error indicating it was unable to communicate with the host but it will also show the host state as Running.
If the register is not 0xA5000001, then the BMC will follow whatever recovery mechanisms are defined for when the host has a failure (most likely a reboot of the host).
It is the responsibility of the host firmware to set this register as applicable during the boot of the system. Host firmware will clear this register in shutdown scenarios. To handle different host crash scenarios, the register will also be cleared by the BMC firmware on power off's, system checkstops, and during Memory Preserving reboots.
One thought was to avoid IPMI/PLDM all together and only use a "lowest common denominator" hardware register of some sort. The problem with that is you start creating your own protocol, and before you know it you have something like IPMI or PLDM anyway, except you created something custom. So 99% of the time the IPMI or PLDM path will be fine, and as a backup option, system owners can put their own custom host-detection applications in.
None
The normal path of IPMI and PLDM will be simple to test. Boot your system and reboot the BMC. Verify the BMC chassis and host states are correct and verify the host continued to run without issue throughout the BMC reset.
The more complicated tests will be error paths: