OpenPower Platform Event Log (PEL) extension

This extension will create PELs for every OpenBMC event log. It is also possible to point to the raw PEL to use in the OpenBMC event, and then that will be used instead of creating one.

Contents

• Passing in data when creating PELs
• Default UserData sections for BMC created PELs
• The PEL Message Registry
• Callouts
• Action Flags and Event Type Rules
• D-Bus Interfaces
• PEL Retention
• Adding python3 modules for PEL UserData and SRC parsing
• Fail Boot on Host Errors
• SBE FFDC
• PEL Archiving
• Handling PELs for hot plugged FRUs

Passing PEL related data within an OpenBMC event log

An error log creator can pass in data that is relevant to a PEL by using certain keywords in the AdditionalData property of the event log.

AdditionalData keywords

RAWPEL

This keyword is used to point to an existing PEL in a binary file that should be associated with this event log. The syntax is:

RAWPEL=<path to PEL File>
e.g.
RAWPEL="/tmp/pels/pel.5"

The code will assign its own error log ID to this PEL, and also update the commit timestamp field to the current time.

POWER_THERMAL_CRITICAL_FAULT

This keyword is used to set the power fault bit in PEL. The syntax is:

POWER_THERMAL_CRITICAL_FAULT=<FLAG>
e.g.
POWER_THERMAL_CRITICAL_FAULT=TRUE

Note that TRUE is the only value supported.

SEVERITY_DETAIL

This is used when the passed in event log severity determines the PEL severity and a more granular PEL severity is needed beyond what the normal event log to PEL severity conversion could give.

The syntax is:

SEVERITY_DETAIL=<SEVERITY_TYPE>
e.g.
SEVERITY_DETAIL=SYSTEM_TERM

Option Supported:

• SYSTEM_TERM, changes the Severity value from 0x50 to 0x51

ESEL

This keyword's data contains a full PEL in string format. This is how hostboot sends down PELs when it is configured in IPMI communication mode. The PEL is handled just like the PEL obtained using the RAWPEL keyword.

The syntax is:

ESEL=
"00 00 df 00 00 00 00 20 00 04 12 01 6f aa 00 00 50 48 00 30 01 00 33 00 00..."

Note that there are 16 bytes of IPMI SEL data before the PEL data starts.

_PID

This keyword that contains the application's PID is added automatically by the phosphor-logging daemon when the commit or report APIs are used to create an event log, but not when the Create D-Bus method is used. If a caller of the Create API wishes to have their PID captured in the PEL this should be used.

This will be added to the PEL in a section of type User Data (UD), along with the application name it corresponds to.

The syntax is:

_PID=<PID of application>
e.g.
_PID="12345"

CALLOUT_INVENTORY_PATH

This is used to pass in an inventory item to use as a callout. See here for details

CALLOUT_PRIORITY

This can be used along with CALLOUT_INVENTORY_PATH to specify the priority of that FRU callout. If not specified, the default priority is "H"/High Priority.

The possible values are:

• "H": High Priority
• "M": Medium Priority
• "L": Low Priority

See here for details

CALLOUT_DEVICE_PATH with CALLOUT_ERRNO

This is used to pass in a device path to create callouts from. See here for details

CALLOUT_IIC_BUS with CALLOUT_IIC_ADDR and CALLOUT_ERRNO

This is used to pass in an I2C bus and address to create callouts from. See here for details

PEL_SUBSYSTEM

This keyword is used to pass in the subsystem that should be associated with this event log. The syntax is: PEL_SUBSYSTEM=<subsystem value in hex> e.g. PEL_SUBSYSTEM=0x20

FFDC Intended For UserData PEL sections

When one needs to add FFDC into the PEL UserData sections, the CreateWithFFDCFiles D-Bus method on the xyz.openbmc_project.Logging.Create interface must be used when creating a new event log. This method takes a list of files to store in the PEL UserData sections.

That API is the same as the 'Create' one, except it has a new parameter:

std::vector<std::tuple<enum[FFDCFormat],
                       uint8_t,
                       uint8_t,
                       sdbusplus::message::unix_fd>>

Each entry in the vector contains a file descriptor for a file that will be stored in a unique UserData section. The tuple's arguments are:

• enum[FFDCFormat]: The data format type, the options are:
  • 'JSON'
    • The parser will use nlohmann::json's pretty print
  • 'CBOR'
    • The parser will use nlohmann::json's pretty print
  • 'Text'
    • The parser will output ASCII text
  • 'Custom'
    • The parser will hexdump the data, unless there is a parser registered for this component ID and subtype.
• uint8_t: subType
  • Useful for the 'custom' type. Not used with the other types.
• uint8_t: version
  • The version of the data.
  • Used for the custom type.
  • Not planning on using for JSON/BSON unless a reason to do so appears.
• unixfd - The file descriptor for the opened file that contains the contents. The file descriptor can be closed and the file can be deleted if desired after the method call.

An example of saving JSON data to a file and getting its file descriptor is:

nlohmann::json json = ...;
auto jsonString = json.dump();
FILE* fp = fopen(filename, "w");
fwrite(jsonString.data(), 1, jsonString.size(), fp);
int fd = fileno(fp);

Alternatively, 'open()' can be used to obtain the file descriptor of the file.

Upon receiving this data, the PEL code will create UserData sections for each entry in that vector with the following UserData fields:

• Section header component ID:
  • If the type field from the tuple is "custom", use the component ID from the message registry.
  • Otherwise, set the component ID to the phosphor-logging component ID so that the parser knows to use the built in parsers (e.g. json) for the type.
• Section header subtype: The subtype field from the tuple.
• Section header version: The version field from the tuple.
• Section data: The data from the file.

If there is a peltool parser registered for the custom type (method is TBD), that will be used by peltool to print the data, otherwise it will be hexdumped.

Before adding each of these UserData sections, a check will be done to see if the PEL size will remain under the maximum size of 16KB. If not, the UserData section will be truncated down enough so that it will fit into the 16KB.

Default UserData sections for BMC created PELs

The extension code that creates PELs will add these UserData sections to every PEL:

• The AdditionalData property contents

  • If the AdditionalData property in the OpenBMC event log has anything in it, it will be saved in a UserData section as a JSON string.

• System information

  • This section contains various pieces of system information, such as the full code level and the BMC, chassis, and host state properties.

The PEL Message Registry

The PEL message registry is used to create PELs from OpenBMC event logs. Documentation can be found here.

Callouts

A callout points to a FRU, a symbolic FRU, or an isolation procedure. There can be from zero to ten of them in each PEL, where they are located in the SRC section.

There are a few different ways to add callouts to a PEL. In all cases, the callouts will be sorted from highest to lowest priority within the PEL after they are added.

Passing callouts in with the AdditionalData property

The PEL code can add callouts based on the values of special entries in the AdditionalData event log property. They are:

• CALLOUT_INVENTORY_PATH

  This keyword is used to call out a single FRU by passing in its D-Bus inventory path. When the PEL code sees this, it will create a single FRU callout, using the VPD properties (location code, FN, CCIN) from that inventory item. If that item is not a FRU itself and does not have a location code, it will keep searching its parents until it finds one that is.

  The priority of the FRU callout will be high, unless the CALLOUT_PRIORITY keyword is also present and contains a different priority in which case it will be used instead. This can be useful when a maintenance procedure with a high priority callout is specified for this error in the message registry and the FRU callout needs to have a different priority.

  CALLOUT_INVENTORY_PATH=
  "/xyz/openbmc_project/inventory/system/chassis/motherboard"

• CALLOUT_DEVICE_PATH with CALLOUT_ERRNO

  These keywords are required as a pair to indicate faulty device communication, usually detected by a failure accessing a device at that sysfs path. The PEL code will use a data table generated by the MRW to map these device paths to FRU callout lists. The errno value may influence the callout.

  I2C, FSI, FSI-I2C, and FSI-SPI paths are supported.

  CALLOUT_DEVICE_PATH="/sys/bus/i2c/devices/3-0069"
  CALLOUT_ERRNO="2"

• CALLOUT_IIC_BUS with CALLOUT_IIC_ADDR and CALLOUT_ERRNO

  These 3 keywords can be used to callout a failing I2C device path when the full device path isn't known. It is similar to CALLOUT_DEVICE_PATH in that it will use data tables generated by the MRW to create the callouts.

  CALLOUT_IIC_BUS is in the form "/dev/i2c-X" where X is the bus number, or just the bus number by itself. CALLOUT_IIC_ADDR is the 7 bit address either as a decimal or a hex number if preceded with a "0x".

  CALLOUT_IIC_BUS="/dev/i2c-7"
  CALLOUT_IIC_ADDR="81"
  CALLOUT_ERRNO=62

Defining callouts in the message registry

Callouts can be completely defined inside that error's definition in the PEL message registry. This method allows the callouts to vary based on the system type or on any AdditionalData item.

At a high level, this involves defining a callout section inside the registry entry that contain the location codes or procedure names to use, along with their priority. If these can vary based on system type, the type provided by the entity manager will be one of the keys. If they can also depend on an AdditionalData entry, then that will also be a key.

See the message registry README and schema for the details.

Using the message registry along with CALLOUT_ entries

If the message registry entry contains a callout definition and the event log also contains one of aforementioned CALLOUT keys in the AdditionalData property, then the PEL code will first add the callouts stemming from the CALLOUT items, followed by the callouts from the message registry.

Specifying multiple callouts using JSON format FFDC files

Multiple callouts can be passed in by the creator at the time of PEL creation. This is done by specifying them in a JSON file that is then passed in as an FFDC file. The JSON will still be added into a PEL UserData section for debug.

To specify that an FFDC file contains callouts, the format value for that FFDC entry must be set to JSON, and the subtype field must be set to 0xCA:

using FFDC = std::tuple<CreateIface::FFDCFormat,
                        uint8_t,
                        uint8_t,
                        sdbusplus::message::unix_fd>;

FFDC ffdc{
    CreateIface::FFDCFormat::JSON,
    0xCA, // Callout subtype
    0x01, // Callout version, set to 0x01
    fd};

The JSON contains an array of callouts that must be in order of highest priority to lowest, with a maximum of 10. Any callouts after the 10th will just be thrown away as there is no room for them in the PEL. The format looks like:

[
    {
        // First callout
    },
    {
        // Second callout
    },
    {
        // Nth callout
    }
]

A callout entry can be a normal hardware callout, a maintenance procedure callout, or a symbolic FRU callout. Each callout must contain a Priority field, where the possible values are:

• "H" = High
• "M" = Medium
• "A" = Medium Group A
• "B" = Medium Group B
• "C" = Medium Group C
• "L" = Low

Either unexpanded location codes or D-Bus inventory object paths can be used to specify the called out part. An unexpanded location code does not have the system VPD information embedded in it, and the 'Ufcs-' prefix is optional (so can be either Ufcs-P1 or just P1).

Normal hardware FRU callout

Normal hardware callouts must contain either the location code or inventory path, and priority. Even though the PEL code doesn't do any guarding or deconfiguring itself, it needs to know if either of those things occurred as there are status bits in the PEL to reflect them. The Guarded and Deconfigured fields are used for this. Those fields are optional and if omitted then their values will be false.

When the inventory path of a sub-FRU is passed in, the PEL code will put the location code of the parent FRU into the callout.

{
    "LocationCode": "P0-C1",
    "Priority": "H"
}

{
    "InventoryPath": "/xyz/openbmc_project/inventory/motherboard/cpu0/core5",
    "Priority": "H",
    "Deconfigured": true,
    "Guarded": true
}

MRUs (Manufacturing Replaceable Units) are 4 byte numbers that can optionally be added to callouts to specify failing devices on a FRU. These may be used during the manufacturing test process, where there may be the ability to do these replacements. There can be up to 15 MRUs, each with its own priority, embedded in a callout. The possible priority values match the FRU priority values.

Note that since JSON only supports numbers in decimal and not in hex, MRU IDs will show up as decimal when visually inspecting the JSON.

{
  "LocationCode": "P0-C1",
  "Priority": "H",
  "MRUs": [
    {
      "ID": 1234,
      "Priority": "H"
    },
    {
      "ID": 5678,
      "Priority": "H"
    }
  ]
}

Maintenance procedure callout

The LocationCode field is not used with procedure callouts. Only the first 7 characters of the Procedure field will be used by the PEL.

{
  "Procedure": "PRONAME",
  "Priority": "H"
}

Symbolic FRU callout

Only the first seven characters of the SymbolicFRU field will be used by the PEL.

If the TrustedLocationCode field is present and set to true, this means the location code may be used to turn on service indicators, so the LocationCode field is required. If TrustedLocationCode is false or missing, then the LocationCode field is optional.

{
  "TrustedLocationCode": true,
  "Location Code": "P0-C1",
  "Priority": "H",
  "SymbolicFRU": "FRUNAME"
}

Action Flags and Event Type Rules

The Action Flags and Event Type PEL fields are optional in the message registry, and if not present the code will set them based on certain rules layed out in the PEL spec.

These rules are:

1. Always set the Report flag, unless the Do Not Report flag is already on.
2. Always clear the SP Call Home flag, as that feature isn't supported.
3. If the severity is Non-error Event:
   • Clear the Service Action flag.
   • Clear the Call Home flag.
   • If the Event Type field is Not Applicable, change it to Information Only.
   • If the Event Type field is Information Only or Tracing, set the Hidden flag.
4. If the severity is Recovered:
   • Set the Hidden flag.
   • Clear the Service Action flag.
   • Clear the Call Home flag.
5. For all other severities:
   • Clear the Hidden flag.
   • Set the Service Action flag.
   • Set the Call Home flag.

Additional rules may be added in the future if necessary.

D-Bus Interfaces

See the org.open_power.Logging.PEL interface definition for the most up to date information.

PEL Retention

The PEL repository is allocated a set amount of space on the BMC. When that space gets close to being full, the code will remove a percentage of PELs to make room for new ones. In addition, the code will keep a cap on the total number of PELs allowed. Note that removing a PEL will also remove the corresponding OpenBMC event log.

The disk capacity limit is set to 20MB, and the number limit is 3000.

The rules used to remove logs are listed below. The checks will be run after a PEL has been added and the method to create the PEL has returned to the caller, i.e. run when control gets back to the event loop.

Removal Algorithm

If the size used is 95% or under of the allocated space and under the limit on the number of PELs, nothing further needs to be done, otherwise continue and run all 5 of the following steps. Each step itself only deletes PELs until it meets its requirement and then it stops.

The steps are:

1. Remove BMC created informational PELs until they take up 15% or less of the allocated space.

2. Remove BMC created non-informational PELs until they take up 30% or less of the allocated space.

3. Remove non-BMC created informational PELs until they take up 15% or less of the allocated space.

4. Remove non-BMC created non-informational PELs until they take up 30% or less of the allocated space.

5. After the previous 4 steps are complete, if there are still more than the maximum number of PELs, remove PELs down to 80% of the maximum.

PELs with associated guard records will never be deleted. Each step above makes the following 4 passes, stopping as soon as its limit is reached:

• Pass 1. Remove HMC acknowledged PELs.
• Pass 2. Remove OS acknowledged PELs.
• Pass 3. Remove PHYP acknowledged PELs.
• Pass 4. Remove all PELs.

After all these steps, disk capacity will be at most 90% (15% + 30% + 15% + 30%).

Adding python3 modules for PEL UserData and SRC parsing

In order to support python3 modules for the parsing of PEL User Data sections and to decode SRC data, setuptools is used to import python3 packages from external repos to be included in the OpenBMC image.

Sample layout for setuptools:

setup.py src/usr/scom/plugins/ebmc/b0300.py src/usr/i2c/plugins/ebmc/b0700.py src/build/tools/ebmc/errludP_Helpers.py

setup.py is the build script for setuptools. It contains information about the package (such as the name and version) as well as which code files to include.

The setup.py template to be used for eBMC User Data parsers:

import os.path
from setuptools import setup

# To update this dict with new key/value pair for every component added
# Key: The package name to be installed as
# Value: The path containing the package's python modules
dirmap = {
    "b0300": "src/usr/scom/plugins/ebmc",
    "b0700": "src/usr/i2c/plugins/ebmc",
    "helpers": "src/build/tools/ebmc"
}

# All packages will be installed under 'udparsers' namespace
def get_package_name(dirmap_key):
    return "udparsers.{}".format(dirmap_key)

def get_package_dirent(dirmap_item):
    package_name = get_package_name(dirmap_item[0])
    package_dir = dirmap_item[1]
    return (package_name, package_dir)

def get_packages():
    return map(get_package_name, dirmap.keys())

def get_package_dirs():
    return map(get_package_dirent, dirmap.items())

setup(
        name="Hostboot",
        version="0.1",
        packages=list(get_packages()),
        package_dir=dict(get_package_dirs())
)

• User Data parser module

  • Module name: xzzzz.py, where x is the Creator Subsystem from the Private Header section (in ASCII) and zzzz is the 2 byte Component ID from the User Data section itself (in HEX). All should be converted to lowercase.

    • For example: b0100.py for Hostboot created UserData with CompID 0x0100

  • Function to provide: parseUDToJson

    • Argument list:
      1. (int) Sub-section type
      2. (int) Section version
      3. (memoryview): Data
    • Return data:
      1. (str) JSON string

  • Sample User Data parser module:

    import json
    def parseUDToJson(subType, ver, data):
        d = dict()
        ...
        # Parse and populate data into dictionary
        ...
        jsonStr = json.dumps(d)
        return jsonStr

• SRC parser module

  • Module name: xsrc.py, where x is the Creator Subsystem from the Private Header section (in ASCII, converted to lowercase).

    • For example: bsrc.py for Hostboot generated SRCs

  • Function to provide: parseSRCToJson

    • Argument list:
      1. (str) Refcode ASCII string
      2. (str) Hexword 2
      3. (str) Hexword 3
      4. (str) Hexword 4
      5. (str) Hexword 5
      6. (str) Hexword 6
      7. (str) Hexword 7
      8. (str) Hexword 8
      9. (str) Hexword 9
    • Return data:
      1. (str) JSON string

  • Sample SRC parser module:

    import json
    def parseSRCToJson(ascii_str, word2, word3, word4, word5, word6, word7, \
                       word8, word9):
        d = dict({'A': 1, 'B': 2})
        ...
        # Decode SRC data into dictionary
        ...
        jsonStr = json.dumps(d)
        return jsonStr
    

Fail Boot on Host Errors

The fail boot on hw error design provides a function where a system owner can tell the firmware to fail the boot of a system if a BMC phosphor-logging event has a hardware callout in it.

It is required that when this fail boot on hardware error setting is enabled, that the BMC fail the boot for any error from the host which satisfies the following criteria:

• not SeverityType::nonError
• has a callout of any kind from the FailingComponentType structure

Self Boot Engine First Failure Data Capture Support

During SBE chip-op failure SBE creates FFDC with custom data format. SBE FFDC contains different packets, which include SBE internal failure related Trace and user data also Hardware procedure failure FFDC created by FAPI infrastructure. PEL infrastructure provides support to process SBE FFDC packets created by FAPI infrastructure during hardware procedure execution failures, also add callouts, user data section information based on FAPI processing in case non FAPI based failure, just keeps the raw FFDC data in the user section to support SBE parser plugins.

CreatePELWithFFDCFiles D-Bus method on the org.open_power.Logging.PEL interface must be used when creating a new event log.

To specify that an FFDC file contains SBE FFDC, the format value for that FFDC entry must be set to "custom", and the subtype field must be set to 0xCB:

using FFDC = std::tuple<CreateIface::FFDCFormat,
                        uint8_t,
                        uint8_t,
                        sdbusplus::message::unix_fd>;

FFDC ffdc{
     CreateIface::FFDCFormat::custom,
     0xCB, // SBE FFDC subtype
     0x01, // SBE FFDC version, set to 0x01
     fd};

"SRC6" Keyword in the additional data section should be populated with below.

• [0:15] chip position (hex)
• [16:23] command class (hex)
• [24:31] command (hex)

e.g for GetSCOM

SRC6="0002A201"

Note: "phal" build-time configure option should be "enabled" to enable this feature.

PEL Archiving

When an OpenBMC event log is deleted its corresponding PEL is moved to an archive folder. These archived PELs will be available in BMC dump. The archive path: /var/lib/phosphor-logging/extensions/pels/logs/archive.

Highlighted points are:

• PELs whose corresponding event logs have been deleted will be available in the archive folder.
• Archive folder size is tracked along with logs folder size and if combined size exceeds warning size all archived PELs will be deleted.
• Archived PEL logs can be viewed using peltool with flag --archive.
• If a PEL is deleted using peltool its not archived.

Handling PELs for hot plugged FRUs

The degraded mode reporting functionality (i.e. nag) implemented by IBM creates periodic degraded mode reports when a system is running in degraded mode. This includes when hardware has been deconfigured or guarded by hostboot, and also when there is a fault on a redundant fan, VRM, or power supply. The report includes the PELs created for the fails leading to the degraded mode. These PELs can be identified by the 'deconfig' or 'guard' flags set in the SRC of BMC or hostboot PELs.

Fans and power supplies can be hot plugged when they are replaced, and as that FRU is no longer considered degraded the PELs that led to its replacement no longer need be picked up in the degraded mode report.

To handle this, the PEL daemon will watch the inventory D-Bus objects that have the 'xyz.openbmc_project.Inventory.Item.Fan' or 'xyz.openbmc_project.Inventory.Item.PowerSupply' interface. When the 'Present' property on the 'xyz.openbmc_project.Inventory.Item' interface on these paths change to true, the code will find all fan/PS PELs with the deconfig bit set and that location code in the callout list and clear the deconfig flag in the PEL. That way, when the code that does the report searches for PELs, it will no longer find them.