Document Purpose: How to add a new system to the OpenBMC distribution
Audience: Programmer familiar with OpenBMC
Prerequisites: Completed Development Environment Setup Document
Please note: This document is no longer officially supported by the OpenBMC team. It still contains a lot of useful information so it has been left here. Ideally this guide would become a standalone document (outside of the development tree) and would cover all of the different areas that must be updated to add a new system.
This document will describe the following:
The OpenBMC distribution is based on Yocto. Yocto is a project that allows developers to create custom Linux distributions. OpenBMC uses Yocto to create their embedded Linux distribution to run on a variety of devices.
Yocto has a concept of hierarchical layers. When you build a Yocto-based distribution, you define a set of layers for that distribution. OpenBMC uses many common layers from Yocto as well as some of its own layers. The layers defined within OpenBMC can be found with the meta-* directories in OpenBMC GitHub.
Yocto layers are a combination of different files that define packages to incorporate in that layer. One of the key file types used in these layers is BitBake recipes.
BitBake is a fully functional language in itself. For this lesson, we will focus on only the aspects of BitBake required to understand the process of adding a new system.
For this work, you will need to have allocated at least 100GB of space to your development environment and as much memory and CPU as possible. The initial build of an OpenBMC distribution can take hours. Once that first build is done though, future builds will use cached data from the first build, speeding the process up by orders of magnitude.
So before we do anything else, let's get that first build going.
Follow the direction on the OpenBMC GitHub page for the Romulus system (steps 2-4).
While the BitBake operation is going above, let's start creating our new system. Similar to previous lessons, we'll be using Romulus as our reference. Our new system will be called romulus-prime.
From your openbmc repository you cloned above, the Romulus layer is defined within meta-ibm/meta-romulus/. The Romulus layer is defined within the conf subdirectory. Under conf you will see a layout like this:
meta-ibm/meta-romulus/conf/
├── bblayers.conf.sample
├── conf-notes.txt
├── layer.conf
├── local.conf.sample
└── machine
└── romulus.conf
To create our new romulus-prime system we are going to start out by copying our romulus layer.
cp -R meta-ibm/meta-romulus meta-ibm/meta-romulus-prime
Let's review and modify each file needed in our new layer
meta-ibm/meta-romulus-prime/conf/bblayers.conf.sample
This file defines the layers to pull into the meta-romulus-prime distribution. You can see in it a variety of Yocto layers (meta, meta-poky, meta-openembedded/meta-oe, ...). It also has OpenBMC layers like meta-phosphor, meta-openpower, meta-ibm, and meta-ibm/meta-romulus.
The only change you need in this file is to change the two instances of meta-romulus to meta-romulus-prime. This will ensure your new layer is used when building your new system.
meta-ibm/meta-romulus-prime/conf/conf-notes.txt
This file simply states the default target the user will build when working with your new layer. This remains the same as it is common for all OpenBMC systems.
meta-ibm/meta-romulus-prime/conf/layer.conf
The main purpose of this file is to tell BitBake where to look for recipes (*.bb files). Recipe files end with a .bb extension and are what contain all of the packaging logic for the different layers. .bbappend files are also recipe files but provide a way to append onto .bb files. .bbappend files are commonly used to add or remove something from a corresponding .bb file in a different layer.
The only change you need in here is to find/replace the "romulus-layer" to "romulus-prime-layer"
meta-ibm/meta-romulus-prime/conf/local.conf.sample
This file is where all local configuration settings go for your layer. The documentation in it is well done and it's worth a read.
The only change required in here is to change the MACHINE to romulus-prime.
meta-ibm/meta-romulus-prime/conf/machine/romulus.conf
This file describes the specifics for your machine. You define the kernel device tree to use, any overrides to specific features you will be pulling in, and other system specific pointers. This file is a good reference for the different things you need to change when creating a new system (kernel device tree, MRW, LED settings, inventory access, ...)
The first thing you need to do is rename the file to romulus-prime.conf.
Note If our new system really was just a variant of Romulus, with the same hardware configuration, then we could have just created a new machine in the Romulus layer. Any customizations for that system could be included in the corresponding .conf file for that new machine. For the purposes of this exercise we are assuming our romulus-prime system has at least a few hardware changes requiring us to create this new layer.
This will not initially compile but it's good to verify a few things from the initial setup are done correctly.
Do not start this step until the build we started at the beginning of this lesson has completed.
Modify the conf for your current build
Within the shell you did the initial "bitbake" operation you need to reset the conf file for your build. You can manually copy in the new files or just remove it and let BitBake do it for you:
cd .. rm -r ./build/conf . setup romulus-prime
Run your "bitbake" command.
Nothing RPROVIDES 'romulus-prime-config'
This will be your first error after running "bitbake obmc-phosphor-image" against your new system.
The openbmc/skeleton repository was used for initial prototyping of OpenBMC. Within this repository is a configs directory.
The majority of this config data is no longer used but until it is all completely removed, you need to provide it.
Since this repository and file are on there way out, we'll simply do a quick workaround for this issue.
Create a config files as follows:
cp meta-ibm/meta-romulus-prime/recipes-phosphor/workbook/romulus-config.bb meta-ibm/meta-romulus-prime/recipes-phosphor/workbook/romulus-prime-config.bb
vi meta-ibm/meta-romulus-prime/recipes-phosphor/workbook/romulus-prime-config.bb
SUMMARY = "Romulus board wiring"
DESCRIPTION = "Board wiring information for the Romulus OpenPOWER system."
PR = "r1"
inherit config-in-skeleton
#Use Romulus config
do_make_setup() {
cp ${S}/Romulus.py \
${S}/obmc_system_config.py
cat <<EOF > ${S}/setup.py
from distutils.core import setup
setup(name='${BPN}',
version='${PR}',
py_modules=['obmc_system_config'],
)
EOF
}
Re-run your "bitbake" command.
Fetcher failure for URL: file://romulus.cfg
This is the config file required by the kernel. It's where you can put some additional kernel config parameters. For our purposes here, just modify romulus-prime to use the romulus.cfg file. We just need to add the -prime to the prepend path.
vi ./meta-ibm/meta-romulus-prime/recipes-kernel/linux/linux-aspeed_%.bbappend
FILESEXTRAPATHS_prepend_romulus-prime := "${THISDIR}/${PN}:"
SRC_URI += "file://romulus.cfg"
Re-run your "bitbake" command.
No rule to make target arch/arm/boot/dts/aspeed-bmc-opp-romulus-prime.dtb
The .dtb file is a device tree blob file. It is generated during the Linux kernel build based on its corresponding .dts file. When you introduce a new OpenBMC system, you need to send these kernel updates upstream. The linked email thread is an example of this process. Upstreaming to the kernel is a lesson in itself. For this lesson, we will simply use the Romulus kernel config files.
vi ./meta-ibm/meta-romulus-prime/conf/machine/romulus-prime.conf
# Replace the ${MACHINE} variable in the KERNEL_DEVICETREE
# Use romulus device tree
KERNEL_DEVICETREE = "${KMACHINE}-bmc-opp-romulus.dtb"
Re-run your "bitbake" command.
And you've finally built your new system's image! There are more customizations to be done but let's first verify what you have boots.
Your new image will be in the following location from where you ran your "bitbake" command:
./tmp/deploy/images/romulus-prime/obmc-phosphor-image-romulus-prime.static.mtd
Copy this image to where you've set up your QEMU session and re-run the command to start QEMU (qemu-system-arm command from dev-environment.md), giving your new file as input.
Once booted, you should see the following for the login:
romulus-prime login:
There you go! You've done the basics of creating, booting, and building a new system. This is by no means a complete system but you now have the base for the customizations you'll need to do for your new system.
There are a lot of other areas to customize when creating a new system.
This section describes how you can make changes to the kernel to port OpenBMC to a new machine. The device tree is in https://github.com/openbmc/linux/tree/dev-4.13/arch/arm/boot/dts. For examples, see aspeed-bmc-opp-romulus.dts or a similar machine. Complete the following steps to make kernel changes:
openbmc-flash-layout.dtsi.Note:
dev-4.10, there is common and machine-specific initialization code in arch/arm/mach-aspeed/aspeed.c which is used to do common initializations and perform specific settings in each machine. Starting in branch dev-4.13, there is no such initialization code. Most of the inits are done with the upstream clock and reset driver.In legacy OpenBMC, there is a "workbook" to describe the machine's services, sensors, FRUs, etc. This workbook is a python configuration file and it is used by other services in skeleton. In the latest OpenBMC, the skeleton services are mostly replaced by phosphor-xxx services and thus skeleton is deprecated. But the workbook is still needed for now to make the build.
meta-quanta is an example that defines its own config in OpenBMC tree, so that it does not rely on skeleton repo, although it is kind of dummy.
Before e0e69be, or before v2.4 tag, OpenPOWER machines use several configurations related to GPIO. For example, in Romulus.py, the configuration details are as follows:
GPIO_CONFIG['BMC_POWER_UP'] = \ {'gpio_pin': 'D1', 'direction': 'out'} GPIO_CONFIG['SYS_PWROK_BUFF'] = \ {'gpio_pin': 'D2', 'direction': 'in'} GPIO_CONFIGS = { 'power_config' : { 'power_good_in' : 'SYS_PWROK_BUFF', 'power_up_outs' : [ ('BMC_POWER_UP', True), ], 'reset_outs' : [ ], }, }
The PowerUp and PowerOK GPIOs are needed for the build to power on the chassis and check the power state.
After that, the GPIO related configs are removed from the workbook, and replaced by gpio_defs.json, e.g. 2a80da2 introduces the GPIO json config for Romulus.
{ "gpio_configs": { "power_config": { "power_good_in": "SYS_PWROK_BUFF", "power_up_outs": [ { "name": "SOFTWARE_PGOOD", "polarity": true}, { "name": "BMC_POWER_UP", "polarity": true} ], "reset_outs": [ ] } }, "gpio_definitions": [ { "name": "SOFTWARE_PGOOD", "pin": "R1", "direction": "out" }, { "name": "BMC_POWER_UP", "pin": "D1", "direction": "out" }, ... }
Each machine shall define the similar json config to describe the GPIO configurations.
Hwmon sensors include sensors on board (e.g. temperature sensors, fans) and OCC sensors. The config files path and name shall match the devices in device tree.
There is detailed document in openbmc doc/architecture/sensor-architecture.
Here let's take Romulus as an example. The config files are in meta-romulus/recipes-phosphor/sensors which includes sensors on board and sensors of OCC, where on board sensors are via i2c and occ sensors are via FSI.
w83773 temperature sensor containing 3 temperatures:LABEL_temp1 = "outlet" ... LABEL_temp2 = "inlet_cpu" ... LABEL_temp3 = "inlet_io"This device is defined in its device tree as w83773g@4c. When BMC starts, the udev rule will start
phosphor-hwmon and it will create temperature sensors on below DBus objects based on its sysfs attributes./xyz/openbmc_project/sensors/temperature/outlet /xyz/openbmc_project/sensors/temperature/inlet_cpu /xyz/openbmc_project/sensors/temperature/inlet_io
fan_tach sensors.phosphor-hwmon to read the label instead of directly getting the index of the sensor, because CPU cores and DIMMs could be dynamic, e.g. CPU cores could be disabled, DIMMs could be pulled out.MODE_temp1 = "label" MODE_temp2 = "label" ... MODE_temp31 = "label" MODE_temp32 = "label" LABEL_temp91 = "p0_core0_temp" LABEL_temp92 = "p0_core1_temp" ... LABEL_temp33 = "dimm6_temp" LABEL_temp34 = "dimm7_temp" LABEL_power2 = "p0_power" ...
MODE_temp* = "label" tells that if it sees tempX, it shall read the label which is the sensor id.LABEL_temp* = "xxx" tells the sensor name for the corresponding sensor id.temp1_input is 37000 and temp1_label is 91 in sysfs, phosphor-hwmon knows temp1_input is for sensor id 91, which is p0_core0_temp, so it creates /xyz/openbmc_project/sensors/temperature/p0_core0_temp with sensor value 37000.function_id instead of directly getting the index of the sensor.Several parts are involved for LED.
In kernel dts, LEDs shall be described, e.g. romulus dts describes 3 LEDs, fault, identify and power.
leds {
compatible = "gpio-leds";
fault {
gpios = <&gpio ASPEED_GPIO(N, 2) GPIO_ACTIVE_LOW>;
};
identify {
gpios = <&gpio ASPEED_GPIO(N, 4) GPIO_ACTIVE_HIGH>;
};
power {
gpios = <&gpio ASPEED_GPIO(R, 5) GPIO_ACTIVE_LOW>;
};
};
In machine layer, LEDs shall be configured via yaml to describe how it functions, e.g. Romulus led yaml:
bmc_booted:
power:
Action: 'Blink'
DutyOn: 50
Period: 1000
Priority: 'On'
power_on:
power:
Action: 'On'
DutyOn: 50
Period: 0
Priority: 'On'
...
It tells the LED manager to set the power LED to blink when BMC is ready and booted, and set it on when host is powered on.
At runtime, LED manager automatically set LEDs on/off/blink based on the above yaml config.
LED can be accessed manually via /xyz/openbmc_project/led/, e.g.
curl -b cjar -k https://$bmc/xyz/openbmc_project/led/physical/identify
curl -b cjar -k -X PUT -H "Content-Type: application/json" -d '{"data": "xyz.openbmc_project.Led.Physical.Action.Blink" }' https://$bmc/xyz/openbmc_project/led/physical/identify/attr/State
When an error related to a FRU occurs, an event log is created in logging with a CALLOUT path. phosphor-fru-fault-monitor monitors the logs:
Note: This yaml config can be automatically generated by phosphor-mrw-tools from its MRW, see Witherspoon example.
Inventories, other sensors (e.g. CPU/DIMM temperature), and FRUs are defined in ipmi's yaml config files.
E.g. meta-romulus/recipes-phosphor/ipmi
romulus-ipmi-inventory-map defines regular inventories, e.g. CPU, memory, motherboard.phosphor-ipmi-fru-properties defines extra properties of the inventories.phosphor-ipmi-sensor-inventory defines the sensors from IPMI.romulus-ipmi-inventory-sel defines inventories used for IPMI SEL.For inventory map and fru-properties, they are similar between different systems, you can refer to this example and make one for your system.
For ipmi-sensor-inventory, the sensors from IPMI are different between systems, so you need to define your own sensors, e.g.
0x08: sensorType: 0x07 path: /org/open_power/control/occ0 ... 0x1e: sensorType: 0x0C path: /system/chassis/motherboard/dimm0 ... 0x22: sensorType: 0x07 path: /system/chassis/motherboard/cpu0/core0
The first value 0x08, 0x1e and 0x22 are the sensor id of IPMI, which is defined in MRW. You should follow the system's MRW to define the above config.
Note: The yaml configs can be automatically generated by phosphor-mrw-tools from its MRW, see Witherspoon example.
phosphor-fan-presence manages all the services about fan:
phosphor-fan-presence checks if a fan is present, creates the fan DBus objects in inventory and update the Present property.phosphor-fan-monitor checks if a fan is functional, and update the Functional property of the fan Dbus object.phosphor-fan-control controls the fan speed by setting the fan speed target based on conditions, e.g. temperatures.phosphor-cooling-type checks and sets if the system is air-cooled or water-cooled by setting properties of /xyz/openbmc_project/inventory/system/chassis object.All the above services are configurable, e.g. by yaml config. So the machine specific configs shall be written when porting OpenBMC to a new machine.
Taking Romulus as an example, it is air-cooled and has 3 fans without GPIO presence detection.
Romulus has no GPIO detection for fans, so it checks fan tach sensor:
- name: fan0
path: /system/chassis/motherboard/fan0
methods:
- type: tach
sensors:
- fan0
The yaml config tells that
/system/chassis/motherboard/fan0 object in inventory./sensors/fan_tach/fan0) to set Present property on the fan0 object.Romulus fans use pwm to control the fan speed, where pwm ranges from 0 to 255, and the fan speed ranges from 0 to about 7000. So it needs a factor and offset to mapping the pwm to fan speed:
- inventory: /system/chassis/motherboard/fan0
allowed_out_of_range_time: 30
deviation: 15
num_sensors_nonfunc_for_fan_nonfunc: 1
sensors:
- name: fan0
has_target: true
target_interface: xyz.openbmc_project.Control.FanPwm
factor: 21
offset: 1600
The yaml config tells that:
FanPwm as target interface of the tach sensor.target * 21 + 1600.15%, so if the fan speed is out of the expected range for more than 30 seconds, fan0 shall be set as non-functional.The fan control service requires 4 yaml configuration files:
zone-condition defines the cooling zone conditions. Romulus is always air-cooled, so this config is as simple as defining an air_cooled_chassis condition based on the cooling type property.- name: air_cooled_chassis
type: getProperty
properties:
- property: WaterCooled
interface: xyz.openbmc_project.Inventory.Decorator.CoolingType
path: /xyz/openbmc_project/inventory/system/chassis
type: bool
value: false
zone-config defines the cooling zones. Romulus has only one zone:zones: - zone: 0 full_speed: 255 default_floor: 195 increase_delay: 5 decrease_interval: 30It defines that the zone full speed and default floor speed for the fans, so the fan pwm will be set to 255 if it is in full speed, and set to 195 if fans are in default floor speed.
fan-config defines which fans are controlled in which zone and which target interface shall be used, e.g. below yaml config defines fan0 shall be controlled in zone0 and it shall use FanPwm interface.- inventory: /system/chassis/motherboard/fan0 cooling_zone: 0 sensors: - fan0 target_interface: xyz.openbmc_project.Control.FanPwm ...
events-config defines the various events and its handlers, e.g. which fan targets shall be set in which temperature. This config is a bit complicated, the example event yaml provides documents and examples. Romulus example: - name: set_air_cooled_speed_boundaries_based_on_ambient
groups:
- name: zone0_ambient
interface: xyz.openbmc_project.Sensor.Value
property:
name: Value
type: int64_t
matches:
- name: propertiesChanged
actions:
- name: set_floor_from_average_sensor_value
map:
value:
- 27000: 85
- 32000: 112
- 37000: 126
- 40000: 141
type: std::map<int64_t, uint64_t>
- name: set_ceiling_from_average_sensor_value
map:
value:
- 25000: 175
- 27000: 255
type: std::map<int64_t, uint64_t>
The above yaml config defines the fan floor and ceiling speed in zone0_ambient's different temperatures. E.g.With above configs, phosphor-fan will run the fan presence/monitor/control logic as configured specifically for the machine.
Note: Romulus fans are simple. For a more complicated example, refer to Witherspoon fan configurations. The following are the additional functions of Witherspoon fan configuration:
This section mainly focuses on the GPIOs in device tree that shall be monitored. E.g.:
They are categorized as phosphor-gpio-presence for checking presences of a device, and phosphor-gpio-monitor for monitoring a GPIO.
All the GPIOs to be monitored shall be described in the device tree. E.g.
gpio-keys {
compatible = "gpio-keys";
checkstop {
label = "checkstop";
gpios = <&gpio ASPEED_GPIO(J, 2) GPIO_ACTIVE_LOW>;
linux,code = <ASPEED_GPIO(J, 2)>;
};
id-button {
label = "id-button";
gpios = <&gpio ASPEED_GPIO(Q, 7) GPIO_ACTIVE_LOW>;
linux,code = <ASPEED_GPIO(Q, 7)>;
};
};
The following code describes two GPIO keys, one for checkstop and the other for id-button, where the key code is calculated from aspeed-gpio.h:
#define ASPEED_GPIO_PORT_A 0 #define ASPEED_GPIO_PORT_B 1 ... #define ASPEED_GPIO_PORT_Y 24 #define ASPEED_GPIO_PORT_Z 25 #define ASPEED_GPIO_PORT_AA 26 ... #define ASPEED_GPIO(port, offset) \ ((ASPEED_GPIO_PORT_##port * 8) + offset)
Witherspoon and Zaius have examples for gpio presence.
INVENTORY=/system/chassis/motherboard/powersupply0 DEVPATH=/dev/input/by-path/platform-gpio-keys-event KEY=104 NAME=powersupply0 DRIVERS=/sys/bus/i2c/drivers/ibm-cffps,3-0069It checks GPIO key 104 for
powersupply0's presence, creates the inventory object and bind or unbind the driver.INVENTORY=/system/chassis/pcie_card_e2b DEVPATH=/dev/input/by-path/platform-gpio-keys-event KEY=39 NAME=pcie_card_e2bIt checks GPIO key 39 for
pcie_card_e2b's presence, and creates the inventory object.Typical usage of GPIO monitor is to monitor the checkstop event from the host, or button presses.
DEVPATH=/dev/input/by-path/platform-gpio-keys-event KEY=74 POLARITY=1 TARGET=obmc-host-crash@0.targetBy default it monitors GPIO key 74, and if it is triggered, it tells systemd to start
obmc-host-crash@0.target. For systems using a different GPIO pin for checkstop, it simply overrides the default one by specifying its own config file in meta-machine layer. E.g. Zaius's checkstop config. Note: when the key is pressed, phosphor-gpio-monitor starts the target unit and exits.DEVPATH=/dev/input/by-path/platform-gpio-keys-event KEY=135 POLARITY=1 TARGET=id-button-pressed.service EXTRA_ARGS=--continueIt monitors GPIO key 135 for the button press and starts
id-button-pressed.service, that handles the event by setting the identify LED group's Assert property. Note: It has an extra argument, --continue, that tells phosphor-gpio-monitor to not exit and continue running when the key is pressed.