Memory preserving reboot and system dump high level flow.
This documents explains the flow of memory preserving reboot
and system dump extraction with interaction with various
components involved in this operation.
Signed-off-by: Dhruvaraj Subhashchandran <dhruvaraj@in.ibm.com>
Change-Id: I966e0b09bca2127b2944aa3d2647d7efb806a2fe
diff --git a/designs/power-systems-memory-preserving-reboot.md b/designs/power-systems-memory-preserving-reboot.md
new file mode 100644
index 0000000..611f82f
--- /dev/null
+++ b/designs/power-systems-memory-preserving-reboot.md
@@ -0,0 +1,245 @@
+# Memory preserving reboot and System Dump extraction flow on POWER Systems.
+
+ Author: Dhruvaraj S <dhruvaraj@in.ibm.com>
+
+ Primary Assignee: Dhruvaraj S
+
+ Created: 11/06/2019
+
+## Problem Description
+
+On POWER based servers, a hypervisor firmware manages and allocates
+resources to the logical partitions running on the server. If this hypervisor
+encounters an error and cannot continue with management operations, the server
+needs to be restarted. A typical server reboot will erase the content of the
+main memory with the current running configuration of the logical partitions
+and the data required for debugging the fault. Some hypervisors on the POWER
+based systems don't have access to a non-volatile storage to store this
+content after a failure. A warm reboot with preserving the main memory is needed
+on the POWER based servers to create a memory dump required for the
+debugging. This document explains the high-level flow of warm reboot and
+extraction of the resulting dump from the hypervisor memory.
+
+
+## Glossary
+
+- **Boot**: The process of initializing hardware components in a computer system
+and loading the operating system.
+
+- **Hostboot**: The firmware runs on the host processors and performs all
+processor, bus, and memory initialization on POWER based servers.
+[read more](https://github.com/open-power/docs/blob/master/hostboot/HostBoot_PG.md)
+
+- **Self Boot Engine (SBE)**: A microcontroller built into the host processors
+of POWER systems to assist in initializing the processor during the boot.
+It also acts as an entry point for several hardware access operations to the
+processor. [read more](https://sched.co/SPZP)
+
+- **Master Processor**: The processor which gets initialized first to execute
+boot firmware.
+
+- **POWER Hardware Abstraction Layer (PHAL)**: A software component on the BMC
+providing access to the POWER hardware.
+
+- **Hypervisor**: A hypervisor (or virtual machine monitor, VMM) is a computer
+software, firmware, or hardware that creates and runs virtual machines
+[read more](https://en.wikipedia.org/wiki/Hypervisor)
+
+- **System Dump**: A dump of main memory and hardware states for debugging the
+faults in hypervisor.
+
+- **Memory Preserving Reboot (MPR)**: A method of reboot with preserving the
+contents of the volatile memory.
+
+- **Terminate Immediate (TI)**: A condition when the hypervisor encountered
+a fatal error and cannot continue with the normal operations.
+
+- **Attention**: The signal generated by the hardware or the firmware for
+a specific event.
+
+- **Redfish**: The Redfish standard is a suite of specifications that deliver
+an industry-standard protocol providing a RESTful interface for the management
+of servers, storage, networking, and converged infrastructure.
+[Read More](https://en.wikipedia.org/wiki/Redfish_(specification))
+
+- **OCC**: An On-Chip Controller (OCC) is a co-processor that is embedded
+directly on the die of POWER processors. The OCC can be used to controls
+the processor frequency, power consumption, and temperature to maximize
+performance and minimize energy usage.
+
+[Read More](https://openpowerfoundation.org/on-chip-controller-occ/)
+- **Checkstop**: A severe error inside a processor core that causes a processor
+core to stop all processing activities.
+
+- **PNOR**: PNOR is a host NOR flash where the firmware is stored.
+
+## Background and References
+When the POWER based server encounters a fault and needs a restart,
+it alerts BMC to initiate a memory preserving reboot. BMC starts the reboot
+by informing the SBE on each of the processors. SBE stops the running cores and
+collects the hardware states in a specified format and store into the host
+memory. Once the data is collected, the SBE returns control to the BMC. BMC then
+initiates a memory preserved reboot. Once the system finished booting,
+the hypervisor collects the hardware data and memory contents to create
+a dump file in the host memory.
+
+## Requirements
+
+### Primary Requirements
+
+- System dump should be collected irrespective of the availability of an
+ external entity to offload it at the time of a failure.
+
+- It should provide a mechanism for the user to request a system dump.
+
+- The server should boot back to runtime
+
+- The hypervisor should send a special attention to BMC to notify about
+ a severe fault.
+
+- BMC should receive special TI attention from hypervisor
+
+- BMC should change the host state to 'DiagnosticMode.'
+
+- BMC should inform SBE to start the memory preserving reboot and
+ collect the hardware data.
+
+- Error log associated with dump needs to be part of the dump package
+
+- A dump summary should be created with size and other details of the dump
+
+- Once the dump is generated, the hypervisor should notify BMC.
+
+- Hypervisor should offload the dump to BMC to transfer to an external client.
+
+- Provide Redfish interfaces to manage dump
+
+- A tool to collect the dump from the server.
+
+- A method to parse the content of the dump.
+
+## Proposed Design
+
+### The flow
+The flow of the memory preserving reboot and system dump offloading
+
+
+#### 1 - Server fault and notification to BMC
+When there is a fault, the hypervisor generates attention. The attention
+listener on the BMC detects the attention. In the case of OpenPOWER based Linux
+systems, an additional s0 interrupt will be sent to SBE to stop the cores
+immediately.
+
+#### 2 - Analyze the error data.
+The attention listener on the BMC calls a chip-op to analyze the reason for the
+attention.
+
+#### 3 - Initiate System Dump
+Attention on the BMC sets the Diagnostic target for reboot to initiate a
+memory preserving reboot.
+
+#### 4 - Initiate Memory preserve transition
+following steps are executed as part of the reboot target
+ - Set the system state to DiagnosticMode
+ - Stop OCC
+ - Disable checkstop monitoring
+ - Issue enter_mpipl chip-op to each SBE
+
+#### 5 - SBE collects the hardware data
+Each SBE collects the architected states and stores it into a pre-defined
+location.
+
+#### 6 - BMC Start warm boot
+Once the SBE finishes the hardware collection, it does following to boot the
+system with preserving the memory.
+ - Reset VPNOR
+ - Enable watchdog
+ - Enable checkstop monitoring
+ - Run istep proc_select_boot_master
+ - Run istep sbe_config_update
+ - Issue continue_mpipl chip-op instead of start_cbs on the
+ master processor
+
+#### 7 - Hostboot booting
+Once SBE is started, it starts hostboot, hostboot copies the architected states
+to the right location, move the memory contents to create the dump.
+
+#### 8 - Hypervisor Formats dump and sends notification to BMC
+Once the hypervisor is started, it formats the dump and sends a notification to
+BMC through PLDM and with the dump size PLDM calls the dump manager
+interface to notify the dump. Dump manager creates a dBus object for the
+new dump, with status not offloaded and dump size.
+BMC web catches the object creation signal and notifies HMC.
+
+#### 9 - HMC send request to dump offload
+Once HMC is ready to offload, it creates NBD server and send dump offload
+request to BMC. BMCWeb creates an NBD client and NBD proxy to
+offload the dump. BMC dump manager make a PLDM call with dump id provided
+by hypervisor and the NBD device id. PLDM sends the offload request to the
+hypervisor with the dump id.
+
+#### 10 - Hypervisor starts dump offload
+Hypervisor start sending down the dump packets through DMA
+PLDM reads the DUMP and write to the NBD client endpoint
+The data reaches the NBD server on the HMC and get written to a dump file.
+
+#### 11 - Hypervisor sends down offload complete message
+Hypervisor sends down offload complete message to BMC and BMC sends it to HMC.
+The NBD endpoints are cleared.
+
+#### 12 - HMC verifies dump and send dump DELETE to BMC.
+HMC verifies the dump and send dump delete request to BMC
+BMC sends the dump delete message to hypervisor
+Hypervisor deletes dump in host memory.
+
+### Memory preserve reboot sequence.
+
+
+### Dump offload sequence
+
+
+## Alternatives Considered
+Offload the dump from hypervisor to external dump collection application instead
+of offloading through BMC. But offloading though BMC is selected due to following
+reasons.
+ - BMC provides a common point for offloading all dumps
+ - During the prototyping, it is found that the offloading
+ through BMC gave better performance.
+ - Offloading through BMC has less development impact on the host.
+
+## Impacts
+- PLDM on BMC and Host - Extensions to PLDM implementation to pass type of dump,
+ and notification of new dump file to dump manager. [PLDM Design]([https://github.com/openbmc/docs/blob/7c8847e95203ebfaeddb82a6e1b9888bc6a11311/designs/pldm-stack.md])
+
+- Dump manager on BMC - BMC dump manager supports dump stored on BMC and that
+ needs to expanded to support host dumps.
+
+- External dump offloading application needs to support NBD based offload
+
+- Proposing a new redfish schema for dump operations. [Redfish Dump Proposal](https://lists.ozlabs.org/pipermail/openbmc/2019-December/019827.html)
+
+- BMC Web needs to implement new redfish specification for dump.
+
+- Add support to openpower-hw-diags to catch special attention and initiate
+ memory preserving reboot.
+
+- SBE needs to support a new operation to analyze the attention received
+ from the host. The interface update is yet to be published.
+
+## Testing
+- Unit test plans
+ - Test dump manager interfaces using busctl
+ - Test reboot by setting the diag mode target
+ - Test the SBE chip on using standalone calls
+ - Test PLDM by using hypervisor debug commands
+ - Test BMCWeb interfaces using curl
+
+- Integration testing by
+ - User-initiated dump testing, which invokes a memory preserving reboot
+ to collect dump.
+ - Initiate memory preserving reboot by injecting host error
+ - Offload dump collected in host.
+
+- System Dump test plan
+ - Automated tests to initiate and offload dump as part of test bucket.
+ - Both user-initiated and error injection should be attempted.