blob: c2086e73182ace00c173e80bf7f56b8df352ce7f [file] [log] [blame]
*** Settings ***
Documentation Verify EEH recovery on the integrated storage controller.
Library SSHLibrary
Library String
Library ../lib/bmc_ssh_utils.py
Resource ../lib/resource.robot
Resource ../syslib/utils_os.robot
# Test Parameters:
# TYPE EEH error function to use, default is 4.
# OPENBMC_HOST The BMC host name or IP address.
# OS_HOST The OS host name or IP Address.
# OS_USERNAME The OS login userid (usually root).
# OS_PASSWORD The password for the OS login.
# HTX_DURATION Duration of HTX run, for example, 8 hours, or
# 30 minutes.
# HTX_INTERVAL The time delay between consecutive checks of HTX
# status, for example, 30s.
# In summary: Run HTX for $HTX_DURATION, checking
# every $HTX_INTERVAL.
# This test case verifies the correct operation of EEH on the integrated
# storage controller. It should be verified with all supported distros.
# The injection command:
# echo <pe_num>:<mode>:<type>:<address>:<mask> > /sys/kernel/debug/powerpc/PCIxxxx/err_injct
# Input arguments to injection command:
# pe_num: determined from the output of
# 'cat /sys/bus/pci/devices/xxxx:xx:xx.x/eeh_pe_config_addr'
# where the xxxx.xx.xx.x is the PCI location of the device you want.
# mode: 0 for 32-bit BARs and 64-bit non-prefetchable BARs
# 1 for 64-bit prefetchable BARs
# type: error function to use:
# 0 to inject on an MMIO load
# 4 to inject on a config load--Not recommended for standard
# injection trials.
# 6 to inject on an MMIO writes
# 10 to inject on a CFG write
# address, mask: if you do not care to specify which endpoint under the
# slot is to be frozen, use a value of '0' for both the address and mask.
# xxxx: the PCI domain location of the device.
# Example:
# echo 5:0:6:3fe280810000:fffffffffffff800 > /sys/kernel/debug/powerpc/PCI0005/err_injct
# Expected output:
# HTX Runs mdt.hdbuster and doesn't log errors after error injections.
# Glossary:
# EEH:
# Enhanced I/O Error Handling is an error-recovery mechanism for errors that
# occur during load and store operations on the PCI bus.
# MDT:
# Master device table is a collection of hardware devices on the system for
# which HTX exercisers can be run.
# mdt.hdbuster:
# MDT for disk storage device testing. Uses 0s and FFs for data patterns.
# HTX error log file /tmp/htxerr
# Records all the errors that occur. If there's no error during the test, it
# should be empty.
# TODO: Check if monitoring dmesg entries adds value.
Suite Setup Suite Setup Execution
Suite Teardown Collect HTX Log Files
Test Teardown FFDC On Test Case Fail
*** Variables ***
${HTX_DURATION} 4 hours
${HTX_INTERVAL} 15 minutes
${TYPE} 4
${lspci_cmd} lspci -D | grep Marvell
*** Test Cases ***
Test EEH Operation
[Documentation] Inject EEH Errors and check if errors are logged in htx.
[Tags] Test_EEH_Operation
Run MDT Profile
Repeat Keyword ${HTX_DURATION} Run Keywords
... Inject EEH Error AND Check HTX Run Status AND Sleep ${HTX_INTERVAL}
Shutdown HTX Exerciser
*** Keywords ***
Inject EEH Error
[Documentation] Inject EEH error to a PE (Partitionable Endpoint).
Log Building the injection command.
${pci}= Get PCI
${sub_cmd}= Make Injection Command ${pci}
${pci}= Fetch From Left ${pci} :
${cmd}= Convert To String
... echo ${sub_cmd} > /sys/kernel/debug/powerpc/PCI${pci}/err_injct
Log Proceeding with injection:\n${cmd}\n
${output} ${stderr} ${rc}= OS Execute Command ${cmd}
Log ${output}
Get PCI
[Documentation] Get the PCI ID for the Marvell adapter.
${output} ${stderr} ${rc}= OS Execute Command
... ${lspci_cmd} | cut -d " " -f1 | head -n 1
[Return] ${output}
Get Pe Num
[Documentation] Get the PE (partitionable endpoint) configuration
... address for the specified PCI.
[Arguments] ${pci}=${EMPTY}
# Description of argument(s):
# pci PCI address as domain number (0 to ffff), bus (0 to ff),
# slot (0 to 1f) and function (0 to 7).
# (e.g. "0000:00:1f.2").
${output} ${stderr} ${rc}= OS Execute Command
... cut -d "x" -f2 /sys/bus/pci/devices/${pci}/eeh_pe_config_addr
[Return] ${output}
Get Mode
[Documentation] Determine the 'mode' field value, by checking the device's
... memory information.
[Arguments] ${pci}=${EMPTY}
# Description of argument(s):
# pci PCI address as domain number (0 to ffff), bus (0 to ff),
# slot (0 to 1f) and function (0 to 7).
# (e.g. "0000:00:1f.2").
${cmd_buf}= Catenate lspci -bvs ${pci} | grep "Memory at " |
... grep "64-bit, prefetchable" | wc -l
${output} ${stderr} ${rc}= OS Execute Command ${cmd_buf}
${mode}= Set Variable If '${output}' == '0' 0 1
[Return] ${mode}
Get Address
[Documentation] Determine the PE address field.
[Arguments] ${pci}=${EMPTY}
# Description of argument(s):
# pci PCI address as domain number (0 to ffff), bus (0 to ff),
# slot (0 to 1f) and function (0 to 7).
# (e.g. "0000:00:1f.2").
${output} ${stderr} ${rc}= OS Execute Command
... lspci -vv -s ${pci} | grep "Memory at " | cut -d " " -f5 | head -n 1
[Return] ${output}
Get Mask
[Documentation] Determine the adress' mask field.
[Arguments] ${pci}=${EMPTY}
# Description of argument(s):
# pci PCI address as domain number (0 to ffff), bus (0 to ff),
# slot (0 to 1f) and function (0 to 7).
# (e.g. "0000:00:1f.2").
${output} ${stderr} ${rc}= OS Execute Command
... lspci -vv -s ${pci} | grep "Memory at" | head -n 1
${size_left}= Fetch From Right ${output} size=
${size_right}= Fetch From Left ${size_left} K
${size1}= Convert To Integer ${size_right}
${size}= Evaluate ${size1}*1024
${size_hex}= Convert To Hex ${size}
${mask}= Evaluate "{:f>16}".format(${size_hex})
[Return] ${mask}
Make Injection Command
[Documentation] Concatenate the fields to form a valid injection command.
[Arguments] ${pci}=${EMPTY}
# Description of argument(s):
# pci PCI address as domain number (0 to ffff), bus (0 to ff),
# slot (0 to 1f) and function (0 to 7).
# (e.g. "0000:00:1f.2").
${pe_num}= Get Pe Num ${pci}
${mode}= Get Mode ${pci}
${addr}= Get Address ${pci}
${mask}= Get Mask ${pci}
${res}= Catenate SEPARATOR=:
... ${pe_num} ${mode} ${TYPE} ${addr} ${mask}
[Return] ${res}
Suite Setup Execution
[Documentation] Do suite setup tasks.
Boot To OS
Tool Exist lspci
Tool Exist htxcmdline
# HTX_MDT_PROFILE: MDT to use in the 'Run MDT Profile' external keyword.
# It's set to mdt.hdbuster in the Suite Setup, so it's not a -v argument.
Set Suite Variable ${HTX_MDT_PROFILE} mdt.hdbuster children=true
Create Default MDT Profile