systest/test_io_storage_eeh.robot

*** Settings ***
Documentation  Verify EEH recovery on the integrated storage controller.
Library     SSHLibrary
Library     String
Library     ../lib/bmc_ssh_utils.py
Resource    ../lib/resource.txt
Resource    ../syslib/utils_os.robot

# Test Parameters:
# TYPE                EEH error function to use, default is 4.
# OPENBMC_HOST        The BMC host name or IP address.
# OS_HOST             The OS host name or IP Address.
# OS_USERNAME         The OS login userid (usually root).
# OS_PASSWORD         The password for the OS login.
# HTX_DURATION        Duration of HTX run, for example, 8 hours, or
#                     30 minutes.
# HTX_INTERVAL        The time delay between consecutive checks of HTX
#                     status, for example, 30s.
#                     In summary: Run HTX for $HTX_DURATION, checking
#                     every $HTX_INTERVAL.
# This test case verifies the correct operation of EEH on the integrated
# storage controller. It should be verified with all supported distros.
# The injection command:
# echo <pe_num>:<mode>:<type>:<address>:<mask> > /sys/kernel/debug/powerpc/PCIxxxx/err_injct
# Input arguments to injection command:
#   pe_num: determined from the output of
#          'cat /sys/bus/pci/devices/xxxx:xx:xx.x/eeh_pe_config_addr'
#          where the xxxx.xx.xx.x is the PCI location of the device you want.
#   mode: 0 for 32-bit BARs and 64-bit non-prefetchable BARs
#         1 for 64-bit prefetchable BARs
#   type: error function to use:
#          0 to inject on an MMIO load
#          4 to inject on a config load--Not recommended for standard
#            injection trials.
#          6 to inject on an MMIO writes
#         10 to inject on a CFG write
#   address, mask: if you do not care to specify which endpoint under the
#      slot is to be frozen, use a value of '0' for both the address and mask.
#   xxxx: the PCI domain location of the device.
# Example:
# echo 5:0:6:3fe280810000:fffffffffffff800 > /sys/kernel/debug/powerpc/PCI0005/err_injct
# Expected output:
# HTX Runs mdt.hdbuster and doesn't log errors after error injections.
# Glossary:
# EEH:
#   Enhanced I/O Error Handling is an error-recovery mechanism for errors that
#   occur during load and store operations on the PCI bus.
# MDT:
#   Master device table is a collection of hardware devices on the system for
#   which HTX exercisers can be run.
# mdt.hdbuster:
#   MDT for disk storage device testing. Uses 0s and FFs for data patterns.
# HTX error log file /tmp/htxerr
#   Records all the errors that occur. If there's no error during the test, it
#   should be empty.
#   TODO: Check if monitoring dmesg entries adds value.

Suite Setup  Suite Setup Execution
Suite Teardown  Collect HTX Log Files
Test Teardown  FFDC On Test Case Fail

*** Variables ***
${HTX_DURATION}  4 hours
${HTX_INTERVAL}  15 minutes
${TYPE}  4
${lspci_cmd}  lspci -D | grep Marvell

*** Test Cases ***
Test EEH Operation
    [Documentation]  Inject EEH Errors and check if errors are logged in htx.
    [Tags]  Test_EEH_Operation

    Run MDT Profile
    Repeat Keyword  ${HTX_DURATION}  Run Keywords
    ...  Inject EEH Error  AND  Check HTX Run Status  AND  Sleep  ${HTX_INTERVAL}
    Shutdown HTX Exerciser

*** Keywords ***
Inject EEH Error
    [Documentation]  Inject EEH error to a PE (Partitionable Endpoint).

    Log  Building the injection command.
    ${pci}=  Get PCI
    ${sub_cmd}=  Make Injection Command  ${pci}
    ${pci}=  Fetch From Left  ${pci}  :
    ${cmd}=  Convert To String
    ...  echo ${sub_cmd} > /sys/kernel/debug/powerpc/PCI${pci}/err_injct
    Log  Proceeding with injection:\n${cmd}\n
    ${output}  ${stderr}  ${rc}=  OS Execute Command  ${cmd}
    Log  ${output}

Get PCI
    [Documentation]  Get the PCI ID for the Marvell adapter.

    ${output}  ${stderr}  ${rc}=  OS Execute Command
    ...  ${lspci_cmd} | cut -d " " -f1 | head -n 1
    [Return]  ${output}

Get Pe Num
    [Documentation]  Get the PE (partitionable endpoint) configuration
    ...              address for the specified PCI.
    [Arguments]  ${pci}=${EMPTY}
    # Description of argument(s):
    # pci   PCI address as domain number (0 to ffff), bus (0 to ff),
    #       slot (0 to 1f) and function (0 to 7).
    #       (e.g. "0000:00:1f.2").

    ${output}  ${stderr}  ${rc}=  OS Execute Command
    ...  cut -d "x" -f2 /sys/bus/pci/devices/${pci}/eeh_pe_config_addr
    [Return]  ${output}

Get Mode
    [Documentation]  Determine the 'mode' field value, by checking the device's
    ...  memory information.
    [Arguments]  ${pci}=${EMPTY}
    # Description of argument(s):
    # pci   PCI address as domain number (0 to ffff), bus (0 to ff),
    #       slot (0 to 1f) and function (0 to 7).
    #       (e.g. "0000:00:1f.2").

    ${cmd_buf}=  Catenate  lspci -bvs ${pci} | grep "Memory at " |
    ...  grep "64-bit, prefetchable" | wc -l
    ${output}  ${stderr}  ${rc}=  OS Execute Command  ${cmd_buf}
    ${mode}=  Set Variable If  '${output}' == '0'  0  1
    [Return]  ${mode}

Get Address
    [Documentation]  Determine the PE address field.
    [Arguments]  ${pci}=${EMPTY}
    # Description of argument(s):
    # pci   PCI address as domain number (0 to ffff), bus (0 to ff),
    #       slot (0 to 1f) and function (0 to 7).
    #       (e.g. "0000:00:1f.2").

    ${output}  ${stderr}  ${rc}=  OS Execute Command
    ...  lspci -vv -s ${pci} | grep "Memory at " | cut -d " " -f5 | head -n 1
    [Return]  ${output}

Get Mask
    [Documentation]  Determine the adress' mask field.
    [Arguments]  ${pci}=${EMPTY}
    # Description of argument(s):
    # pci   PCI address as domain number (0 to ffff), bus (0 to ff),
    #       slot (0 to 1f) and function (0 to 7).
    #       (e.g. "0000:00:1f.2").

    ${output}  ${stderr}  ${rc}=  OS Execute Command
    ...  lspci -vv -s ${pci} | grep "Memory at" | head -n 1
    ${size_left}=  Fetch From Right  ${output}  size=
    ${size_right}=  Fetch From Left  ${size_left}  K
    ${size1}=  Convert To Integer  ${size_right}
    ${size}=  Evaluate  ${size1}*1024
    ${size_hex}=  Convert To Hex  ${size}
    ${mask}=  Evaluate  "{:f>16}".format(${size_hex})
    [Return]  ${mask}

Make Injection Command
    [Documentation]  Concatenate the fields to form a valid injection command.
    [Arguments]  ${pci}=${EMPTY}
    # Description of argument(s):
    # pci   PCI address as domain number (0 to ffff), bus (0 to ff),
    #       slot (0 to 1f) and function (0 to 7).
    #       (e.g. "0000:00:1f.2").

    ${pe_num}=  Get Pe Num  ${pci}
    ${mode}=  Get Mode  ${pci}
    ${addr}=  Get Address  ${pci}
    ${mask}=  Get Mask  ${pci}
    ${res}=  Catenate  SEPARATOR=:
    ...  ${pe_num}  ${mode}  ${TYPE}  ${addr}  ${mask}
    [Return]  ${res}

Suite Setup Execution
    [Documentation]  Do suite setup tasks.

    Boot To OS
    Tool Exist  lspci
    Tool Exist  htxcmdline
    # HTX_MDT_PROFILE:   MDT to use in the 'Run MDT Profile' external keyword.
    # It's set to mdt.hdbuster in the Suite Setup, so it's not a -v argument.
    Set Suite Variable  ${HTX_MDT_PROFILE}  mdt.hdbuster  children=true
    Create Default MDT Profile