systest/test_io_eeh.robot

*** Settings ***
Documentation  Verify the EEH recovery on the controllers connected to the
...  PCI. This injects an EEH error to every controller installed on the
...  server.

# Test Parameters:

# TYPE                EEH error function to use.
# OPENBMC_HOST        The BMC host name or IP address.
# OS_HOST             The OS host name or IP Address.
# OS_USERNAME         The OS login userid (usually root).
# OS_PASSWORD         The password for the OS login.
# HTX_DURATION        Duration of HTX run (e.g. "8 hours", "30 minutes").
# HTX_INTERVAL        The time delay between consecutive checks of HTX
#                     status, for example, 30s.
#                     In summary: Run HTX for $HTX_DURATION, checking
#                     every $HTX_INTERVAL.

# The injection command:
# echo <pe_num>:<mode>:<type>:<address>:<mask> > /sys/kernel/debug/powerpc/PCIxxxx/err_injct
# Input arguments to injection command:
#   pe_num:  Determined from the output of
#            'cat /sys/bus/pci/devices/xxxx:xx:xx.x/eeh_pe_config_addr'
#            where the xxxx.xx.xx.x is the PCI location of the device you want.
#   mode:    0 for 32-bit BARs and 64-bit non-prefetchable BARs
#            1 for 64-bit prefetchable BARs
#   type:    Error function to use:
#            0 to inject on an MMIO load
#            4 to inject on a config load--Not recommended for standard
#            injection trials.
#            6 to inject on an MMIO writes
#            10 to inject on a CFG write
#   address: If you do not care to specify which endpoint under the
#            slot is to be frozen, use a value of '0'.
#   mask:    If you do not care to specify which endpoint under the
#            slot is to be frozen, use a value of '0'.
#   xxxx:    The PCI domain location of the device.

# Example:
# echo 5:0:6:3fe280810000:fffffffffffff800 > /sys/kernel/debug/powerpc/PCI0005/err_injct
# Expected output:
# HTX Runs mdt.hdbuster or net.mdt and doesn't log errors after error injections.

# Glossary:
# EEH:
#   Enhanced I/O Error Handling is an error-recovery mechanism for errors that
#   occur during load and store operations on the PCI bus.
# MDT:
#   Master device table is a collection of hardware devices on the system for
#   which HTX exercisers can be run.
# mdt.hdbuster:
#   MDT for disk storage device testing. Uses 0s and FFs for data patterns.
# net.mdt:
#   MDT used to test network adapters.
# HTX error log file /tmp/htxerr
#   Records all the errors that occur. If there's no error during the test, it
#   should be empty.
#   TODO: Check if monitoring dmesg entries adds value.

Library         SSHLibrary
Library         String
Library         ../lib/bmc_ssh_utils.py
Resource        ../lib/resource.robot
Resource        ../syslib/utils_os.robot

Suite Setup     Suite Setup Execution
Suite Teardown  Collect HTX Log Files
Test Teardown   FFDC On Test Case Fail

*** Variables ***
${HTX_DURATION}  8 horus
${HTX_INTERVAL}  15 minutes
${TYPE}  0

*** Test Cases ***
Test Integrated IO Adapters EEH
    [Documentation]  Inject EEH errors in every integrated ethernet controller
    ...  and check if errors are logged.
    [Tags]  Test_IO_Adapters_EEH

    # Setting HTX_MDT_PROFILE to be used in 'Run MDT Profile' keyword
    Set Suite Variable  ${HTX_MDT_PROFILE}  net.mdt
    # Setting lspci_cmd to be used by 'Get PCI' keyword.
    Set Test Variable  ${lspci_cmd}  lspci -D | grep "Ethernet controller"
    @{pci_list}=  Get PCI
    @{pci_list}=  Split Integrated Or External PCI  ${TRUE}  ${pci_list}
    #Preconfigure Net MDT
    Run MDT Profile
    Repeat Keyword  ${HTX_DURATION}  Cycle Through PCIs  ${pci_list}
    Shutdown HTX Exerciser

Test External IO Adapters EEH
    [Documentation]  Inject EEH errors in every external ethernet controller
    ...  and check if errors are logged.
    [Tags]  Test_IO_Adapters_EEH

    # Setting HTX_MDT_PROFILE to be used in 'Run MDT Profile' keyword
    Set Suite Variable  ${HTX_MDT_PROFILE}  net.mdt
    # Setting lspci_cmd to be used by 'Get PCI' keyword.
    Set Test Variable  ${lspci_cmd}  lspci -D | grep "Ethernet controller"
    @{pci_list}=  Get PCI
    @{pci_list}=  Split Integrated Or External PCI  ${FALSE}  ${pci_list}
    Preconfigure Net MDT
    Run MDT Profile
    Repeat Keyword  ${HTX_DURATION}  Cycle Through PCIs  ${pci_list}
    Shutdown HTX Exerciser

Test IO Storage EEH
    [Documentation]  Inject EEH Errors and check if errors are logged in htx.
    [Tags]  Test_IO_Storage_EEH

    # Setting HTX_MDT_PROFILE to be used in 'Run MDT Profile' keyword
    Set Suite Variable  ${HTX_MDT_PROFILE}  mdt.hdbuster  children=true
    # Setting lspci_cmd to be used by 'Get PCI' keyword.
    Set Test Variable  ${lspci_cmd}  lspci -D | grep Marvell
    # Setting pci_list to be used in 'Cycle Through PCIs' keyword
    @{pci_list}=  Get PCI
    Set Test Variable  @{pci_list}
    Run MDT Profile
    Repeat Keyword  ${HTX_DURATION}  Cycle Through PCIs  ${pci_list}
    Shutdown HTX Exerciser

*** Keywords ***
Cycle Through PCIs
    [Documentation]  Run a cycle to make error injections to every PCI
    ...  in the pci_list.
    [Arguments]  ${pci_list}=${EMPTY}
    # Descroption of argument(s):
    # pci_list   A list that contains the PCIs selected (integrated/external).
    #            Every object in the list is the PCI address as domain number
    #            (0 to ffff), bus (0 to ff), slot (0 to 1f) and function
    #            (0 to 7). (e.g. "0005:01:00.1").

    :FOR  ${pci}  IN  @{pci_list}
    \  Inject EEH Error  ${pci}
    \  Check HTX Run Status
    \  Run Key U  sleep \ ${HTX_INTERVAL}

Preconfigure Net MDT
    [Documentation]  Run build_net to preconfigure the interfaces to be
    ...  injected with the EEH error.

    OS Execute Command  build_net help y y
    ${output}  ${stderr}  ${rc}=  OS Execute Command
    ...  pingum | egrep "All networks ping Ok"
    Should Contain  ${output}  All networks ping Ok

Inject EEH Error
    [Documentation]  Build and inject the EEH error command.
    [Arguments]  ${pci}=${EMPTY}
    # Description of argument(s):
    # pci   PCI address as domain number (0 to ffff), bus (0 to ff),
    #       slot (0 to 1f) and function (0 to 7).
    #       (e.g. "0000:00:1f.2").

    ${sub_cmd}=  Make Injection Command  ${pci}
    ${pci}=  Fetch From Left  ${pci}  :
    ${cmd}=  Catenate  echo ${sub_cmd} >
    ...  /sys/kernel/debug/powerpc/PCI${pci}/err_injct_inboundA
    ${output}  ${stderr}  ${rc}=  OS Execute Command  ${cmd}

Get PCI
    [Documentation]  Obtain the PCI IDs for every ethernet controller.
    ${output}  ${stderr}  ${rc}=  OS Execute Command
    ...  ${lspci_cmd} | cut -d " " -f1
    @{pci_list}=  Split To Lines  ${output}
    [Return]  @{pci_list}

Split Integrated Or External PCI
    [Documentation]  Separate a list of internal or external PCIs according to
    ...  the test case.
    [Arguments]  ${integrated}=${TRUE}  ${pci_list}=${EMPTY}
    # Description of argument(s):
    # integrated    Boolean variable to determine if the adapters are external
    #               or integrated to the system.
    # tmp_pci_list  List of PCIs (e.g. ["0000:00:00.0", "0001:00:00.0"]).

    @{new_pci_list}=  Create List
    :FOR  ${pci}  IN  @{pci_list}
    \  ${output}  ${stderr}  ${rc}=  OS Execute Command
    ...  lscfg -vl ${pci} | grep "Location Code"
    \  ${status}  ${err_result}=  Run Keyword And Ignore Error
    ...  Should Contain  ${output}  SLOT
    \  Run Keyword If  '${status}' == 'PASS' and '${integrated}' == '${FALSE}'
    ...  Append To List  ${new_pci_list}  ${pci}
    ...  ELSE IF  '${status}' == 'FAIL' and '${integrated}' == '${TRUE}'
    ...  Append To List  ${new_pci_list}  ${pci}
    [Return]  ${new_pci_list}

Make Injection Command
    [Documentation]  Create the string that will inject the EEH error.
    [Tags]  Create_Injection_Command
    [Arguments]  ${pci}=${EMPTY}
    # Description of argument(s):
    # pci   PCI address as domain number (0 to ffff), bus (0 to ff),
    #       slot (0 to 1f) and function (0 to 7).
    #       (e.g. "0000:00:1f.2").

    ${pe_num}=  Get PE Num  ${pci}
    ${mode}=  Get Mode  ${pci}
    ${address}=  Get Address  ${pci}
    ${mask}=  Get Mask  ${pci}
    ${result}=  Catenate  SEPARATOR=:
    ...  ${pe_num}  ${mode}  ${TYPE}  ${address}  ${mask}
    [Return]  ${result}

Get PE Num
    [Documentation]  Return the PE configuration address of the PCI sent.
    [Tags]  Get_PE_Number
    [Arguments]  ${pci}=${EMPTY}
    # Description of argument(s):
    # pci   PCI address as domain number (0 to ffff), bus (0 to ff),
    #       slot (0 to 1f) and function (0 to 7).
    #       (e.g. "0000:00:1f.2").

    ${output}  ${stderr}  ${rc}=  OS Execute Command
    ...  cut -d "x" -f2 /sys/bus/pci/devices/${pci}/eeh_pe_config_addr
    [Return]  ${output}

Get Mode
    [Documentation]  Return the "mode" field value.
    [Tags]  Get_Mode
    [Arguments]  ${pci}=${EMPTY}
    # Description of argument(s):
    # pci   PCI address as domain number (0 to ffff), bus (0 to ff),
    #       slot (0 to 1f) and function (0 to 7).
    #       (e.g. "0000:00:1f.2").

    ${cmd_buf}=  Catenate  lspci -bvs ${pci} | grep "Memory at" |
    ...  grep "64-bit" | wc -l
    ${output}  ${stderr}  ${rc}=  OS Execute Command  ${cmd_buf}
    ${mode}=  Set Variable If  '${output}'=='0'  0  1
    [Return]  ${mode}

Get Address
    [Documentation]  Obtain the PCI address.
    [Arguments]  ${pci}=${EMPTY}
    # Description of argument(s):
    # pci   PCI address as domain number (0 to ffff), bus (0 to ff),
    #       slot (0 to 1f) and function (0 to 7).
    #       (e.g. "0000:00:1f.2").

    ${cmd_buf}=  Catenate  lspci -vv -s ${pci} | grep "Memory at"
    ...  | cut -d " " -f5 | head -n 1
    ${output}  ${stderr}  ${rc}=  OS Execute Command  ${cmd_buf}
    [Return]  ${output}

Get Mask
    [Documentation]  Return the selected PCI mask.
    [Arguments]  ${pci}=${EMPTY}
    # Description of argument(s):
    # pci   PCI address as domain number (0 to ffff), bus (0 to ff),
    #       slot (0 to 1f) and function (0 to 7).
    #       (e.g. "0000:00:1f.2").

    ${cmd_buf}=  Catenate  lspci -vv -s ${pci} | grep "Memory at"
    ...    | head -n 1 | cut -d "=" -f2
    ${stdout}  ${stderr}  ${rc}=  OS Execute Command  ${cmd_buf}
    ${sze}=  Get Substring  ${stdout}  -2  -1
    ${size}=  Get Substring  ${stdout}  0  -2
    ${size}=  Convert To Integer  ${size}
    ${size}=  Run Keyword If  '${sze}' == 'M'
    ...    Evaluate  ${size}*1024*1024
    ...  ELSE IF  '${sze}' == 'K'
    ...    Evaluate  ${size}*1024
    Log To Console  ${size}
    ${size}=  Convert To Hex  ${size}
    ${mask}=  Evaluate  "{:f>16}".format(${size})
    [Return]  ${mask}

Suite Setup Execution
    [Documentation]  Do suite setup tasks.

    Boot To OS
    Tool Exist  lspci
    Tool Exist  htxcmdline
    Create Default MDT Profile