"Test IO Adapters EEH" test case modified
- Split into integrated and external IO adapters.
- EEH error injection tests in IO adapters must be done
separatedly for integrated and external adapters since
there are two different test cases.
New keyword created: "Separate Integrated Or External PCI"
- Separate internal adapters from externals (connected via PCI).
- Used in the Test Internal/External IO Adapters EEH test cases.
Documentation improvements
- Unnecessary blank spaces erased.
- Comments added for arguments in keywords.
- Comments added on suite and test variables.
Change-Id: Iba538510fba4de383bc83e8a1b383cd728796e05
Signed-off-by: Daniel Gonzalez <Luis.Daniel.Gonzalez@ibm.com>
diff --git a/systest/test_io_eeh.robot b/systest/test_io_eeh.robot
new file mode 100644
index 0000000..03f9d5b
--- /dev/null
+++ b/systest/test_io_eeh.robot
@@ -0,0 +1,277 @@
+*** Settings ***
+Documentation Verify the EEH recovery on the controllers connected to the
+... PCI. This injects an EEH error to every controller installed on the
+... server.
+
+# Test Parameters:
+
+# TYPE EEH error function to use.
+# OPENBMC_HOST The BMC host name or IP address.
+# OS_HOST The OS host name or IP Address.
+# OS_USERNAME The OS login userid (usually root).
+# OS_PASSWORD The password for the OS login.
+# HTX_DURATION Duration of HTX run (e.g. "8 hours", "30 minutes").
+# HTX_INTERVAL The time delay between consecutive checks of HTX
+# status, for example, 30s.
+# In summary: Run HTX for $HTX_DURATION, checking
+# every $HTX_INTERVAL.
+
+# The injection command:
+# echo <pe_num>:<mode>:<type>:<address>:<mask> > /sys/kernel/debug/powerpc/PCIxxxx/err_injct
+# Input arguments to injection command:
+# pe_num: Determined from the output of
+# 'cat /sys/bus/pci/devices/xxxx:xx:xx.x/eeh_pe_config_addr'
+# where the xxxx.xx.xx.x is the PCI location of the device you want.
+# mode: 0 for 32-bit BARs and 64-bit non-prefetchable BARs
+# 1 for 64-bit prefetchable BARs
+# type: Error function to use:
+# 0 to inject on an MMIO load
+# 4 to inject on a config load--Not recommended for standard
+# injection trials.
+# 6 to inject on an MMIO writes
+# 10 to inject on a CFG write
+# address: If you do not care to specify which endpoint under the
+# slot is to be frozen, use a value of '0'.
+# mask: If you do not care to specify which endpoint under the
+# slot is to be frozen, use a value of '0'.
+# xxxx: The PCI domain location of the device.
+
+# Example:
+# echo 5:0:6:3fe280810000:fffffffffffff800 > /sys/kernel/debug/powerpc/PCI0005/err_injct
+# Expected output:
+# HTX Runs mdt.hdbuster or net.mdt and doesn't log errors after error injections.
+
+# Glossary:
+# EEH:
+# Enhanced I/O Error Handling is an error-recovery mechanism for errors that
+# occur during load and store operations on the PCI bus.
+# MDT:
+# Master device table is a collection of hardware devices on the system for
+# which HTX exercisers can be run.
+# mdt.hdbuster:
+# MDT for disk storage device testing. Uses 0s and FFs for data patterns.
+# net.mdt:
+# MDT used to test network adapters.
+# HTX error log file /tmp/htxerr
+# Records all the errors that occur. If there's no error during the test, it
+# should be empty.
+# TODO: Check if monitoring dmesg entries adds value.
+
+Library SSHLibrary
+Library String
+Library ../lib/bmc_ssh_utils.py
+Resource ../lib/resource.txt
+Resource ../syslib/utils_os.robot
+
+Suite Setup Suite Setup Execution
+Suite Teardown Collect HTX Log Files
+Test Teardown FFDC On Test Case Fail
+
+*** Variables ***
+${HTX_DURATION} 8 horus
+${HTX_INTERVAL} 15 minutes
+${TYPE} 0
+
+*** Test Cases ***
+Test Integrated IO Adapters EEH
+ [Documentation] Inject EEH errors in every integrated ethernet controller
+ ... and check if errors are logged.
+ [Tags] Test_IO_Adapters_EEH
+
+ # Setting HTX_MDT_PROFILE to be used in 'Run MDT Profile' keyword
+ Set Suite Variable ${HTX_MDT_PROFILE} net.mdt
+ # Setting lspci_cmd to be used by 'Get PCI' keyword.
+ Set Test Variable ${lspci_cmd} lspci -D | grep "Ethernet controller"
+ @{pci_list}= Get PCI
+ @{pci_list}= Split Integrated Or External PCI ${TRUE} ${pci_list}
+ #Preconfigure Net MDT
+ Run MDT Profile
+ Repeat Keyword ${HTX_DURATION} Cycle Through PCIs ${pci_list}
+ Shutdown HTX Exerciser
+
+Test External IO Adapters EEH
+ [Documentation] Inject EEH errors in every external ethernet controller
+ ... and check if errors are logged.
+ [Tags] Test_IO_Adapters_EEH
+
+ # Setting HTX_MDT_PROFILE to be used in 'Run MDT Profile' keyword
+ Set Suite Variable ${HTX_MDT_PROFILE} net.mdt
+ # Setting lspci_cmd to be used by 'Get PCI' keyword.
+ Set Test Variable ${lspci_cmd} lspci -D | grep "Ethernet controller"
+ @{pci_list}= Get PCI
+ @{pci_list}= Split Integrated Or External PCI ${FALSE} ${pci_list}
+ Preconfigure Net MDT
+ Run MDT Profile
+ Repeat Keyword ${HTX_DURATION} Cycle Through PCIs ${pci_list}
+ Shutdown HTX Exerciser
+
+Test IO Storage EEH
+ [Documentation] Inject EEH Errors and check if errors are logged in htx.
+ [Tags] Test_IO_Storage_EEH
+
+ # Setting HTX_MDT_PROFILE to be used in 'Run MDT Profile' keyword
+ Set Suite Variable ${HTX_MDT_PROFILE} mdt.hdbuster children=true
+ # Setting lspci_cmd to be used by 'Get PCI' keyword.
+ Set Test Variable ${lspci_cmd} lspci -D | grep Marvell
+ # Setting pci_list to be used in 'Cycle Through PCIs' keyword
+ @{pci_list}= Get PCI
+ Set Test Variable @{pci_list}
+ Run MDT Profile
+ Repeat Keyword ${HTX_DURATION} Cycle Through PCIs ${pci_list}
+ Shutdown HTX Exerciser
+
+*** Keywords ***
+Cycle Through PCIs
+ [Documentation] Run a cycle to make error injections to every PCI
+ ... in the pci_list.
+ [Arguments] ${pci_list}=${EMPTY}
+ # Descroption of argument(s):
+ # pci_list A list that contains the PCIs selected (integrated/external).
+ # Every object in the list is the PCI address as domain number
+ # (0 to ffff), bus (0 to ff), slot (0 to 1f) and function
+ # (0 to 7). (e.g. "0005:01:00.1").
+
+ :FOR ${pci} IN @{pci_list}
+ \ Inject EEH Error ${pci}
+ \ Check HTX Run Status
+ \ Run Key U sleep \ ${HTX_INTERVAL}
+
+Preconfigure Net MDT
+ [Documentation] Run build_net to preconfigure the interfaces to be
+ ... injected with the EEH error.
+
+ OS Execute Command build_net help y y
+ ${output} ${stderr} ${rc}= OS Execute Command
+ ... pingum | egrep "All networks ping Ok"
+ Should Contain ${output} All networks ping Ok
+
+Inject EEH Error
+ [Documentation] Build and inject the EEH error command.
+ [Arguments] ${pci}=${EMPTY}
+ # Description of argument(s):
+ # pci PCI address as domain number (0 to ffff), bus (0 to ff),
+ # slot (0 to 1f) and function (0 to 7).
+ # (e.g. "0000:00:1f.2").
+
+ ${sub_cmd}= Make Injection Command ${pci}
+ ${pci}= Fetch From Left ${pci} :
+ ${cmd}= Catenate echo ${sub_cmd} >
+ ... /sys/kernel/debug/powerpc/PCI${pci}/err_injct_inboundA
+ ${output} ${stderr} ${rc}= OS Execute Command ${cmd}
+
+Get PCI
+ [Documentation] Obtain the PCI IDs for every ethernet controller.
+ ${output} ${stderr} ${rc}= OS Execute Command
+ ... ${lspci_cmd} | cut -d " " -f1
+ @{pci_list}= Split To Lines ${output}
+ [Return] @{pci_list}
+
+Split Integrated Or External PCI
+ [Documentation] Separate a list of internal or external PCIs according to
+ ... the test case.
+ [Arguments] ${integrated}=${TRUE} ${pci_list}=${EMPTY}
+ # Description of argument(s):
+ # integrated Boolean variable to determine if the adapters are external
+ # or integrated to the system.
+ # tmp_pci_list List of PCIs (e.g. ["0000:00:00.0", "0001:00:00.0"]).
+
+ @{new_pci_list}= Create List
+ :FOR ${pci} IN @{pci_list}
+ \ ${output} ${stderr} ${rc}= OS Execute Command
+ ... lscfg -vl ${pci} | grep "Location Code"
+ \ ${status} ${err_result}= Run Keyword And Ignore Error
+ ... Should Contain ${output} SLOT
+ \ Run Keyword If '${status}' == 'PASS' and '${integrated}' == '${FALSE}'
+ ... Append To List ${new_pci_list} ${pci}
+ ... ELSE IF '${status}' == 'FAIL' and '${integrated}' == '${TRUE}'
+ ... Append To List ${new_pci_list} ${pci}
+ [Return] ${new_pci_list}
+
+Make Injection Command
+ [Documentation] Create the string that will inject the EEH error.
+ [Tags] Create_Injection_Command
+ [Arguments] ${pci}=${EMPTY}
+ # Description of argument(s):
+ # pci PCI address as domain number (0 to ffff), bus (0 to ff),
+ # slot (0 to 1f) and function (0 to 7).
+ # (e.g. "0000:00:1f.2").
+
+ ${pe_num}= Get PE Num ${pci}
+ ${mode}= Get Mode ${pci}
+ ${address}= Get Address ${pci}
+ ${mask}= Get Mask ${pci}
+ ${result}= Catenate SEPARATOR=:
+ ... ${pe_num} ${mode} ${TYPE} ${address} ${mask}
+ [Return] ${result}
+
+Get PE Num
+ [Documentation] Return the PE configuration address of the PCI sent.
+ [Tags] Get_PE_Number
+ [Arguments] ${pci}=${EMPTY}
+ # Description of argument(s):
+ # pci PCI address as domain number (0 to ffff), bus (0 to ff),
+ # slot (0 to 1f) and function (0 to 7).
+ # (e.g. "0000:00:1f.2").
+
+ ${output} ${stderr} ${rc}= OS Execute Command
+ ... cut -d "x" -f2 /sys/bus/pci/devices/${pci}/eeh_pe_config_addr
+ [Return] ${output}
+
+Get Mode
+ [Documentation] Return the "mode" field value.
+ [Tags] Get_Mode
+ [Arguments] ${pci}=${EMPTY}
+ # Description of argument(s):
+ # pci PCI address as domain number (0 to ffff), bus (0 to ff),
+ # slot (0 to 1f) and function (0 to 7).
+ # (e.g. "0000:00:1f.2").
+
+ ${cmd_buf}= Catenate lspci -bvs ${pci} | grep "Memory at" |
+ ... grep "64-bit" | wc -l
+ ${output} ${stderr} ${rc}= OS Execute Command ${cmd_buf}
+ ${mode}= Set Variable If '${output}'=='0' 0 1
+ [Return] ${mode}
+
+Get Address
+ [Documentation] Obtain the PCI address.
+ [Arguments] ${pci}=${EMPTY}
+ # Description of argument(s):
+ # pci PCI address as domain number (0 to ffff), bus (0 to ff),
+ # slot (0 to 1f) and function (0 to 7).
+ # (e.g. "0000:00:1f.2").
+
+ ${cmd_buf}= Catenate lspci -vv -s ${pci} | grep "Memory at"
+ ... | cut -d " " -f5 | head -n 1
+ ${output} ${stderr} ${rc}= OS Execute Command ${cmd_buf}
+ [Return] ${output}
+
+Get Mask
+ [Documentation] Return the selected PCI mask.
+ [Arguments] ${pci}=${EMPTY}
+ # Description of argument(s):
+ # pci PCI address as domain number (0 to ffff), bus (0 to ff),
+ # slot (0 to 1f) and function (0 to 7).
+ # (e.g. "0000:00:1f.2").
+
+ ${cmd_buf}= Catenate lspci -vv -s ${pci} | grep "Memory at"
+ ... | head -n 1 | cut -d "=" -f2
+ ${stdout} ${stderr} ${rc}= OS Execute Command ${cmd_buf}
+ ${sze}= Get Substring ${stdout} -2 -1
+ ${size}= Get Substring ${stdout} 0 -2
+ ${size}= Convert To Integer ${size}
+ ${size}= Run Keyword If '${sze}' == 'M'
+ ... Evaluate ${size}*1024*1024
+ ... ELSE IF '${sze}' == 'K'
+ ... Evaluate ${size}*1024
+ Log To Console ${size}
+ ${size}= Convert To Hex ${size}
+ ${mask}= Evaluate "{:f>16}".format(${size})
+ [Return] ${mask}
+
+Suite Setup Execution
+ [Documentation] Do suite setup tasks.
+
+ Boot To OS
+ Tool Exist lspci
+ Tool Exist htxcmdline
+ Create Default MDT Profile
\ No newline at end of file