Test Witherspoon RAS use cases using pdbg tool.
Added:
- unrecoverable test scenarios on different targets
(MCA, MCI, CXA, NXDMAENG, L2FIR, L3FIR, NCUFIR, CoreFIR).
Change-Id: Ibf6929098528b4365662150739da77d2370934a7
Signed-off-by: Sridevi Ramesh <sridevra@in.ibm.com>
diff --git a/openpower/ras/ras_utils.robot b/openpower/ras/ras_utils.robot
index 3bed3ca..2170b0b 100755
--- a/openpower/ras/ras_utils.robot
+++ b/openpower/ras/ras_utils.robot
@@ -1,9 +1,6 @@
*** Settings ***
Documentation Utility for RAS test scenarios through HOST & BMC.
Resource ../../lib/utils.robot
-Resource ../../lib/openbmc_ffdc.robot
-Resource ../../lib/openbmc_ffdc_utils.robot
-Resource ../../lib/openbmc_ffdc_methods.robot
Resource ../../lib/ras/host_utils.robot
Resource ../../lib/resource.robot
Resource ../../lib/state_manager.robot
@@ -49,12 +46,12 @@
Inject Recoverable Error With Threshold Limit
[Documentation] Inject and verify recoverable error on processor through
- ... host.
+ ... BMC/HOST.
... Test sequence:
- ... 1. Enable Auto Reboot Setting.
- ... 2. Inject Error on processor/centaur.
- ... 3. Check If HOST is running.
- ... 4. Verify error log entry & signature description.
+ ... 1. Inject recoverable error on a given target
+ ... (e.g: Processor core, CAPP MCA) through BMC/HOST.
+ ... 2. Check If HOST is running.
+ ... 3. Verify error log entry & signature description.
... 4. Verify & clear gard records.
[Arguments] ${interface_type} ${fir} ${chip_address} ${threshold_limit}
... ${signature_desc} ${log_prefix}
@@ -66,7 +63,6 @@
# signature_desc Error log signature description.
# log_prefix Log path prefix.
- Set Auto Reboot 1
Run Keyword If '${interface_type}' == 'HOST'
... Inject Error Through HOST ${fir} ${chip_address} ${threshold_limit}
... ${master_proc_chip}
@@ -81,19 +77,20 @@
# TODO: Verify SOL console logs.
-Inject Unrecoverable Error Through Host
- [Documentation] Inject and verify recoverable error on processor through
- ... host.
+Inject Unrecoverable Error
+ [Documentation] Inject and verify unrecoverable error on processor through
+ ... BMC/HOST.
... Test sequence:
- ... 1. Enable Auto Reboot Setting.
- ... 2. Inject Error on processor/centaur.
- ... 3. Check If HOST is rebooted.
- ... 4. Verify & clear gard records.
- ... 5. Verify error log entry & signature description.
- ... 6. Verify & clear dump entry.
- [Arguments] ${fir} ${chip_address} ${threshold_limit}
+ ... 1. Inject unrecoverable error on a given target
+ ... (e.g: Processor core, CAPP MCA) through BMC/HOST.
+ ... 2. Check If HOST is rebooted.
+ ... 3. Verify & clear gard records.
+ ... 4. Verify error log entry & signature description.
+ ... 5. Verify & clear dump entry.
+ [Arguments] ${interface_type} ${fir} ${chip_address} ${threshold_limit}
... ${signature_desc} ${log_prefix}
# Description of argument(s):
+ # interface_type Inject error through 'BMC' or 'HOST'.
# fir FIR (Fault isolation register) value (e.g. 2011400).
# chip_address Chip address (e.g 2000000000000000).
# threshold_limit Threshold limit (e.g 1, 5, 32).
@@ -101,9 +98,13 @@
# (e.g 'mcs(n0p0c0) (MCFIR[0]) mc internal recoverable')
# log_prefix Log path prefix.
- Set Auto Reboot 1
- Inject Error Through HOST ${fir} ${chip_address} ${threshold_limit}
- ... ${master_proc_chip}
+ Run Keyword If '${interface_type}' == 'HOST'
+ ... Inject Error Through HOST ${fir} ${chip_address} ${threshold_limit}
+ ... ${master_proc_chip}
+ ... ELSE
+ ... Inject Error Through BMC ${fir} ${chip_address} ${threshold_limit}
+ ... ${master_proc_chip}
+
Wait Until Keyword Succeeds 500 sec 20 sec Is Host Rebooted
Wait for OS
Verify Error Log Entry ${signature_desc} ${log_prefix}
diff --git a/openpower/ras/test_bmc_ras.robot b/openpower/ras/test_bmc_ras.robot
index 960d2a7..2cfa1e3 100755
--- a/openpower/ras/test_bmc_ras.robot
+++ b/openpower/ras/test_bmc_ras.robot
@@ -2,14 +2,9 @@
Documentation This suite tests checkstop operations through BMC using
pdbg utility.
-Resource ../../lib/utils.robot
Resource ../../lib/openbmc_ffdc.robot
-Resource ../../lib/ras/host_utils.robot
-Resource ../../lib/resource.robot
-Resource ../../lib/state_manager.robot
Resource ../../lib/openbmc_ffdc_methods.robot
-Resource ../../lib/boot_utils.robot
-Resource ../../lib/dump_utils.robot
+Resource ../../lib/openbmc_ffdc_utils.robot
Resource ../../openpower/ras/ras_utils.robot
Variables ../../lib/ras/variables.py
Variables ../../data/variables.py
@@ -54,6 +49,16 @@
Inject Recoverable Error With Threshold Limit
... BMC ${value[0]} ${value[1]} 32 ${value[2]} ${err_log_path}
+Verify Pdbg Unrecoverable Callout Handling For MCA
+ [Documentation] Verify unrecoverable callout handling for MCACALIFIR
+ ... with pdbg tool.
+ [Tags] Verify_Pdbg_Unrecoverable_Callout_Handling_For_MCA
+
+ ${value}= Get From Dictionary ${ERROR_INJECT_DICT} MCACALIFIR_UE
+ ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}mcacalfir
+ Inject Unrecoverable Error BMC
+ ... ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
+
# Memory controller Interface (MCIFIR) related error injection.
Verify Pdbg Recoverable Callout Handling For MCI With Threshold 1
@@ -66,6 +71,17 @@
Inject Recoverable Error With Threshold Limit
... BMC ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
+Verify Pdbg Unrecoverable Callout Handling For MCI
+ [Documentation] Verify unrecoverable callout handling for mci
+ ... with pdbg tool.
+ [Tags] Verify_Pdbg_Unrecoverable_Callout_Handling_For_MCI
+
+ ${value}= Get From Dictionary ${ERROR_INJECT_DICT} MCI_UE
+ ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}mcifir
+ Inject Unrecoverable Error BMC
+ ... ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
+
+
# CAPP accelerator (CXAFIR) related error injection.
Verify Pdbg Recoverable Callout Handling For CXA With Threshold 5
@@ -88,6 +104,15 @@
Inject Recoverable Error With Threshold Limit
... BMC ${value[0]} ${value[1]} 32 ${value[2]} ${err_log_path}
+Verify Pdbg Unrecoverable Callout Handling For CXA
+ [Documentation] Verify unrecoverable callout handling for CXAFIR
+ ... with pdbg tool.
+ [Tags] Verify_Pdbg_Unrecoverable_Callout_Handling_For_CXA
+
+ ${value}= Get From Dictionary ${ERROR_INJECT_DICT} CXA_UE
+ ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}cxafir_ue
+ Inject Unrecoverable Error BMC
+ ... ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
# Optical BUS (OBUSFIR) related error injection.
@@ -136,6 +161,15 @@
Inject Recoverable Error With Threshold Limit
... BMC ${value[0]} ${value[1]} 32 ${value[2]} ${err_log_path}
+Verify Pdbg Unrecoverable Callout Handling For NXDMAENG
+ [Documentation] Verify unrecoverable callout handling for NXDMAENG
+ ... with pdbg tool.
+ [Tags] Verify_Pdbg_Unrecoverable_Callout_Handling_For_NXDMAENG
+
+ ${value}= Get From Dictionary ${ERROR_INJECT_DICT} NX_UE
+ ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}nxfir_ue
+ Inject Unrecoverable Error BMC
+ ... ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
# L2FIR related error injection.
@@ -161,6 +195,17 @@
Inject Recoverable Error With Threshold Limit
... BMC ${translated_fir} ${value[1]} 32 ${value[2]} ${err_log_path}
+Verify Pdbg Unrecoverable Callout Handling For L2FIR
+ [Documentation] Verify unrecoverable callout handling for L2FIR
+ ... with pdbg tool.
+ [Tags] Verify_Pdbg_Unrecoverable_Callout_Handling_For_L2FIR
+
+ ${value}= Get From Dictionary ${ERROR_INJECT_DICT} L2FIR_UE
+ ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
+ ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}l2fir_ue
+ Inject Unrecoverable Error BMC
+ ... ${translated_fir} ${value[1]} 1 ${value[2]} ${err_log_path}
+
# L3FIR related error injection.
Verify Pdbg Recoverable Callout Handling For L3FIR With Threshold 1
@@ -185,6 +230,17 @@
Inject Recoverable Error With Threshold Limit
... BMC ${translated_fir} ${value[1]} 32 ${value[2]} ${err_log_path}
+Verify Pdbg Unrecoverable Callout Handling For L3FIR
+ [Documentation] Verify unrecoverable callout handling for L3FIR
+ ... with pdbg tool.
+ [Tags] Verify_Pdbg_Unrecoverable_Callout_Handling_For_L3FIR
+
+ ${value}= Get From Dictionary ${ERROR_INJECT_DICT} L3FIR_UE
+ ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
+ ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}l3fir_ue
+ Inject Unrecoverable Error BMC
+ ... ${translated_fir} ${value[1]} 1 ${value[2]} ${err_log_path}
+
# On chip controller (OCCFIR) related error injection.
Verify Pdbg Recoverable Callout Handling For OCC With Threshold 1
@@ -223,6 +279,17 @@
Inject Recoverable Error With Threshold Limit
... BMC ${translated_fir} ${value[1]} 1 ${value[2]} ${err_log_path}
+Verify Pdbg Unrecoverable Callout Handling For NCUFIR
+ [Documentation] Verify unrecoverable callout handling for NCUFIR
+ ... with pdbg tool.
+ [Tags] Verify_Pdbg_Unrecoverable_Callout_Handling_For_NCUFIR
+
+ ${value}= Get From Dictionary ${ERROR_INJECT_DICT} NCUFIR_UE
+ ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
+ ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}ncufir_ue
+ Inject Unrecoverable Error BMC
+ ... ${translated_fir} ${value[1]} 1 ${value[2]} ${err_log_path}
+
# Core FIR related error injection.
Verify Pdbg Recoverable Callout Handling For CoreFIR With Threshold 5
@@ -249,6 +316,18 @@
Inject Recoverable Error With Threshold Limit
... BMC ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
+Verify Pdbg Unrecoverable Callout Handling For CoreFIR
+ [Documentation] Verify unrecoverable callout handling for CoreFIR
+ ... with pdbg tool.
+ [Tags] Verify_Pdbg_Unrecoverable_Callout_Handling_For_CoreFIR
+
+ ${value}= Get From Dictionary ${ERROR_INJECT_DICT} COREFIR_UE
+ ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
+ Disable CPU States Through HOST
+ ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}corefir_ue
+ Inject Unrecoverable Error BMC
+ ... ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
+
Verify Pdbg Recoverable Callout Handling For EQFIR With Threshold 32
[Documentation] Verify recoverable callout handling for L3FIR
... with threshold 32 using pdbg tool.
diff --git a/openpower/ras/test_host_ras.robot b/openpower/ras/test_host_ras.robot
index c8906f3..c7edfd9 100755
--- a/openpower/ras/test_host_ras.robot
+++ b/openpower/ras/test_host_ras.robot
@@ -1,13 +1,9 @@
*** Settings ***
Documentation This suite tests checkstop operations through HOST.
-Resource ../../lib/utils.robot
+
Resource ../../lib/openbmc_ffdc.robot
-Resource ../../lib/ras/host_utils.robot
-Resource ../../lib/resource.robot
-Resource ../../lib/state_manager.robot
+Resource ../../lib/openbmc_ffdc_utils.robot
Resource ../../lib/openbmc_ffdc_methods.robot
-Resource ../../lib/boot_utils.robot
-Resource ../../lib/dump_utils.robot
Resource ../../openpower/ras/ras_utils.robot
Variables ../../lib/ras/variables.py
Variables ../../data/variables.py
@@ -23,6 +19,7 @@
Suite Teardown RAS Suite Cleanup
Force Tags Host_RAS
+
*** Variables ***
${stack_mode} normal
@@ -55,7 +52,7 @@
${value}= Get From Dictionary ${ERROR_INJECT_DICT} MCACALIFIR_UE
${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}mcacalfir
- Inject Unrecoverable Error Through Host
+ Inject Unrecoverable Error HOST
... ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
# Memory buffer (MCIFIR) related error injection.
@@ -76,7 +73,7 @@
${value}= Get From Dictionary ${ERROR_INJECT_DICT} MCI_UE
${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}mcifir
- Inject Unrecoverable Error Through Host
+ Inject Unrecoverable Error HOST
... ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
# CAPP accelerator (CXAFIR) related error injection.
@@ -107,7 +104,7 @@
${value}= Get From Dictionary ${ERROR_INJECT_DICT} CXA_UE
${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}cxafir_ue
- Inject Unrecoverable Error Through Host
+ Inject Unrecoverable Error HOST
... ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
@@ -164,7 +161,7 @@
${value}= Get From Dictionary ${ERROR_INJECT_DICT} NX_UE
${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}nxfir_ue
- Inject Unrecoverable Error Through Host
+ Inject Unrecoverable Error HOST
... ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
# L2FIR related error injection.
@@ -198,7 +195,7 @@
${value}= Get From Dictionary ${ERROR_INJECT_DICT} L2FIR_UE
${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}l2fir_ue
- Inject Unrecoverable Error Through Host
+ Inject Unrecoverable Error HOST
... ${translated_fir} ${value[1]} 1 ${value[2]} ${err_log_path}
@@ -233,7 +230,7 @@
${value}= Get From Dictionary ${ERROR_INJECT_DICT} L3FIR_UE
${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}l3fir_ue
- Inject Unrecoverable Error Through Host
+ Inject Unrecoverable Error HOST
... ${translated_fir} ${value[1]} 1 ${value[2]} ${err_log_path}
@@ -282,7 +279,7 @@
${value}= Get From Dictionary ${ERROR_INJECT_DICT} NCUFIR_UE
${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}ncufir_ue
- Inject Unrecoverable Error Through Host
+ Inject Unrecoverable Error HOST
... ${translated_fir} ${value[1]} 1 ${value[2]} ${err_log_path}
@@ -320,7 +317,7 @@
${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
Disable CPU States Through HOST
${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}corefir_ue
- Inject Unrecoverable Error Through Host
+ Inject Unrecoverable Error HOST
... ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
Verify Recoverable Callout Handling For EQFIR With Threshold 32