Error injection through OS related automation [stage 3].
Automating error injection scenarios as shown below
RE TH limit 1 ---> NCUFIR
RE TH limit 32 --> EQFIR
UE ---> L2, L3, CXA
Resolves openbmc/openbmc-test-automation#1081
Change-Id: Ie87688fbf7fb456fda98b96085117c1395777ee2
Signed-off-by: Sridevi Ramesh <sridevra@in.ibm.com>
diff --git a/extended/test_host_ras.robot b/extended/test_host_ras.robot
index 9f9cae2..8d3b518 100755
--- a/extended/test_host_ras.robot
+++ b/extended/test_host_ras.robot
@@ -23,7 +23,6 @@
*** Variables ***
${stack_mode} normal
-
*** Test Cases ***
# Memory channel (MCACALIFIR) related error injection.
@@ -100,6 +99,15 @@
Inject Recoverable Error With Threshold Limit Through Host
... ${value[0]} ${value[1]} 32 ${value[2]} ${err_log_path}
+Verify Unrecoverable Callout Handling For CXA
+ [Documentation] Verify unrecoverable callout handling for CXAFIR.
+ [Tags] Verify_Unrecoverable_Callout_Handling_For_CXA
+
+ ${value}= Get From Dictionary ${ERROR_INJECT_DICT} CXA_UE
+ ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}cxafir_ue
+ Inject Unrecoverable Error Through Host
+ ... ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
+
# OBUSFIR related error injection.
Verify Recoverable Callout Handling For OBUS With Threshold 32
@@ -170,6 +178,27 @@
Inject Recoverable Error With Threshold Limit Through Host
... ${translated_fir} ${value[1]} 1 ${value[2]} ${err_log_path}
+Verify Recoverable Callout Handling For L2FIR With Threshold 32
+ [Documentation] Verify recoverable callout handling for L2FIR with
+ ... threshold 32.
+ [Tags] Verify_Recoverable_Callout_Handling_For_L2FIR_With_Threshold_32
+
+ ${value}= Get From Dictionary ${ERROR_INJECT_DICT} L2FIR_RECV32
+ ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
+ ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}l2fir_th32
+ Inject Recoverable Error With Threshold Limit Through Host
+ ... ${translated_fir} ${value[1]} 32 ${value[2]} ${err_log_path}
+
+Verify Unrecoverable Callout Handling For L2FIR
+ [Documentation] Verify unrecoverable callout handling for L2FIR.
+ [Tags] Verify_Unrecoverable_Callout_Handling_For_L2FIR
+
+ ${value}= Get From Dictionary ${ERROR_INJECT_DICT} L2FIR_UE
+ ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
+ ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}l2fir_ue
+ Inject Unrecoverable Error Through Host
+ ... ${translated_fir} ${value[1]} 1 ${value[2]} ${err_log_path}
+
# L3FIR related error injection.
Verify Recoverable Callout Handling For L3FIR With Threshold 1
@@ -194,6 +223,16 @@
Inject Recoverable Error With Threshold Limit Through Host
... ${translated_fir} ${value[1]} 32 ${value[2]} ${err_log_path}
+Verify Unrecoverable Callout Handling For L3FIR
+ [Documentation] Verify unrecoverable callout handling for L3FIR.
+ [Tags] Verify_Unrecoverable_Callout_Handling_For_L3FIR
+
+ ${value}= Get From Dictionary ${ERROR_INJECT_DICT} L3FIR_UE
+ ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
+ ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}l3fir_ue
+ Inject Unrecoverable Error Through Host
+ ... ${translated_fir} ${value[1]} 1 ${value[2]} ${err_log_path}
+
# On chip controller (OCCFIR) related error injection.
Verify Recoverable Callout Handling For OCC With Threshold 1
@@ -219,6 +258,16 @@
Inject Recoverable Error With Threshold Limit Through Host
... ${translated_fir} ${value[1]} 1 ${value[2]} ${err_log_path}
+Verify Recoverable Callout Handling For NCUFIR With Threshold 1
+ [Documentation] Verify recoverable callout handling for NCUFIR with
+ ... threshold 1.
+ [Tags] Verify_Recoverable_Callout_Handling_For_NCUFIR_With_Threshold_1
+
+ ${value}= Get From Dictionary ${ERROR_INJECT_DICT} NCUFIR_RECV1
+ ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
+ ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}ncufir_th1
+ Inject Recoverable Error With Threshold Limit Through Host
+ ... ${translated_fir} ${value[1]} 1 ${value[2]} ${err_log_path}
*** Keywords ***
@@ -226,7 +275,7 @@
[Documentation] Verify And Clear gard records on HOST.
${output}= Gard Operations On OS list
- Should Not Contain ${output} 'No GARD entries to display'
+ Should Not Contain ${output} No GARD
Gard Operations On OS clear all
Verify Error Log Entry
@@ -256,14 +305,12 @@
... 4. Verify & clear gard records.
[Arguments] ${fir} ${chip_address} ${threshold_limit}
... ${signature_desc} ${log_prefix}
- ... ${master_proc_chip}=True
# Description of argument(s):
# fir FIR (Fault isolation register) value (e.g. 2011400).
# chip_address Chip address (e.g 2000000000000000).
# threshold_limit Threshold limit (e.g 1, 5, 32).
# signature_desc Error log signature description.
# log_prefix Log path prefix.
- # master_proc_chip Processor chip type ('True' or 'False').
Set Auto Reboot 1
Inject Error Through HOST ${fir} ${chip_address} ${threshold_limit}
@@ -286,7 +333,6 @@
... 4. Verify & clear gard records.
[Arguments] ${fir} ${chip_address} ${threshold_limit}
... ${signature_desc} ${log_prefix}
- ... ${master_proc_chip}=True
# Description of argument(s):
# fir FIR (Fault isolation register) value (e.g. 2011400).
# chip_address Chip address (e.g 2000000000000000).
@@ -294,7 +340,6 @@
# signature_desc Error Log signature description.
# (e.g 'mcs(n0p0c0) (MCFIR[0]) mc internal recoverable')
# log_prefix Log path prefix.
- # master_proc_chip Processor chip type ('True' or 'False').
Set Auto Reboot 1
Inject Error Through HOST ${fir} ${chip_address} ${threshold_limit}
@@ -306,12 +351,11 @@
Fetch FIR Address Translation Value
[Documentation] Fetch FIR address translation value through HOST.
- [Arguments] ${fir} ${target_type} ${master_proc_chip}=True
+ [Arguments] ${fir} ${target_type}
# Description of argument(s):
# fir FIR (Fault isolation register) value (e.g. 2011400).
# core_id Core ID (e.g. 9).
# target_type Target type (e.g. 'EX', 'EQ', 'C').
- # master_proc_chip Processor chip type ('True' or 'False').
Login To OS Host
Copy Address Translation Utils To HOST OS
@@ -356,6 +400,8 @@
${RAS_LOG_DIR_PATH}= Catenate ${EXECDIR}/RAS_logs/
Set Suite Variable ${RAS_LOG_DIR_PATH}
+ Set Suite Variable ${master_proc_chip} False
+
Create Directory ${RAS_LOG_DIR_PATH}
OperatingSystem.Directory Should Exist ${RAS_LOG_DIR_PATH}
Empty Directory ${RAS_LOG_DIR_PATH}
diff --git a/lib/ras/variables.py b/lib/ras/variables.py
index 1bf97c8..0964259 100644
--- a/lib/ras/variables.py
+++ b/lib/ras/variables.py
@@ -29,7 +29,7 @@
DES_L3_RECV1 = "'L3FIR[^17].*Received addr_error cresp'"
DES_L3_RECV32 = "'L3FIR[^7].*L3 cache write data CE'"
-DES_L3_UE = "'L3FIR[^8].*L3 cache write data UE'"
+DES_L3_UE = "'L3FIR[^16].*addr_error cresp for mem'"
DES_OCC_RECV1 = "'OCCFIR[^45].*C405_ECC_CE'"
DES_CME_RECV1 = "'CMEFIR[^7].*PPE SRAM Uncorrectable Err'"
@@ -63,7 +63,7 @@
'L2FIR_UE': ['10010800', '0040000000000000', DES_L2_UE],
'L3FIR_RECV1': ['10011800','0000400000000000', DES_L3_RECV1],
'L3FIR_RECV32': ['10011800', '0100000000000000', DES_L3_RECV32],
- 'L3FIR_UE': ['10011800', '0080000000000000', DES_L3_UE],
+ 'L3FIR_UE': ['10011800', '0000800000000000', DES_L3_UE],
'OCCFIR_RECV1': ['01010800', '0000000000040000', DES_OCC_RECV1],
'CMEFIR_RECV1': ['10012000', '0100000000000000', DES_CME_RECV1],
'EQFIR_RECV32': ['1004000A', '8000000000000000', DES_EQ_RECV32],