blob: fa82d9414b7047e8db8aa45e8a3f1e01e9b2c16b [file] [log] [blame]
Sridevi Ramesh1d85af02019-02-22 04:08:15 -06001*** Settings ***
2Documentation Utility for RAS test scenarios through HOST & BMC.
3Resource ../../lib/utils.robot
Sridevi Ramesh1d85af02019-02-22 04:08:15 -06004Resource ../../lib/ras/host_utils.robot
5Resource ../../lib/resource.robot
6Resource ../../lib/state_manager.robot
7Resource ../../lib/boot_utils.robot
8Variables ../../lib/ras/variables.py
9Variables ../../data/variables.py
10Resource ../../lib/dump_utils.robot
11
12Library DateTime
13Library OperatingSystem
14Library random
15Library Collections
16
17*** Variables ***
18${stack_mode} normal
19
20*** Keywords ***
21
22Verify And Clear Gard Records On HOST
23 [Documentation] Verify And Clear gard records on HOST.
24
25 ${output}= Gard Operations On OS list
26 Should Not Contain ${output} No GARD
27 Gard Operations On OS clear all
28
29Verify Error Log Entry
30 [Documentation] Verify error log entry & signature description.
31 [Arguments] ${signature_desc} ${log_prefix}
32 # Description of argument(s):
33 # signature_desc Error log signature description.
34 # log_prefix Log path prefix.
35
36 # TODO: Need to move this keyword to common utility.
37
38 Error Logs Should Exist
39
40 Collect eSEL Log ${log_prefix}
41 ${error_log_file_path}= Catenate ${log_prefix}esel.txt
42 ${rc} ${output}= Run and Return RC and Output
43 ... grep -i ${signature_desc} ${error_log_file_path}
44 Should Be Equal ${rc} ${0}
45 Should Not Be Empty ${output}
46
47Inject Recoverable Error With Threshold Limit
48 [Documentation] Inject and verify recoverable error on processor through
Sridevi Ramesh9c6ec282019-03-25 03:35:46 -050049 ... BMC/HOST.
Sridevi Ramesh1d85af02019-02-22 04:08:15 -060050 ... Test sequence:
Sridevi Ramesh9c6ec282019-03-25 03:35:46 -050051 ... 1. Inject recoverable error on a given target
Sridevi Ramesh3e2a3bd2019-05-09 05:30:53 -050052 ... (e.g: Processor core, CAPP, MCA) through BMC/HOST.
Sridevi Ramesh9c6ec282019-03-25 03:35:46 -050053 ... 2. Check If HOST is running.
54 ... 3. Verify error log entry & signature description.
Sridevi Ramesh1d85af02019-02-22 04:08:15 -060055 ... 4. Verify & clear gard records.
56 [Arguments] ${interface_type} ${fir} ${chip_address} ${threshold_limit}
57 ... ${signature_desc} ${log_prefix}
58 # Description of argument(s):
59 # interface_type Inject error through 'BMC' or 'HOST'.
60 # fir FIR (Fault isolation register) value (e.g. 2011400).
61 # chip_address Chip address (e.g 2000000000000000).
62 # threshold_limit Threshold limit (e.g 1, 5, 32).
63 # signature_desc Error log signature description.
64 # log_prefix Log path prefix.
65
Sridevi Ramesh1d85af02019-02-22 04:08:15 -060066 Run Keyword If '${interface_type}' == 'HOST'
67 ... Inject Error Through HOST ${fir} ${chip_address} ${threshold_limit}
68 ... ${master_proc_chip}
69 ... ELSE
70 ... Inject Error Through BMC ${fir} ${chip_address} ${threshold_limit}
71 ... ${master_proc_chip}
72
73 Is Host Running
74 ${output}= Gard Operations On OS list
75 Should Contain ${output} No GARD
76 Verify Error Log Entry ${signature_desc} ${log_prefix}
77 # TODO: Verify SOL console logs.
78
79
Sridevi Ramesh9c6ec282019-03-25 03:35:46 -050080Inject Unrecoverable Error
81 [Documentation] Inject and verify unrecoverable error on processor through
82 ... BMC/HOST.
Sridevi Ramesh1d85af02019-02-22 04:08:15 -060083 ... Test sequence:
Sridevi Ramesh9c6ec282019-03-25 03:35:46 -050084 ... 1. Inject unrecoverable error on a given target
Sridevi Ramesh3e2a3bd2019-05-09 05:30:53 -050085 ... (e.g: Processor core, CAPP, MCA) through BMC/HOST.
Sridevi Ramesh9c6ec282019-03-25 03:35:46 -050086 ... 2. Check If HOST is rebooted.
87 ... 3. Verify & clear gard records.
88 ... 4. Verify error log entry & signature description.
89 ... 5. Verify & clear dump entry.
90 [Arguments] ${interface_type} ${fir} ${chip_address} ${threshold_limit}
Sridevi Ramesh1d85af02019-02-22 04:08:15 -060091 ... ${signature_desc} ${log_prefix}
92 # Description of argument(s):
Sridevi Ramesh9c6ec282019-03-25 03:35:46 -050093 # interface_type Inject error through 'BMC' or 'HOST'.
Sridevi Ramesh1d85af02019-02-22 04:08:15 -060094 # fir FIR (Fault isolation register) value (e.g. 2011400).
95 # chip_address Chip address (e.g 2000000000000000).
96 # threshold_limit Threshold limit (e.g 1, 5, 32).
97 # signature_desc Error Log signature description.
98 # (e.g 'mcs(n0p0c0) (MCFIR[0]) mc internal recoverable')
99 # log_prefix Log path prefix.
100
Sridevi Ramesh9c6ec282019-03-25 03:35:46 -0500101 Run Keyword If '${interface_type}' == 'HOST'
102 ... Inject Error Through HOST ${fir} ${chip_address} ${threshold_limit}
103 ... ${master_proc_chip}
104 ... ELSE
105 ... Inject Error Through BMC ${fir} ${chip_address} ${threshold_limit}
106 ... ${master_proc_chip}
107
Sridevi Ramesh1d85af02019-02-22 04:08:15 -0600108 Wait Until Keyword Succeeds 500 sec 20 sec Is Host Rebooted
109 Wait for OS
110 Verify Error Log Entry ${signature_desc} ${log_prefix}
111 ${resp}= OpenBMC Get Request ${DUMP_ENTRY_URI}list
112 Should Not Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND}
113 Delete All BMC Dump
114 Verify And Clear Gard Records On HOST
115
116Fetch FIR Address Translation Value
117 [Documentation] Fetch FIR address translation value through HOST.
118 [Arguments] ${fir} ${target_type}
119 # Description of argument(s):
120 # fir FIR (Fault isolation register) value (e.g. '2011400').
121 # core_id Core ID (e.g. '9').
122 # target_type Target type (e.g. 'EX', 'EQ', 'C').
123
124 Login To OS Host
125 Copy Address Translation Utils To HOST OS
126
127 # Fetch processor chip IDs.
128 ${proc_chip_id}= Get ProcChipId From OS Processor ${master_proc_chip}
129 # Example output:
130 # 00000000
131
132 ${core_ids}= Get Core IDs From OS ${proc_chip_id[-1]}
133 # Example output:
134 #./probe_cpus.sh | grep 'CHIP ID: 0' | cut -c21-22
135 # ['14', '15', '16', '17']
136
137 # Ignoring master core ID.
138 ${output}= Get Slice From List ${core_ids} 1
139 # Feth random non-master core ID.
140 ${core_ids_sub_list}= Evaluate random.sample(${core_ids}, 1) random
141 ${core_id}= Get From List ${core_ids_sub_list} 0
142 ${translated_fir_addr}= FIR Address Translation Through HOST
143 ... ${fir} ${core_id} ${target_type}
144
145 [Return] ${translated_fir_addr}
146
147RAS Test SetUp
148 [Documentation] Validates input parameters.
149
150 Should Not Be Empty
151 ... ${OS_HOST} msg=You must provide DNS name/IP of the OS host.
152 Should Not Be Empty
153 ... ${OS_USERNAME} msg=You must provide OS host user name.
154 Should Not Be Empty
155 ... ${OS_PASSWORD} msg=You must provide OS host user password.
156
157 # Boot to OS.
158 REST Power On quiet=${1}
159 # Adding delay after host bring up.
160 Sleep 60s
161
162RAS Suite Setup
163 [Documentation] Create RAS log directory to store all RAS test logs.
164
165 ${RAS_LOG_DIR_PATH}= Catenate ${EXECDIR}/RAS_logs/
166 Set Suite Variable ${RAS_LOG_DIR_PATH}
167 Set Suite Variable ${master_proc_chip} False
168
169 Create Directory ${RAS_LOG_DIR_PATH}
170 OperatingSystem.Directory Should Exist ${RAS_LOG_DIR_PATH}
171 Empty Directory ${RAS_LOG_DIR_PATH}
172
173 Should Not Be Empty ${ESEL_BIN_PATH}
174 Set Environment Variable PATH %{PATH}:${ESEL_BIN_PATH}
175
176 # Boot to Os.
177 REST Power On quiet=${1}
178
179 # Check Opal-PRD service enabled on host.
180 ${opal_prd_state}= Is Opal-PRD Service Enabled
181 Run Keyword If '${opal_prd_state}' == 'disabled'
182 ... Enable Opal-PRD Service On HOST
183
184RAS Suite Cleanup
185 [Documentation] Perform RAS suite cleanup and verify that host
186 ... boots after test suite run.
187
188 # Boot to OS.
189 REST Power On quiet=${1}
190 Delete Error Logs
191 Gard Operations On OS clear all
Sridevi Ramesh3e2a3bd2019-05-09 05:30:53 -0500192
193
194Inject Error At HOST Boot Path
195
196 [Documentation] Inject and verify recoverable error on processor through
197 ... BMC using pdbg tool at HOST Boot path.
198 ... Test sequence:
199 ... 1. Inject error on a given target
200 ... (e.g: Processor core, CAPP, MCA) through BMC using
201 ... pdbg tool at HOST Boot path.
202 ... 2. Check If HOST is rebooted and running.
203 ... 3. Verify error log entry & signature description.
204 ... 4. Verify & clear gard records.
205 [Arguments] ${fir} ${chip_address} ${signature_desc} ${log_prefix}
206 # Description of argument(s):
207 # fir FIR (Fault isolation register) value (e.g. 2011400).
208 # chip_address Chip address (e.g 2000000000000000).
209 # signature_desc Error log signature description.
210 # log_prefix Log path prefix.
211
212 Inject Error Through BMC At HOST Boot ${fir} ${chip_address}
213
214 Wait Until Keyword Succeeds 500 sec 20 sec Is Host Rebooted
215 Wait for OS
216 Verify Error Log Entry ${signature_desc} ${log_prefix}
217 Verify And Clear Gard Records On HOST