blob: c7edfd9694312559c24948b7e18af6ba48f6288a [file] [log] [blame]
Sridevi Ramesh3ed96342018-01-15 05:14:37 -06001*** Settings ***
2Documentation This suite tests checkstop operations through HOST.
Sridevi Ramesh9c6ec282019-03-25 03:35:46 -05003
George Keishingbba33902018-06-06 13:39:27 -05004Resource ../../lib/openbmc_ffdc.robot
Sridevi Ramesh9c6ec282019-03-25 03:35:46 -05005Resource ../../lib/openbmc_ffdc_utils.robot
George Keishingbba33902018-06-06 13:39:27 -05006Resource ../../lib/openbmc_ffdc_methods.robot
Sridevi Ramesh1d85af02019-02-22 04:08:15 -06007Resource ../../openpower/ras/ras_utils.robot
Sridevi Ramesh4adb3f22019-03-12 03:31:35 -05008Variables ../../lib/ras/variables.py
9Variables ../../data/variables.py
Sridevi Ramesh3ed96342018-01-15 05:14:37 -060010
11Library DateTime
12Library OperatingSystem
13Library random
14Library Collections
15
16Suite Setup RAS Suite Setup
17Test Setup RAS Test Setup
18Test Teardown FFDC On Test Case Fail
19Suite Teardown RAS Suite Cleanup
20
21Force Tags Host_RAS
Sridevi Ramesh9c6ec282019-03-25 03:35:46 -050022
Sridevi Ramesh3ed96342018-01-15 05:14:37 -060023*** Variables ***
24${stack_mode} normal
25
26*** Test Cases ***
27# Memory channel (MCACALIFIR) related error injection.
28
29Verify Recoverable Callout Handling For MCA With Threshold 1
30 [Documentation] Verify recoverable callout handling for MCACALIFIR with
31 ... threshold 1.
32 [Tags] Verify_Recoverable_Callout_Handling_For_MCA_With_Threshold_1
33
34 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} MCACALIFIR_RECV1
35 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}mcacalfir_th1
Sridevi Ramesh1d85af02019-02-22 04:08:15 -060036 Inject Recoverable Error With Threshold Limit
37 ... HOST ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
Sridevi Ramesh3ed96342018-01-15 05:14:37 -060038
39Verify Recoverable Callout Handling For MCA With Threshold 32
40 [Documentation] Verify recoverable callout handling for MCACALIFIR with
41 ... threshold 32.
42 [Tags] Verify_Recoverable_Callout_Handling_For_MCA_With_Threshold_32
43
44 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} MCACALIFIR_RECV32
45 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}mcacalfir_th32
Sridevi Ramesh1d85af02019-02-22 04:08:15 -060046 Inject Recoverable Error With Threshold Limit
47 ... HOST ${value[0]} ${value[1]} 32 ${value[2]} ${err_log_path}
Sridevi Ramesh3ed96342018-01-15 05:14:37 -060048
Sridevi Ramesh5f9fee52019-03-21 06:25:45 -050049Verify Unrecoverable Callout Handling For MCA
50 [Documentation] Verify unrecoverable callout handling for MCACALIFIR.
51 [Tags] Verify_Unrecoverable_Callout_Handling_For_MCA
52
53 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} MCACALIFIR_UE
54 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}mcacalfir
Sridevi Ramesh9c6ec282019-03-25 03:35:46 -050055 Inject Unrecoverable Error HOST
Sridevi Ramesh5f9fee52019-03-21 06:25:45 -050056 ... ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
57
Sridevi Ramesh3ed96342018-01-15 05:14:37 -060058# Memory buffer (MCIFIR) related error injection.
59
60Verify Recoverable Callout Handling For MCI With Threshold 1
61 [Documentation] Verify recoverable callout handling for mci with
62 ... threshold 1.
63 [Tags] Verify_Recoverable_Callout_Handling_For_MCI_With_Threshold_1
64
Sridevi Ramesh4adb3f22019-03-12 03:31:35 -050065 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} MCI_RECV1
Sridevi Ramesh3ed96342018-01-15 05:14:37 -060066 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}mcifir_th1
Sridevi Ramesh1d85af02019-02-22 04:08:15 -060067 Inject Recoverable Error With Threshold Limit
68 ... HOST ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
Sridevi Ramesh3ed96342018-01-15 05:14:37 -060069
Sridevi Ramesh5f9fee52019-03-21 06:25:45 -050070Verify Unrecoverable Callout Handling For MCI
71 [Documentation] Verify unrecoverable callout handling for mci.
72 [Tags] Verify_Unrecoverable_Callout_Handling_For_MCI
73
Sridevi Ramesh4adb3f22019-03-12 03:31:35 -050074 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} MCI_UE
Sridevi Ramesh5f9fee52019-03-21 06:25:45 -050075 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}mcifir
Sridevi Ramesh9c6ec282019-03-25 03:35:46 -050076 Inject Unrecoverable Error HOST
Sridevi Ramesh5f9fee52019-03-21 06:25:45 -050077 ... ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
78
Sridevi Ramesh3ed96342018-01-15 05:14:37 -060079# CAPP accelerator (CXAFIR) related error injection.
80
81Verify Recoverable Callout Handling For CXA With Threshold 5
82 [Documentation] Verify recoverable callout handling for CXA with
83 ... threshold 5.
84 [Tags] Verify_Recoverable_Callout_Handling_For_CXA_With_Threshold_5
85
86 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} CXA_RECV5
87 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}cxafir_th5
Sridevi Ramesh1d85af02019-02-22 04:08:15 -060088 Inject Recoverable Error With Threshold Limit
89 ... HOST ${value[0]} ${value[1]} 5 ${value[2]} ${err_log_path}
Sridevi Ramesh3ed96342018-01-15 05:14:37 -060090
91Verify Recoverable Callout Handling For CXA With Threshold 32
92 [Documentation] Verify recoverable callout handling for CXA with
93 ... threshold 32.
94 [Tags] Verify_Recoverable_Callout_Handling_For_CXA_With_Threshold_32
95
96 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} CXA_RECV32
97 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}cxafir_th32
Sridevi Ramesh1d85af02019-02-22 04:08:15 -060098 Inject Recoverable Error With Threshold Limit
99 ... HOST ${value[0]} ${value[1]} 32 ${value[2]} ${err_log_path}
Sridevi Ramesh3ed96342018-01-15 05:14:37 -0600100
Sridevi Ramesh5f9fee52019-03-21 06:25:45 -0500101Verify Unrecoverable Callout Handling For CXA
102 [Documentation] Verify unrecoverable callout handling for CXAFIR.
103 [Tags] Verify_Unrecoverable_Callout_Handling_For_CXA
104
105 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} CXA_UE
106 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}cxafir_ue
Sridevi Ramesh9c6ec282019-03-25 03:35:46 -0500107 Inject Unrecoverable Error HOST
Sridevi Ramesh5f9fee52019-03-21 06:25:45 -0500108 ... ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
109
Sridevi Ramesh3ed96342018-01-15 05:14:37 -0600110
111# OBUSFIR related error injection.
112
113Verify Recoverable Callout Handling For OBUS With Threshold 32
114 [Documentation] Verify recoverable callout handling for OBUS with
115 ... threshold 32.
116 [Tags] Verify_Recoverable_Callout_Handling_For_OBUS_With_Threshold_32
117
118 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} OBUS_RECV32
119 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}obusfir_th32
Sridevi Ramesh1d85af02019-02-22 04:08:15 -0600120 Inject Recoverable Error With Threshold Limit
121 ... HOST ${value[0]} ${value[1]} 32 ${value[2]} ${err_log_path}
Sridevi Ramesh3ed96342018-01-15 05:14:37 -0600122
123# Nvidia graphics processing units (NPU0FIR) related error injection.
124
125Verify Recoverable Callout Handling For NPU0 With Threshold 32
126 [Documentation] Verify recoverable callout handling for NPU0 with
127 ... threshold 32.
128 [Tags] Verify_Recoverable_Callout_Handling_For_NPU0_With_Threshold_32
129
130 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} NPU0_RECV32
131 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}npu0fir_th32
Sridevi Ramesh1d85af02019-02-22 04:08:15 -0600132 Inject Recoverable Error With Threshold Limit
133 ... HOST ${value[0]} ${value[1]} 32 ${value[2]} ${err_log_path}
Sridevi Ramesh3ed96342018-01-15 05:14:37 -0600134
135# Nest accelerator NXDMAENGFIR related error injection.
136
137Verify Recoverable Callout Handling For NXDMAENG With Threshold 1
138 [Documentation] Verify recoverable callout handling for NXDMAENG with
139 ... threshold 1.
140 [Tags] Verify_Recoverable_Callout_Handling_For_NXDMAENG_With_Threshold_1
141
142 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} NX_RECV1
143 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}nxfir_th1
Sridevi Ramesh1d85af02019-02-22 04:08:15 -0600144 Inject Recoverable Error With Threshold Limit
145 ... HOST ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
Sridevi Ramesh3ed96342018-01-15 05:14:37 -0600146
147
148Verify Recoverable Callout Handling For NXDMAENG With Threshold 32
149 [Documentation] Verify recoverable callout handling for NXDMAENG with
150 ... threshold 32.
151 [Tags] Verify_Recoverable_Callout_Handling_For_NXDMAENG_With_Threshold_32
152
153 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} NX_RECV32
154 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}nxfir_th32
Sridevi Ramesh1d85af02019-02-22 04:08:15 -0600155 Inject Recoverable Error With Threshold Limit
156 ... HOST ${value[0]} ${value[1]} 32 ${value[2]} ${err_log_path}
Sridevi Ramesh3ed96342018-01-15 05:14:37 -0600157
Sridevi Ramesh5f9fee52019-03-21 06:25:45 -0500158Verify Unrecoverable Callout Handling For NXDMAENG
159 [Documentation] Verify unrecoverable callout handling for NXDMAENG.
160 [Tags] Verify_Unrecoverable_Callout_Handling_For_NXDMAENG
161
162 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} NX_UE
163 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}nxfir_ue
Sridevi Ramesh9c6ec282019-03-25 03:35:46 -0500164 Inject Unrecoverable Error HOST
Sridevi Ramesh5f9fee52019-03-21 06:25:45 -0500165 ... ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
Sridevi Ramesh3ed96342018-01-15 05:14:37 -0600166
167# L2FIR related error injection.
168
169Verify Recoverable Callout Handling For L2FIR With Threshold 1
170 [Documentation] Verify recoverable callout handling for L2FIR with
171 ... threshold 1.
172 [Tags] Verify_Recoverable_Callout_Handling_For_L2FIR_With_Threshold_1
173
174 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} L2FIR_RECV1
175 ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
176 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}l2fir_th1
Sridevi Ramesh1d85af02019-02-22 04:08:15 -0600177 Inject Recoverable Error With Threshold Limit
178 ... HOST ${translated_fir} ${value[1]} 1 ${value[2]} ${err_log_path}
Sridevi Ramesh3ed96342018-01-15 05:14:37 -0600179
180Verify Recoverable Callout Handling For L2FIR With Threshold 32
181 [Documentation] Verify recoverable callout handling for L2FIR with
182 ... threshold 32.
183 [Tags] Verify_Recoverable_Callout_Handling_For_L2FIR_With_Threshold_32
184
185 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} L2FIR_RECV32
186 ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
187 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}l2fir_th32
Sridevi Ramesh1d85af02019-02-22 04:08:15 -0600188 Inject Recoverable Error With Threshold Limit
189 ... HOST ${translated_fir} ${value[1]} 32 ${value[2]} ${err_log_path}
Sridevi Ramesh3ed96342018-01-15 05:14:37 -0600190
Sridevi Ramesh5f9fee52019-03-21 06:25:45 -0500191Verify Unrecoverable Callout Handling For L2FIR
192 [Documentation] Verify unrecoverable callout handling for L2FIR.
193 [Tags] Verify_Unrecoverable_Callout_Handling_For_L2FIR
194
195 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} L2FIR_UE
196 ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
197 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}l2fir_ue
Sridevi Ramesh9c6ec282019-03-25 03:35:46 -0500198 Inject Unrecoverable Error HOST
Sridevi Ramesh5f9fee52019-03-21 06:25:45 -0500199 ... ${translated_fir} ${value[1]} 1 ${value[2]} ${err_log_path}
200
Sridevi Ramesh3ed96342018-01-15 05:14:37 -0600201
202# L3FIR related error injection.
203
204Verify Recoverable Callout Handling For L3FIR With Threshold 1
205 [Documentation] Verify recoverable callout handling for L3FIR with
206 ... threshold 1.
207 [Tags] Verify_Recoverable_Callout_Handling_For_L3FIR_With_Threshold_1
208
209 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} L3FIR_RECV1
210 ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
211 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}l3fir_th1
Sridevi Ramesh1d85af02019-02-22 04:08:15 -0600212 Inject Recoverable Error With Threshold Limit
213 ... HOST ${translated_fir} ${value[1]} 1 ${value[2]} ${err_log_path}
Sridevi Ramesh3ed96342018-01-15 05:14:37 -0600214
215Verify Recoverable Callout Handling For L3FIR With Threshold 32
216 [Documentation] Verify recoverable callout handling for L3FIR with
217 ... threshold 32.
218 [Tags] Verify_Recoverable_Callout_Handling_For_L3FIR_With_Threshold_32
219
220 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} L3FIR_RECV32
221 ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
222 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}l3fir_th32
Sridevi Ramesh1d85af02019-02-22 04:08:15 -0600223 Inject Recoverable Error With Threshold Limit
224 ... HOST ${translated_fir} ${value[1]} 32 ${value[2]} ${err_log_path}
Sridevi Ramesh3ed96342018-01-15 05:14:37 -0600225
Sridevi Ramesh5f9fee52019-03-21 06:25:45 -0500226Verify Unrecoverable Callout Handling For L3FIR
227 [Documentation] Verify unrecoverable callout handling for L3FIR.
228 [Tags] Verify_Unrecoverable_Callout_Handling_For_L3FIR
229
230 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} L3FIR_UE
231 ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
232 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}l3fir_ue
Sridevi Ramesh9c6ec282019-03-25 03:35:46 -0500233 Inject Unrecoverable Error HOST
Sridevi Ramesh5f9fee52019-03-21 06:25:45 -0500234 ... ${translated_fir} ${value[1]} 1 ${value[2]} ${err_log_path}
235
Sridevi Ramesh3ed96342018-01-15 05:14:37 -0600236
237# On chip controller (OCCFIR) related error injection.
238
239Verify Recoverable Callout Handling For OCC With Threshold 1
240 [Documentation] Verify recoverable callout handling for OCCFIR with
241 ... threshold 1.
242 [Tags] Verify_Recoverable_Callout_Handling_For_OCC_With_Threshold_1
243
244 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} OCCFIR_RECV1
245 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}occfir_th1
Sridevi Ramesh1d85af02019-02-22 04:08:15 -0600246 Inject Recoverable Error With Threshold Limit
247 ... HOST ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
Sridevi Ramesh3ed96342018-01-15 05:14:37 -0600248
249# Core management engine (CMEFIR) related error injection.
250
251Verify Recoverable Callout Handling For CMEFIR With Threshold 1
252 [Documentation] Verify recoverable callout handling for CMEFIR with
253 ... threshold 1.
254 [Tags] Verify_Recoverable_Callout_Handling_For_CMEFIR_With_Threshold_1
255
256 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} CMEFIR_RECV1
257 ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
258 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}cmefir_th1
Sridevi Ramesh1d85af02019-02-22 04:08:15 -0600259 Inject Recoverable Error With Threshold Limit
260 ... HOST ${translated_fir} ${value[1]} 1 ${value[2]} ${err_log_path}
Sridevi Ramesh3ed96342018-01-15 05:14:37 -0600261
262# Nest control vunit (NCUFIR) related error injection.
263
264Verify Recoverable Callout Handling For NCUFIR With Threshold 1
265 [Documentation] Verify recoverable callout handling for NCUFIR with
266 ... threshold 1.
267 [Tags] Verify_Recoverable_Callout_Handling_For_NCUFIR_With_Threshold_1
268
269 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} NCUFIR_RECV1
270 ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
271 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}ncufir_th1
Sridevi Ramesh1d85af02019-02-22 04:08:15 -0600272 Inject Recoverable Error With Threshold Limit
273 ... HOST ${translated_fir} ${value[1]} 1 ${value[2]} ${err_log_path}
Sridevi Ramesh3ed96342018-01-15 05:14:37 -0600274
Sridevi Ramesh5f9fee52019-03-21 06:25:45 -0500275Verify Unrecoverable Callout Handling For NCUFIR
276 [Documentation] Verify unrecoverable callout handling for NCUFIR.
277 [Tags] Verify_Unrecoverable_Callout_Handling_For_NCUFIR
278
279 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} NCUFIR_UE
280 ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
281 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}ncufir_ue
Sridevi Ramesh9c6ec282019-03-25 03:35:46 -0500282 Inject Unrecoverable Error HOST
Sridevi Ramesh5f9fee52019-03-21 06:25:45 -0500283 ... ${translated_fir} ${value[1]} 1 ${value[2]} ${err_log_path}
284
Sridevi Ramesh3ed96342018-01-15 05:14:37 -0600285
286# Core FIR related error injection.
287
288Verify Recoverable Callout Handling For CoreFIR With Threshold 5
289 [Documentation] Verify recoverable callout handling for CoreFIR with
290 ... threshold 5.
291 [Tags] Verify_Recoverable_Callout_Handling_For_CoreFIR_With_Threshold_5
292
293 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} COREFIR_RECV5
294 ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
295 Disable CPU States Through HOST
296 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}corefir_th5
Sridevi Ramesh1d85af02019-02-22 04:08:15 -0600297 Inject Recoverable Error With Threshold Limit
298 ... HOST ${value[0]} ${value[1]} 5 ${value[2]} ${err_log_path}
Sridevi Ramesh3ed96342018-01-15 05:14:37 -0600299
300Verify Recoverable Callout Handling For CoreFIR With Threshold 1
301 [Documentation] Verify recoverable callout handling for CoreFIR with
302 ... threshold 1.
303 [Tags] Verify_Recoverable_Callout_CoreFIR_Handling_For_With_Threshold_1
304
305 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} COREFIR_RECV1
306 ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
307 Disable CPU States Through HOST
308 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}corefir_th1
Sridevi Ramesh1d85af02019-02-22 04:08:15 -0600309 Inject Recoverable Error With Threshold Limit
310 ... HOST ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
Sridevi Ramesh3ed96342018-01-15 05:14:37 -0600311
Sridevi Ramesh5f9fee52019-03-21 06:25:45 -0500312Verify Unrecoverable Callout Handling For CoreFIR
313 [Documentation] Verify unrecoverable callout handling for CoreFIR.
314 [Tags] Verify_Unrecoverable_Callout_Handling_For_CoreFIR
315
316 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} COREFIR_UE
317 ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EX
318 Disable CPU States Through HOST
319 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}corefir_ue
Sridevi Ramesh9c6ec282019-03-25 03:35:46 -0500320 Inject Unrecoverable Error HOST
Sridevi Ramesh5f9fee52019-03-21 06:25:45 -0500321 ... ${value[0]} ${value[1]} 1 ${value[2]} ${err_log_path}
322
Sridevi Ramesh3ed96342018-01-15 05:14:37 -0600323Verify Recoverable Callout Handling For EQFIR With Threshold 32
324 [Documentation] Verify recoverable callout handling for L3FIR with
325 ... threshold 32.
326 [Tags] Verify_Recoverable_Callout_Handling_For_EQFIR_With_Threshold_32
327
328 ${value}= Get From Dictionary ${ERROR_INJECT_DICT} EQFIR_RECV32
329 ${translated_fir}= Fetch FIR Address Translation Value ${value[0]} EQ
330 ${err_log_path}= Catenate ${RAS_LOG_DIR_PATH}eqfir_th32
Sridevi Ramesh1d85af02019-02-22 04:08:15 -0600331 Inject Recoverable Error With Threshold Limit
332 ... HOST ${translated_fir} ${value[1]} 32 ${value[2]} ${err_log_path}