blob: 0e212b2f4fc457af6249fdc58b71550c9d7058ac [file] [log] [blame]
Rahul Maheshwari2568f872020-09-23 01:03:39 -05001*** Settings ***
2
3Documentation Test BMC dump functionality of OpenBMC.
4
George Keishing88ec2802021-08-04 03:04:22 -05005Resource ../../lib/bmc_redfish_resource.robot
6Resource ../../lib/boot_utils.robot
Rahul Maheshwari2568f872020-09-23 01:03:39 -05007Resource ../../lib/dump_utils.robot
George Keishing88ec2802021-08-04 03:04:22 -05008Resource ../../lib/openbmc_ffdc.robot
manashsarma7e1f2c52023-02-03 03:18:40 -06009Variables ../../data/pel_variables.py
Rahul Maheshwari2568f872020-09-23 01:03:39 -050010
George Keishingf514ccb2021-02-02 23:55:56 -060011Suite Setup Redfish.Login
Rahul Maheshwari2568f872020-09-23 01:03:39 -050012Test Setup Redfish Delete All BMC Dumps
13Test Teardown Test Teardown Execution
14
Tim Leede2d1302021-06-18 01:24:11 +080015*** Variables ***
16
17# Total size of the dump in kilo bytes
18${BMC_DUMP_TOTAL_SIZE} ${1024}
19
20# Minimum space required for one bmc dump in kilo bytes
21${BMC_DUMP_MIN_SPACE_REQD} ${20}
manashsarmabd346b32022-07-20 06:22:58 -050022${MAX_DUMP_COUNT} ${20}
manashsarma395f8802022-11-17 03:09:02 -060023${BMC_DUMP_COLLECTOR_PATH} /var/lib/phosphor-debug-collector/dumps
Rahul Maheshwari2568f872020-09-23 01:03:39 -050024
25*** Test Cases ***
26
manashsarma4b5b1fa2022-11-16 02:36:58 -060027Verify Error Response For Already Deleted Dump Id
28 [Documentation] Delete non existing BMC dump and expect an error.
29 [Tags] Verify_Error_Response_For_Already_Deleted_Dump_Id
30
manashsarma9dcabad2023-02-02 03:55:17 -060031 Redfish Power Off stack_mode=skip
manashsarma4b5b1fa2022-11-16 02:36:58 -060032 ${dump_id}= Create User Initiated BMC Dump Via Redfish
33 Redfish Delete BMC Dump ${dump_id}
34 Run Keyword And Expect Error ValueError: * Redfish Delete BMC Dump ${dump_id}
35
36
Rahul Maheshwari2568f872020-09-23 01:03:39 -050037Verify User Initiated BMC Dump When Host Powered Off
38 [Documentation] Create user initiated BMC dump at host off state and
39 ... verify dump entry for it.
40 [Tags] Verify_User_Initiated_BMC_Dump_When_Host_Powered_Off
41
42 Redfish Power Off stack_mode=skip
Rahul Maheshwaribd38aa42021-08-04 01:24:39 -050043 ${dump_id}= Create User Initiated BMC Dump Via Redfish
Rahul Maheshwaribcefdf22020-10-16 07:51:34 -050044 ${dump_entries}= Get BMC Dump Entries
Rahul Maheshwari2568f872020-09-23 01:03:39 -050045 Length Should Be ${dump_entries} 1
46 List Should Contain Value ${dump_entries} ${dump_id}
47
Rahul Maheshwari2568f872020-09-23 01:03:39 -050048
Rahul Maheshwarie8e9d0c2020-10-30 02:39:01 -050049Verify User Initiated BMC Dump Size
manashsarma7831dc22022-03-08 01:49:36 -060050 [Documentation] Verify user initiated BMC dump size is under 20 MB.
Rahul Maheshwarie8e9d0c2020-10-30 02:39:01 -050051 [Tags] Verify_User_Initiated_BMC_Dump_Size
52
Rahul Maheshwaribd38aa42021-08-04 01:24:39 -050053 ${dump_id}= Create User Initiated BMC Dump Via Redfish
Rahul Maheshwarie8e9d0c2020-10-30 02:39:01 -050054 ${resp}= Redfish.Get Properties /redfish/v1/Managers/bmc/LogServices/Dump/Entries/${dump_id}
55
56 # Example of response from above Redfish GET request.
57 # "@odata.type": "#LogEntry.v1_7_0.LogEntry",
58 # "AdditionalDataSizeBytes": 31644,
59 # "AdditionalDataURI": "/redfish/v1/Managers/bmc/LogServices/Dump/attachment/9",
60 # "Created": "2020-10-23T06:32:53+00:00",
61 # "DiagnosticDataType": "Manager",
62 # "EntryType": "Event",
63 # "Id": "9",
64 # "Name": "BMC Dump Entry"
65
manashsarma7831dc22022-03-08 01:49:36 -060066 # Max size for dump is 20 MB = 20x1024x1024 Byte.
67 Should Be True 0 < ${resp["AdditionalDataSizeBytes"]} < 20971520
Rahul Maheshwarie8e9d0c2020-10-30 02:39:01 -050068
69
manashsarma7e1f2c52023-02-03 03:18:40 -060070Verify Internal Failure Initiated BMC Dump Size
71 [Documentation] Verify that the internal failure initiated BMC dump size is under 20 MB.
72 [Tags] Verify_Internal_Failure_Initiated_BMC_Dump_Size
73
74 Redfish Delete All BMC Dumps
75
76 # Create an internal failure error log.
77 BMC Execute Command ${CMD_INTERNAL_FAILURE}
78
79 # Wait for BMC dump to get generated after injecting internal failure.
80 Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available
81
82 # Verify that only one BMC dump is generated after injecting error.
83 ${dump_entries}= Get BMC Dump Entries
84 ${length}= Get length ${dump_entries}
85 Should Be Equal As Integers ${length} ${1}
86
87 # Max size for dump is 20 MB = 20x1024x1024 Byte.
88 ${resp}= Redfish.Get Properties
89 ... /redfish/v1/Managers/bmc/LogServices/Dump/Entries/${dump_entries[0]}
90 Should Be True 0 < ${resp["AdditionalDataSizeBytes"]} < 20971520
91
92
manashsarmabbc68bc2022-12-05 09:55:17 -060093Verify Multiple BMC Dump Creation
94 [Documentation] Verify that multiple BMC dumps can be created one after
95 ... another successfully.
96 [Tags] Verify_Multiple_BMC_Dump_Creation
97
98 ${dump_count}= Evaluate random.randint(5, 10) modules=random
99 FOR ${INDEX} IN 1 ${dump_count}
100 Create User Initiated BMC Dump Via Redfish
101 END
102
103
manashsarma395f8802022-11-17 03:09:02 -0600104Verify BMC Dump Default Location In BMC
105 [Documentation] Verify that BMC dump is created in its default location of BMC.
George Keishing224e4c72022-12-14 05:07:58 -0600106 [Tags] Verify_BMC_Dump_Default_Location_In_BMC
manashsarma395f8802022-11-17 03:09:02 -0600107
108 Redfish Delete All BMC Dumps
109 ${dump_id}= Create User Initiated BMC Dump Via Redfish
110 ${dump_file} ${stderr} ${rc}= BMC Execute Command
111 ... ls ${BMC_DUMP_COLLECTOR_PATH}/${dump_id}
112 Should Be True ${rc} == 0
manashsarmac30e02a2023-02-20 06:46:15 -0600113 Should Contain Any ${dump_file} BMCDUMP obmcdump
manashsarma395f8802022-11-17 03:09:02 -0600114
115
George Keishing3d850982022-07-19 11:03:03 -0500116Verify User Initiated BMC Dump When Host Booted
117 [Documentation] Create user initiated BMC dump at host booted state and
118 ... verify dump entry for it.
119 [Tags] Verify_User_Initiated_BMC_Dump_When_Host_Booted
120
121 Redfish Power On stack_mode=skip
122 ${dump_id}= Create User Initiated BMC Dump Via Redfish
123 ${dump_entries}= Get BMC Dump Entries
124 Length Should Be ${dump_entries} 1
125 List Should Contain Value ${dump_entries} ${dump_id}
126
127
manashsarma4b5d6ac2023-02-08 00:04:03 -0600128Verify User Initiated BMC Dump At Host Booting
129 [Documentation] Create and verify user initiated BMC dump during Host is powwering on
130 ... or when host booting is in progress.
131 [Tags] Verify_User_Initiated_BMC_Dump_At_Host_Booting
132
133 Redfish Delete All BMC Dumps
134
135 # Initiate power on.
136 Redfish Power Operation On
137 Wait Until Keyword Succeeds 2 min 5 sec Is Boot Progress Changed
138
139 # Create user initiated BMC dump and verify only one dump is available.
140 Create User Initiated BMC Dump Via Redfish
141 ${dump_entries}= Get BMC Dump Entries
142 Length Should Be ${dump_entries} 1
143
144
Rahul Maheshwari2568f872020-09-23 01:03:39 -0500145Verify Dump Persistency On Dump Service Restart
146 [Documentation] Create user dump, restart dump manager service and verify dump
147 ... persistency.
148 [Tags] Verify_Dump_Persistency_On_Dump_Service_Restart
149
Rahul Maheshwaribd38aa42021-08-04 01:24:39 -0500150 Create User Initiated BMC Dump Via Redfish
Rahul Maheshwari2568f872020-09-23 01:03:39 -0500151 ${dump_entries_before}= redfish_utils.get_member_list /redfish/v1/Managers/bmc/LogServices/Dump/Entries
152
153 # Restart dump service.
154 BMC Execute Command systemctl restart xyz.openbmc_project.Dump.Manager.service
155 Sleep 10s reason=Wait for BMC dump service to restart properly
156
157 ${dump_entries_after}= redfish_utils.get_member_list /redfish/v1/Managers/bmc/LogServices/Dump/Entries
158 Lists Should Be Equal ${dump_entries_before} ${dump_entries_after}
159
160
161Verify Dump Persistency On BMC Reset
162 [Documentation] Create user dump, reset BMC and verify dump persistency.
163 [Tags] Verify_Dump_Persistency_On_BMC_Reset
164
manashsarma8fa9c892022-10-03 06:20:58 -0500165 # Power off host so that dump is not offloaded to host OS.
166 Redfish Power Off stack_mode=skip
167
Rahul Maheshwaribd38aa42021-08-04 01:24:39 -0500168 Create User Initiated BMC Dump Via Redfish
Rahul Maheshwari2568f872020-09-23 01:03:39 -0500169 ${dump_entries_before}= redfish_utils.get_member_list /redfish/v1/Managers/bmc/LogServices/Dump/Entries
170
171 # Reset BMC.
George Keishing88ec2802021-08-04 03:04:22 -0500172 OBMC Reboot (off) stack_mode=skip
Rahul Maheshwari2568f872020-09-23 01:03:39 -0500173
174 ${dump_entries_after}= redfish_utils.get_member_list /redfish/v1/Managers/bmc/LogServices/Dump/Entries
175 Lists Should Be Equal ${dump_entries_before} ${dump_entries_after}
176
177
Rahul Maheshwari95cbceb2020-10-21 23:25:08 -0500178Delete User Initiated BMC Dump And Verify
179 [Documentation] Delete user initiated BMC dump and verify.
180 [Tags] Delete_User_Initiated_BMC_Dump_And_Verify
181
Rahul Maheshwaribd38aa42021-08-04 01:24:39 -0500182 ${dump_id}= Create User Initiated BMC Dump Via Redfish
Rahul Maheshwari95cbceb2020-10-21 23:25:08 -0500183 Redfish Delete BMC Dump ${dump_id}
184
185 ${dump_entries}= Get BMC Dump Entries
186 Should Be Empty ${dump_entries}
187
188
189Delete All User Initiated BMC Dumps And Verify
190 [Documentation] Delete all user initiated BMC dumps and verify.
191 [Tags] Delete_All_User_Initiated_BMC_Dumps_And_Verify
192
manashsarma70d840e2022-10-18 02:34:45 -0500193 # Power off host so that dump is not offloaded to host OS.
194 Redfish Power Off stack_mode=skip
195
Rahul Maheshwari95cbceb2020-10-21 23:25:08 -0500196 # Create some BMC dump.
Rahul Maheshwaribd38aa42021-08-04 01:24:39 -0500197 Create User Initiated BMC Dump Via Redfish
198 Create User Initiated BMC Dump Via Redfish
Rahul Maheshwari95cbceb2020-10-21 23:25:08 -0500199
200 Redfish Delete All BMC Dumps
201 ${dump_entries}= Get BMC Dump Entries
202 Should Be Empty ${dump_entries}
203
204
Rahul Maheshwari4f338ab2020-10-21 23:28:40 -0500205Create Two User Initiated BMC Dumps
206 [Documentation] Create two user initiated BMC dumps.
207 [Tags] Create_Two_User_Initiated_BMC_Dumps
208
Rahul Maheshwaribd38aa42021-08-04 01:24:39 -0500209 ${dump_id1}= Create User Initiated BMC Dump Via Redfish
210 ${dump_id2}= Create User Initiated BMC Dump Via Redfish
Rahul Maheshwari4f338ab2020-10-21 23:28:40 -0500211
212 ${dump_entries}= Get BMC Dump Entries
213 Length Should Be ${dump_entries} 2
214 Should Contain ${dump_entries} ${dump_id1}
215 Should Contain ${dump_entries} ${dump_id2}
216
217
218Create Two User Initiated BMC Dumps And Delete One
219 [Documentation] Create two dumps and delete the first.
220 [Tags] Create_Two_User_Initiated_BMC_Dumps_And_Delete_One
221
Rahul Maheshwaribd38aa42021-08-04 01:24:39 -0500222 ${dump_id1}= Create User Initiated BMC Dump Via Redfish
223 ${dump_id2}= Create User Initiated BMC Dump Via Redfish
Rahul Maheshwari4f338ab2020-10-21 23:28:40 -0500224
225 Redfish Delete BMC Dump ${dump_id1}
226
227 ${dump_entries}= Get BMC Dump Entries
228 Length Should Be ${dump_entries} 1
229 List Should Contain Value ${dump_entries} ${dump_id2}
230
231
232Create And Delete User Initiated BMC Dump Multiple Times
233 [Documentation] Create and delete user initiated BMC dump multiple times.
234 [Tags] Create_And_Delete_User_Initiated_BMC_Dump_Multiple_Times
235
236 FOR ${INDEX} IN 1 10
Rahul Maheshwaribd38aa42021-08-04 01:24:39 -0500237 ${dump_id}= Create User Initiated BMC Dump Via Redfish
Rahul Maheshwari4f338ab2020-10-21 23:28:40 -0500238 Redfish Delete BMC Dump ${dump_id}
239 END
240
241
Rahul Maheshwarid709c8d2020-11-01 23:03:51 -0600242Verify Maximum BMC Dump Creation
243 [Documentation] Create maximum BMC dump and verify error when dump runs out of space.
244 [Tags] Verify_Maximum_BMC_Dump_Creation
245 [Teardown] Redfish Delete All BMC Dumps
246
247 # Maximum allowed space for dump is 1024 KB. BMC typically hold 8-14 dumps
248 # before running out of this dump space. So trying to create dumps in 20
249 # iterations to run out of space.
manashsarmabd346b32022-07-20 06:22:58 -0500250 # User can key in the Maximum allowed space for bmc dump and how many iteration.
251 FOR ${n} IN RANGE 0 ${MAX_DUMP_COUNT}
Rahul Maheshwaribd38aa42021-08-04 01:24:39 -0500252 Create User Initiated BMC Dump Via Redfish
Rahul Maheshwarid709c8d2020-11-01 23:03:51 -0600253 ${dump_space}= Get Disk Usage For Dumps
Tim Leede2d1302021-06-18 01:24:11 +0800254 Exit For Loop If ${dump_space} >= (${BMC_DUMP_TOTAL_SIZE} - ${BMC_DUMP_MIN_SPACE_REQD})
Rahul Maheshwarid709c8d2020-11-01 23:03:51 -0600255 END
256
257 # Check error while creating dump when dump size is full.
258 ${payload}= Create Dictionary DiagnosticDataType=Manager
259 Redfish.Post /redfish/v1/Managers/bmc/LogServices/Dump/Actions/LogService.CollectDiagnosticData
260 ... body=${payload} valid_status_codes=[${HTTP_INTERNAL_SERVER_ERROR}]
261
262
manashsarma633cac92022-11-25 03:27:03 -0600263Verify BMC Core Dump When Host Powered Off
264 [Documentation] Verify BMC core dump after application crash at host powered off state.
265 [Tags] Verify_BMC_Core_Dump_When_Host_Powered_Off
266
267 Redfish Power Off stack_mode=skip
268
269 # Ensure all dumps are cleaned out.
270 Redfish Delete All BMC Dumps
271 Trigger Core Dump
272
273 # Verify that BMC dump is available.
274 Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available
275
276
manashsarma849189c2022-11-28 04:16:51 -0600277Verify Core Dump Size
278 [Documentation] Verify BMC core dump size is under 20 MB.
279 [Tags] Verify_Core_Dump_Size
280
281 Redfish Power Off stack_mode=skip
282
283 # Ensure all dumps are cleaned out.
284 Redfish Delete All BMC Dumps
285 Trigger Core Dump
286
287 # Verify that BMC dump is available.
288 Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available
289 ${dump_entries}= Get BMC Dump Entries
290 ${resp}= Redfish.Get Properties
291 ... /redfish/v1/Managers/bmc/LogServices/Dump/Entries/${dump_entries[0]}
292
293 # Max size for dump is 20 MB = 20x1024x1024 Byte.
294 Should Be True 0 < ${resp["AdditionalDataSizeBytes"]} < 20971520
295
296
manashsarmafcbfdf62022-12-07 06:36:28 -0600297Verify Error While Initiating BMC Dump During Dumping State
298 [Documentation] Verify error while initiating BMC dump during dumping state.
299 [Tags] Verify_Error_While_Initiating_BMC_Dump_During_Dumping_State
300
Nandish-Mattiaf5bc152023-01-30 00:49:59 -0600301 ${task_id}= Create User Initiated BMC Dump Via Redfish ${1}
manashsarmafcbfdf62022-12-07 06:36:28 -0600302
303 # Check error while initiating BMC dump while dump in progress.
304 ${payload}= Create Dictionary DiagnosticDataType=Manager
305 Redfish.Post
306 ... /redfish/v1/Managers/bmc/LogServices/Dump/Actions/LogService.CollectDiagnosticData
307 ... body=${payload} valid_status_codes=[${HTTP_SERVICE_UNAVAILABLE}]
308
Nandish-Mattiaf5bc152023-01-30 00:49:59 -0600309 # Wait for above initiated dump to complete. Otherwise, on going dump would impact next test.
310 Wait Until Keyword Succeeds 5 min 15 sec Check Task Completion ${task_id}
311
manashsarmafcbfdf62022-12-07 06:36:28 -0600312
manashsarmaaa845b72022-12-12 05:41:32 -0600313Verify BMC Dump Create Errors While Another BMC Dump In Progress
314 [Documentation] Verify BMC dump creation error until older BMC dump completion.
315 [Tags] Verify_BMC_Dump_Create_Errors_While_Another_BMC_Dump_In_Progress
316
317 # Initiate a BMC dump that returns without completion.
318 ${task_id}= Create User Initiated BMC Dump Via Redfish ${1}
319
320 # Now continue to initiate multiple dump request which is not expected to be accepted
321 # till earlier BMC dump task is completed. A limit is set to avoid risk of infinite loop.
322 ${payload}= Create Dictionary DiagnosticDataType=Manager
323 WHILE True limit=1000
324 ${task_dict}= Redfish.Get Properties /redfish/v1/TaskService/Tasks/${task_id}
325 IF '${task_dict['TaskState']}' == 'Completed' BREAK
326 Redfish.Post
327 ... /redfish/v1/Managers/bmc/LogServices/Dump/Actions/LogService.CollectDiagnosticData
328 ... body=${payload} valid_status_codes=[${HTTP_SERVICE_UNAVAILABLE}]
329 END
330
331 # The next BMC dump initiation request should be accepted as earlier dump is completed.
332 ${resp}= Redfish.Post
333 ... /redfish/v1/Managers/bmc/LogServices/Dump/Actions/LogService.CollectDiagnosticData
334 ... body=${payload} valid_status_codes=[${HTTP_ACCEPTED}]
335
336 # Wait for above initiated dump to complete. Otherwise, on going dump would impact next test.
337 Wait Until Keyword Succeeds 5 min 15 sec Check Task Completion ${resp.dict['Id']}
338
339
manashsarmaadabd3b2023-02-08 04:51:32 -0600340Verify Core Dump After Terminating Dump Manager Service
341 [Documentation] Verify initiate core dumps and kill Phosphor-dump-manager.
342 [Tags] Verify_Core_Dump_After_Terminating_Dump_Manager_Service
343
344 Redfish Power Off stack_mode=skip
345
346 # Remove all available dumps in BMC.
347 Redfish Delete All BMC Dumps
348
349 # Find the pid of the phosphor-dump-manage process and kill it.
350 ${cmd_buf}= Catenate kill -s SEGV $(pgrep phosphor-dump-manager)
351 ${cmd_output} ${stderr} ${rc}= BMC Execute Command ${cmd_buf}
352 Should Be Equal As Integers ${rc} ${0}
353
354 # Verify that BMC dump is available.
355 Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available
356
George Keishing4611b812023-02-16 09:44:33 -0600357 # Verifying that there is only one dump.
manashsarmaadabd3b2023-02-08 04:51:32 -0600358 ${dump_entries}= Get BMC Dump Entries
359 ${length}= Get length ${dump_entries}
360 Should Be Equal As Integers ${length} ${1}
361
362
manashsarma7bf47d22023-02-08 01:10:18 -0600363Verify Error Log And Dump For Internal Failure
364 [Documentation] Verify error log and dump for internal failure.
365 [Tags] Verify_Error_Log_And_Dump_For_Internal_Failure
366
367 Redfish Purge Event Log
368 Redfish Delete All BMC Dumps
369
370 # Create an internal failure error log.
371 BMC Execute Command ${CMD_INTERNAL_FAILURE}
372
373 # With internal failure, an error log file is generated. Check if
374 # BMC has only one error log for this internal failure.
375 ${resp}= Redfish.Get /redfish/v1/Systems/system/LogServices/CELog/Entries
376 Should Be True ${resp.dict["Members@odata.count"]} == ${1}
377
378 # Wait for the BMC dump to become available and verify its presence.
379 Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available
380 ${dump_entries}= Get BMC Dump Entries
381 ${length}= Get length ${dump_entries}
382 Should Be Equal As Integers ${length} ${1}
383
384
Rahul Maheshwari2568f872020-09-23 01:03:39 -0500385*** Keywords ***
386
Rahul Maheshwaribcefdf22020-10-16 07:51:34 -0500387Get BMC Dump Entries
388 [Documentation] Return BMC dump ids list.
389
390 ${dump_uris}= redfish_utils.get_member_list /redfish/v1/Managers/bmc/LogServices/Dump/Entries
391 ${dump_ids}= Create List
392
393 FOR ${dump_uri} IN @{dump_uris}
394 ${dump_id}= Fetch From Right ${dump_uri} /
395 Append To List ${dump_ids} ${dump_id}
396 END
397
398 [Return] ${dump_ids}
399
400
manashsarma633cac92022-11-25 03:27:03 -0600401Is BMC Dump Available
402 [Documentation] Verify if BMC dump is available.
403
404 ${dump_entries}= Get BMC Dump Entries
405
406 # Verifying that BMC dump is available.
407 ${length}= Get length ${dump_entries}
408 Should Be True 0 < ${length}
409
410
Rahul Maheshwarid709c8d2020-11-01 23:03:51 -0600411Get Disk Usage For Dumps
412 [Documentation] Return disk usage in kilobyte for BMC dumps.
413
George Keishing952bb0f2022-12-12 03:13:35 -0600414 ${usage_output} ${stderr} ${rc}= BMC Execute Command du -s ${BMC_DUMP_COLLECTOR_PATH}
Rahul Maheshwarid709c8d2020-11-01 23:03:51 -0600415
416 # Example of output from above BMC cli command.
417 # $ du -s /var/lib/phosphor-debug-collector/dumps
418 # 516 /var/lib/phosphor-debug-collector/dumps
419
420 ${usage_output}= Fetch From Left ${usage_output} /
421 ${usage_output}= Convert To Integer ${usage_output}
422
423 [return] ${usage_output}
424
425
Rahul Maheshwari2568f872020-09-23 01:03:39 -0500426Test Teardown Execution
427 [Documentation] Do test teardown operation.
428
429 FFDC On Test Case Fail
430 Close All Connections