blob: 339bc8ecf84d7ae3ee3d77c7b1888d2091cc0fa4 [file] [log] [blame]
George Keishing69e6f712016-09-12 06:30:09 -05001*** Settings ***
2Documentation Methods to execute commands on BMC and collect
3... data to a list of FFDC files
4
Michael Walsh265ae412018-11-16 15:32:03 -06005Resource openbmc_ffdc_utils.robot
6Resource rest_client.robot
7Resource utils.robot
8Resource list_utils.robot
9Resource logging_utils.robot
Steven Sombar2da175f2019-02-04 10:32:58 -060010Resource bmc_redfish_resource.robot
Michael Walsh265ae412018-11-16 15:32:03 -060011Library SSHLibrary
12Library OperatingSystem
13Library Collections
14Library String
15Library gen_print.py
16Library gen_cmd.py
17Library gen_robot_keyword.py
18Library dump_utils.py
19Library logging_utils.py
Gunnar Mills7e2cda22016-10-11 15:37:34 -050020
Michael Walshde5e9372018-11-15 11:28:23 -060021*** Variables ***
22
Michael Walsh265ae412018-11-16 15:32:03 -060023${FFDC_CMD_TIMEOUT} 240
George Keishingbe3d1df2022-03-29 08:49:10 -050024${FFDC_BMC_FILES_CLEANUP} rm -rf /tmp/BMC_* /tmp/PEL_* /tmp/PLDM_*
25... /tmp/OCC_* /tmp/fan_* /tmp/GUARD_* /tmp/DEVTREE
Michael Walshde5e9372018-11-15 11:28:23 -060026
George Keishing69e6f712016-09-12 06:30:09 -050027*** Keywords ***
28
George Keishing69e6f712016-09-12 06:30:09 -050029# Method : Call FFDC Methods #
30# Execute the user define keywords from the FFDC List #
31# Unlike any other keywords this will call into the #
32# list of keywords defined in the FFDC list at one go #
George Keishing69e6f712016-09-12 06:30:09 -050033
34Call FFDC Methods
Michael Walshe844e9a2017-04-20 16:51:10 -050035 [Documentation] Call into FFDC Keyword index list.
36 [Arguments] ${ffdc_function_list}=${EMPTY}
George Keishing69e6f712016-09-12 06:30:09 -050037
Michael Walshe844e9a2017-04-20 16:51:10 -050038 # Description of argument(s):
39 # ffdc_function_list A colon-delimited list naming the kinds of FFDC that
Michael Walsh63b70c02017-10-30 15:02:09 -050040 # are to be collected
Michael Walshe844e9a2017-04-20 16:51:10 -050041 # (e.g. "FFDC Generic Report:BMC Specific Files").
42 # Acceptable values can be found in the description
43 # field of FFDC_METHOD_CALL in
44 # lib/openbmc_ffdc_list.py. Those values can be
45 # obtained via a call to 'Get FFDC Method Desc' (also
46 # from lib/openbmc_ffdc_list.py).
47
48 @{entries}= Get FFDC Method Index
Michael Walsh63b70c02017-10-30 15:02:09 -050049 # Example entries:
50 # entries:
51 # entries[0]: BMC LOGS
52
53 @{ffdc_file_list}= Create List
Sushil Singh19888552020-06-08 02:14:52 -050054 FOR ${index} IN @{entries}
55 ${ffdc_file_sub_list}= Method Call Keyword List ${index} ${ffdc_function_list}
56 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
57 END
Michael Walsh63b70c02017-10-30 15:02:09 -050058
Michael Walsh6bad8412017-06-27 14:09:36 -050059 Run Key U SSHLibrary.Close All Connections
George Keishing69e6f712016-09-12 06:30:09 -050060
Michael Walsh63b70c02017-10-30 15:02:09 -050061 [Return] ${ffdc_file_list}
62
Steven Sombar2da175f2019-02-04 10:32:58 -060063
George Keishing69e6f712016-09-12 06:30:09 -050064Method Call Keyword List
Michael Walsh63b70c02017-10-30 15:02:09 -050065 [Documentation] Process FFDC request and return a list of generated files.
Michael Walshe844e9a2017-04-20 16:51:10 -050066 [Arguments] ${index} ${ffdc_function_list}=${EMPTY}
George Keishing69e6f712016-09-12 06:30:09 -050067
Michael Walshe844e9a2017-04-20 16:51:10 -050068 # Description of argument(s):
69 # index The index into the FFDC_METHOD_CALL dictionary (e.g.
70 # 'BMC LOGS').
71 # ffdc_function_list See ffdc_function_list description in
72 # "Call FFDC Methods" (above).
73
Michael Walsh63b70c02017-10-30 15:02:09 -050074 @{method_list}= Get FFDC Method Call ${index}
75 # Example method_list:
76 # method_list:
77 # method_list[0]:
78 # method_list[0][0]: FFDC Generic Report
79 # method_list[0][1]: BMC FFDC Manifest
80 # method_list[1]:
81 # method_list[1][0]: Get Request FFDC
82 # method_list[1][1]: BMC FFDC Get Requests
83 # (etc.)
Michael Walshe844e9a2017-04-20 16:51:10 -050084
85 # If function list is empty assign default (i.e. a list of all allowable
86 # values). In either case, convert ffdc_function_list from a string to
87 # a list.
88 @{ffdc_function_list}=
89 ... Run Keyword If '${ffdc_function_list}' == '${EMPTY}'
90 ... Get FFDC Method Desc ${index}
91 ... ELSE
92 ... Split String ${ffdc_function_list} separator=:
93
Michael Walsh63b70c02017-10-30 15:02:09 -050094 @{ffdc_file_list}= Create List
Sushil Singh19888552020-06-08 02:14:52 -050095 FOR ${method} IN @{method_list}
96 ${ffdc_file_sub_list}= Execute Keyword Method ${method[0]} ${method[1]} @{ffdc_function_list}
97 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
98 END
Michael Walsh63b70c02017-10-30 15:02:09 -050099
100 [Return] ${ffdc_file_list}
101
George Keishing69e6f712016-09-12 06:30:09 -0500102
103Execute Keyword Method
Michael Walshe844e9a2017-04-20 16:51:10 -0500104 [Documentation] Call into BMC method keywords. Don't let one
105 ... failure skip the remaining. Get whatever data
106 ... it could gather at worse case scenario.
107 [Arguments] ${description} ${keyword_name} @{ffdc_function_list}
George Keishing69e6f712016-09-12 06:30:09 -0500108
Michael Walshe844e9a2017-04-20 16:51:10 -0500109 # Description of argument(s):
110 # description The description of the FFDC to be collected. This
111 # would be any value returned by
112 # 'Get FFDC Method Desc' (e.g. "FFDC Generic Report").
113 # keyword_name The name of the keyword to call to collect the FFDC
114 # data (again, see FFDC_METHOD_CALL).
115 # ffdc_function_list See ffdc_function_list description in
116 # "Call FFDC Methods" (above). The only difference is
117 # in this case, it should be a list rather than a
118 # colon-delimited value.
George Keishing69e6f712016-09-12 06:30:09 -0500119
Michael Walsh63b70c02017-10-30 15:02:09 -0500120 @{ffdc_file_list}= Create List
Michael Walshe844e9a2017-04-20 16:51:10 -0500121
Michael Walsh63b70c02017-10-30 15:02:09 -0500122 ${index}= Get Index From List ${ffdc_function_list} ${description}
123 Run Keyword If '${index}' == '${-1}' Return from Keyword
124 ... ${ffdc_file_list}
125
126 ${status} ${ffdc_file_list}= Run Key ${keyword_name} ignore=1
127 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500128
George Keishingbe3d1df2022-03-29 08:49:10 -0500129
130BMC FFDC Cleanup
131 [Documentation] Run the ssh commands from FFDC_BMC_FILES_CLEANUP.
132
133 Log To Console BMC FFDC Files clean up: ${FFDC_BMC_FILES_CLEANUP}
134 BMC Execute Command ${FFDC_BMC_FILES_CLEANUP} ignore_err=1
135
136
George Keishing69e6f712016-09-12 06:30:09 -0500137# Method : BMC FFDC Manifest #
138# Execute command on BMC and write to ffdc_report.txt #
George Keishing69e6f712016-09-12 06:30:09 -0500139
140BMC FFDC Manifest
Michael Walsh63b70c02017-10-30 15:02:09 -0500141 [Documentation] Run the ssh commands from FFDC_BMC_CMD and return a list
142 ... of generated files.
George Keishing69e6f712016-09-12 06:30:09 -0500143
Michael Walsh63b70c02017-10-30 15:02:09 -0500144 @{ffdc_file_list}= Create List ${FFDC_FILE_PATH}
145 @{entries}= Get FFDC Cmd Index
Sushil Singh19888552020-06-08 02:14:52 -0500146
147 FOR ${index} IN @{entries}
148 Iterate BMC Command List Pairs ${index}
149 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500150
151 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500152
153
154Iterate BMC Command List Pairs
155 [Documentation] Feed in key pair list from dictionary to execute
156 [Arguments] ${key_index}
157
158 @{cmd_list}= Get ffdc bmc cmd ${key_index}
159 Set Suite Variable ${ENTRY_INDEX} ${key_index}
George Keishing69e6f712016-09-12 06:30:09 -0500160
Sushil Singh19888552020-06-08 02:14:52 -0500161 FOR ${cmd} IN @{cmd_list}
162 Execute Command and Write FFDC ${cmd[0]} ${cmd[1]}
163 END
George Keishing69e6f712016-09-12 06:30:09 -0500164
165Execute Command and Write FFDC
Michael Walsh63b70c02017-10-30 15:02:09 -0500166 [Documentation] Run a command on the BMC or OS, write the output to the
167 ... specified file and return a list of generated files.
168 [Arguments] ${key_index} ${cmd} ${logpath}=${FFDC_FILE_PATH}
169 ... ${target}=BMC
George Keishing69e6f712016-09-12 06:30:09 -0500170
Michael Walsh63b70c02017-10-30 15:02:09 -0500171 Run Keyword If '${logpath}' == '${FFDC_FILE_PATH}'
172 ... Write Cmd Output to FFDC File ${key_index} ${cmd}
173
174 @{ffdc_file_list}= Create List ${log_path}
George Keishing69e6f712016-09-12 06:30:09 -0500175
Michael Walshe53e47a2017-06-30 17:03:24 -0500176 ${cmd_buf}= Catenate ${target} Execute Command \ ${cmd} \ ignore_err=${1}
Michael Walsh265ae412018-11-16 15:32:03 -0600177 ... \ time_out=${FFDC_CMD_TIMEOUT}
Michael Walshe53e47a2017-06-30 17:03:24 -0500178 ${status} ${ret_values}= Run Key ${cmd_buf} ignore=${1}
Michael Walshde5e9372018-11-15 11:28:23 -0600179 # If the command times out, status will be 'FAIL'.
180 Return From Keyword If '${status}' == 'FAIL' ${ffdc_file_list}
Michael Walshe53e47a2017-06-30 17:03:24 -0500181
Sushil Singh19888552020-06-08 02:14:52 -0500182 ${stdout}= Set Variable ${ret_values}[0]
183 ${stderr}= Set Variable ${ret_values}[1]
George Keishing69e6f712016-09-12 06:30:09 -0500184
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600185 # Write stdout on success and stderr/stdout to the file on failure.
186 Run Keyword If $stderr == '${EMPTY}'
Sweta Potthuri15e6d2f2017-02-28 03:10:02 -0600187 ... Write Data To File ${stdout}${\n} ${logpath}
188 ... ELSE Write Data To File
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600189 ... ERROR output:${\n}${stderr}${\n}Output:${\n}${stdout}${\n}
190 ... ${logpath}
George Keishing69e6f712016-09-12 06:30:09 -0500191
Michael Walsh63b70c02017-10-30 15:02:09 -0500192 [Return] ${ffdc_file_list}
193
George Keishing69e6f712016-09-12 06:30:09 -0500194
George Keishing69e6f712016-09-12 06:30:09 -0500195# Method : BMC FFDC Files #
196# Execute command on BMC and write to individual file #
197# based on the file name pre-defined in the list #
George Keishing69e6f712016-09-12 06:30:09 -0500198
199BMC FFDC Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500200 [Documentation] Run the commands from FFDC_BMC_FILE and return a list of
201 ... generated files.
202
203 @{entries}= Get FFDC File Index
204 # Example of entries:
205 # entries:
206 # entries[0]: BMC FILES
207
Steven Sombara31f4c82019-08-01 16:00:23 -0500208 scp.Open Connection
vinaybs6b8110942020-01-29 23:15:29 -0600209 ... ${OPENBMC_HOST} username=${OPENBMC_USERNAME} password=${OPENBMC_PASSWORD} port=${SSH_PORT}
Steven Sombara31f4c82019-08-01 16:00:23 -0500210
Michael Walsh63b70c02017-10-30 15:02:09 -0500211 @{ffdc_file_list}= Create List
Sushil Singh19888552020-06-08 02:14:52 -0500212
213 FOR ${index} IN @{entries}
214 ${ffdc_file_sub_list}= Create File and Write Data ${index}
215 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
216 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500217
Michael Walsh17eaafe2019-08-22 11:05:20 -0500218 BMC Execute Command rm -rf /tmp/BMC_*
Steven Sombara31f4c82019-08-01 16:00:23 -0500219 scp.Close Connection
220
Michael Walsh63b70c02017-10-30 15:02:09 -0500221 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500222
223
224Create File and Write Data
Michael Walsh63b70c02017-10-30 15:02:09 -0500225 [Documentation] Run commands from FFDC_BMC_FILE to create FFDC files and
226 ... return a list of generated files.
227 [Arguments] ${key_index}
George Keishing69e6f712016-09-12 06:30:09 -0500228
Michael Walsh63b70c02017-10-30 15:02:09 -0500229 # Description of argument(s):
230 # key_index The index into the FFDC_BMC_FILE dictionary.
231
232 @{ffdc_file_list}= Create List
233 @{cmd_list}= Get FFDC BMC File ${key_index}
Sushil Singh19888552020-06-08 02:14:52 -0500234
235 FOR ${cmd} IN @{cmd_list}
236 ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}
237 ${ffdc_file_sub_list}= Execute Command and Write FFDC ${cmd[0]} ${cmd[1]} ${logpath}
George Keishing89974c92022-02-14 10:43:03 -0600238 Run Key U scp.Get File \ /tmp/${cmd[0]} \ ${LOG_PREFIX}${cmd[0]} ignore=1
Sushil Singh19888552020-06-08 02:14:52 -0500239 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
240 END
Sivas SRR0d657e82017-03-13 01:00:34 -0500241
Michael Walsh63b70c02017-10-30 15:02:09 -0500242 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500243
244
George Keishing69e6f712016-09-12 06:30:09 -0500245# Method : Log Test Case Status #
246# Creates test result history footprint for reference #
George Keishing69e6f712016-09-12 06:30:09 -0500247
248Log Test Case Status
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600249 [Documentation] Test case execution result history.
250 ... Create once and append to this file
251 ... logs/test_history.txt
252 ... Format Date:Test suite:Test case:Status
253 ... 20160909214053719992:Test Warmreset:Test WarmReset via REST:FAIL
Michael Walshfa9f70f2017-04-21 16:00:18 -0500254
255 ${FFDC_DIR_PATH_STYLE}= Get Variable Value ${FFDC_DIR_PATH_STYLE}
256 ... ${EMPTY}
257 ${FFDC_DIR_PATH}= Get Variable Value ${FFDC_DIR_PATH} ${EMPTY}
258
259 Run Keyword If '${FFDC_DIR_PATH}' == '${EMPTY}' Set FFDC Defaults
260
261 Run Keyword If '${FFDC_DIR_PATH_STYLE}' == '${1}' Run Keywords
262 ... Set Global Variable ${FFDC_LOG_PATH} ${FFDC_DIR_PATH} AND
263 ... Set Global Variable ${TEST_HISTORY} ${FFDC_DIR_PATH}test_history.txt
264
George Keishing69e6f712016-09-12 06:30:09 -0500265 Create Directory ${FFDC_LOG_PATH}
266
267 ${exist}= Run Keyword and Return Status
268 ... OperatingSystem.File Should Exist ${TEST_HISTORY}
269
270 Run Keyword If '${exist}' == '${False}'
271 ... Create File ${TEST_HISTORY}
272
Michael Walshfa9f70f2017-04-21 16:00:18 -0500273 Rpvars TEST_HISTORY
274
George Keishing69e6f712016-09-12 06:30:09 -0500275 ${cur_time}= Get Current Time Stamp
276
277 Append To File ${TEST_HISTORY}
278 ... ${cur_time}:${SUITE_NAME}:${TEST_NAME}:${TEST_STATUS}${\n}
279
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500280
281Log FFDC Get Requests
Michael Walsh63b70c02017-10-30 15:02:09 -0500282 [Documentation] Run the get requests associated with the key and return a
283 ... list of generated files.
284 [Arguments] ${key_index}
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500285
Michael Walsh63b70c02017-10-30 15:02:09 -0500286 # Note: Output will be in JSON pretty_print format.
287
288 # Description of argument(s):
289 # key_index The key to the FFDC_GET_REQUEST dictionary that contains the
290 # get requests that are to be run.
291
292 @{ffdc_file_list}= Create List
293 @{cmd_list}= Get FFDC Get Request ${key_index}
Sushil Singh19888552020-06-08 02:14:52 -0500294
295 FOR ${cmd} IN @{cmd_list}
296 ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}
George Keishingccd20e12020-07-29 11:14:26 -0500297 ${resp}= OpenBMC Get Request ${cmd[1]} quiet=${1} timeout=${30}
Sushil Singh19888552020-06-08 02:14:52 -0500298 ${status}= Run Keyword and Return Status Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
299 Run Keyword If '${status}' == '${False}' Continue For Loop
George Keishingfbd67002022-08-01 11:24:03 -0500300 Write Data To File ${\n}${resp.json()}${\n} ${logpath}
Sushil Singh19888552020-06-08 02:14:52 -0500301 Append To List ${ffdc_file_list} ${logpath}
302 END
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500303
Michael Walsh63b70c02017-10-30 15:02:09 -0500304 [Return] ${ffdc_file_list}
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500305
George Keishing8c42bbc2021-09-28 14:52:54 -0500306
307Log FFDC Get Redfish Requests
308 [Documentation] Run the get requests associated with the key and return a
309 ... list of generated files.
310 [Arguments] ${key_index}
311
312 # Note: Output will be in JSON pretty_print format.
313
314 # Description of argument(s):
315 # key_index The key to the FFDC_GET_REDFISH_REQUEST dictionary that contains the
316 # get requests that are to be run.
317
318 @{ffdc_file_list}= Create List
319 @{cmd_list}= Get FFDC Get Redfish Request ${key_index}
320
321 FOR ${cmd} IN @{cmd_list}
322 ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}
323 ${resp}= Redfish.Get ${cmd[1]}
324 Write Data To File ${\n}${resp}${\n} ${logpath}
325 Append To List ${ffdc_file_list} ${logpath}
326 END
327
328 [Return] ${ffdc_file_list}
329
330
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500331BMC FFDC Get Requests
Michael Walsh63b70c02017-10-30 15:02:09 -0500332 [Documentation] Iterate over get request list and return a list of
333 ... generated files.
334
335 @{ffdc_file_list}= Create List
336
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500337 @{entries}= Get ffdc get request index
Michael Walsh63b70c02017-10-30 15:02:09 -0500338 # Example of entries:
339 # entries:
340 # entries[0]: GET REQUESTS
Sushil Singh19888552020-06-08 02:14:52 -0500341
342 FOR ${index} IN @{entries}
343 ${ffdc_file_sub_list}= Log FFDC Get Requests ${index}
344 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
345 END
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500346
Michael Walsh63b70c02017-10-30 15:02:09 -0500347 [Return] ${ffdc_file_list}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500348
George Keishing8c42bbc2021-09-28 14:52:54 -0500349
350BMC FFDC Get Redfish Requests
351 [Documentation] Iterate over get request list and return a list of
352 ... generated files.
353
354 @{ffdc_file_list}= Create List
355
356 @{entries}= Get ffdc get redfish request index
357 # Example of entries:
358 # entries:
359 # entries[0]: GET REQUESTS
360
361 FOR ${index} IN @{entries}
362 ${ffdc_file_sub_list}= Log FFDC Get Redfish Requests ${index}
363 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
364 END
365
366 [Return] ${ffdc_file_list}
367
368
Michael Walsh63b70c02017-10-30 15:02:09 -0500369Log OS All distros FFDC
370 [Documentation] Run commands from FFDC_OS_ALL_DISTROS_FILE to create FFDC
371 ... files and return a list of generated files.
372 [Arguments] ${key_index}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500373
Michael Walsh63b70c02017-10-30 15:02:09 -0500374 # Description of argument(s):
375 # key_index The index into the FFDC_OS_ALL_DISTROS_FILE dictionary.
376
377 @{ffdc_file_list}= Create List
378
379 @{cmd_list}= Get FFDC OS All Distros Call ${key_index}
Sushil Singh19888552020-06-08 02:14:52 -0500380
381 FOR ${cmd} IN @{cmd_list}
382 ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}
383 ${ffdc_file_sub_list}= Execute Command and Write FFDC ${cmd[0]} ${cmd[1]} ${logpath} target=OS
384 # scp it to the LOG_PREFIX ffdc directory.
George Keishing89974c92022-02-14 10:43:03 -0600385 Run Key U scp.Get File \ /tmp/${cmd[0]} \ ${LOG_PREFIX}${cmd[0]} ignore=1
Sushil Singh19888552020-06-08 02:14:52 -0500386 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
387 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500388
389 [Return] ${ffdc_file_list}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500390
391
Gunnar Millscce185d2016-10-17 17:04:15 -0500392Log OS SPECIFIC DISTRO FFDC
Michael Walsh63b70c02017-10-30 15:02:09 -0500393 [Documentation] Run commands from the FFDC_OS_<distro>_FILE to create FFDC
394 ... files and return a list of generated files.
395 [Arguments] ${key_index} ${linux_distro}
Gunnar Millscce185d2016-10-17 17:04:15 -0500396
Michael Walsh63b70c02017-10-30 15:02:09 -0500397 # Description of argument(s):
398 # key_index The index into the FFDC_OS_<distro>_FILE dictionary.
399 # linux_distro Your OS's linux distro (e.g. "UBUNTU", "RHEL", etc).
400
George Keishing671dd662021-05-28 03:26:30 -0500401 Log To Console Collecting log for ${linux_distro}
402
Michael Walsh63b70c02017-10-30 15:02:09 -0500403 @{ffdc_file_list}= Create List
404
405 @{cmd_list}= Get FFDC OS Distro Call ${key_index} ${linux_distro}
Sushil Singh19888552020-06-08 02:14:52 -0500406
407 FOR ${cmd} IN @{cmd_list}
408 ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}
409 ${ffdc_file_sub_list}= Execute Command and Write FFDC ${cmd[0]} ${cmd[1]} ${logpath} target=OS
George Keishing89974c92022-02-14 10:43:03 -0600410 Run Key U scp.Get File \ /tmp/${cmd[0]} \ ${LOG_PREFIX}${cmd[0]} ignore=1
Sushil Singh19888552020-06-08 02:14:52 -0500411 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
George Keishing4609d2f2021-05-28 04:40:58 -0500412 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500413
Steven Sombare70acbe2019-08-20 14:37:47 -0500414 # Get the name of the sosreport file.
415 ${sosreport_file_path} ${stderr} ${rc}= OS Execute Command
416 ... ls /tmp/sosreport*FFDC*tar.xz ignore_err=${True}
417 # Example: sosreport_file_path="/tmp/sosreport-myhost-FFDC-2019-08-20-pbuaqtk.tar.xz".
418
419 # Return if there is no sosreport file.
420 Return From Keyword If ${rc} != ${0} ${ffdc_file_list}
421
422 ${sosreport_dir_path} ${sosreport_file_name}= Split Path ${sosreport_file_path}
423 # Example: sosreport_dir_path="/tmp",
424 # sosreport_file_name="sosreport-myhost-FFDC-2019-08-20-pbuaqtk.tar.xz".
425
426 # Location where the sosreport file will be copied to.
427 ${local_sosreport_file_path}= Set Variable ${LOG_PREFIX}OS_${sosreport_file_name}
428
429 # Change file permissions otherwise scp will not see the file.
430 OS Execute Command chmod 644 ${sosreport_file_path}
431
432 # SCP the sosreport file from the OS.
George Keishing89974c92022-02-14 10:43:03 -0600433 Run Key U scp.Get File \ ${sosreport_file_path} \ ${local_sosreport_file_path} ignore=1
Steven Sombare70acbe2019-08-20 14:37:47 -0500434
435 # Add the file location to the ffdc_file_list.
436 Append To List ${ffdc_file_list} ${local_sosreport_file_path}
437
Michael Walsh63b70c02017-10-30 15:02:09 -0500438 [Return] ${ffdc_file_list}
Gunnar Millscce185d2016-10-17 17:04:15 -0500439
440
George Keishingca8c4c62016-10-14 10:08:40 -0500441OS FFDC Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500442 [Documentation] Run the commands from FFDC_OS_ALL_DISTROS_FILE to create
443 ... FFDC files and return a list of generated files.
Gunnar Millsaca140d2016-10-26 13:05:10 -0500444 [Arguments] ${OS_HOST}=${OS_HOST} ${OS_USERNAME}=${OS_USERNAME}
Michael Walsh63b70c02017-10-30 15:02:09 -0500445 ... ${OS_PASSWORD}=${OS_PASSWORD}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500446
Michael Walsh63b70c02017-10-30 15:02:09 -0500447 @{ffdc_file_list}= Create List
Gunnar Millsaca140d2016-10-26 13:05:10 -0500448
Michael Walsh63b70c02017-10-30 15:02:09 -0500449 Run Keyword If '${OS_HOST}' == '${EMPTY}' Run Keywords
450 ... Print Timen No OS Host provided so no OS FFDC will be done. AND
451 ... Return From Keyword ${ffdc_file_list}
Gunnar Millsaca140d2016-10-26 13:05:10 -0500452
Michael Walsh63b70c02017-10-30 15:02:09 -0500453 ${match_state}= Create Dictionary os_ping=^1$ os_login=^1$
454 ... os_run_cmd=^1$
455 ${status} ${ret_values}= Run Keyword and Ignore Error Check State
456 ... ${match_state} quiet=0
457
458 Run Keyword If '${status}' == 'FAIL' Run Keywords
Michael Walshc3b28d72017-11-16 10:43:14 -0600459 ... Print Timen The OS is not communicating so no OS FFDC will be done.\n
Michael Walsh63b70c02017-10-30 15:02:09 -0500460 ... AND
461 ... Return From Keyword ${ffdc_file_list}
Gunnar Millscce185d2016-10-17 17:04:15 -0500462
George Keishing671dd662021-05-28 03:26:30 -0500463 ${stdout}= OS Distro Type
464
Michael Walshe53e47a2017-06-30 17:03:24 -0500465 Set Global Variable ${linux_distro} ${stdout}
Gunnar Millsaca140d2016-10-26 13:05:10 -0500466
Michael Walshe53e47a2017-06-30 17:03:24 -0500467 Rpvars linux_distro
Gunnar Millsaca140d2016-10-26 13:05:10 -0500468
Steven Sombarfc024cb2019-04-22 11:16:16 -0500469 scp.Open Connection
470 ... ${OS_HOST} username=${OS_USERNAME} password=${OS_PASSWORD}
471
Michael Walsh63b70c02017-10-30 15:02:09 -0500472 @{entries}= Get FFDC OS All Distros Index
Sushil Singh19888552020-06-08 02:14:52 -0500473
474 FOR ${index} IN @{entries}
475 ${ffdc_file_sub_list}= Log OS All distros FFDC ${index}
476 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
477 END
Gunnar Millscce185d2016-10-17 17:04:15 -0500478
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600479 Return From Keyword If
480 ... '${linux_distro}' == '${EMPTY}' or '${linux_distro}' == 'None'
Michael Walsh63b70c02017-10-30 15:02:09 -0500481 ... ${ffdc_file_list}
Gunnar Millscce185d2016-10-17 17:04:15 -0500482
483 @{entries}= Get ffdc os distro index ${linux_distro}
Sushil Singh19888552020-06-08 02:14:52 -0500484
485 FOR ${index} IN @{entries}
486 ${ffdc_file_sub_list}= Log OS SPECIFIC DISTRO FFDC ${index} ${linux_distro}
487 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
488 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500489
Steven Sombarfc024cb2019-04-22 11:16:16 -0500490 # Delete ffdc files still on OS and close scp.
Steven Sombare70acbe2019-08-20 14:37:47 -0500491 OS Execute Command rm -rf /tmp/OS_* /tmp/sosreport*FFDC* ignore_err=${True}
Steven Sombarfc024cb2019-04-22 11:16:16 -0500492 scp.Close Connection
493
Michael Walsh63b70c02017-10-30 15:02:09 -0500494 [Return] ${ffdc_file_list}
George Keishing7a520222017-02-27 09:44:30 -0600495
496
Steven Sombar96de5732017-08-17 09:04:52 -0500497System Inventory Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500498 [Documentation] Copy systest os_inventory files and return a list of
499 ... generated files..
Steven Sombar96de5732017-08-17 09:04:52 -0500500 # The os_inventory files are the result of running
501 # systest/htx_hardbootme_test. If these files exist
502 # they are copied to the FFDC directory.
503 # Global variable ffdc_dir_path is the path name of the
504 # directory they are copied to.
Michael Walsh63b70c02017-10-30 15:02:09 -0500505
506 @{ffdc_file_list}= Create List
507
508 ${globex}= Set Variable os_inventory_*.json
509
510 @{file_list}= OperatingSystem.List Files In Directory . ${globex}
511
512 Copy Files ${globex} ${ffdc_dir_path}
513
Sushil Singh19888552020-06-08 02:14:52 -0500514 FOR ${file_name} IN @{file_list}
515 Append To List ${ffdc_file_list} ${ffdc_dir_path}${file_name}
516 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500517
518 Run Keyword and Ignore Error Remove Files ${globex}
519
520 [Return] ${ffdc_file_list}
Steven Sombar96de5732017-08-17 09:04:52 -0500521
522
George Keishing7a520222017-02-27 09:44:30 -0600523SCP Coredump Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500524 [Documentation] Copy core dump files from BMC to local system and return a
525 ... list of generated file names.
526
527 @{ffdc_file_list}= Create List
Michael Walshe53e47a2017-06-30 17:03:24 -0500528
George Keishing7a520222017-02-27 09:44:30 -0600529 # Check if core dump exist in the /tmp
Michael Walshe53e47a2017-06-30 17:03:24 -0500530 ${core_files} ${stderr} ${rc}= BMC Execute Command ls /tmp/core_*
Michael Walsh63b70c02017-10-30 15:02:09 -0500531 ... ignore_err=${1}
532 Run Keyword If '${rc}' != '${0}' Return From Keyword ${ffdc_file_list}
533
534 @{core_list}= Split String ${core_files}
George Keishing7a520222017-02-27 09:44:30 -0600535 # Copy the core files
Michael Walsh6bad8412017-06-27 14:09:36 -0500536 Run Key U Open Connection for SCP
Sushil Singh19888552020-06-08 02:14:52 -0500537
538 FOR ${index} IN @{core_list}
539 ${ffdc_file_path}= Catenate ${LOG_PREFIX}${index.lstrip("/tmp/")}
540 ${status}= Run Keyword and Return Status scp.Get File ${index} ${ffdc_file_path}
541 Run Keyword If '${status}' == '${False}' Continue For Loop
542 Append To List ${ffdc_file_list} ${ffdc_file_path}
543
544 # Remove the file from remote to avoid re-copying on next FFDC call
545
546 BMC Execute Command rm ${index} ignore_err=${1}
547 # I can't find a way to do this: scp.Close Connection
548
549 END
George Keishing7a520222017-02-27 09:44:30 -0600550
Michael Walsh63b70c02017-10-30 15:02:09 -0500551 [Return] ${ffdc_file_list}
552
Steven Sombar2da175f2019-02-04 10:32:58 -0600553
Sweta Potthuria82efd62017-10-18 05:34:59 -0500554SCP Dump Files
555 [Documentation] Copy all dump files from BMC to local system.
556
557 # Check if dumps exist
558 ${ffdc_file_list}= Scp Dumps ${FFDC_DIR_PATH} ${FFDC_PREFIX}
559
560 [Return] ${ffdc_file_list}
561
Steven Sombar2da175f2019-02-04 10:32:58 -0600562
Sweta Potthuria82efd62017-10-18 05:34:59 -0500563Collect Dump Log
564 [Documentation] Collect dumps from dump entry.
565 [Arguments] ${log_prefix_path}=${LOG_PREFIX}
566
George Keishing752b02b2022-03-30 01:37:22 -0500567 Return From Keyword If ${REDFISH_SUPPORT_TRANS_STATE} == ${1}
568
Rahul Maheshwaria89ff9e2020-09-25 05:04:33 -0500569 ${resp}= OpenBMC Get Request ${DUMP_URI}
570 Run Keyword If '${resp.status_code}' == '${HTTP_NOT_FOUND}'
571 ... Set Test Variable ${DUMP_ENTRY_URI} /xyz/openbmc_project/dump/entry/
572
George Keishingccd20e12020-07-29 11:14:26 -0500573 ${data}= Read Properties ${DUMP_ENTRY_URI}enumerate quiet=${1} timeout=${30}
Sweta Potthuria82efd62017-10-18 05:34:59 -0500574
575 # Grab the list of entries from dump/entry/
576 # The data shown below is the result of the "Get Dictionary Keys".
577 # Example:
578 # /xyz/openbmc_project/dump/entry/1
579 # /xyz/openbmc_project/dump/entry/2
580
581 ${dump_list}= Get Dictionary Keys ${data}
George Keishingeb7b4bc2017-03-13 12:04:28 -0500582
Michael Walsh02c70602018-10-04 15:24:30 -0500583
George Keishingbe396b82020-03-23 10:23:58 -0500584Collect PEL Log
585 [Documentation] Collect PEL files from from BMC.
586
587 Create Directory ${FFDC_DIR_PATH}${/}pel_files/
588 scp.Get File /var/lib/phosphor-logging/extensions/pels/logs/
589 ... ${FFDC_DIR_PATH}${/}pel_files recursive=True
590
591
Steven Sombar2da175f2019-02-04 10:32:58 -0600592Enumerate Redfish Resources
593 [Documentation] Enumerate /redfish/v1 resources and properties to
594 ... a file. Return a list which contains the file name.
George Keishing5942d202021-10-04 13:09:24 -0500595 [Arguments] ${log_prefix_path}=${LOG_PREFIX} ${enum_uri}=/redfish/v1
596 ... ${file_enum_name}=redfish_resource_properties.txt
Steven Sombar2da175f2019-02-04 10:32:58 -0600597
598 # Description of argument(s):
599 # log_prefix_path The location specifying where to create FFDC file(s).
600
601 # Login is needed to fetch Redfish information.
602 # If login fails, return from keyword.
George Keishing97c93942019-03-04 12:45:07 -0600603 ${status}= Run Keyword And Return Status Redfish.Login
Steven Sombar2da175f2019-02-04 10:32:58 -0600604 Return From Keyword If ${status} == ${False}
605
606 # Get the Redfish resources and properties.
George Keishing5942d202021-10-04 13:09:24 -0500607 ${json_data}= redfish_utils.Enumerate Request ${enum_uri}
Steven Sombar2da175f2019-02-04 10:32:58 -0600608 # Typical output:
609 # {
610 # "@odata.id": "/redfish/v1",
611 # "@odata.type": "#ServiceRoot.v1_1_1.ServiceRoot",
612 # "AccountService": {
613 # "@odata.id": "/redfish/v1/AccountService"
614 # },
615 # "Chassis": {
616 # "@odata.id": "/redfish/v1/Chassis"
617 # },
618 # "Id": "RootService",
619 # "JsonSchemas": {
620 # "@odata.id": "/redfish/v1/JsonSchemas"
621 # },
622 # ..etc...
623 # }
624
625 @{ffdc_file_list}= Create List
George Keishing5942d202021-10-04 13:09:24 -0500626 ${logpath}= Catenate SEPARATOR= ${log_prefix_path} ${file_enum_name}
Steven Sombar2da175f2019-02-04 10:32:58 -0600627 Create File ${logpath}
628 Write Data To File "${\n}${json_data}${\n}" ${logpath}
629
630 Append To List ${ffdc_file_list} ${logpath}
631
632 [Return] ${ffdc_file_list}
633
634
George Keishing985659a2021-06-23 02:52:13 -0500635Enumerate Redfish OEM Resources
636 [Documentation] Enumerate /<oem>/v1 resources and properties to
637 ... a file. Return a list which contains the file name.
638 [Arguments] ${log_prefix_path}=${LOG_PREFIX}
639
640 # Description of argument(s):
641 # log_prefix_path The location specifying where to create FFDC file(s).
642
643 # No-op by default if input is not supplied from command line.
644 Return From Keyword If "${OEM_REDFISH_PATH}" == "${EMPTY}"
645
646 # Login is needed to fetch Redfish information.
647 # If login fails, return from keyword.
648 ${status}= Run Keyword And Return Status Redfish.Login
649 Return From Keyword If ${status} == ${False}
650
651 # Get the Redfish resources and properties.
652 ${json_data}= redfish_utils.Enumerate Request ${OEM_REDFISH_PATH}
653
654 @{ffdc_file_list}= Create List
655 ${logpath}= Catenate SEPARATOR= ${log_prefix_path}
656 ... redfish_oem_resource_properties.txt
657 Create File ${logpath}
658 Write Data To File "${\n}${json_data}${\n}" ${logpath}
659
660 Append To List ${ffdc_file_list} ${logpath}
661
662 [Return] ${ffdc_file_list}
663
664
George Keishingeb7b4bc2017-03-13 12:04:28 -0500665Collect eSEL Log
Michael Walsh02c70602018-10-04 15:24:30 -0500666 [Documentation] Create raw and formatted eSEL files.
Sridevi Rameshb180c9f2017-08-06 10:27:41 -0500667 [Arguments] ${log_prefix_path}=${LOG_PREFIX}
668
Michael Walsh02c70602018-10-04 15:24:30 -0500669 # NOTE: If no eSEL.pl program can be located, then no formatted eSEL file
670 # will be generated.
671
672 # Description of argument(s):
673 # log_prefix_path The path prefix to be used in creating
674 # eSEL file path names. For example, if
675 # log_prefix_path is
676 # "/tmp/user1/dummy.181001.120000.", then
677 # this keyword will create
678 # /tmp/user1/dummy.181001.120000.esel (raw)
679 # and
680 # /tmp/user1/dummy.181001.120000.esel.txt
681 # (formatted).
682
Michael Walsh63b70c02017-10-30 15:02:09 -0500683 @{ffdc_file_list}= Create List
684
Michael Walsh02c70602018-10-04 15:24:30 -0500685 ${esels}= Get Esels
686 ${num_esels}= Evaluate len(${esels})
687 Rprint Vars num_esels
688 Return From Keyword If ${num_esels} == ${0} ${ffdc_file_list}
George Keishingeb7b4bc2017-03-13 12:04:28 -0500689
Sridevi Rameshb180c9f2017-08-06 10:27:41 -0500690 ${logpath}= Catenate SEPARATOR= ${log_prefix_path} esel
George Keishingeb7b4bc2017-03-13 12:04:28 -0500691 Create File ${logpath}
George Keishingeb7b4bc2017-03-13 12:04:28 -0500692
Sushil Singh19888552020-06-08 02:14:52 -0500693 FOR ${esel} IN @{esels}
694 Write Data To File "${esel}"${\n} ${logpath}
695 END
George Keishingb1ca69b2017-03-31 14:36:42 -0500696
Michael Walsh63b70c02017-10-30 15:02:09 -0500697 Append To List ${ffdc_file_list} ${logpath}
Michael Walsh02c70602018-10-04 15:24:30 -0500698
699 ${rc} ${output}= Shell Cmd which eSEL.pl show_err=0
700 Return From Keyword If ${rc} != ${0} ${ffdc_file_list}
701
702 Convert eSEL To Elog Format ${logpath}
Michael Walsh0192e9b2017-11-14 15:14:15 -0600703 Append To List ${ffdc_file_list} ${logpath}.txt
Michael Walsh63b70c02017-10-30 15:02:09 -0500704
705 [Return] ${ffdc_file_list}
706
George Keishingb1ca69b2017-03-31 14:36:42 -0500707
George Keishingb1ca69b2017-03-31 14:36:42 -0500708Convert eSEL To Elog Format
709 [Documentation] Execute parser tool on the eSEL data file to generate
Michael Walsha30dac72017-04-04 17:56:23 -0500710 ... formatted error log.
George Keishingb1ca69b2017-03-31 14:36:42 -0500711 [Arguments] ${esel_file_path}
Michael Walsh02c70602018-10-04 15:24:30 -0500712 # Description of argument(s):
George Keishing7c6342b2018-11-02 14:27:13 -0500713 # esel_file_path The path to the file containing raw eSEL
Michael Walsh02c70602018-10-04 15:24:30 -0500714 # data (e.g.
715 # "/tmp/user1/dummy.181001.120000.esel").
George Keishingb1ca69b2017-03-31 14:36:42 -0500716
Michael Walsha30dac72017-04-04 17:56:23 -0500717 # Note: The only way to get eSEL.pl to put the output in a particular
718 # directory is to cd to that directory.
719 ${cmd_buf}= Catenate cd $(dirname ${esel_file_path}) ; eSEL.pl -l
720 ... ${esel_file_path} -p decode_obmc_data
Michael Walsh02c70602018-10-04 15:24:30 -0500721 Qprint Issuing ${cmd_buf}
Michael Walsha30dac72017-04-04 17:56:23 -0500722 Run ${cmd_buf}
Michael Walsh02c70602018-10-04 15:24:30 -0500723 # The .binary file, which is generated by eSEL.pl, is of no use to us.
724 Remove File ${esel_file_path}.binary
George Keishing671dd662021-05-28 03:26:30 -0500725
726
727OS Distro Type
728 [Documentation] Determine the host partition distro type
729
730 ${stdout} ${stderr} ${rc}= OS Execute Command
731 ... . /etc/os-release; echo $ID ignore_err=${1}
732
733 Return From Keyword If ${rc} == ${0} ${stdout}
734
George Keishing33d15032021-06-07 00:49:56 -0500735 # If linux distro doesn't have os-release, check for uname.
George Keishing671dd662021-05-28 03:26:30 -0500736 ${stdout} ${stderr} ${rc}= OS Execute Command uname ignore_err=${0}
737
738 [Return] ${stdout}