blob: fe68c9f0f163316e6132d3d5c0d0b77605f1a9c3 [file] [log] [blame]
George Keishing69e6f712016-09-12 06:30:09 -05001*** Settings ***
2Documentation Methods to execute commands on BMC and collect
3... data to a list of FFDC files
4
Michael Walsh265ae412018-11-16 15:32:03 -06005Resource openbmc_ffdc_utils.robot
6Resource rest_client.robot
7Resource utils.robot
8Resource list_utils.robot
9Resource logging_utils.robot
Steven Sombar2da175f2019-02-04 10:32:58 -060010Resource bmc_redfish_resource.robot
Michael Walsh265ae412018-11-16 15:32:03 -060011Library SSHLibrary
12Library OperatingSystem
13Library Collections
14Library String
15Library gen_print.py
16Library gen_cmd.py
17Library gen_robot_keyword.py
18Library dump_utils.py
19Library logging_utils.py
Gunnar Mills7e2cda22016-10-11 15:37:34 -050020
Michael Walshde5e9372018-11-15 11:28:23 -060021*** Variables ***
22
Michael Walsh265ae412018-11-16 15:32:03 -060023${FFDC_CMD_TIMEOUT} 240
George Keishingbe3d1df2022-03-29 08:49:10 -050024${FFDC_BMC_FILES_CLEANUP} rm -rf /tmp/BMC_* /tmp/PEL_* /tmp/PLDM_*
25... /tmp/OCC_* /tmp/fan_* /tmp/GUARD_* /tmp/DEVTREE
Michael Walshde5e9372018-11-15 11:28:23 -060026
George Keishing69e6f712016-09-12 06:30:09 -050027*** Keywords ***
28
George Keishing69e6f712016-09-12 06:30:09 -050029# Method : Call FFDC Methods #
30# Execute the user define keywords from the FFDC List #
31# Unlike any other keywords this will call into the #
32# list of keywords defined in the FFDC list at one go #
George Keishing69e6f712016-09-12 06:30:09 -050033
34Call FFDC Methods
Michael Walshe844e9a2017-04-20 16:51:10 -050035 [Documentation] Call into FFDC Keyword index list.
36 [Arguments] ${ffdc_function_list}=${EMPTY}
George Keishing69e6f712016-09-12 06:30:09 -050037
Michael Walshe844e9a2017-04-20 16:51:10 -050038 # Description of argument(s):
39 # ffdc_function_list A colon-delimited list naming the kinds of FFDC that
Michael Walsh63b70c02017-10-30 15:02:09 -050040 # are to be collected
Michael Walshe844e9a2017-04-20 16:51:10 -050041 # (e.g. "FFDC Generic Report:BMC Specific Files").
42 # Acceptable values can be found in the description
43 # field of FFDC_METHOD_CALL in
44 # lib/openbmc_ffdc_list.py. Those values can be
45 # obtained via a call to 'Get FFDC Method Desc' (also
46 # from lib/openbmc_ffdc_list.py).
47
48 @{entries}= Get FFDC Method Index
Michael Walsh63b70c02017-10-30 15:02:09 -050049 # Example entries:
50 # entries:
51 # entries[0]: BMC LOGS
52
53 @{ffdc_file_list}= Create List
Sushil Singh19888552020-06-08 02:14:52 -050054 FOR ${index} IN @{entries}
55 ${ffdc_file_sub_list}= Method Call Keyword List ${index} ${ffdc_function_list}
56 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
57 END
Michael Walsh63b70c02017-10-30 15:02:09 -050058
Michael Walsh6bad8412017-06-27 14:09:36 -050059 Run Key U SSHLibrary.Close All Connections
George Keishing69e6f712016-09-12 06:30:09 -050060
Michael Walsh63b70c02017-10-30 15:02:09 -050061 [Return] ${ffdc_file_list}
62
Steven Sombar2da175f2019-02-04 10:32:58 -060063
George Keishing69e6f712016-09-12 06:30:09 -050064Method Call Keyword List
Michael Walsh63b70c02017-10-30 15:02:09 -050065 [Documentation] Process FFDC request and return a list of generated files.
Michael Walshe844e9a2017-04-20 16:51:10 -050066 [Arguments] ${index} ${ffdc_function_list}=${EMPTY}
George Keishing69e6f712016-09-12 06:30:09 -050067
Michael Walshe844e9a2017-04-20 16:51:10 -050068 # Description of argument(s):
69 # index The index into the FFDC_METHOD_CALL dictionary (e.g.
70 # 'BMC LOGS').
71 # ffdc_function_list See ffdc_function_list description in
72 # "Call FFDC Methods" (above).
73
Michael Walsh63b70c02017-10-30 15:02:09 -050074 @{method_list}= Get FFDC Method Call ${index}
75 # Example method_list:
76 # method_list:
77 # method_list[0]:
78 # method_list[0][0]: FFDC Generic Report
79 # method_list[0][1]: BMC FFDC Manifest
80 # method_list[1]:
81 # method_list[1][0]: Get Request FFDC
82 # method_list[1][1]: BMC FFDC Get Requests
83 # (etc.)
Michael Walshe844e9a2017-04-20 16:51:10 -050084
85 # If function list is empty assign default (i.e. a list of all allowable
86 # values). In either case, convert ffdc_function_list from a string to
87 # a list.
88 @{ffdc_function_list}=
89 ... Run Keyword If '${ffdc_function_list}' == '${EMPTY}'
90 ... Get FFDC Method Desc ${index}
91 ... ELSE
92 ... Split String ${ffdc_function_list} separator=:
93
Michael Walsh63b70c02017-10-30 15:02:09 -050094 @{ffdc_file_list}= Create List
Sushil Singh19888552020-06-08 02:14:52 -050095 FOR ${method} IN @{method_list}
96 ${ffdc_file_sub_list}= Execute Keyword Method ${method[0]} ${method[1]} @{ffdc_function_list}
97 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
98 END
Michael Walsh63b70c02017-10-30 15:02:09 -050099
100 [Return] ${ffdc_file_list}
101
George Keishing69e6f712016-09-12 06:30:09 -0500102
103Execute Keyword Method
Michael Walshe844e9a2017-04-20 16:51:10 -0500104 [Documentation] Call into BMC method keywords. Don't let one
105 ... failure skip the remaining. Get whatever data
106 ... it could gather at worse case scenario.
107 [Arguments] ${description} ${keyword_name} @{ffdc_function_list}
George Keishing69e6f712016-09-12 06:30:09 -0500108
Michael Walshe844e9a2017-04-20 16:51:10 -0500109 # Description of argument(s):
110 # description The description of the FFDC to be collected. This
111 # would be any value returned by
112 # 'Get FFDC Method Desc' (e.g. "FFDC Generic Report").
113 # keyword_name The name of the keyword to call to collect the FFDC
114 # data (again, see FFDC_METHOD_CALL).
115 # ffdc_function_list See ffdc_function_list description in
116 # "Call FFDC Methods" (above). The only difference is
117 # in this case, it should be a list rather than a
118 # colon-delimited value.
George Keishing69e6f712016-09-12 06:30:09 -0500119
Michael Walsh63b70c02017-10-30 15:02:09 -0500120 @{ffdc_file_list}= Create List
Michael Walshe844e9a2017-04-20 16:51:10 -0500121
Michael Walsh63b70c02017-10-30 15:02:09 -0500122 ${index}= Get Index From List ${ffdc_function_list} ${description}
123 Run Keyword If '${index}' == '${-1}' Return from Keyword
124 ... ${ffdc_file_list}
125
126 ${status} ${ffdc_file_list}= Run Key ${keyword_name} ignore=1
127 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500128
George Keishingbe3d1df2022-03-29 08:49:10 -0500129
130BMC FFDC Cleanup
131 [Documentation] Run the ssh commands from FFDC_BMC_FILES_CLEANUP.
132
133 Log To Console BMC FFDC Files clean up: ${FFDC_BMC_FILES_CLEANUP}
134 BMC Execute Command ${FFDC_BMC_FILES_CLEANUP} ignore_err=1
135
136
George Keishing69e6f712016-09-12 06:30:09 -0500137# Method : BMC FFDC Manifest #
138# Execute command on BMC and write to ffdc_report.txt #
George Keishing69e6f712016-09-12 06:30:09 -0500139
140BMC FFDC Manifest
Michael Walsh63b70c02017-10-30 15:02:09 -0500141 [Documentation] Run the ssh commands from FFDC_BMC_CMD and return a list
142 ... of generated files.
George Keishing69e6f712016-09-12 06:30:09 -0500143
Michael Walsh63b70c02017-10-30 15:02:09 -0500144 @{ffdc_file_list}= Create List ${FFDC_FILE_PATH}
145 @{entries}= Get FFDC Cmd Index
Sushil Singh19888552020-06-08 02:14:52 -0500146
147 FOR ${index} IN @{entries}
148 Iterate BMC Command List Pairs ${index}
149 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500150
151 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500152
153
154Iterate BMC Command List Pairs
155 [Documentation] Feed in key pair list from dictionary to execute
156 [Arguments] ${key_index}
157
158 @{cmd_list}= Get ffdc bmc cmd ${key_index}
159 Set Suite Variable ${ENTRY_INDEX} ${key_index}
George Keishing69e6f712016-09-12 06:30:09 -0500160
Sushil Singh19888552020-06-08 02:14:52 -0500161 FOR ${cmd} IN @{cmd_list}
162 Execute Command and Write FFDC ${cmd[0]} ${cmd[1]}
163 END
George Keishing69e6f712016-09-12 06:30:09 -0500164
165Execute Command and Write FFDC
Michael Walsh63b70c02017-10-30 15:02:09 -0500166 [Documentation] Run a command on the BMC or OS, write the output to the
167 ... specified file and return a list of generated files.
168 [Arguments] ${key_index} ${cmd} ${logpath}=${FFDC_FILE_PATH}
169 ... ${target}=BMC
George Keishing69e6f712016-09-12 06:30:09 -0500170
Michael Walsh63b70c02017-10-30 15:02:09 -0500171 Run Keyword If '${logpath}' == '${FFDC_FILE_PATH}'
172 ... Write Cmd Output to FFDC File ${key_index} ${cmd}
173
174 @{ffdc_file_list}= Create List ${log_path}
George Keishing69e6f712016-09-12 06:30:09 -0500175
Michael Walshe53e47a2017-06-30 17:03:24 -0500176 ${cmd_buf}= Catenate ${target} Execute Command \ ${cmd} \ ignore_err=${1}
Michael Walsh265ae412018-11-16 15:32:03 -0600177 ... \ time_out=${FFDC_CMD_TIMEOUT}
Michael Walshe53e47a2017-06-30 17:03:24 -0500178 ${status} ${ret_values}= Run Key ${cmd_buf} ignore=${1}
Michael Walshde5e9372018-11-15 11:28:23 -0600179 # If the command times out, status will be 'FAIL'.
180 Return From Keyword If '${status}' == 'FAIL' ${ffdc_file_list}
Michael Walshe53e47a2017-06-30 17:03:24 -0500181
Sushil Singh19888552020-06-08 02:14:52 -0500182 ${stdout}= Set Variable ${ret_values}[0]
183 ${stderr}= Set Variable ${ret_values}[1]
George Keishing69e6f712016-09-12 06:30:09 -0500184
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600185 # Write stdout on success and stderr/stdout to the file on failure.
186 Run Keyword If $stderr == '${EMPTY}'
Sweta Potthuri15e6d2f2017-02-28 03:10:02 -0600187 ... Write Data To File ${stdout}${\n} ${logpath}
188 ... ELSE Write Data To File
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600189 ... ERROR output:${\n}${stderr}${\n}Output:${\n}${stdout}${\n}
190 ... ${logpath}
George Keishing69e6f712016-09-12 06:30:09 -0500191
Michael Walsh63b70c02017-10-30 15:02:09 -0500192 [Return] ${ffdc_file_list}
193
George Keishing69e6f712016-09-12 06:30:09 -0500194
George Keishing69e6f712016-09-12 06:30:09 -0500195# Method : BMC FFDC Files #
196# Execute command on BMC and write to individual file #
197# based on the file name pre-defined in the list #
George Keishing69e6f712016-09-12 06:30:09 -0500198
199BMC FFDC Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500200 [Documentation] Run the commands from FFDC_BMC_FILE and return a list of
201 ... generated files.
202
203 @{entries}= Get FFDC File Index
204 # Example of entries:
205 # entries:
206 # entries[0]: BMC FILES
207
Steven Sombara31f4c82019-08-01 16:00:23 -0500208 scp.Open Connection
vinaybs6b8110942020-01-29 23:15:29 -0600209 ... ${OPENBMC_HOST} username=${OPENBMC_USERNAME} password=${OPENBMC_PASSWORD} port=${SSH_PORT}
Steven Sombara31f4c82019-08-01 16:00:23 -0500210
Michael Walsh63b70c02017-10-30 15:02:09 -0500211 @{ffdc_file_list}= Create List
Sushil Singh19888552020-06-08 02:14:52 -0500212
213 FOR ${index} IN @{entries}
214 ${ffdc_file_sub_list}= Create File and Write Data ${index}
215 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
216 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500217
Michael Walsh17eaafe2019-08-22 11:05:20 -0500218 BMC Execute Command rm -rf /tmp/BMC_*
Steven Sombara31f4c82019-08-01 16:00:23 -0500219 scp.Close Connection
220
Michael Walsh63b70c02017-10-30 15:02:09 -0500221 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500222
223
224Create File and Write Data
Michael Walsh63b70c02017-10-30 15:02:09 -0500225 [Documentation] Run commands from FFDC_BMC_FILE to create FFDC files and
226 ... return a list of generated files.
227 [Arguments] ${key_index}
George Keishing69e6f712016-09-12 06:30:09 -0500228
Michael Walsh63b70c02017-10-30 15:02:09 -0500229 # Description of argument(s):
230 # key_index The index into the FFDC_BMC_FILE dictionary.
231
232 @{ffdc_file_list}= Create List
233 @{cmd_list}= Get FFDC BMC File ${key_index}
Sushil Singh19888552020-06-08 02:14:52 -0500234
235 FOR ${cmd} IN @{cmd_list}
236 ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}
237 ${ffdc_file_sub_list}= Execute Command and Write FFDC ${cmd[0]} ${cmd[1]} ${logpath}
George Keishing89974c92022-02-14 10:43:03 -0600238 Run Key U scp.Get File \ /tmp/${cmd[0]} \ ${LOG_PREFIX}${cmd[0]} ignore=1
Sushil Singh19888552020-06-08 02:14:52 -0500239 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
240 END
Sivas SRR0d657e82017-03-13 01:00:34 -0500241
Michael Walsh63b70c02017-10-30 15:02:09 -0500242 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500243
244
George Keishing69e6f712016-09-12 06:30:09 -0500245# Method : Log Test Case Status #
246# Creates test result history footprint for reference #
George Keishing69e6f712016-09-12 06:30:09 -0500247
248Log Test Case Status
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600249 [Documentation] Test case execution result history.
250 ... Create once and append to this file
251 ... logs/test_history.txt
252 ... Format Date:Test suite:Test case:Status
253 ... 20160909214053719992:Test Warmreset:Test WarmReset via REST:FAIL
Michael Walshfa9f70f2017-04-21 16:00:18 -0500254
255 ${FFDC_DIR_PATH_STYLE}= Get Variable Value ${FFDC_DIR_PATH_STYLE}
256 ... ${EMPTY}
257 ${FFDC_DIR_PATH}= Get Variable Value ${FFDC_DIR_PATH} ${EMPTY}
258
259 Run Keyword If '${FFDC_DIR_PATH}' == '${EMPTY}' Set FFDC Defaults
260
261 Run Keyword If '${FFDC_DIR_PATH_STYLE}' == '${1}' Run Keywords
262 ... Set Global Variable ${FFDC_LOG_PATH} ${FFDC_DIR_PATH} AND
263 ... Set Global Variable ${TEST_HISTORY} ${FFDC_DIR_PATH}test_history.txt
264
George Keishing69e6f712016-09-12 06:30:09 -0500265 Create Directory ${FFDC_LOG_PATH}
266
267 ${exist}= Run Keyword and Return Status
268 ... OperatingSystem.File Should Exist ${TEST_HISTORY}
269
270 Run Keyword If '${exist}' == '${False}'
271 ... Create File ${TEST_HISTORY}
272
Michael Walshfa9f70f2017-04-21 16:00:18 -0500273 Rpvars TEST_HISTORY
274
George Keishing69e6f712016-09-12 06:30:09 -0500275 ${cur_time}= Get Current Time Stamp
276
277 Append To File ${TEST_HISTORY}
278 ... ${cur_time}:${SUITE_NAME}:${TEST_NAME}:${TEST_STATUS}${\n}
279
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500280
281Log FFDC Get Requests
Michael Walsh63b70c02017-10-30 15:02:09 -0500282 [Documentation] Run the get requests associated with the key and return a
283 ... list of generated files.
284 [Arguments] ${key_index}
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500285
Michael Walsh63b70c02017-10-30 15:02:09 -0500286 # Note: Output will be in JSON pretty_print format.
287
288 # Description of argument(s):
289 # key_index The key to the FFDC_GET_REQUEST dictionary that contains the
290 # get requests that are to be run.
291
292 @{ffdc_file_list}= Create List
293 @{cmd_list}= Get FFDC Get Request ${key_index}
Sushil Singh19888552020-06-08 02:14:52 -0500294
295 FOR ${cmd} IN @{cmd_list}
296 ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}
George Keishingccd20e12020-07-29 11:14:26 -0500297 ${resp}= OpenBMC Get Request ${cmd[1]} quiet=${1} timeout=${30}
Sushil Singh19888552020-06-08 02:14:52 -0500298 ${status}= Run Keyword and Return Status Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
299 Run Keyword If '${status}' == '${False}' Continue For Loop
300 ${jsondata}= to json ${resp.content} pretty_print=True
301 Write Data To File ${\n}${jsondata}${\n} ${logpath}
302 Append To List ${ffdc_file_list} ${logpath}
303 END
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500304
Michael Walsh63b70c02017-10-30 15:02:09 -0500305 [Return] ${ffdc_file_list}
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500306
George Keishing8c42bbc2021-09-28 14:52:54 -0500307
308Log FFDC Get Redfish Requests
309 [Documentation] Run the get requests associated with the key and return a
310 ... list of generated files.
311 [Arguments] ${key_index}
312
313 # Note: Output will be in JSON pretty_print format.
314
315 # Description of argument(s):
316 # key_index The key to the FFDC_GET_REDFISH_REQUEST dictionary that contains the
317 # get requests that are to be run.
318
319 @{ffdc_file_list}= Create List
320 @{cmd_list}= Get FFDC Get Redfish Request ${key_index}
321
322 FOR ${cmd} IN @{cmd_list}
323 ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}
324 ${resp}= Redfish.Get ${cmd[1]}
325 Write Data To File ${\n}${resp}${\n} ${logpath}
326 Append To List ${ffdc_file_list} ${logpath}
327 END
328
329 [Return] ${ffdc_file_list}
330
331
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500332BMC FFDC Get Requests
Michael Walsh63b70c02017-10-30 15:02:09 -0500333 [Documentation] Iterate over get request list and return a list of
334 ... generated files.
335
336 @{ffdc_file_list}= Create List
337
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500338 @{entries}= Get ffdc get request index
Michael Walsh63b70c02017-10-30 15:02:09 -0500339 # Example of entries:
340 # entries:
341 # entries[0]: GET REQUESTS
Sushil Singh19888552020-06-08 02:14:52 -0500342
343 FOR ${index} IN @{entries}
344 ${ffdc_file_sub_list}= Log FFDC Get Requests ${index}
345 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
346 END
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500347
Michael Walsh63b70c02017-10-30 15:02:09 -0500348 [Return] ${ffdc_file_list}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500349
George Keishing8c42bbc2021-09-28 14:52:54 -0500350
351BMC FFDC Get Redfish Requests
352 [Documentation] Iterate over get request list and return a list of
353 ... generated files.
354
355 @{ffdc_file_list}= Create List
356
357 @{entries}= Get ffdc get redfish request index
358 # Example of entries:
359 # entries:
360 # entries[0]: GET REQUESTS
361
362 FOR ${index} IN @{entries}
363 ${ffdc_file_sub_list}= Log FFDC Get Redfish Requests ${index}
364 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
365 END
366
367 [Return] ${ffdc_file_list}
368
369
Michael Walsh63b70c02017-10-30 15:02:09 -0500370Log OS All distros FFDC
371 [Documentation] Run commands from FFDC_OS_ALL_DISTROS_FILE to create FFDC
372 ... files and return a list of generated files.
373 [Arguments] ${key_index}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500374
Michael Walsh63b70c02017-10-30 15:02:09 -0500375 # Description of argument(s):
376 # key_index The index into the FFDC_OS_ALL_DISTROS_FILE dictionary.
377
378 @{ffdc_file_list}= Create List
379
380 @{cmd_list}= Get FFDC OS All Distros Call ${key_index}
Sushil Singh19888552020-06-08 02:14:52 -0500381
382 FOR ${cmd} IN @{cmd_list}
383 ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}
384 ${ffdc_file_sub_list}= Execute Command and Write FFDC ${cmd[0]} ${cmd[1]} ${logpath} target=OS
385 # scp it to the LOG_PREFIX ffdc directory.
George Keishing89974c92022-02-14 10:43:03 -0600386 Run Key U scp.Get File \ /tmp/${cmd[0]} \ ${LOG_PREFIX}${cmd[0]} ignore=1
Sushil Singh19888552020-06-08 02:14:52 -0500387 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
388 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500389
390 [Return] ${ffdc_file_list}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500391
392
Gunnar Millscce185d2016-10-17 17:04:15 -0500393Log OS SPECIFIC DISTRO FFDC
Michael Walsh63b70c02017-10-30 15:02:09 -0500394 [Documentation] Run commands from the FFDC_OS_<distro>_FILE to create FFDC
395 ... files and return a list of generated files.
396 [Arguments] ${key_index} ${linux_distro}
Gunnar Millscce185d2016-10-17 17:04:15 -0500397
Michael Walsh63b70c02017-10-30 15:02:09 -0500398 # Description of argument(s):
399 # key_index The index into the FFDC_OS_<distro>_FILE dictionary.
400 # linux_distro Your OS's linux distro (e.g. "UBUNTU", "RHEL", etc).
401
George Keishing671dd662021-05-28 03:26:30 -0500402 Log To Console Collecting log for ${linux_distro}
403
Michael Walsh63b70c02017-10-30 15:02:09 -0500404 @{ffdc_file_list}= Create List
405
406 @{cmd_list}= Get FFDC OS Distro Call ${key_index} ${linux_distro}
Sushil Singh19888552020-06-08 02:14:52 -0500407
408 FOR ${cmd} IN @{cmd_list}
409 ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}
410 ${ffdc_file_sub_list}= Execute Command and Write FFDC ${cmd[0]} ${cmd[1]} ${logpath} target=OS
George Keishing89974c92022-02-14 10:43:03 -0600411 Run Key U scp.Get File \ /tmp/${cmd[0]} \ ${LOG_PREFIX}${cmd[0]} ignore=1
Sushil Singh19888552020-06-08 02:14:52 -0500412 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
George Keishing4609d2f2021-05-28 04:40:58 -0500413 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500414
Steven Sombare70acbe2019-08-20 14:37:47 -0500415 # Get the name of the sosreport file.
416 ${sosreport_file_path} ${stderr} ${rc}= OS Execute Command
417 ... ls /tmp/sosreport*FFDC*tar.xz ignore_err=${True}
418 # Example: sosreport_file_path="/tmp/sosreport-myhost-FFDC-2019-08-20-pbuaqtk.tar.xz".
419
420 # Return if there is no sosreport file.
421 Return From Keyword If ${rc} != ${0} ${ffdc_file_list}
422
423 ${sosreport_dir_path} ${sosreport_file_name}= Split Path ${sosreport_file_path}
424 # Example: sosreport_dir_path="/tmp",
425 # sosreport_file_name="sosreport-myhost-FFDC-2019-08-20-pbuaqtk.tar.xz".
426
427 # Location where the sosreport file will be copied to.
428 ${local_sosreport_file_path}= Set Variable ${LOG_PREFIX}OS_${sosreport_file_name}
429
430 # Change file permissions otherwise scp will not see the file.
431 OS Execute Command chmod 644 ${sosreport_file_path}
432
433 # SCP the sosreport file from the OS.
George Keishing89974c92022-02-14 10:43:03 -0600434 Run Key U scp.Get File \ ${sosreport_file_path} \ ${local_sosreport_file_path} ignore=1
Steven Sombare70acbe2019-08-20 14:37:47 -0500435
436 # Add the file location to the ffdc_file_list.
437 Append To List ${ffdc_file_list} ${local_sosreport_file_path}
438
Michael Walsh63b70c02017-10-30 15:02:09 -0500439 [Return] ${ffdc_file_list}
Gunnar Millscce185d2016-10-17 17:04:15 -0500440
441
George Keishingca8c4c62016-10-14 10:08:40 -0500442OS FFDC Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500443 [Documentation] Run the commands from FFDC_OS_ALL_DISTROS_FILE to create
444 ... FFDC files and return a list of generated files.
Gunnar Millsaca140d2016-10-26 13:05:10 -0500445 [Arguments] ${OS_HOST}=${OS_HOST} ${OS_USERNAME}=${OS_USERNAME}
Michael Walsh63b70c02017-10-30 15:02:09 -0500446 ... ${OS_PASSWORD}=${OS_PASSWORD}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500447
Michael Walsh63b70c02017-10-30 15:02:09 -0500448 @{ffdc_file_list}= Create List
Gunnar Millsaca140d2016-10-26 13:05:10 -0500449
Michael Walsh63b70c02017-10-30 15:02:09 -0500450 Run Keyword If '${OS_HOST}' == '${EMPTY}' Run Keywords
451 ... Print Timen No OS Host provided so no OS FFDC will be done. AND
452 ... Return From Keyword ${ffdc_file_list}
Gunnar Millsaca140d2016-10-26 13:05:10 -0500453
Michael Walsh63b70c02017-10-30 15:02:09 -0500454 ${match_state}= Create Dictionary os_ping=^1$ os_login=^1$
455 ... os_run_cmd=^1$
456 ${status} ${ret_values}= Run Keyword and Ignore Error Check State
457 ... ${match_state} quiet=0
458
459 Run Keyword If '${status}' == 'FAIL' Run Keywords
Michael Walshc3b28d72017-11-16 10:43:14 -0600460 ... Print Timen The OS is not communicating so no OS FFDC will be done.\n
Michael Walsh63b70c02017-10-30 15:02:09 -0500461 ... AND
462 ... Return From Keyword ${ffdc_file_list}
Gunnar Millscce185d2016-10-17 17:04:15 -0500463
George Keishing671dd662021-05-28 03:26:30 -0500464 ${stdout}= OS Distro Type
465
Michael Walshe53e47a2017-06-30 17:03:24 -0500466 Set Global Variable ${linux_distro} ${stdout}
Gunnar Millsaca140d2016-10-26 13:05:10 -0500467
Michael Walshe53e47a2017-06-30 17:03:24 -0500468 Rpvars linux_distro
Gunnar Millsaca140d2016-10-26 13:05:10 -0500469
Steven Sombarfc024cb2019-04-22 11:16:16 -0500470 scp.Open Connection
471 ... ${OS_HOST} username=${OS_USERNAME} password=${OS_PASSWORD}
472
Michael Walsh63b70c02017-10-30 15:02:09 -0500473 @{entries}= Get FFDC OS All Distros Index
Sushil Singh19888552020-06-08 02:14:52 -0500474
475 FOR ${index} IN @{entries}
476 ${ffdc_file_sub_list}= Log OS All distros FFDC ${index}
477 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
478 END
Gunnar Millscce185d2016-10-17 17:04:15 -0500479
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600480 Return From Keyword If
481 ... '${linux_distro}' == '${EMPTY}' or '${linux_distro}' == 'None'
Michael Walsh63b70c02017-10-30 15:02:09 -0500482 ... ${ffdc_file_list}
Gunnar Millscce185d2016-10-17 17:04:15 -0500483
484 @{entries}= Get ffdc os distro index ${linux_distro}
Sushil Singh19888552020-06-08 02:14:52 -0500485
486 FOR ${index} IN @{entries}
487 ${ffdc_file_sub_list}= Log OS SPECIFIC DISTRO FFDC ${index} ${linux_distro}
488 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
489 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500490
Steven Sombarfc024cb2019-04-22 11:16:16 -0500491 # Delete ffdc files still on OS and close scp.
Steven Sombare70acbe2019-08-20 14:37:47 -0500492 OS Execute Command rm -rf /tmp/OS_* /tmp/sosreport*FFDC* ignore_err=${True}
Steven Sombarfc024cb2019-04-22 11:16:16 -0500493 scp.Close Connection
494
Michael Walsh63b70c02017-10-30 15:02:09 -0500495 [Return] ${ffdc_file_list}
George Keishing7a520222017-02-27 09:44:30 -0600496
497
Steven Sombar96de5732017-08-17 09:04:52 -0500498System Inventory Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500499 [Documentation] Copy systest os_inventory files and return a list of
500 ... generated files..
Steven Sombar96de5732017-08-17 09:04:52 -0500501 # The os_inventory files are the result of running
502 # systest/htx_hardbootme_test. If these files exist
503 # they are copied to the FFDC directory.
504 # Global variable ffdc_dir_path is the path name of the
505 # directory they are copied to.
Michael Walsh63b70c02017-10-30 15:02:09 -0500506
507 @{ffdc_file_list}= Create List
508
509 ${globex}= Set Variable os_inventory_*.json
510
511 @{file_list}= OperatingSystem.List Files In Directory . ${globex}
512
513 Copy Files ${globex} ${ffdc_dir_path}
514
Sushil Singh19888552020-06-08 02:14:52 -0500515 FOR ${file_name} IN @{file_list}
516 Append To List ${ffdc_file_list} ${ffdc_dir_path}${file_name}
517 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500518
519 Run Keyword and Ignore Error Remove Files ${globex}
520
521 [Return] ${ffdc_file_list}
Steven Sombar96de5732017-08-17 09:04:52 -0500522
523
George Keishing7a520222017-02-27 09:44:30 -0600524SCP Coredump Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500525 [Documentation] Copy core dump files from BMC to local system and return a
526 ... list of generated file names.
527
528 @{ffdc_file_list}= Create List
Michael Walshe53e47a2017-06-30 17:03:24 -0500529
George Keishing7a520222017-02-27 09:44:30 -0600530 # Check if core dump exist in the /tmp
Michael Walshe53e47a2017-06-30 17:03:24 -0500531 ${core_files} ${stderr} ${rc}= BMC Execute Command ls /tmp/core_*
Michael Walsh63b70c02017-10-30 15:02:09 -0500532 ... ignore_err=${1}
533 Run Keyword If '${rc}' != '${0}' Return From Keyword ${ffdc_file_list}
534
535 @{core_list}= Split String ${core_files}
George Keishing7a520222017-02-27 09:44:30 -0600536 # Copy the core files
Michael Walsh6bad8412017-06-27 14:09:36 -0500537 Run Key U Open Connection for SCP
Sushil Singh19888552020-06-08 02:14:52 -0500538
539 FOR ${index} IN @{core_list}
540 ${ffdc_file_path}= Catenate ${LOG_PREFIX}${index.lstrip("/tmp/")}
541 ${status}= Run Keyword and Return Status scp.Get File ${index} ${ffdc_file_path}
542 Run Keyword If '${status}' == '${False}' Continue For Loop
543 Append To List ${ffdc_file_list} ${ffdc_file_path}
544
545 # Remove the file from remote to avoid re-copying on next FFDC call
546
547 BMC Execute Command rm ${index} ignore_err=${1}
548 # I can't find a way to do this: scp.Close Connection
549
550 END
George Keishing7a520222017-02-27 09:44:30 -0600551
Michael Walsh63b70c02017-10-30 15:02:09 -0500552 [Return] ${ffdc_file_list}
553
Steven Sombar2da175f2019-02-04 10:32:58 -0600554
Sweta Potthuria82efd62017-10-18 05:34:59 -0500555SCP Dump Files
556 [Documentation] Copy all dump files from BMC to local system.
557
558 # Check if dumps exist
559 ${ffdc_file_list}= Scp Dumps ${FFDC_DIR_PATH} ${FFDC_PREFIX}
560
561 [Return] ${ffdc_file_list}
562
Steven Sombar2da175f2019-02-04 10:32:58 -0600563
Sweta Potthuria82efd62017-10-18 05:34:59 -0500564Collect Dump Log
565 [Documentation] Collect dumps from dump entry.
566 [Arguments] ${log_prefix_path}=${LOG_PREFIX}
567
George Keishing752b02b2022-03-30 01:37:22 -0500568 Return From Keyword If ${REDFISH_SUPPORT_TRANS_STATE} == ${1}
569
Rahul Maheshwaria89ff9e2020-09-25 05:04:33 -0500570 ${resp}= OpenBMC Get Request ${DUMP_URI}
571 Run Keyword If '${resp.status_code}' == '${HTTP_NOT_FOUND}'
572 ... Set Test Variable ${DUMP_ENTRY_URI} /xyz/openbmc_project/dump/entry/
573
George Keishingccd20e12020-07-29 11:14:26 -0500574 ${data}= Read Properties ${DUMP_ENTRY_URI}enumerate quiet=${1} timeout=${30}
Sweta Potthuria82efd62017-10-18 05:34:59 -0500575
576 # Grab the list of entries from dump/entry/
577 # The data shown below is the result of the "Get Dictionary Keys".
578 # Example:
579 # /xyz/openbmc_project/dump/entry/1
580 # /xyz/openbmc_project/dump/entry/2
581
582 ${dump_list}= Get Dictionary Keys ${data}
George Keishingeb7b4bc2017-03-13 12:04:28 -0500583
Michael Walsh02c70602018-10-04 15:24:30 -0500584
George Keishingbe396b82020-03-23 10:23:58 -0500585Collect PEL Log
586 [Documentation] Collect PEL files from from BMC.
587
588 Create Directory ${FFDC_DIR_PATH}${/}pel_files/
589 scp.Get File /var/lib/phosphor-logging/extensions/pels/logs/
590 ... ${FFDC_DIR_PATH}${/}pel_files recursive=True
591
592
Steven Sombar2da175f2019-02-04 10:32:58 -0600593Enumerate Redfish Resources
594 [Documentation] Enumerate /redfish/v1 resources and properties to
595 ... a file. Return a list which contains the file name.
George Keishing5942d202021-10-04 13:09:24 -0500596 [Arguments] ${log_prefix_path}=${LOG_PREFIX} ${enum_uri}=/redfish/v1
597 ... ${file_enum_name}=redfish_resource_properties.txt
Steven Sombar2da175f2019-02-04 10:32:58 -0600598
599 # Description of argument(s):
600 # log_prefix_path The location specifying where to create FFDC file(s).
601
602 # Login is needed to fetch Redfish information.
603 # If login fails, return from keyword.
George Keishing97c93942019-03-04 12:45:07 -0600604 ${status}= Run Keyword And Return Status Redfish.Login
Steven Sombar2da175f2019-02-04 10:32:58 -0600605 Return From Keyword If ${status} == ${False}
606
607 # Get the Redfish resources and properties.
George Keishing5942d202021-10-04 13:09:24 -0500608 ${json_data}= redfish_utils.Enumerate Request ${enum_uri}
Steven Sombar2da175f2019-02-04 10:32:58 -0600609 # Typical output:
610 # {
611 # "@odata.id": "/redfish/v1",
612 # "@odata.type": "#ServiceRoot.v1_1_1.ServiceRoot",
613 # "AccountService": {
614 # "@odata.id": "/redfish/v1/AccountService"
615 # },
616 # "Chassis": {
617 # "@odata.id": "/redfish/v1/Chassis"
618 # },
619 # "Id": "RootService",
620 # "JsonSchemas": {
621 # "@odata.id": "/redfish/v1/JsonSchemas"
622 # },
623 # ..etc...
624 # }
625
626 @{ffdc_file_list}= Create List
George Keishing5942d202021-10-04 13:09:24 -0500627 ${logpath}= Catenate SEPARATOR= ${log_prefix_path} ${file_enum_name}
Steven Sombar2da175f2019-02-04 10:32:58 -0600628 Create File ${logpath}
629 Write Data To File "${\n}${json_data}${\n}" ${logpath}
630
631 Append To List ${ffdc_file_list} ${logpath}
632
633 [Return] ${ffdc_file_list}
634
635
George Keishing985659a2021-06-23 02:52:13 -0500636Enumerate Redfish OEM Resources
637 [Documentation] Enumerate /<oem>/v1 resources and properties to
638 ... a file. Return a list which contains the file name.
639 [Arguments] ${log_prefix_path}=${LOG_PREFIX}
640
641 # Description of argument(s):
642 # log_prefix_path The location specifying where to create FFDC file(s).
643
644 # No-op by default if input is not supplied from command line.
645 Return From Keyword If "${OEM_REDFISH_PATH}" == "${EMPTY}"
646
647 # Login is needed to fetch Redfish information.
648 # If login fails, return from keyword.
649 ${status}= Run Keyword And Return Status Redfish.Login
650 Return From Keyword If ${status} == ${False}
651
652 # Get the Redfish resources and properties.
653 ${json_data}= redfish_utils.Enumerate Request ${OEM_REDFISH_PATH}
654
655 @{ffdc_file_list}= Create List
656 ${logpath}= Catenate SEPARATOR= ${log_prefix_path}
657 ... redfish_oem_resource_properties.txt
658 Create File ${logpath}
659 Write Data To File "${\n}${json_data}${\n}" ${logpath}
660
661 Append To List ${ffdc_file_list} ${logpath}
662
663 [Return] ${ffdc_file_list}
664
665
George Keishingeb7b4bc2017-03-13 12:04:28 -0500666Collect eSEL Log
Michael Walsh02c70602018-10-04 15:24:30 -0500667 [Documentation] Create raw and formatted eSEL files.
Sridevi Rameshb180c9f2017-08-06 10:27:41 -0500668 [Arguments] ${log_prefix_path}=${LOG_PREFIX}
669
Michael Walsh02c70602018-10-04 15:24:30 -0500670 # NOTE: If no eSEL.pl program can be located, then no formatted eSEL file
671 # will be generated.
672
673 # Description of argument(s):
674 # log_prefix_path The path prefix to be used in creating
675 # eSEL file path names. For example, if
676 # log_prefix_path is
677 # "/tmp/user1/dummy.181001.120000.", then
678 # this keyword will create
679 # /tmp/user1/dummy.181001.120000.esel (raw)
680 # and
681 # /tmp/user1/dummy.181001.120000.esel.txt
682 # (formatted).
683
Michael Walsh63b70c02017-10-30 15:02:09 -0500684 @{ffdc_file_list}= Create List
685
Michael Walsh02c70602018-10-04 15:24:30 -0500686 ${esels}= Get Esels
687 ${num_esels}= Evaluate len(${esels})
688 Rprint Vars num_esels
689 Return From Keyword If ${num_esels} == ${0} ${ffdc_file_list}
George Keishingeb7b4bc2017-03-13 12:04:28 -0500690
Sridevi Rameshb180c9f2017-08-06 10:27:41 -0500691 ${logpath}= Catenate SEPARATOR= ${log_prefix_path} esel
George Keishingeb7b4bc2017-03-13 12:04:28 -0500692 Create File ${logpath}
George Keishingeb7b4bc2017-03-13 12:04:28 -0500693
Sushil Singh19888552020-06-08 02:14:52 -0500694 FOR ${esel} IN @{esels}
695 Write Data To File "${esel}"${\n} ${logpath}
696 END
George Keishingb1ca69b2017-03-31 14:36:42 -0500697
Michael Walsh63b70c02017-10-30 15:02:09 -0500698 Append To List ${ffdc_file_list} ${logpath}
Michael Walsh02c70602018-10-04 15:24:30 -0500699
700 ${rc} ${output}= Shell Cmd which eSEL.pl show_err=0
701 Return From Keyword If ${rc} != ${0} ${ffdc_file_list}
702
703 Convert eSEL To Elog Format ${logpath}
Michael Walsh0192e9b2017-11-14 15:14:15 -0600704 Append To List ${ffdc_file_list} ${logpath}.txt
Michael Walsh63b70c02017-10-30 15:02:09 -0500705
706 [Return] ${ffdc_file_list}
707
George Keishingb1ca69b2017-03-31 14:36:42 -0500708
George Keishingb1ca69b2017-03-31 14:36:42 -0500709Convert eSEL To Elog Format
710 [Documentation] Execute parser tool on the eSEL data file to generate
Michael Walsha30dac72017-04-04 17:56:23 -0500711 ... formatted error log.
George Keishingb1ca69b2017-03-31 14:36:42 -0500712 [Arguments] ${esel_file_path}
Michael Walsh02c70602018-10-04 15:24:30 -0500713 # Description of argument(s):
George Keishing7c6342b2018-11-02 14:27:13 -0500714 # esel_file_path The path to the file containing raw eSEL
Michael Walsh02c70602018-10-04 15:24:30 -0500715 # data (e.g.
716 # "/tmp/user1/dummy.181001.120000.esel").
George Keishingb1ca69b2017-03-31 14:36:42 -0500717
Michael Walsha30dac72017-04-04 17:56:23 -0500718 # Note: The only way to get eSEL.pl to put the output in a particular
719 # directory is to cd to that directory.
720 ${cmd_buf}= Catenate cd $(dirname ${esel_file_path}) ; eSEL.pl -l
721 ... ${esel_file_path} -p decode_obmc_data
Michael Walsh02c70602018-10-04 15:24:30 -0500722 Qprint Issuing ${cmd_buf}
Michael Walsha30dac72017-04-04 17:56:23 -0500723 Run ${cmd_buf}
Michael Walsh02c70602018-10-04 15:24:30 -0500724 # The .binary file, which is generated by eSEL.pl, is of no use to us.
725 Remove File ${esel_file_path}.binary
George Keishing671dd662021-05-28 03:26:30 -0500726
727
728OS Distro Type
729 [Documentation] Determine the host partition distro type
730
731 ${stdout} ${stderr} ${rc}= OS Execute Command
732 ... . /etc/os-release; echo $ID ignore_err=${1}
733
734 Return From Keyword If ${rc} == ${0} ${stdout}
735
George Keishing33d15032021-06-07 00:49:56 -0500736 # If linux distro doesn't have os-release, check for uname.
George Keishing671dd662021-05-28 03:26:30 -0500737 ${stdout} ${stderr} ${rc}= OS Execute Command uname ignore_err=${0}
738
739 [Return] ${stdout}