blob: ad0bbc002e065f850eae5f4bf1c5487fcf3a83ed [file] [log] [blame]
George Keishing69e6f712016-09-12 06:30:09 -05001*** Settings ***
2Documentation Methods to execute commands on BMC and collect
3... data to a list of FFDC files
4
Michael Walsh265ae412018-11-16 15:32:03 -06005Resource openbmc_ffdc_utils.robot
6Resource rest_client.robot
7Resource utils.robot
8Resource list_utils.robot
9Resource logging_utils.robot
Steven Sombar2da175f2019-02-04 10:32:58 -060010Resource bmc_redfish_resource.robot
Michael Walsh265ae412018-11-16 15:32:03 -060011Library SSHLibrary
12Library OperatingSystem
13Library Collections
14Library String
15Library gen_print.py
16Library gen_cmd.py
17Library gen_robot_keyword.py
18Library dump_utils.py
19Library logging_utils.py
Gunnar Mills7e2cda22016-10-11 15:37:34 -050020
Michael Walshde5e9372018-11-15 11:28:23 -060021*** Variables ***
22
Michael Walsh265ae412018-11-16 15:32:03 -060023${FFDC_CMD_TIMEOUT} 240
Michael Walshde5e9372018-11-15 11:28:23 -060024
George Keishing69e6f712016-09-12 06:30:09 -050025*** Keywords ***
26
George Keishing69e6f712016-09-12 06:30:09 -050027# Method : Call FFDC Methods #
28# Execute the user define keywords from the FFDC List #
29# Unlike any other keywords this will call into the #
30# list of keywords defined in the FFDC list at one go #
George Keishing69e6f712016-09-12 06:30:09 -050031
32Call FFDC Methods
Michael Walshe844e9a2017-04-20 16:51:10 -050033 [Documentation] Call into FFDC Keyword index list.
34 [Arguments] ${ffdc_function_list}=${EMPTY}
George Keishing69e6f712016-09-12 06:30:09 -050035
Michael Walshe844e9a2017-04-20 16:51:10 -050036 # Description of argument(s):
37 # ffdc_function_list A colon-delimited list naming the kinds of FFDC that
Michael Walsh63b70c02017-10-30 15:02:09 -050038 # are to be collected
Michael Walshe844e9a2017-04-20 16:51:10 -050039 # (e.g. "FFDC Generic Report:BMC Specific Files").
40 # Acceptable values can be found in the description
41 # field of FFDC_METHOD_CALL in
42 # lib/openbmc_ffdc_list.py. Those values can be
43 # obtained via a call to 'Get FFDC Method Desc' (also
44 # from lib/openbmc_ffdc_list.py).
45
46 @{entries}= Get FFDC Method Index
Michael Walsh63b70c02017-10-30 15:02:09 -050047 # Example entries:
48 # entries:
49 # entries[0]: BMC LOGS
50
51 @{ffdc_file_list}= Create List
Sushil Singh19888552020-06-08 02:14:52 -050052 FOR ${index} IN @{entries}
53 ${ffdc_file_sub_list}= Method Call Keyword List ${index} ${ffdc_function_list}
54 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
55 END
Michael Walsh63b70c02017-10-30 15:02:09 -050056
Michael Walsh6bad8412017-06-27 14:09:36 -050057 Run Key U SSHLibrary.Close All Connections
George Keishing69e6f712016-09-12 06:30:09 -050058
Michael Walsh63b70c02017-10-30 15:02:09 -050059 [Return] ${ffdc_file_list}
60
Steven Sombar2da175f2019-02-04 10:32:58 -060061
George Keishing69e6f712016-09-12 06:30:09 -050062Method Call Keyword List
Michael Walsh63b70c02017-10-30 15:02:09 -050063 [Documentation] Process FFDC request and return a list of generated files.
Michael Walshe844e9a2017-04-20 16:51:10 -050064 [Arguments] ${index} ${ffdc_function_list}=${EMPTY}
George Keishing69e6f712016-09-12 06:30:09 -050065
Michael Walshe844e9a2017-04-20 16:51:10 -050066 # Description of argument(s):
67 # index The index into the FFDC_METHOD_CALL dictionary (e.g.
68 # 'BMC LOGS').
69 # ffdc_function_list See ffdc_function_list description in
70 # "Call FFDC Methods" (above).
71
Michael Walsh63b70c02017-10-30 15:02:09 -050072 @{method_list}= Get FFDC Method Call ${index}
73 # Example method_list:
74 # method_list:
75 # method_list[0]:
76 # method_list[0][0]: FFDC Generic Report
77 # method_list[0][1]: BMC FFDC Manifest
78 # method_list[1]:
79 # method_list[1][0]: Get Request FFDC
80 # method_list[1][1]: BMC FFDC Get Requests
81 # (etc.)
Michael Walshe844e9a2017-04-20 16:51:10 -050082
83 # If function list is empty assign default (i.e. a list of all allowable
84 # values). In either case, convert ffdc_function_list from a string to
85 # a list.
86 @{ffdc_function_list}=
87 ... Run Keyword If '${ffdc_function_list}' == '${EMPTY}'
88 ... Get FFDC Method Desc ${index}
89 ... ELSE
90 ... Split String ${ffdc_function_list} separator=:
91
Michael Walsh63b70c02017-10-30 15:02:09 -050092 @{ffdc_file_list}= Create List
Sushil Singh19888552020-06-08 02:14:52 -050093 FOR ${method} IN @{method_list}
94 ${ffdc_file_sub_list}= Execute Keyword Method ${method[0]} ${method[1]} @{ffdc_function_list}
95 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
96 END
Michael Walsh63b70c02017-10-30 15:02:09 -050097
98 [Return] ${ffdc_file_list}
99
George Keishing69e6f712016-09-12 06:30:09 -0500100
101Execute Keyword Method
Michael Walshe844e9a2017-04-20 16:51:10 -0500102 [Documentation] Call into BMC method keywords. Don't let one
103 ... failure skip the remaining. Get whatever data
104 ... it could gather at worse case scenario.
105 [Arguments] ${description} ${keyword_name} @{ffdc_function_list}
George Keishing69e6f712016-09-12 06:30:09 -0500106
Michael Walshe844e9a2017-04-20 16:51:10 -0500107 # Description of argument(s):
108 # description The description of the FFDC to be collected. This
109 # would be any value returned by
110 # 'Get FFDC Method Desc' (e.g. "FFDC Generic Report").
111 # keyword_name The name of the keyword to call to collect the FFDC
112 # data (again, see FFDC_METHOD_CALL).
113 # ffdc_function_list See ffdc_function_list description in
114 # "Call FFDC Methods" (above). The only difference is
115 # in this case, it should be a list rather than a
116 # colon-delimited value.
George Keishing69e6f712016-09-12 06:30:09 -0500117
Michael Walsh63b70c02017-10-30 15:02:09 -0500118 @{ffdc_file_list}= Create List
Michael Walshe844e9a2017-04-20 16:51:10 -0500119
Michael Walsh63b70c02017-10-30 15:02:09 -0500120 ${index}= Get Index From List ${ffdc_function_list} ${description}
121 Run Keyword If '${index}' == '${-1}' Return from Keyword
122 ... ${ffdc_file_list}
123
124 ${status} ${ffdc_file_list}= Run Key ${keyword_name} ignore=1
125 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500126
George Keishing69e6f712016-09-12 06:30:09 -0500127# Method : BMC FFDC Manifest #
128# Execute command on BMC and write to ffdc_report.txt #
George Keishing69e6f712016-09-12 06:30:09 -0500129
130BMC FFDC Manifest
Michael Walsh63b70c02017-10-30 15:02:09 -0500131 [Documentation] Run the ssh commands from FFDC_BMC_CMD and return a list
132 ... of generated files.
George Keishing69e6f712016-09-12 06:30:09 -0500133
Michael Walsh63b70c02017-10-30 15:02:09 -0500134 @{ffdc_file_list}= Create List ${FFDC_FILE_PATH}
135 @{entries}= Get FFDC Cmd Index
Sushil Singh19888552020-06-08 02:14:52 -0500136
137 FOR ${index} IN @{entries}
138 Iterate BMC Command List Pairs ${index}
139 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500140
141 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500142
143
144Iterate BMC Command List Pairs
145 [Documentation] Feed in key pair list from dictionary to execute
146 [Arguments] ${key_index}
147
148 @{cmd_list}= Get ffdc bmc cmd ${key_index}
149 Set Suite Variable ${ENTRY_INDEX} ${key_index}
George Keishing69e6f712016-09-12 06:30:09 -0500150
Sushil Singh19888552020-06-08 02:14:52 -0500151 FOR ${cmd} IN @{cmd_list}
152 Execute Command and Write FFDC ${cmd[0]} ${cmd[1]}
153 END
George Keishing69e6f712016-09-12 06:30:09 -0500154
155Execute Command and Write FFDC
Michael Walsh63b70c02017-10-30 15:02:09 -0500156 [Documentation] Run a command on the BMC or OS, write the output to the
157 ... specified file and return a list of generated files.
158 [Arguments] ${key_index} ${cmd} ${logpath}=${FFDC_FILE_PATH}
159 ... ${target}=BMC
George Keishing69e6f712016-09-12 06:30:09 -0500160
Michael Walsh63b70c02017-10-30 15:02:09 -0500161 Run Keyword If '${logpath}' == '${FFDC_FILE_PATH}'
162 ... Write Cmd Output to FFDC File ${key_index} ${cmd}
163
164 @{ffdc_file_list}= Create List ${log_path}
George Keishing69e6f712016-09-12 06:30:09 -0500165
Michael Walshe53e47a2017-06-30 17:03:24 -0500166 ${cmd_buf}= Catenate ${target} Execute Command \ ${cmd} \ ignore_err=${1}
Michael Walsh265ae412018-11-16 15:32:03 -0600167 ... \ time_out=${FFDC_CMD_TIMEOUT}
Michael Walshe53e47a2017-06-30 17:03:24 -0500168 ${status} ${ret_values}= Run Key ${cmd_buf} ignore=${1}
Michael Walshde5e9372018-11-15 11:28:23 -0600169 # If the command times out, status will be 'FAIL'.
170 Return From Keyword If '${status}' == 'FAIL' ${ffdc_file_list}
Michael Walshe53e47a2017-06-30 17:03:24 -0500171
Sushil Singh19888552020-06-08 02:14:52 -0500172 ${stdout}= Set Variable ${ret_values}[0]
173 ${stderr}= Set Variable ${ret_values}[1]
George Keishing69e6f712016-09-12 06:30:09 -0500174
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600175 # Write stdout on success and stderr/stdout to the file on failure.
176 Run Keyword If $stderr == '${EMPTY}'
Sweta Potthuri15e6d2f2017-02-28 03:10:02 -0600177 ... Write Data To File ${stdout}${\n} ${logpath}
178 ... ELSE Write Data To File
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600179 ... ERROR output:${\n}${stderr}${\n}Output:${\n}${stdout}${\n}
180 ... ${logpath}
George Keishing69e6f712016-09-12 06:30:09 -0500181
Michael Walsh63b70c02017-10-30 15:02:09 -0500182 [Return] ${ffdc_file_list}
183
George Keishing69e6f712016-09-12 06:30:09 -0500184
George Keishing69e6f712016-09-12 06:30:09 -0500185# Method : BMC FFDC Files #
186# Execute command on BMC and write to individual file #
187# based on the file name pre-defined in the list #
George Keishing69e6f712016-09-12 06:30:09 -0500188
189BMC FFDC Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500190 [Documentation] Run the commands from FFDC_BMC_FILE and return a list of
191 ... generated files.
192
193 @{entries}= Get FFDC File Index
194 # Example of entries:
195 # entries:
196 # entries[0]: BMC FILES
197
Steven Sombara31f4c82019-08-01 16:00:23 -0500198 scp.Open Connection
vinaybs6b8110942020-01-29 23:15:29 -0600199 ... ${OPENBMC_HOST} username=${OPENBMC_USERNAME} password=${OPENBMC_PASSWORD} port=${SSH_PORT}
Steven Sombara31f4c82019-08-01 16:00:23 -0500200
Michael Walsh63b70c02017-10-30 15:02:09 -0500201 @{ffdc_file_list}= Create List
Sushil Singh19888552020-06-08 02:14:52 -0500202
203 FOR ${index} IN @{entries}
204 ${ffdc_file_sub_list}= Create File and Write Data ${index}
205 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
206 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500207
Michael Walsh17eaafe2019-08-22 11:05:20 -0500208 BMC Execute Command rm -rf /tmp/BMC_*
Steven Sombara31f4c82019-08-01 16:00:23 -0500209 scp.Close Connection
210
Michael Walsh63b70c02017-10-30 15:02:09 -0500211 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500212
213
214Create File and Write Data
Michael Walsh63b70c02017-10-30 15:02:09 -0500215 [Documentation] Run commands from FFDC_BMC_FILE to create FFDC files and
216 ... return a list of generated files.
217 [Arguments] ${key_index}
George Keishing69e6f712016-09-12 06:30:09 -0500218
Michael Walsh63b70c02017-10-30 15:02:09 -0500219 # Description of argument(s):
220 # key_index The index into the FFDC_BMC_FILE dictionary.
221
222 @{ffdc_file_list}= Create List
223 @{cmd_list}= Get FFDC BMC File ${key_index}
Sushil Singh19888552020-06-08 02:14:52 -0500224
225 FOR ${cmd} IN @{cmd_list}
226 ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}
227 ${ffdc_file_sub_list}= Execute Command and Write FFDC ${cmd[0]} ${cmd[1]} ${logpath}
George Keishing89974c92022-02-14 10:43:03 -0600228 Run Key U scp.Get File \ /tmp/${cmd[0]} \ ${LOG_PREFIX}${cmd[0]} ignore=1
Sushil Singh19888552020-06-08 02:14:52 -0500229 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
230 END
Sivas SRR0d657e82017-03-13 01:00:34 -0500231
Michael Walsh63b70c02017-10-30 15:02:09 -0500232 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500233
234
George Keishing69e6f712016-09-12 06:30:09 -0500235# Method : Log Test Case Status #
236# Creates test result history footprint for reference #
George Keishing69e6f712016-09-12 06:30:09 -0500237
238Log Test Case Status
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600239 [Documentation] Test case execution result history.
240 ... Create once and append to this file
241 ... logs/test_history.txt
242 ... Format Date:Test suite:Test case:Status
243 ... 20160909214053719992:Test Warmreset:Test WarmReset via REST:FAIL
Michael Walshfa9f70f2017-04-21 16:00:18 -0500244
245 ${FFDC_DIR_PATH_STYLE}= Get Variable Value ${FFDC_DIR_PATH_STYLE}
246 ... ${EMPTY}
247 ${FFDC_DIR_PATH}= Get Variable Value ${FFDC_DIR_PATH} ${EMPTY}
248
249 Run Keyword If '${FFDC_DIR_PATH}' == '${EMPTY}' Set FFDC Defaults
250
251 Run Keyword If '${FFDC_DIR_PATH_STYLE}' == '${1}' Run Keywords
252 ... Set Global Variable ${FFDC_LOG_PATH} ${FFDC_DIR_PATH} AND
253 ... Set Global Variable ${TEST_HISTORY} ${FFDC_DIR_PATH}test_history.txt
254
George Keishing69e6f712016-09-12 06:30:09 -0500255 Create Directory ${FFDC_LOG_PATH}
256
257 ${exist}= Run Keyword and Return Status
258 ... OperatingSystem.File Should Exist ${TEST_HISTORY}
259
260 Run Keyword If '${exist}' == '${False}'
261 ... Create File ${TEST_HISTORY}
262
Michael Walshfa9f70f2017-04-21 16:00:18 -0500263 Rpvars TEST_HISTORY
264
George Keishing69e6f712016-09-12 06:30:09 -0500265 ${cur_time}= Get Current Time Stamp
266
267 Append To File ${TEST_HISTORY}
268 ... ${cur_time}:${SUITE_NAME}:${TEST_NAME}:${TEST_STATUS}${\n}
269
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500270
271Log FFDC Get Requests
Michael Walsh63b70c02017-10-30 15:02:09 -0500272 [Documentation] Run the get requests associated with the key and return a
273 ... list of generated files.
274 [Arguments] ${key_index}
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500275
Michael Walsh63b70c02017-10-30 15:02:09 -0500276 # Note: Output will be in JSON pretty_print format.
277
278 # Description of argument(s):
279 # key_index The key to the FFDC_GET_REQUEST dictionary that contains the
280 # get requests that are to be run.
281
282 @{ffdc_file_list}= Create List
283 @{cmd_list}= Get FFDC Get Request ${key_index}
Sushil Singh19888552020-06-08 02:14:52 -0500284
285 FOR ${cmd} IN @{cmd_list}
286 ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}
George Keishingccd20e12020-07-29 11:14:26 -0500287 ${resp}= OpenBMC Get Request ${cmd[1]} quiet=${1} timeout=${30}
Sushil Singh19888552020-06-08 02:14:52 -0500288 ${status}= Run Keyword and Return Status Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
289 Run Keyword If '${status}' == '${False}' Continue For Loop
290 ${jsondata}= to json ${resp.content} pretty_print=True
291 Write Data To File ${\n}${jsondata}${\n} ${logpath}
292 Append To List ${ffdc_file_list} ${logpath}
293 END
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500294
Michael Walsh63b70c02017-10-30 15:02:09 -0500295 [Return] ${ffdc_file_list}
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500296
George Keishing8c42bbc2021-09-28 14:52:54 -0500297
298Log FFDC Get Redfish Requests
299 [Documentation] Run the get requests associated with the key and return a
300 ... list of generated files.
301 [Arguments] ${key_index}
302
303 # Note: Output will be in JSON pretty_print format.
304
305 # Description of argument(s):
306 # key_index The key to the FFDC_GET_REDFISH_REQUEST dictionary that contains the
307 # get requests that are to be run.
308
309 @{ffdc_file_list}= Create List
310 @{cmd_list}= Get FFDC Get Redfish Request ${key_index}
311
312 FOR ${cmd} IN @{cmd_list}
313 ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}
314 ${resp}= Redfish.Get ${cmd[1]}
315 Write Data To File ${\n}${resp}${\n} ${logpath}
316 Append To List ${ffdc_file_list} ${logpath}
317 END
318
319 [Return] ${ffdc_file_list}
320
321
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500322BMC FFDC Get Requests
Michael Walsh63b70c02017-10-30 15:02:09 -0500323 [Documentation] Iterate over get request list and return a list of
324 ... generated files.
325
326 @{ffdc_file_list}= Create List
327
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500328 @{entries}= Get ffdc get request index
Michael Walsh63b70c02017-10-30 15:02:09 -0500329 # Example of entries:
330 # entries:
331 # entries[0]: GET REQUESTS
Sushil Singh19888552020-06-08 02:14:52 -0500332
333 FOR ${index} IN @{entries}
334 ${ffdc_file_sub_list}= Log FFDC Get Requests ${index}
335 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
336 END
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500337
Michael Walsh63b70c02017-10-30 15:02:09 -0500338 [Return] ${ffdc_file_list}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500339
George Keishing8c42bbc2021-09-28 14:52:54 -0500340
341BMC FFDC Get Redfish Requests
342 [Documentation] Iterate over get request list and return a list of
343 ... generated files.
344
345 @{ffdc_file_list}= Create List
346
347 @{entries}= Get ffdc get redfish request index
348 # Example of entries:
349 # entries:
350 # entries[0]: GET REQUESTS
351
352 FOR ${index} IN @{entries}
353 ${ffdc_file_sub_list}= Log FFDC Get Redfish Requests ${index}
354 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
355 END
356
357 [Return] ${ffdc_file_list}
358
359
Michael Walsh63b70c02017-10-30 15:02:09 -0500360Log OS All distros FFDC
361 [Documentation] Run commands from FFDC_OS_ALL_DISTROS_FILE to create FFDC
362 ... files and return a list of generated files.
363 [Arguments] ${key_index}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500364
Michael Walsh63b70c02017-10-30 15:02:09 -0500365 # Description of argument(s):
366 # key_index The index into the FFDC_OS_ALL_DISTROS_FILE dictionary.
367
368 @{ffdc_file_list}= Create List
369
370 @{cmd_list}= Get FFDC OS All Distros Call ${key_index}
Sushil Singh19888552020-06-08 02:14:52 -0500371
372 FOR ${cmd} IN @{cmd_list}
373 ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}
374 ${ffdc_file_sub_list}= Execute Command and Write FFDC ${cmd[0]} ${cmd[1]} ${logpath} target=OS
375 # scp it to the LOG_PREFIX ffdc directory.
George Keishing89974c92022-02-14 10:43:03 -0600376 Run Key U scp.Get File \ /tmp/${cmd[0]} \ ${LOG_PREFIX}${cmd[0]} ignore=1
Sushil Singh19888552020-06-08 02:14:52 -0500377 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
378 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500379
380 [Return] ${ffdc_file_list}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500381
382
Gunnar Millscce185d2016-10-17 17:04:15 -0500383Log OS SPECIFIC DISTRO FFDC
Michael Walsh63b70c02017-10-30 15:02:09 -0500384 [Documentation] Run commands from the FFDC_OS_<distro>_FILE to create FFDC
385 ... files and return a list of generated files.
386 [Arguments] ${key_index} ${linux_distro}
Gunnar Millscce185d2016-10-17 17:04:15 -0500387
Michael Walsh63b70c02017-10-30 15:02:09 -0500388 # Description of argument(s):
389 # key_index The index into the FFDC_OS_<distro>_FILE dictionary.
390 # linux_distro Your OS's linux distro (e.g. "UBUNTU", "RHEL", etc).
391
George Keishing671dd662021-05-28 03:26:30 -0500392 Log To Console Collecting log for ${linux_distro}
393
Michael Walsh63b70c02017-10-30 15:02:09 -0500394 @{ffdc_file_list}= Create List
395
396 @{cmd_list}= Get FFDC OS Distro Call ${key_index} ${linux_distro}
Sushil Singh19888552020-06-08 02:14:52 -0500397
398 FOR ${cmd} IN @{cmd_list}
399 ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}
400 ${ffdc_file_sub_list}= Execute Command and Write FFDC ${cmd[0]} ${cmd[1]} ${logpath} target=OS
George Keishing89974c92022-02-14 10:43:03 -0600401 Run Key U scp.Get File \ /tmp/${cmd[0]} \ ${LOG_PREFIX}${cmd[0]} ignore=1
Sushil Singh19888552020-06-08 02:14:52 -0500402 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
George Keishing4609d2f2021-05-28 04:40:58 -0500403 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500404
Steven Sombare70acbe2019-08-20 14:37:47 -0500405 # Get the name of the sosreport file.
406 ${sosreport_file_path} ${stderr} ${rc}= OS Execute Command
407 ... ls /tmp/sosreport*FFDC*tar.xz ignore_err=${True}
408 # Example: sosreport_file_path="/tmp/sosreport-myhost-FFDC-2019-08-20-pbuaqtk.tar.xz".
409
410 # Return if there is no sosreport file.
411 Return From Keyword If ${rc} != ${0} ${ffdc_file_list}
412
413 ${sosreport_dir_path} ${sosreport_file_name}= Split Path ${sosreport_file_path}
414 # Example: sosreport_dir_path="/tmp",
415 # sosreport_file_name="sosreport-myhost-FFDC-2019-08-20-pbuaqtk.tar.xz".
416
417 # Location where the sosreport file will be copied to.
418 ${local_sosreport_file_path}= Set Variable ${LOG_PREFIX}OS_${sosreport_file_name}
419
420 # Change file permissions otherwise scp will not see the file.
421 OS Execute Command chmod 644 ${sosreport_file_path}
422
423 # SCP the sosreport file from the OS.
George Keishing89974c92022-02-14 10:43:03 -0600424 Run Key U scp.Get File \ ${sosreport_file_path} \ ${local_sosreport_file_path} ignore=1
Steven Sombare70acbe2019-08-20 14:37:47 -0500425
426 # Add the file location to the ffdc_file_list.
427 Append To List ${ffdc_file_list} ${local_sosreport_file_path}
428
Michael Walsh63b70c02017-10-30 15:02:09 -0500429 [Return] ${ffdc_file_list}
Gunnar Millscce185d2016-10-17 17:04:15 -0500430
431
George Keishingca8c4c62016-10-14 10:08:40 -0500432OS FFDC Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500433 [Documentation] Run the commands from FFDC_OS_ALL_DISTROS_FILE to create
434 ... FFDC files and return a list of generated files.
Gunnar Millsaca140d2016-10-26 13:05:10 -0500435 [Arguments] ${OS_HOST}=${OS_HOST} ${OS_USERNAME}=${OS_USERNAME}
Michael Walsh63b70c02017-10-30 15:02:09 -0500436 ... ${OS_PASSWORD}=${OS_PASSWORD}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500437
Michael Walsh63b70c02017-10-30 15:02:09 -0500438 @{ffdc_file_list}= Create List
Gunnar Millsaca140d2016-10-26 13:05:10 -0500439
Michael Walsh63b70c02017-10-30 15:02:09 -0500440 Run Keyword If '${OS_HOST}' == '${EMPTY}' Run Keywords
441 ... Print Timen No OS Host provided so no OS FFDC will be done. AND
442 ... Return From Keyword ${ffdc_file_list}
Gunnar Millsaca140d2016-10-26 13:05:10 -0500443
Michael Walsh63b70c02017-10-30 15:02:09 -0500444 ${match_state}= Create Dictionary os_ping=^1$ os_login=^1$
445 ... os_run_cmd=^1$
446 ${status} ${ret_values}= Run Keyword and Ignore Error Check State
447 ... ${match_state} quiet=0
448
449 Run Keyword If '${status}' == 'FAIL' Run Keywords
Michael Walshc3b28d72017-11-16 10:43:14 -0600450 ... Print Timen The OS is not communicating so no OS FFDC will be done.\n
Michael Walsh63b70c02017-10-30 15:02:09 -0500451 ... AND
452 ... Return From Keyword ${ffdc_file_list}
Gunnar Millscce185d2016-10-17 17:04:15 -0500453
George Keishing671dd662021-05-28 03:26:30 -0500454 ${stdout}= OS Distro Type
455
Michael Walshe53e47a2017-06-30 17:03:24 -0500456 Set Global Variable ${linux_distro} ${stdout}
Gunnar Millsaca140d2016-10-26 13:05:10 -0500457
Michael Walshe53e47a2017-06-30 17:03:24 -0500458 Rpvars linux_distro
Gunnar Millsaca140d2016-10-26 13:05:10 -0500459
Steven Sombarfc024cb2019-04-22 11:16:16 -0500460 scp.Open Connection
461 ... ${OS_HOST} username=${OS_USERNAME} password=${OS_PASSWORD}
462
Michael Walsh63b70c02017-10-30 15:02:09 -0500463 @{entries}= Get FFDC OS All Distros Index
Sushil Singh19888552020-06-08 02:14:52 -0500464
465 FOR ${index} IN @{entries}
466 ${ffdc_file_sub_list}= Log OS All distros FFDC ${index}
467 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
468 END
Gunnar Millscce185d2016-10-17 17:04:15 -0500469
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600470 Return From Keyword If
471 ... '${linux_distro}' == '${EMPTY}' or '${linux_distro}' == 'None'
Michael Walsh63b70c02017-10-30 15:02:09 -0500472 ... ${ffdc_file_list}
Gunnar Millscce185d2016-10-17 17:04:15 -0500473
474 @{entries}= Get ffdc os distro index ${linux_distro}
Sushil Singh19888552020-06-08 02:14:52 -0500475
476 FOR ${index} IN @{entries}
477 ${ffdc_file_sub_list}= Log OS SPECIFIC DISTRO FFDC ${index} ${linux_distro}
478 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
479 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500480
Steven Sombarfc024cb2019-04-22 11:16:16 -0500481 # Delete ffdc files still on OS and close scp.
Steven Sombare70acbe2019-08-20 14:37:47 -0500482 OS Execute Command rm -rf /tmp/OS_* /tmp/sosreport*FFDC* ignore_err=${True}
Steven Sombarfc024cb2019-04-22 11:16:16 -0500483 scp.Close Connection
484
Michael Walsh63b70c02017-10-30 15:02:09 -0500485 [Return] ${ffdc_file_list}
George Keishing7a520222017-02-27 09:44:30 -0600486
487
Steven Sombar96de5732017-08-17 09:04:52 -0500488System Inventory Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500489 [Documentation] Copy systest os_inventory files and return a list of
490 ... generated files..
Steven Sombar96de5732017-08-17 09:04:52 -0500491 # The os_inventory files are the result of running
492 # systest/htx_hardbootme_test. If these files exist
493 # they are copied to the FFDC directory.
494 # Global variable ffdc_dir_path is the path name of the
495 # directory they are copied to.
Michael Walsh63b70c02017-10-30 15:02:09 -0500496
497 @{ffdc_file_list}= Create List
498
499 ${globex}= Set Variable os_inventory_*.json
500
501 @{file_list}= OperatingSystem.List Files In Directory . ${globex}
502
503 Copy Files ${globex} ${ffdc_dir_path}
504
Sushil Singh19888552020-06-08 02:14:52 -0500505 FOR ${file_name} IN @{file_list}
506 Append To List ${ffdc_file_list} ${ffdc_dir_path}${file_name}
507 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500508
509 Run Keyword and Ignore Error Remove Files ${globex}
510
511 [Return] ${ffdc_file_list}
Steven Sombar96de5732017-08-17 09:04:52 -0500512
513
George Keishing7a520222017-02-27 09:44:30 -0600514SCP Coredump Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500515 [Documentation] Copy core dump files from BMC to local system and return a
516 ... list of generated file names.
517
518 @{ffdc_file_list}= Create List
Michael Walshe53e47a2017-06-30 17:03:24 -0500519
George Keishing7a520222017-02-27 09:44:30 -0600520 # Check if core dump exist in the /tmp
Michael Walshe53e47a2017-06-30 17:03:24 -0500521 ${core_files} ${stderr} ${rc}= BMC Execute Command ls /tmp/core_*
Michael Walsh63b70c02017-10-30 15:02:09 -0500522 ... ignore_err=${1}
523 Run Keyword If '${rc}' != '${0}' Return From Keyword ${ffdc_file_list}
524
525 @{core_list}= Split String ${core_files}
George Keishing7a520222017-02-27 09:44:30 -0600526 # Copy the core files
Michael Walsh6bad8412017-06-27 14:09:36 -0500527 Run Key U Open Connection for SCP
Sushil Singh19888552020-06-08 02:14:52 -0500528
529 FOR ${index} IN @{core_list}
530 ${ffdc_file_path}= Catenate ${LOG_PREFIX}${index.lstrip("/tmp/")}
531 ${status}= Run Keyword and Return Status scp.Get File ${index} ${ffdc_file_path}
532 Run Keyword If '${status}' == '${False}' Continue For Loop
533 Append To List ${ffdc_file_list} ${ffdc_file_path}
534
535 # Remove the file from remote to avoid re-copying on next FFDC call
536
537 BMC Execute Command rm ${index} ignore_err=${1}
538 # I can't find a way to do this: scp.Close Connection
539
540 END
George Keishing7a520222017-02-27 09:44:30 -0600541
Michael Walsh63b70c02017-10-30 15:02:09 -0500542 [Return] ${ffdc_file_list}
543
Steven Sombar2da175f2019-02-04 10:32:58 -0600544
Sweta Potthuria82efd62017-10-18 05:34:59 -0500545SCP Dump Files
546 [Documentation] Copy all dump files from BMC to local system.
547
548 # Check if dumps exist
549 ${ffdc_file_list}= Scp Dumps ${FFDC_DIR_PATH} ${FFDC_PREFIX}
550
551 [Return] ${ffdc_file_list}
552
Steven Sombar2da175f2019-02-04 10:32:58 -0600553
George Keishing1165a022021-05-19 06:23:13 -0500554SCP Dump HB Files
555 [Documentation] Copy all HB dump files from BMC to local system.
556
557 # Check if dumps exist
558 ${ffdc_file_list}= Scp Dumps HB ${FFDC_DIR_PATH} ${FFDC_PREFIX}
559
560 [Return] ${ffdc_file_list}
561
562
Sweta Potthuria82efd62017-10-18 05:34:59 -0500563Collect Dump Log
564 [Documentation] Collect dumps from dump entry.
565 [Arguments] ${log_prefix_path}=${LOG_PREFIX}
566
Rahul Maheshwaria89ff9e2020-09-25 05:04:33 -0500567 ${resp}= OpenBMC Get Request ${DUMP_URI}
568 Run Keyword If '${resp.status_code}' == '${HTTP_NOT_FOUND}'
569 ... Set Test Variable ${DUMP_ENTRY_URI} /xyz/openbmc_project/dump/entry/
570
George Keishingccd20e12020-07-29 11:14:26 -0500571 ${data}= Read Properties ${DUMP_ENTRY_URI}enumerate quiet=${1} timeout=${30}
Sweta Potthuria82efd62017-10-18 05:34:59 -0500572
573 # Grab the list of entries from dump/entry/
574 # The data shown below is the result of the "Get Dictionary Keys".
575 # Example:
576 # /xyz/openbmc_project/dump/entry/1
577 # /xyz/openbmc_project/dump/entry/2
578
579 ${dump_list}= Get Dictionary Keys ${data}
George Keishingeb7b4bc2017-03-13 12:04:28 -0500580
Michael Walsh02c70602018-10-04 15:24:30 -0500581
George Keishingbe396b82020-03-23 10:23:58 -0500582Collect PEL Log
583 [Documentation] Collect PEL files from from BMC.
584
585 Create Directory ${FFDC_DIR_PATH}${/}pel_files/
586 scp.Get File /var/lib/phosphor-logging/extensions/pels/logs/
587 ... ${FFDC_DIR_PATH}${/}pel_files recursive=True
588
589
Steven Sombar2da175f2019-02-04 10:32:58 -0600590Enumerate Redfish Resources
591 [Documentation] Enumerate /redfish/v1 resources and properties to
592 ... a file. Return a list which contains the file name.
George Keishing5942d202021-10-04 13:09:24 -0500593 [Arguments] ${log_prefix_path}=${LOG_PREFIX} ${enum_uri}=/redfish/v1
594 ... ${file_enum_name}=redfish_resource_properties.txt
Steven Sombar2da175f2019-02-04 10:32:58 -0600595
596 # Description of argument(s):
597 # log_prefix_path The location specifying where to create FFDC file(s).
598
599 # Login is needed to fetch Redfish information.
600 # If login fails, return from keyword.
George Keishing97c93942019-03-04 12:45:07 -0600601 ${status}= Run Keyword And Return Status Redfish.Login
Steven Sombar2da175f2019-02-04 10:32:58 -0600602 Return From Keyword If ${status} == ${False}
603
604 # Get the Redfish resources and properties.
George Keishing5942d202021-10-04 13:09:24 -0500605 ${json_data}= redfish_utils.Enumerate Request ${enum_uri}
Steven Sombar2da175f2019-02-04 10:32:58 -0600606 # Typical output:
607 # {
608 # "@odata.id": "/redfish/v1",
609 # "@odata.type": "#ServiceRoot.v1_1_1.ServiceRoot",
610 # "AccountService": {
611 # "@odata.id": "/redfish/v1/AccountService"
612 # },
613 # "Chassis": {
614 # "@odata.id": "/redfish/v1/Chassis"
615 # },
616 # "Id": "RootService",
617 # "JsonSchemas": {
618 # "@odata.id": "/redfish/v1/JsonSchemas"
619 # },
620 # ..etc...
621 # }
622
623 @{ffdc_file_list}= Create List
George Keishing5942d202021-10-04 13:09:24 -0500624 ${logpath}= Catenate SEPARATOR= ${log_prefix_path} ${file_enum_name}
Steven Sombar2da175f2019-02-04 10:32:58 -0600625 Create File ${logpath}
626 Write Data To File "${\n}${json_data}${\n}" ${logpath}
627
628 Append To List ${ffdc_file_list} ${logpath}
629
630 [Return] ${ffdc_file_list}
631
632
George Keishing985659a2021-06-23 02:52:13 -0500633Enumerate Redfish OEM Resources
634 [Documentation] Enumerate /<oem>/v1 resources and properties to
635 ... a file. Return a list which contains the file name.
636 [Arguments] ${log_prefix_path}=${LOG_PREFIX}
637
638 # Description of argument(s):
639 # log_prefix_path The location specifying where to create FFDC file(s).
640
641 # No-op by default if input is not supplied from command line.
642 Return From Keyword If "${OEM_REDFISH_PATH}" == "${EMPTY}"
643
644 # Login is needed to fetch Redfish information.
645 # If login fails, return from keyword.
646 ${status}= Run Keyword And Return Status Redfish.Login
647 Return From Keyword If ${status} == ${False}
648
649 # Get the Redfish resources and properties.
650 ${json_data}= redfish_utils.Enumerate Request ${OEM_REDFISH_PATH}
651
652 @{ffdc_file_list}= Create List
653 ${logpath}= Catenate SEPARATOR= ${log_prefix_path}
654 ... redfish_oem_resource_properties.txt
655 Create File ${logpath}
656 Write Data To File "${\n}${json_data}${\n}" ${logpath}
657
658 Append To List ${ffdc_file_list} ${logpath}
659
660 [Return] ${ffdc_file_list}
661
662
George Keishingeb7b4bc2017-03-13 12:04:28 -0500663Collect eSEL Log
Michael Walsh02c70602018-10-04 15:24:30 -0500664 [Documentation] Create raw and formatted eSEL files.
Sridevi Rameshb180c9f2017-08-06 10:27:41 -0500665 [Arguments] ${log_prefix_path}=${LOG_PREFIX}
666
Michael Walsh02c70602018-10-04 15:24:30 -0500667 # NOTE: If no eSEL.pl program can be located, then no formatted eSEL file
668 # will be generated.
669
670 # Description of argument(s):
671 # log_prefix_path The path prefix to be used in creating
672 # eSEL file path names. For example, if
673 # log_prefix_path is
674 # "/tmp/user1/dummy.181001.120000.", then
675 # this keyword will create
676 # /tmp/user1/dummy.181001.120000.esel (raw)
677 # and
678 # /tmp/user1/dummy.181001.120000.esel.txt
679 # (formatted).
680
Michael Walsh63b70c02017-10-30 15:02:09 -0500681 @{ffdc_file_list}= Create List
682
Michael Walsh02c70602018-10-04 15:24:30 -0500683 ${esels}= Get Esels
684 ${num_esels}= Evaluate len(${esels})
685 Rprint Vars num_esels
686 Return From Keyword If ${num_esels} == ${0} ${ffdc_file_list}
George Keishingeb7b4bc2017-03-13 12:04:28 -0500687
Sridevi Rameshb180c9f2017-08-06 10:27:41 -0500688 ${logpath}= Catenate SEPARATOR= ${log_prefix_path} esel
George Keishingeb7b4bc2017-03-13 12:04:28 -0500689 Create File ${logpath}
George Keishingeb7b4bc2017-03-13 12:04:28 -0500690
Sushil Singh19888552020-06-08 02:14:52 -0500691 FOR ${esel} IN @{esels}
692 Write Data To File "${esel}"${\n} ${logpath}
693 END
George Keishingb1ca69b2017-03-31 14:36:42 -0500694
Michael Walsh63b70c02017-10-30 15:02:09 -0500695 Append To List ${ffdc_file_list} ${logpath}
Michael Walsh02c70602018-10-04 15:24:30 -0500696
697 ${rc} ${output}= Shell Cmd which eSEL.pl show_err=0
698 Return From Keyword If ${rc} != ${0} ${ffdc_file_list}
699
700 Convert eSEL To Elog Format ${logpath}
Michael Walsh0192e9b2017-11-14 15:14:15 -0600701 Append To List ${ffdc_file_list} ${logpath}.txt
Michael Walsh63b70c02017-10-30 15:02:09 -0500702
703 [Return] ${ffdc_file_list}
704
George Keishingb1ca69b2017-03-31 14:36:42 -0500705
George Keishingb1ca69b2017-03-31 14:36:42 -0500706Convert eSEL To Elog Format
707 [Documentation] Execute parser tool on the eSEL data file to generate
Michael Walsha30dac72017-04-04 17:56:23 -0500708 ... formatted error log.
George Keishingb1ca69b2017-03-31 14:36:42 -0500709 [Arguments] ${esel_file_path}
Michael Walsh02c70602018-10-04 15:24:30 -0500710 # Description of argument(s):
George Keishing7c6342b2018-11-02 14:27:13 -0500711 # esel_file_path The path to the file containing raw eSEL
Michael Walsh02c70602018-10-04 15:24:30 -0500712 # data (e.g.
713 # "/tmp/user1/dummy.181001.120000.esel").
George Keishingb1ca69b2017-03-31 14:36:42 -0500714
Michael Walsha30dac72017-04-04 17:56:23 -0500715 # Note: The only way to get eSEL.pl to put the output in a particular
716 # directory is to cd to that directory.
717 ${cmd_buf}= Catenate cd $(dirname ${esel_file_path}) ; eSEL.pl -l
718 ... ${esel_file_path} -p decode_obmc_data
Michael Walsh02c70602018-10-04 15:24:30 -0500719 Qprint Issuing ${cmd_buf}
Michael Walsha30dac72017-04-04 17:56:23 -0500720 Run ${cmd_buf}
Michael Walsh02c70602018-10-04 15:24:30 -0500721 # The .binary file, which is generated by eSEL.pl, is of no use to us.
722 Remove File ${esel_file_path}.binary
George Keishing671dd662021-05-28 03:26:30 -0500723
724
725OS Distro Type
726 [Documentation] Determine the host partition distro type
727
728 ${stdout} ${stderr} ${rc}= OS Execute Command
729 ... . /etc/os-release; echo $ID ignore_err=${1}
730
731 Return From Keyword If ${rc} == ${0} ${stdout}
732
George Keishing33d15032021-06-07 00:49:56 -0500733 # If linux distro doesn't have os-release, check for uname.
George Keishing671dd662021-05-28 03:26:30 -0500734 ${stdout} ${stderr} ${rc}= OS Execute Command uname ignore_err=${0}
735
736 [Return] ${stdout}