blob: 817ec0643e45efb932495830745f03c5de1cf156 [file] [log] [blame]
George Keishing69e6f712016-09-12 06:30:09 -05001*** Settings ***
2Documentation Methods to execute commands on BMC and collect
3... data to a list of FFDC files
4
Michael Walsh265ae412018-11-16 15:32:03 -06005Resource openbmc_ffdc_utils.robot
6Resource rest_client.robot
7Resource utils.robot
8Resource list_utils.robot
9Resource logging_utils.robot
Steven Sombar2da175f2019-02-04 10:32:58 -060010Resource bmc_redfish_resource.robot
Michael Walsh265ae412018-11-16 15:32:03 -060011Library SSHLibrary
12Library OperatingSystem
13Library Collections
14Library String
15Library gen_print.py
16Library gen_cmd.py
17Library gen_robot_keyword.py
18Library dump_utils.py
19Library logging_utils.py
Gunnar Mills7e2cda22016-10-11 15:37:34 -050020
Michael Walshde5e9372018-11-15 11:28:23 -060021*** Variables ***
22
Michael Walsh265ae412018-11-16 15:32:03 -060023${FFDC_CMD_TIMEOUT} 240
Michael Walshde5e9372018-11-15 11:28:23 -060024
George Keishing69e6f712016-09-12 06:30:09 -050025*** Keywords ***
26
George Keishing69e6f712016-09-12 06:30:09 -050027# Method : Call FFDC Methods #
28# Execute the user define keywords from the FFDC List #
29# Unlike any other keywords this will call into the #
30# list of keywords defined in the FFDC list at one go #
George Keishing69e6f712016-09-12 06:30:09 -050031
32Call FFDC Methods
Michael Walshe844e9a2017-04-20 16:51:10 -050033 [Documentation] Call into FFDC Keyword index list.
34 [Arguments] ${ffdc_function_list}=${EMPTY}
George Keishing69e6f712016-09-12 06:30:09 -050035
Michael Walshe844e9a2017-04-20 16:51:10 -050036 # Description of argument(s):
37 # ffdc_function_list A colon-delimited list naming the kinds of FFDC that
Michael Walsh63b70c02017-10-30 15:02:09 -050038 # are to be collected
Michael Walshe844e9a2017-04-20 16:51:10 -050039 # (e.g. "FFDC Generic Report:BMC Specific Files").
40 # Acceptable values can be found in the description
41 # field of FFDC_METHOD_CALL in
42 # lib/openbmc_ffdc_list.py. Those values can be
43 # obtained via a call to 'Get FFDC Method Desc' (also
44 # from lib/openbmc_ffdc_list.py).
45
46 @{entries}= Get FFDC Method Index
Michael Walsh63b70c02017-10-30 15:02:09 -050047 # Example entries:
48 # entries:
49 # entries[0]: BMC LOGS
50
51 @{ffdc_file_list}= Create List
Sushil Singh19888552020-06-08 02:14:52 -050052 FOR ${index} IN @{entries}
53 ${ffdc_file_sub_list}= Method Call Keyword List ${index} ${ffdc_function_list}
54 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
55 END
Michael Walsh63b70c02017-10-30 15:02:09 -050056
Michael Walsh6bad8412017-06-27 14:09:36 -050057 Run Key U SSHLibrary.Close All Connections
George Keishing69e6f712016-09-12 06:30:09 -050058
Michael Walsh63b70c02017-10-30 15:02:09 -050059 [Return] ${ffdc_file_list}
60
Steven Sombar2da175f2019-02-04 10:32:58 -060061
George Keishing69e6f712016-09-12 06:30:09 -050062Method Call Keyword List
Michael Walsh63b70c02017-10-30 15:02:09 -050063 [Documentation] Process FFDC request and return a list of generated files.
Michael Walshe844e9a2017-04-20 16:51:10 -050064 [Arguments] ${index} ${ffdc_function_list}=${EMPTY}
George Keishing69e6f712016-09-12 06:30:09 -050065
Michael Walshe844e9a2017-04-20 16:51:10 -050066 # Description of argument(s):
67 # index The index into the FFDC_METHOD_CALL dictionary (e.g.
68 # 'BMC LOGS').
69 # ffdc_function_list See ffdc_function_list description in
70 # "Call FFDC Methods" (above).
71
Michael Walsh63b70c02017-10-30 15:02:09 -050072 @{method_list}= Get FFDC Method Call ${index}
73 # Example method_list:
74 # method_list:
75 # method_list[0]:
76 # method_list[0][0]: FFDC Generic Report
77 # method_list[0][1]: BMC FFDC Manifest
78 # method_list[1]:
79 # method_list[1][0]: Get Request FFDC
80 # method_list[1][1]: BMC FFDC Get Requests
81 # (etc.)
Michael Walshe844e9a2017-04-20 16:51:10 -050082
83 # If function list is empty assign default (i.e. a list of all allowable
84 # values). In either case, convert ffdc_function_list from a string to
85 # a list.
86 @{ffdc_function_list}=
87 ... Run Keyword If '${ffdc_function_list}' == '${EMPTY}'
88 ... Get FFDC Method Desc ${index}
89 ... ELSE
90 ... Split String ${ffdc_function_list} separator=:
91
Michael Walsh63b70c02017-10-30 15:02:09 -050092 @{ffdc_file_list}= Create List
Sushil Singh19888552020-06-08 02:14:52 -050093 FOR ${method} IN @{method_list}
94 ${ffdc_file_sub_list}= Execute Keyword Method ${method[0]} ${method[1]} @{ffdc_function_list}
95 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
96 END
Michael Walsh63b70c02017-10-30 15:02:09 -050097
98 [Return] ${ffdc_file_list}
99
George Keishing69e6f712016-09-12 06:30:09 -0500100
101Execute Keyword Method
Michael Walshe844e9a2017-04-20 16:51:10 -0500102 [Documentation] Call into BMC method keywords. Don't let one
103 ... failure skip the remaining. Get whatever data
104 ... it could gather at worse case scenario.
105 [Arguments] ${description} ${keyword_name} @{ffdc_function_list}
George Keishing69e6f712016-09-12 06:30:09 -0500106
Michael Walshe844e9a2017-04-20 16:51:10 -0500107 # Description of argument(s):
108 # description The description of the FFDC to be collected. This
109 # would be any value returned by
110 # 'Get FFDC Method Desc' (e.g. "FFDC Generic Report").
111 # keyword_name The name of the keyword to call to collect the FFDC
112 # data (again, see FFDC_METHOD_CALL).
113 # ffdc_function_list See ffdc_function_list description in
114 # "Call FFDC Methods" (above). The only difference is
115 # in this case, it should be a list rather than a
116 # colon-delimited value.
George Keishing69e6f712016-09-12 06:30:09 -0500117
Michael Walsh63b70c02017-10-30 15:02:09 -0500118 @{ffdc_file_list}= Create List
Michael Walshe844e9a2017-04-20 16:51:10 -0500119
Michael Walsh63b70c02017-10-30 15:02:09 -0500120 ${index}= Get Index From List ${ffdc_function_list} ${description}
121 Run Keyword If '${index}' == '${-1}' Return from Keyword
122 ... ${ffdc_file_list}
123
124 ${status} ${ffdc_file_list}= Run Key ${keyword_name} ignore=1
125 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500126
George Keishing69e6f712016-09-12 06:30:09 -0500127# Method : BMC FFDC Manifest #
128# Execute command on BMC and write to ffdc_report.txt #
George Keishing69e6f712016-09-12 06:30:09 -0500129
130BMC FFDC Manifest
Michael Walsh63b70c02017-10-30 15:02:09 -0500131 [Documentation] Run the ssh commands from FFDC_BMC_CMD and return a list
132 ... of generated files.
George Keishing69e6f712016-09-12 06:30:09 -0500133
Michael Walsh63b70c02017-10-30 15:02:09 -0500134 @{ffdc_file_list}= Create List ${FFDC_FILE_PATH}
135 @{entries}= Get FFDC Cmd Index
Sushil Singh19888552020-06-08 02:14:52 -0500136
137 FOR ${index} IN @{entries}
138 Iterate BMC Command List Pairs ${index}
139 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500140
141 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500142
143
144Iterate BMC Command List Pairs
145 [Documentation] Feed in key pair list from dictionary to execute
146 [Arguments] ${key_index}
147
148 @{cmd_list}= Get ffdc bmc cmd ${key_index}
149 Set Suite Variable ${ENTRY_INDEX} ${key_index}
George Keishing69e6f712016-09-12 06:30:09 -0500150
Sushil Singh19888552020-06-08 02:14:52 -0500151 FOR ${cmd} IN @{cmd_list}
152 Execute Command and Write FFDC ${cmd[0]} ${cmd[1]}
153 END
George Keishing69e6f712016-09-12 06:30:09 -0500154
155Execute Command and Write FFDC
Michael Walsh63b70c02017-10-30 15:02:09 -0500156 [Documentation] Run a command on the BMC or OS, write the output to the
157 ... specified file and return a list of generated files.
158 [Arguments] ${key_index} ${cmd} ${logpath}=${FFDC_FILE_PATH}
159 ... ${target}=BMC
George Keishing69e6f712016-09-12 06:30:09 -0500160
Michael Walsh63b70c02017-10-30 15:02:09 -0500161 Run Keyword If '${logpath}' == '${FFDC_FILE_PATH}'
162 ... Write Cmd Output to FFDC File ${key_index} ${cmd}
163
164 @{ffdc_file_list}= Create List ${log_path}
George Keishing69e6f712016-09-12 06:30:09 -0500165
Michael Walshe53e47a2017-06-30 17:03:24 -0500166 ${cmd_buf}= Catenate ${target} Execute Command \ ${cmd} \ ignore_err=${1}
Michael Walsh265ae412018-11-16 15:32:03 -0600167 ... \ time_out=${FFDC_CMD_TIMEOUT}
Michael Walshe53e47a2017-06-30 17:03:24 -0500168 ${status} ${ret_values}= Run Key ${cmd_buf} ignore=${1}
Michael Walshde5e9372018-11-15 11:28:23 -0600169 # If the command times out, status will be 'FAIL'.
170 Return From Keyword If '${status}' == 'FAIL' ${ffdc_file_list}
Michael Walshe53e47a2017-06-30 17:03:24 -0500171
Sushil Singh19888552020-06-08 02:14:52 -0500172 ${stdout}= Set Variable ${ret_values}[0]
173 ${stderr}= Set Variable ${ret_values}[1]
George Keishing69e6f712016-09-12 06:30:09 -0500174
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600175 # Write stdout on success and stderr/stdout to the file on failure.
176 Run Keyword If $stderr == '${EMPTY}'
Sweta Potthuri15e6d2f2017-02-28 03:10:02 -0600177 ... Write Data To File ${stdout}${\n} ${logpath}
178 ... ELSE Write Data To File
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600179 ... ERROR output:${\n}${stderr}${\n}Output:${\n}${stdout}${\n}
180 ... ${logpath}
George Keishing69e6f712016-09-12 06:30:09 -0500181
Michael Walsh63b70c02017-10-30 15:02:09 -0500182 [Return] ${ffdc_file_list}
183
George Keishing69e6f712016-09-12 06:30:09 -0500184
George Keishing69e6f712016-09-12 06:30:09 -0500185# Method : BMC FFDC Files #
186# Execute command on BMC and write to individual file #
187# based on the file name pre-defined in the list #
George Keishing69e6f712016-09-12 06:30:09 -0500188
189BMC FFDC Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500190 [Documentation] Run the commands from FFDC_BMC_FILE and return a list of
191 ... generated files.
192
193 @{entries}= Get FFDC File Index
194 # Example of entries:
195 # entries:
196 # entries[0]: BMC FILES
197
Steven Sombara31f4c82019-08-01 16:00:23 -0500198 scp.Open Connection
vinaybs6b8110942020-01-29 23:15:29 -0600199 ... ${OPENBMC_HOST} username=${OPENBMC_USERNAME} password=${OPENBMC_PASSWORD} port=${SSH_PORT}
Steven Sombara31f4c82019-08-01 16:00:23 -0500200
Michael Walsh63b70c02017-10-30 15:02:09 -0500201 @{ffdc_file_list}= Create List
Sushil Singh19888552020-06-08 02:14:52 -0500202
203 FOR ${index} IN @{entries}
204 ${ffdc_file_sub_list}= Create File and Write Data ${index}
205 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
206 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500207
Michael Walsh17eaafe2019-08-22 11:05:20 -0500208 BMC Execute Command rm -rf /tmp/BMC_*
Steven Sombara31f4c82019-08-01 16:00:23 -0500209 scp.Close Connection
210
Michael Walsh63b70c02017-10-30 15:02:09 -0500211 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500212
213
214Create File and Write Data
Michael Walsh63b70c02017-10-30 15:02:09 -0500215 [Documentation] Run commands from FFDC_BMC_FILE to create FFDC files and
216 ... return a list of generated files.
217 [Arguments] ${key_index}
George Keishing69e6f712016-09-12 06:30:09 -0500218
Michael Walsh63b70c02017-10-30 15:02:09 -0500219 # Description of argument(s):
220 # key_index The index into the FFDC_BMC_FILE dictionary.
221
222 @{ffdc_file_list}= Create List
223 @{cmd_list}= Get FFDC BMC File ${key_index}
Sushil Singh19888552020-06-08 02:14:52 -0500224
225 FOR ${cmd} IN @{cmd_list}
226 ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}
227 ${ffdc_file_sub_list}= Execute Command and Write FFDC ${cmd[0]} ${cmd[1]} ${logpath}
228 Run Key U scp.Get File \ /tmp/${cmd[0]} \ ${LOG_PREFIX}${cmd[0]}
229 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
230 END
Sivas SRR0d657e82017-03-13 01:00:34 -0500231
Michael Walsh63b70c02017-10-30 15:02:09 -0500232 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500233
234
George Keishing69e6f712016-09-12 06:30:09 -0500235# Method : Log Test Case Status #
236# Creates test result history footprint for reference #
George Keishing69e6f712016-09-12 06:30:09 -0500237
238Log Test Case Status
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600239 [Documentation] Test case execution result history.
240 ... Create once and append to this file
241 ... logs/test_history.txt
242 ... Format Date:Test suite:Test case:Status
243 ... 20160909214053719992:Test Warmreset:Test WarmReset via REST:FAIL
Michael Walshfa9f70f2017-04-21 16:00:18 -0500244
245 ${FFDC_DIR_PATH_STYLE}= Get Variable Value ${FFDC_DIR_PATH_STYLE}
246 ... ${EMPTY}
247 ${FFDC_DIR_PATH}= Get Variable Value ${FFDC_DIR_PATH} ${EMPTY}
248
249 Run Keyword If '${FFDC_DIR_PATH}' == '${EMPTY}' Set FFDC Defaults
250
251 Run Keyword If '${FFDC_DIR_PATH_STYLE}' == '${1}' Run Keywords
252 ... Set Global Variable ${FFDC_LOG_PATH} ${FFDC_DIR_PATH} AND
253 ... Set Global Variable ${TEST_HISTORY} ${FFDC_DIR_PATH}test_history.txt
254
George Keishing69e6f712016-09-12 06:30:09 -0500255 Create Directory ${FFDC_LOG_PATH}
256
257 ${exist}= Run Keyword and Return Status
258 ... OperatingSystem.File Should Exist ${TEST_HISTORY}
259
260 Run Keyword If '${exist}' == '${False}'
261 ... Create File ${TEST_HISTORY}
262
Michael Walshfa9f70f2017-04-21 16:00:18 -0500263 Rpvars TEST_HISTORY
264
George Keishing69e6f712016-09-12 06:30:09 -0500265 ${cur_time}= Get Current Time Stamp
266
267 Append To File ${TEST_HISTORY}
268 ... ${cur_time}:${SUITE_NAME}:${TEST_NAME}:${TEST_STATUS}${\n}
269
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500270
271Log FFDC Get Requests
Michael Walsh63b70c02017-10-30 15:02:09 -0500272 [Documentation] Run the get requests associated with the key and return a
273 ... list of generated files.
274 [Arguments] ${key_index}
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500275
Michael Walsh63b70c02017-10-30 15:02:09 -0500276 # Note: Output will be in JSON pretty_print format.
277
278 # Description of argument(s):
279 # key_index The key to the FFDC_GET_REQUEST dictionary that contains the
280 # get requests that are to be run.
281
282 @{ffdc_file_list}= Create List
283 @{cmd_list}= Get FFDC Get Request ${key_index}
Sushil Singh19888552020-06-08 02:14:52 -0500284
285 FOR ${cmd} IN @{cmd_list}
286 ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}
George Keishingccd20e12020-07-29 11:14:26 -0500287 ${resp}= OpenBMC Get Request ${cmd[1]} quiet=${1} timeout=${30}
Sushil Singh19888552020-06-08 02:14:52 -0500288 ${status}= Run Keyword and Return Status Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
289 Run Keyword If '${status}' == '${False}' Continue For Loop
290 ${jsondata}= to json ${resp.content} pretty_print=True
291 Write Data To File ${\n}${jsondata}${\n} ${logpath}
292 Append To List ${ffdc_file_list} ${logpath}
293 END
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500294
Michael Walsh63b70c02017-10-30 15:02:09 -0500295 [Return] ${ffdc_file_list}
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500296
297BMC FFDC Get Requests
Michael Walsh63b70c02017-10-30 15:02:09 -0500298 [Documentation] Iterate over get request list and return a list of
299 ... generated files.
300
301 @{ffdc_file_list}= Create List
302
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500303 @{entries}= Get ffdc get request index
Michael Walsh63b70c02017-10-30 15:02:09 -0500304 # Example of entries:
305 # entries:
306 # entries[0]: GET REQUESTS
Sushil Singh19888552020-06-08 02:14:52 -0500307
308 FOR ${index} IN @{entries}
309 ${ffdc_file_sub_list}= Log FFDC Get Requests ${index}
310 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
311 END
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500312
Michael Walsh63b70c02017-10-30 15:02:09 -0500313 [Return] ${ffdc_file_list}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500314
Michael Walsh63b70c02017-10-30 15:02:09 -0500315Log OS All distros FFDC
316 [Documentation] Run commands from FFDC_OS_ALL_DISTROS_FILE to create FFDC
317 ... files and return a list of generated files.
318 [Arguments] ${key_index}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500319
Michael Walsh63b70c02017-10-30 15:02:09 -0500320 # Description of argument(s):
321 # key_index The index into the FFDC_OS_ALL_DISTROS_FILE dictionary.
322
323 @{ffdc_file_list}= Create List
324
325 @{cmd_list}= Get FFDC OS All Distros Call ${key_index}
Sushil Singh19888552020-06-08 02:14:52 -0500326
327 FOR ${cmd} IN @{cmd_list}
328 ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}
329 ${ffdc_file_sub_list}= Execute Command and Write FFDC ${cmd[0]} ${cmd[1]} ${logpath} target=OS
330 # scp it to the LOG_PREFIX ffdc directory.
331 Run Key U scp.Get File \ /tmp/${cmd[0]} \ ${LOG_PREFIX}${cmd[0]}
332 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
333 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500334
335 [Return] ${ffdc_file_list}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500336
337
Gunnar Millscce185d2016-10-17 17:04:15 -0500338Log OS SPECIFIC DISTRO FFDC
Michael Walsh63b70c02017-10-30 15:02:09 -0500339 [Documentation] Run commands from the FFDC_OS_<distro>_FILE to create FFDC
340 ... files and return a list of generated files.
341 [Arguments] ${key_index} ${linux_distro}
Gunnar Millscce185d2016-10-17 17:04:15 -0500342
Michael Walsh63b70c02017-10-30 15:02:09 -0500343 # Description of argument(s):
344 # key_index The index into the FFDC_OS_<distro>_FILE dictionary.
345 # linux_distro Your OS's linux distro (e.g. "UBUNTU", "RHEL", etc).
346
347 @{ffdc_file_list}= Create List
348
349 @{cmd_list}= Get FFDC OS Distro Call ${key_index} ${linux_distro}
Sushil Singh19888552020-06-08 02:14:52 -0500350
351 FOR ${cmd} IN @{cmd_list}
352 ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}
353 ${ffdc_file_sub_list}= Execute Command and Write FFDC ${cmd[0]} ${cmd[1]} ${logpath} target=OS
354 Run Key U scp.Get File \ /tmp/${cmd[0]} \ ${LOG_PREFIX}${cmd[0]}
355 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
Michael Walsh63b70c02017-10-30 15:02:09 -0500356
Steven Sombare70acbe2019-08-20 14:37:47 -0500357 # Get the name of the sosreport file.
358 ${sosreport_file_path} ${stderr} ${rc}= OS Execute Command
359 ... ls /tmp/sosreport*FFDC*tar.xz ignore_err=${True}
360 # Example: sosreport_file_path="/tmp/sosreport-myhost-FFDC-2019-08-20-pbuaqtk.tar.xz".
361
362 # Return if there is no sosreport file.
363 Return From Keyword If ${rc} != ${0} ${ffdc_file_list}
364
365 ${sosreport_dir_path} ${sosreport_file_name}= Split Path ${sosreport_file_path}
366 # Example: sosreport_dir_path="/tmp",
367 # sosreport_file_name="sosreport-myhost-FFDC-2019-08-20-pbuaqtk.tar.xz".
368
369 # Location where the sosreport file will be copied to.
370 ${local_sosreport_file_path}= Set Variable ${LOG_PREFIX}OS_${sosreport_file_name}
371
372 # Change file permissions otherwise scp will not see the file.
373 OS Execute Command chmod 644 ${sosreport_file_path}
374
375 # SCP the sosreport file from the OS.
376 Run Key U scp.Get File \ ${sosreport_file_path} \ ${local_sosreport_file_path}
377
378 # Add the file location to the ffdc_file_list.
379 Append To List ${ffdc_file_list} ${local_sosreport_file_path}
380
Michael Walsh63b70c02017-10-30 15:02:09 -0500381 [Return] ${ffdc_file_list}
Gunnar Millscce185d2016-10-17 17:04:15 -0500382
383
George Keishingca8c4c62016-10-14 10:08:40 -0500384OS FFDC Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500385 [Documentation] Run the commands from FFDC_OS_ALL_DISTROS_FILE to create
386 ... FFDC files and return a list of generated files.
Gunnar Millsaca140d2016-10-26 13:05:10 -0500387 [Arguments] ${OS_HOST}=${OS_HOST} ${OS_USERNAME}=${OS_USERNAME}
Michael Walsh63b70c02017-10-30 15:02:09 -0500388 ... ${OS_PASSWORD}=${OS_PASSWORD}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500389
Michael Walsh63b70c02017-10-30 15:02:09 -0500390 @{ffdc_file_list}= Create List
Gunnar Millsaca140d2016-10-26 13:05:10 -0500391
Michael Walsh63b70c02017-10-30 15:02:09 -0500392 Run Keyword If '${OS_HOST}' == '${EMPTY}' Run Keywords
393 ... Print Timen No OS Host provided so no OS FFDC will be done. AND
394 ... Return From Keyword ${ffdc_file_list}
Gunnar Millsaca140d2016-10-26 13:05:10 -0500395
Michael Walsh63b70c02017-10-30 15:02:09 -0500396 ${match_state}= Create Dictionary os_ping=^1$ os_login=^1$
397 ... os_run_cmd=^1$
398 ${status} ${ret_values}= Run Keyword and Ignore Error Check State
399 ... ${match_state} quiet=0
400
401 Run Keyword If '${status}' == 'FAIL' Run Keywords
Michael Walshc3b28d72017-11-16 10:43:14 -0600402 ... Print Timen The OS is not communicating so no OS FFDC will be done.\n
Michael Walsh63b70c02017-10-30 15:02:09 -0500403 ... AND
404 ... Return From Keyword ${ffdc_file_list}
Gunnar Millscce185d2016-10-17 17:04:15 -0500405
Michael Walshe53e47a2017-06-30 17:03:24 -0500406 ${stdout} ${stderr} ${rc}= OS Execute Command
407 ... . /etc/os-release; echo $ID ignore_err=${0}
408 Set Global Variable ${linux_distro} ${stdout}
Gunnar Millsaca140d2016-10-26 13:05:10 -0500409
Michael Walshe53e47a2017-06-30 17:03:24 -0500410 Rpvars linux_distro
Gunnar Millsaca140d2016-10-26 13:05:10 -0500411
Steven Sombarfc024cb2019-04-22 11:16:16 -0500412 scp.Open Connection
413 ... ${OS_HOST} username=${OS_USERNAME} password=${OS_PASSWORD}
414
Michael Walsh63b70c02017-10-30 15:02:09 -0500415 @{entries}= Get FFDC OS All Distros Index
Sushil Singh19888552020-06-08 02:14:52 -0500416
417 FOR ${index} IN @{entries}
418 ${ffdc_file_sub_list}= Log OS All distros FFDC ${index}
419 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
420 END
Gunnar Millscce185d2016-10-17 17:04:15 -0500421
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600422 Return From Keyword If
423 ... '${linux_distro}' == '${EMPTY}' or '${linux_distro}' == 'None'
Michael Walsh63b70c02017-10-30 15:02:09 -0500424 ... ${ffdc_file_list}
Gunnar Millscce185d2016-10-17 17:04:15 -0500425
426 @{entries}= Get ffdc os distro index ${linux_distro}
Sushil Singh19888552020-06-08 02:14:52 -0500427
428 FOR ${index} IN @{entries}
429 ${ffdc_file_sub_list}= Log OS SPECIFIC DISTRO FFDC ${index} ${linux_distro}
430 ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list} ${ffdc_file_sub_list}
431 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500432
Steven Sombarfc024cb2019-04-22 11:16:16 -0500433 # Delete ffdc files still on OS and close scp.
Steven Sombare70acbe2019-08-20 14:37:47 -0500434 OS Execute Command rm -rf /tmp/OS_* /tmp/sosreport*FFDC* ignore_err=${True}
Steven Sombarfc024cb2019-04-22 11:16:16 -0500435 scp.Close Connection
436
Michael Walsh63b70c02017-10-30 15:02:09 -0500437 [Return] ${ffdc_file_list}
George Keishing7a520222017-02-27 09:44:30 -0600438
439
Steven Sombar96de5732017-08-17 09:04:52 -0500440System Inventory Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500441 [Documentation] Copy systest os_inventory files and return a list of
442 ... generated files..
Steven Sombar96de5732017-08-17 09:04:52 -0500443 # The os_inventory files are the result of running
444 # systest/htx_hardbootme_test. If these files exist
445 # they are copied to the FFDC directory.
446 # Global variable ffdc_dir_path is the path name of the
447 # directory they are copied to.
Michael Walsh63b70c02017-10-30 15:02:09 -0500448
449 @{ffdc_file_list}= Create List
450
451 ${globex}= Set Variable os_inventory_*.json
452
453 @{file_list}= OperatingSystem.List Files In Directory . ${globex}
454
455 Copy Files ${globex} ${ffdc_dir_path}
456
Sushil Singh19888552020-06-08 02:14:52 -0500457 FOR ${file_name} IN @{file_list}
458 Append To List ${ffdc_file_list} ${ffdc_dir_path}${file_name}
459 END
Michael Walsh63b70c02017-10-30 15:02:09 -0500460
461 Run Keyword and Ignore Error Remove Files ${globex}
462
463 [Return] ${ffdc_file_list}
Steven Sombar96de5732017-08-17 09:04:52 -0500464
465
George Keishing7a520222017-02-27 09:44:30 -0600466SCP Coredump Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500467 [Documentation] Copy core dump files from BMC to local system and return a
468 ... list of generated file names.
469
470 @{ffdc_file_list}= Create List
Michael Walshe53e47a2017-06-30 17:03:24 -0500471
George Keishing7a520222017-02-27 09:44:30 -0600472 # Check if core dump exist in the /tmp
Michael Walshe53e47a2017-06-30 17:03:24 -0500473 ${core_files} ${stderr} ${rc}= BMC Execute Command ls /tmp/core_*
Michael Walsh63b70c02017-10-30 15:02:09 -0500474 ... ignore_err=${1}
475 Run Keyword If '${rc}' != '${0}' Return From Keyword ${ffdc_file_list}
476
477 @{core_list}= Split String ${core_files}
George Keishing7a520222017-02-27 09:44:30 -0600478 # Copy the core files
Michael Walsh6bad8412017-06-27 14:09:36 -0500479 Run Key U Open Connection for SCP
Sushil Singh19888552020-06-08 02:14:52 -0500480
481 FOR ${index} IN @{core_list}
482 ${ffdc_file_path}= Catenate ${LOG_PREFIX}${index.lstrip("/tmp/")}
483 ${status}= Run Keyword and Return Status scp.Get File ${index} ${ffdc_file_path}
484 Run Keyword If '${status}' == '${False}' Continue For Loop
485 Append To List ${ffdc_file_list} ${ffdc_file_path}
486
487 # Remove the file from remote to avoid re-copying on next FFDC call
488
489 BMC Execute Command rm ${index} ignore_err=${1}
490 # I can't find a way to do this: scp.Close Connection
491
492 END
George Keishing7a520222017-02-27 09:44:30 -0600493
Michael Walsh63b70c02017-10-30 15:02:09 -0500494 [Return] ${ffdc_file_list}
495
Steven Sombar2da175f2019-02-04 10:32:58 -0600496
Sweta Potthuria82efd62017-10-18 05:34:59 -0500497SCP Dump Files
498 [Documentation] Copy all dump files from BMC to local system.
499
500 # Check if dumps exist
501 ${ffdc_file_list}= Scp Dumps ${FFDC_DIR_PATH} ${FFDC_PREFIX}
502
503 [Return] ${ffdc_file_list}
504
Steven Sombar2da175f2019-02-04 10:32:58 -0600505
Sweta Potthuria82efd62017-10-18 05:34:59 -0500506Collect Dump Log
507 [Documentation] Collect dumps from dump entry.
508 [Arguments] ${log_prefix_path}=${LOG_PREFIX}
509
Rahul Maheshwaria89ff9e2020-09-25 05:04:33 -0500510 ${resp}= OpenBMC Get Request ${DUMP_URI}
511 Run Keyword If '${resp.status_code}' == '${HTTP_NOT_FOUND}'
512 ... Set Test Variable ${DUMP_ENTRY_URI} /xyz/openbmc_project/dump/entry/
513
George Keishingccd20e12020-07-29 11:14:26 -0500514 ${data}= Read Properties ${DUMP_ENTRY_URI}enumerate quiet=${1} timeout=${30}
Sweta Potthuria82efd62017-10-18 05:34:59 -0500515
516 # Grab the list of entries from dump/entry/
517 # The data shown below is the result of the "Get Dictionary Keys".
518 # Example:
519 # /xyz/openbmc_project/dump/entry/1
520 # /xyz/openbmc_project/dump/entry/2
521
522 ${dump_list}= Get Dictionary Keys ${data}
George Keishingeb7b4bc2017-03-13 12:04:28 -0500523
Michael Walsh02c70602018-10-04 15:24:30 -0500524
George Keishingbe396b82020-03-23 10:23:58 -0500525Collect PEL Log
526 [Documentation] Collect PEL files from from BMC.
527
528 Create Directory ${FFDC_DIR_PATH}${/}pel_files/
529 scp.Get File /var/lib/phosphor-logging/extensions/pels/logs/
530 ... ${FFDC_DIR_PATH}${/}pel_files recursive=True
531
532
Steven Sombar2da175f2019-02-04 10:32:58 -0600533Enumerate Redfish Resources
534 [Documentation] Enumerate /redfish/v1 resources and properties to
535 ... a file. Return a list which contains the file name.
536 [Arguments] ${log_prefix_path}=${LOG_PREFIX}
537
538 # Description of argument(s):
539 # log_prefix_path The location specifying where to create FFDC file(s).
540
541 # Login is needed to fetch Redfish information.
542 # If login fails, return from keyword.
George Keishing97c93942019-03-04 12:45:07 -0600543 ${status}= Run Keyword And Return Status Redfish.Login
Steven Sombar2da175f2019-02-04 10:32:58 -0600544 Return From Keyword If ${status} == ${False}
545
546 # Get the Redfish resources and properties.
George Keishingf2613b72019-02-13 12:45:59 -0600547 ${json_data}= redfish_utils.Enumerate Request /redfish/v1
Steven Sombar2da175f2019-02-04 10:32:58 -0600548 # Typical output:
549 # {
550 # "@odata.id": "/redfish/v1",
551 # "@odata.type": "#ServiceRoot.v1_1_1.ServiceRoot",
552 # "AccountService": {
553 # "@odata.id": "/redfish/v1/AccountService"
554 # },
555 # "Chassis": {
556 # "@odata.id": "/redfish/v1/Chassis"
557 # },
558 # "Id": "RootService",
559 # "JsonSchemas": {
560 # "@odata.id": "/redfish/v1/JsonSchemas"
561 # },
562 # ..etc...
563 # }
564
565 @{ffdc_file_list}= Create List
566 ${logpath}= Catenate SEPARATOR= ${log_prefix_path}
567 ... redfish_resource_properties.txt
568 Create File ${logpath}
569 Write Data To File "${\n}${json_data}${\n}" ${logpath}
570
571 Append To List ${ffdc_file_list} ${logpath}
572
573 [Return] ${ffdc_file_list}
574
575
George Keishingeb7b4bc2017-03-13 12:04:28 -0500576Collect eSEL Log
Michael Walsh02c70602018-10-04 15:24:30 -0500577 [Documentation] Create raw and formatted eSEL files.
Sridevi Rameshb180c9f2017-08-06 10:27:41 -0500578 [Arguments] ${log_prefix_path}=${LOG_PREFIX}
579
Michael Walsh02c70602018-10-04 15:24:30 -0500580 # NOTE: If no eSEL.pl program can be located, then no formatted eSEL file
581 # will be generated.
582
583 # Description of argument(s):
584 # log_prefix_path The path prefix to be used in creating
585 # eSEL file path names. For example, if
586 # log_prefix_path is
587 # "/tmp/user1/dummy.181001.120000.", then
588 # this keyword will create
589 # /tmp/user1/dummy.181001.120000.esel (raw)
590 # and
591 # /tmp/user1/dummy.181001.120000.esel.txt
592 # (formatted).
593
Michael Walsh63b70c02017-10-30 15:02:09 -0500594 @{ffdc_file_list}= Create List
595
Michael Walsh02c70602018-10-04 15:24:30 -0500596 ${esels}= Get Esels
597 ${num_esels}= Evaluate len(${esels})
598 Rprint Vars num_esels
599 Return From Keyword If ${num_esels} == ${0} ${ffdc_file_list}
George Keishingeb7b4bc2017-03-13 12:04:28 -0500600
Sridevi Rameshb180c9f2017-08-06 10:27:41 -0500601 ${logpath}= Catenate SEPARATOR= ${log_prefix_path} esel
George Keishingeb7b4bc2017-03-13 12:04:28 -0500602 Create File ${logpath}
George Keishingeb7b4bc2017-03-13 12:04:28 -0500603
Sushil Singh19888552020-06-08 02:14:52 -0500604 FOR ${esel} IN @{esels}
605 Write Data To File "${esel}"${\n} ${logpath}
606 END
George Keishingb1ca69b2017-03-31 14:36:42 -0500607
Michael Walsh63b70c02017-10-30 15:02:09 -0500608 Append To List ${ffdc_file_list} ${logpath}
Michael Walsh02c70602018-10-04 15:24:30 -0500609
610 ${rc} ${output}= Shell Cmd which eSEL.pl show_err=0
611 Return From Keyword If ${rc} != ${0} ${ffdc_file_list}
612
613 Convert eSEL To Elog Format ${logpath}
Michael Walsh0192e9b2017-11-14 15:14:15 -0600614 Append To List ${ffdc_file_list} ${logpath}.txt
Michael Walsh63b70c02017-10-30 15:02:09 -0500615
616 [Return] ${ffdc_file_list}
617
George Keishingb1ca69b2017-03-31 14:36:42 -0500618
George Keishingb1ca69b2017-03-31 14:36:42 -0500619Convert eSEL To Elog Format
620 [Documentation] Execute parser tool on the eSEL data file to generate
Michael Walsha30dac72017-04-04 17:56:23 -0500621 ... formatted error log.
George Keishingb1ca69b2017-03-31 14:36:42 -0500622 [Arguments] ${esel_file_path}
Michael Walsh02c70602018-10-04 15:24:30 -0500623 # Description of argument(s):
George Keishing7c6342b2018-11-02 14:27:13 -0500624 # esel_file_path The path to the file containing raw eSEL
Michael Walsh02c70602018-10-04 15:24:30 -0500625 # data (e.g.
626 # "/tmp/user1/dummy.181001.120000.esel").
George Keishingb1ca69b2017-03-31 14:36:42 -0500627
Michael Walsha30dac72017-04-04 17:56:23 -0500628 # Note: The only way to get eSEL.pl to put the output in a particular
629 # directory is to cd to that directory.
630 ${cmd_buf}= Catenate cd $(dirname ${esel_file_path}) ; eSEL.pl -l
631 ... ${esel_file_path} -p decode_obmc_data
Michael Walsh02c70602018-10-04 15:24:30 -0500632 Qprint Issuing ${cmd_buf}
Michael Walsha30dac72017-04-04 17:56:23 -0500633 Run ${cmd_buf}
Michael Walsh02c70602018-10-04 15:24:30 -0500634 # The .binary file, which is generated by eSEL.pl, is of no use to us.
635 Remove File ${esel_file_path}.binary