blob: fa2fc0431cdcfe3b6daca1e2fb38c4b99908bcf2 [file] [log] [blame]
George Keishing69e6f712016-09-12 06:30:09 -05001*** Settings ***
2Documentation Methods to execute commands on BMC and collect
3... data to a list of FFDC files
4
Michael Walsh265ae412018-11-16 15:32:03 -06005Resource openbmc_ffdc_utils.robot
6Resource rest_client.robot
7Resource utils.robot
8Resource list_utils.robot
9Resource logging_utils.robot
Steven Sombar2da175f2019-02-04 10:32:58 -060010Resource bmc_redfish_resource.robot
Michael Walsh265ae412018-11-16 15:32:03 -060011Library SSHLibrary
12Library OperatingSystem
13Library Collections
14Library String
15Library gen_print.py
16Library gen_cmd.py
17Library gen_robot_keyword.py
18Library dump_utils.py
19Library logging_utils.py
Gunnar Mills7e2cda22016-10-11 15:37:34 -050020
Michael Walshde5e9372018-11-15 11:28:23 -060021*** Variables ***
22
Michael Walsh265ae412018-11-16 15:32:03 -060023${FFDC_CMD_TIMEOUT} 240
Michael Walshde5e9372018-11-15 11:28:23 -060024
George Keishing69e6f712016-09-12 06:30:09 -050025*** Keywords ***
26
George Keishing69e6f712016-09-12 06:30:09 -050027# Method : Call FFDC Methods #
28# Execute the user define keywords from the FFDC List #
29# Unlike any other keywords this will call into the #
30# list of keywords defined in the FFDC list at one go #
George Keishing69e6f712016-09-12 06:30:09 -050031
32Call FFDC Methods
Michael Walshe844e9a2017-04-20 16:51:10 -050033 [Documentation] Call into FFDC Keyword index list.
34 [Arguments] ${ffdc_function_list}=${EMPTY}
George Keishing69e6f712016-09-12 06:30:09 -050035
Michael Walshe844e9a2017-04-20 16:51:10 -050036 # Description of argument(s):
37 # ffdc_function_list A colon-delimited list naming the kinds of FFDC that
Michael Walsh63b70c02017-10-30 15:02:09 -050038 # are to be collected
Michael Walshe844e9a2017-04-20 16:51:10 -050039 # (e.g. "FFDC Generic Report:BMC Specific Files").
40 # Acceptable values can be found in the description
41 # field of FFDC_METHOD_CALL in
42 # lib/openbmc_ffdc_list.py. Those values can be
43 # obtained via a call to 'Get FFDC Method Desc' (also
44 # from lib/openbmc_ffdc_list.py).
45
46 @{entries}= Get FFDC Method Index
Michael Walsh63b70c02017-10-30 15:02:09 -050047 # Example entries:
48 # entries:
49 # entries[0]: BMC LOGS
50
51 @{ffdc_file_list}= Create List
Michael Walshe844e9a2017-04-20 16:51:10 -050052 :FOR ${index} IN @{entries}
Michael Walsh63b70c02017-10-30 15:02:09 -050053 \ ${ffdc_file_sub_list}= Method Call Keyword List ${index}
54 ... ${ffdc_function_list}
55 \ ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list}
56 ... ${ffdc_file_sub_list}
57
Michael Walsh6bad8412017-06-27 14:09:36 -050058 Run Key U SSHLibrary.Close All Connections
George Keishing69e6f712016-09-12 06:30:09 -050059
Michael Walsh63b70c02017-10-30 15:02:09 -050060 [Return] ${ffdc_file_list}
61
Steven Sombar2da175f2019-02-04 10:32:58 -060062
George Keishing69e6f712016-09-12 06:30:09 -050063Method Call Keyword List
Michael Walsh63b70c02017-10-30 15:02:09 -050064 [Documentation] Process FFDC request and return a list of generated files.
Michael Walshe844e9a2017-04-20 16:51:10 -050065 [Arguments] ${index} ${ffdc_function_list}=${EMPTY}
George Keishing69e6f712016-09-12 06:30:09 -050066
Michael Walshe844e9a2017-04-20 16:51:10 -050067 # Description of argument(s):
68 # index The index into the FFDC_METHOD_CALL dictionary (e.g.
69 # 'BMC LOGS').
70 # ffdc_function_list See ffdc_function_list description in
71 # "Call FFDC Methods" (above).
72
Michael Walsh63b70c02017-10-30 15:02:09 -050073 @{method_list}= Get FFDC Method Call ${index}
74 # Example method_list:
75 # method_list:
76 # method_list[0]:
77 # method_list[0][0]: FFDC Generic Report
78 # method_list[0][1]: BMC FFDC Manifest
79 # method_list[1]:
80 # method_list[1][0]: Get Request FFDC
81 # method_list[1][1]: BMC FFDC Get Requests
82 # (etc.)
Michael Walshe844e9a2017-04-20 16:51:10 -050083
84 # If function list is empty assign default (i.e. a list of all allowable
85 # values). In either case, convert ffdc_function_list from a string to
86 # a list.
87 @{ffdc_function_list}=
88 ... Run Keyword If '${ffdc_function_list}' == '${EMPTY}'
89 ... Get FFDC Method Desc ${index}
90 ... ELSE
91 ... Split String ${ffdc_function_list} separator=:
92
Michael Walsh63b70c02017-10-30 15:02:09 -050093 @{ffdc_file_list}= Create List
George Keishing69e6f712016-09-12 06:30:09 -050094 :FOR ${method} IN @{method_list}
Michael Walsh63b70c02017-10-30 15:02:09 -050095 \ ${ffdc_file_sub_list}= Execute Keyword Method ${method[0]}
96 ... ${method[1]} @{ffdc_function_list}
97 \ ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list}
98 ... ${ffdc_file_sub_list}
99
100 [Return] ${ffdc_file_list}
101
George Keishing69e6f712016-09-12 06:30:09 -0500102
103Execute Keyword Method
Michael Walshe844e9a2017-04-20 16:51:10 -0500104 [Documentation] Call into BMC method keywords. Don't let one
105 ... failure skip the remaining. Get whatever data
106 ... it could gather at worse case scenario.
107 [Arguments] ${description} ${keyword_name} @{ffdc_function_list}
George Keishing69e6f712016-09-12 06:30:09 -0500108
Michael Walshe844e9a2017-04-20 16:51:10 -0500109 # Description of argument(s):
110 # description The description of the FFDC to be collected. This
111 # would be any value returned by
112 # 'Get FFDC Method Desc' (e.g. "FFDC Generic Report").
113 # keyword_name The name of the keyword to call to collect the FFDC
114 # data (again, see FFDC_METHOD_CALL).
115 # ffdc_function_list See ffdc_function_list description in
116 # "Call FFDC Methods" (above). The only difference is
117 # in this case, it should be a list rather than a
118 # colon-delimited value.
George Keishing69e6f712016-09-12 06:30:09 -0500119
Michael Walsh63b70c02017-10-30 15:02:09 -0500120 @{ffdc_file_list}= Create List
Michael Walshe844e9a2017-04-20 16:51:10 -0500121
Michael Walsh63b70c02017-10-30 15:02:09 -0500122 ${index}= Get Index From List ${ffdc_function_list} ${description}
123 Run Keyword If '${index}' == '${-1}' Return from Keyword
124 ... ${ffdc_file_list}
125
126 ${status} ${ffdc_file_list}= Run Key ${keyword_name} ignore=1
127 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500128
George Keishing69e6f712016-09-12 06:30:09 -0500129# Method : BMC FFDC Manifest #
130# Execute command on BMC and write to ffdc_report.txt #
George Keishing69e6f712016-09-12 06:30:09 -0500131
132BMC FFDC Manifest
Michael Walsh63b70c02017-10-30 15:02:09 -0500133 [Documentation] Run the ssh commands from FFDC_BMC_CMD and return a list
134 ... of generated files.
George Keishing69e6f712016-09-12 06:30:09 -0500135
Michael Walsh63b70c02017-10-30 15:02:09 -0500136 @{ffdc_file_list}= Create List ${FFDC_FILE_PATH}
137 @{entries}= Get FFDC Cmd Index
138 :FOR ${index} IN @{entries}
139 \ Iterate BMC Command List Pairs ${index}
140
141 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500142
143
144Iterate BMC Command List Pairs
145 [Documentation] Feed in key pair list from dictionary to execute
146 [Arguments] ${key_index}
147
148 @{cmd_list}= Get ffdc bmc cmd ${key_index}
149 Set Suite Variable ${ENTRY_INDEX} ${key_index}
150 :FOR ${cmd} IN @{cmd_list}
151 \ Execute Command and Write FFDC ${cmd[0]} ${cmd[1]}
152
153
154Execute Command and Write FFDC
Michael Walsh63b70c02017-10-30 15:02:09 -0500155 [Documentation] Run a command on the BMC or OS, write the output to the
156 ... specified file and return a list of generated files.
157 [Arguments] ${key_index} ${cmd} ${logpath}=${FFDC_FILE_PATH}
158 ... ${target}=BMC
George Keishing69e6f712016-09-12 06:30:09 -0500159
Michael Walsh63b70c02017-10-30 15:02:09 -0500160 Run Keyword If '${logpath}' == '${FFDC_FILE_PATH}'
161 ... Write Cmd Output to FFDC File ${key_index} ${cmd}
162
163 @{ffdc_file_list}= Create List ${log_path}
George Keishing69e6f712016-09-12 06:30:09 -0500164
Michael Walshe53e47a2017-06-30 17:03:24 -0500165 ${cmd_buf}= Catenate ${target} Execute Command \ ${cmd} \ ignore_err=${1}
Michael Walsh265ae412018-11-16 15:32:03 -0600166 ... \ time_out=${FFDC_CMD_TIMEOUT}
Michael Walshe53e47a2017-06-30 17:03:24 -0500167 ${status} ${ret_values}= Run Key ${cmd_buf} ignore=${1}
Michael Walshde5e9372018-11-15 11:28:23 -0600168 # If the command times out, status will be 'FAIL'.
169 Return From Keyword If '${status}' == 'FAIL' ${ffdc_file_list}
Michael Walshe53e47a2017-06-30 17:03:24 -0500170
171 ${stdout}= Set Variable @{ret_values}[0]
172 ${stderr}= Set Variable @{ret_values}[1]
George Keishing69e6f712016-09-12 06:30:09 -0500173
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600174 # Write stdout on success and stderr/stdout to the file on failure.
175 Run Keyword If $stderr == '${EMPTY}'
Sweta Potthuri15e6d2f2017-02-28 03:10:02 -0600176 ... Write Data To File ${stdout}${\n} ${logpath}
177 ... ELSE Write Data To File
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600178 ... ERROR output:${\n}${stderr}${\n}Output:${\n}${stdout}${\n}
179 ... ${logpath}
George Keishing69e6f712016-09-12 06:30:09 -0500180
Michael Walsh63b70c02017-10-30 15:02:09 -0500181 [Return] ${ffdc_file_list}
182
George Keishing69e6f712016-09-12 06:30:09 -0500183
George Keishing69e6f712016-09-12 06:30:09 -0500184# Method : BMC FFDC Files #
185# Execute command on BMC and write to individual file #
186# based on the file name pre-defined in the list #
George Keishing69e6f712016-09-12 06:30:09 -0500187
188BMC FFDC Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500189 [Documentation] Run the commands from FFDC_BMC_FILE and return a list of
190 ... generated files.
191
192 @{entries}= Get FFDC File Index
193 # Example of entries:
194 # entries:
195 # entries[0]: BMC FILES
196
Steven Sombara31f4c82019-08-01 16:00:23 -0500197 scp.Open Connection
198 ... ${OPENBMC_HOST} username=${OPENBMC_USERNAME} password=${OPENBMC_PASSWORD}
199
Michael Walsh63b70c02017-10-30 15:02:09 -0500200 @{ffdc_file_list}= Create List
201 :FOR ${index} IN @{entries}
202 \ ${ffdc_file_sub_list}= Create File and Write Data ${index}
203 \ ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list}
204 ... ${ffdc_file_sub_list}
205
Michael Walsh17eaafe2019-08-22 11:05:20 -0500206 BMC Execute Command rm -rf /tmp/BMC_*
Steven Sombara31f4c82019-08-01 16:00:23 -0500207 scp.Close Connection
208
Michael Walsh63b70c02017-10-30 15:02:09 -0500209 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500210
211
212Create File and Write Data
Michael Walsh63b70c02017-10-30 15:02:09 -0500213 [Documentation] Run commands from FFDC_BMC_FILE to create FFDC files and
214 ... return a list of generated files.
215 [Arguments] ${key_index}
George Keishing69e6f712016-09-12 06:30:09 -0500216
Michael Walsh63b70c02017-10-30 15:02:09 -0500217 # Description of argument(s):
218 # key_index The index into the FFDC_BMC_FILE dictionary.
219
220 @{ffdc_file_list}= Create List
221 @{cmd_list}= Get FFDC BMC File ${key_index}
George Keishing69e6f712016-09-12 06:30:09 -0500222 :FOR ${cmd} IN @{cmd_list}
Michael Walsh63b70c02017-10-30 15:02:09 -0500223 \ ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}.txt
224 \ ${ffdc_file_sub_list}= Execute Command and Write FFDC ${cmd[0]}
225 ... ${cmd[1]} ${logpath}
Steven Sombara31f4c82019-08-01 16:00:23 -0500226 \ Run Key U scp.Get File \ /tmp/${cmd[0]}.txt \ ${LOG_PREFIX}${cmd[0]}.txt
Michael Walsh63b70c02017-10-30 15:02:09 -0500227 \ ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list}
228 ... ${ffdc_file_sub_list}
Sivas SRR0d657e82017-03-13 01:00:34 -0500229
Michael Walsh63b70c02017-10-30 15:02:09 -0500230 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500231
232
George Keishing69e6f712016-09-12 06:30:09 -0500233# Method : Log Test Case Status #
234# Creates test result history footprint for reference #
George Keishing69e6f712016-09-12 06:30:09 -0500235
236Log Test Case Status
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600237 [Documentation] Test case execution result history.
238 ... Create once and append to this file
239 ... logs/test_history.txt
240 ... Format Date:Test suite:Test case:Status
241 ... 20160909214053719992:Test Warmreset:Test WarmReset via REST:FAIL
Michael Walshfa9f70f2017-04-21 16:00:18 -0500242
243 ${FFDC_DIR_PATH_STYLE}= Get Variable Value ${FFDC_DIR_PATH_STYLE}
244 ... ${EMPTY}
245 ${FFDC_DIR_PATH}= Get Variable Value ${FFDC_DIR_PATH} ${EMPTY}
246
247 Run Keyword If '${FFDC_DIR_PATH}' == '${EMPTY}' Set FFDC Defaults
248
249 Run Keyword If '${FFDC_DIR_PATH_STYLE}' == '${1}' Run Keywords
250 ... Set Global Variable ${FFDC_LOG_PATH} ${FFDC_DIR_PATH} AND
251 ... Set Global Variable ${TEST_HISTORY} ${FFDC_DIR_PATH}test_history.txt
252
George Keishing69e6f712016-09-12 06:30:09 -0500253 Create Directory ${FFDC_LOG_PATH}
254
255 ${exist}= Run Keyword and Return Status
256 ... OperatingSystem.File Should Exist ${TEST_HISTORY}
257
258 Run Keyword If '${exist}' == '${False}'
259 ... Create File ${TEST_HISTORY}
260
Michael Walshfa9f70f2017-04-21 16:00:18 -0500261 Rpvars TEST_HISTORY
262
George Keishing69e6f712016-09-12 06:30:09 -0500263 ${cur_time}= Get Current Time Stamp
264
265 Append To File ${TEST_HISTORY}
266 ... ${cur_time}:${SUITE_NAME}:${TEST_NAME}:${TEST_STATUS}${\n}
267
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500268
269Log FFDC Get Requests
Michael Walsh63b70c02017-10-30 15:02:09 -0500270 [Documentation] Run the get requests associated with the key and return a
271 ... list of generated files.
272 [Arguments] ${key_index}
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500273
Michael Walsh63b70c02017-10-30 15:02:09 -0500274 # Note: Output will be in JSON pretty_print format.
275
276 # Description of argument(s):
277 # key_index The key to the FFDC_GET_REQUEST dictionary that contains the
278 # get requests that are to be run.
279
280 @{ffdc_file_list}= Create List
281 @{cmd_list}= Get FFDC Get Request ${key_index}
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500282 :FOR ${cmd} IN @{cmd_list}
Michael Walsh63b70c02017-10-30 15:02:09 -0500283 \ ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}.txt
284 \ ${resp}= OpenBMC Get Request ${cmd[1]} quiet=${1}
285 \ ${status}= Run Keyword and Return Status
286 ... Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
287 \ Run Keyword If '${status}' == '${False}' Continue For Loop
288 \ ${jsondata}= to json ${resp.content} pretty_print=True
289 \ Write Data To File ${\n}${jsondata}${\n} ${logpath}
290 \ Append To List ${ffdc_file_list} ${logpath}
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500291
Michael Walsh63b70c02017-10-30 15:02:09 -0500292 [Return] ${ffdc_file_list}
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500293
294BMC FFDC Get Requests
Michael Walsh63b70c02017-10-30 15:02:09 -0500295 [Documentation] Iterate over get request list and return a list of
296 ... generated files.
297
298 @{ffdc_file_list}= Create List
299
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500300 @{entries}= Get ffdc get request index
Michael Walsh63b70c02017-10-30 15:02:09 -0500301 # Example of entries:
302 # entries:
303 # entries[0]: GET REQUESTS
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500304 :FOR ${index} IN @{entries}
Michael Walsh63b70c02017-10-30 15:02:09 -0500305 \ ${ffdc_file_sub_list}= Log FFDC Get Requests ${index}
306 \ ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list}
307 ... ${ffdc_file_sub_list}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500308
Michael Walsh63b70c02017-10-30 15:02:09 -0500309 [Return] ${ffdc_file_list}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500310
Michael Walsh63b70c02017-10-30 15:02:09 -0500311Log OS All distros FFDC
312 [Documentation] Run commands from FFDC_OS_ALL_DISTROS_FILE to create FFDC
313 ... files and return a list of generated files.
314 [Arguments] ${key_index}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500315
Michael Walsh63b70c02017-10-30 15:02:09 -0500316 # Description of argument(s):
317 # key_index The index into the FFDC_OS_ALL_DISTROS_FILE dictionary.
318
319 @{ffdc_file_list}= Create List
320
321 @{cmd_list}= Get FFDC OS All Distros Call ${key_index}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500322 :FOR ${cmd} IN @{cmd_list}
Michael Walsh63b70c02017-10-30 15:02:09 -0500323 \ ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}.txt
324 \ ${ffdc_file_sub_list}= Execute Command and Write FFDC ${cmd[0]}
Steven Sombarfc024cb2019-04-22 11:16:16 -0500325 ... ${cmd[1]} ${logpath} target=OS
326 \ # scp it to the LOG_PREFIX ffdc directory.
Steven Sombara31f4c82019-08-01 16:00:23 -0500327 \ Run Key U scp.Get File \ /tmp/${cmd[0]}.txt \ ${LOG_PREFIX}${cmd[0]}.txt
Michael Walsh63b70c02017-10-30 15:02:09 -0500328 \ ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list}
329 ... ${ffdc_file_sub_list}
330
331 [Return] ${ffdc_file_list}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500332
333
Gunnar Millscce185d2016-10-17 17:04:15 -0500334Log OS SPECIFIC DISTRO FFDC
Michael Walsh63b70c02017-10-30 15:02:09 -0500335 [Documentation] Run commands from the FFDC_OS_<distro>_FILE to create FFDC
336 ... files and return a list of generated files.
337 [Arguments] ${key_index} ${linux_distro}
Gunnar Millscce185d2016-10-17 17:04:15 -0500338
Michael Walsh63b70c02017-10-30 15:02:09 -0500339 # Description of argument(s):
340 # key_index The index into the FFDC_OS_<distro>_FILE dictionary.
341 # linux_distro Your OS's linux distro (e.g. "UBUNTU", "RHEL", etc).
342
343 @{ffdc_file_list}= Create List
344
345 @{cmd_list}= Get FFDC OS Distro Call ${key_index} ${linux_distro}
Gunnar Millscce185d2016-10-17 17:04:15 -0500346 :FOR ${cmd} IN @{cmd_list}
Michael Walsh63b70c02017-10-30 15:02:09 -0500347 \ ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}.txt
348 \ ${ffdc_file_sub_list}= Execute Command and Write FFDC ${cmd[0]}
349 ... ${cmd[1]} ${logpath} target=OS
Steven Sombara31f4c82019-08-01 16:00:23 -0500350 \ Run Key U scp.Get File \ /tmp/${cmd[0]}.txt \ ${LOG_PREFIX}${cmd[0]}.txt
Michael Walsh63b70c02017-10-30 15:02:09 -0500351 \ ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list}
352 ... ${ffdc_file_sub_list}
353
Steven Sombare70acbe2019-08-20 14:37:47 -0500354 # Get the name of the sosreport file.
355 ${sosreport_file_path} ${stderr} ${rc}= OS Execute Command
356 ... ls /tmp/sosreport*FFDC*tar.xz ignore_err=${True}
357 # Example: sosreport_file_path="/tmp/sosreport-myhost-FFDC-2019-08-20-pbuaqtk.tar.xz".
358
359 # Return if there is no sosreport file.
360 Return From Keyword If ${rc} != ${0} ${ffdc_file_list}
361
362 ${sosreport_dir_path} ${sosreport_file_name}= Split Path ${sosreport_file_path}
363 # Example: sosreport_dir_path="/tmp",
364 # sosreport_file_name="sosreport-myhost-FFDC-2019-08-20-pbuaqtk.tar.xz".
365
366 # Location where the sosreport file will be copied to.
367 ${local_sosreport_file_path}= Set Variable ${LOG_PREFIX}OS_${sosreport_file_name}
368
369 # Change file permissions otherwise scp will not see the file.
370 OS Execute Command chmod 644 ${sosreport_file_path}
371
372 # SCP the sosreport file from the OS.
373 Run Key U scp.Get File \ ${sosreport_file_path} \ ${local_sosreport_file_path}
374
375 # Add the file location to the ffdc_file_list.
376 Append To List ${ffdc_file_list} ${local_sosreport_file_path}
377
Michael Walsh63b70c02017-10-30 15:02:09 -0500378 [Return] ${ffdc_file_list}
Gunnar Millscce185d2016-10-17 17:04:15 -0500379
380
George Keishingca8c4c62016-10-14 10:08:40 -0500381OS FFDC Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500382 [Documentation] Run the commands from FFDC_OS_ALL_DISTROS_FILE to create
383 ... FFDC files and return a list of generated files.
Gunnar Millsaca140d2016-10-26 13:05:10 -0500384 [Arguments] ${OS_HOST}=${OS_HOST} ${OS_USERNAME}=${OS_USERNAME}
Michael Walsh63b70c02017-10-30 15:02:09 -0500385 ... ${OS_PASSWORD}=${OS_PASSWORD}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500386
Michael Walsh63b70c02017-10-30 15:02:09 -0500387 @{ffdc_file_list}= Create List
Gunnar Millsaca140d2016-10-26 13:05:10 -0500388
Michael Walsh63b70c02017-10-30 15:02:09 -0500389 Run Keyword If '${OS_HOST}' == '${EMPTY}' Run Keywords
390 ... Print Timen No OS Host provided so no OS FFDC will be done. AND
391 ... Return From Keyword ${ffdc_file_list}
Gunnar Millsaca140d2016-10-26 13:05:10 -0500392
Michael Walsh63b70c02017-10-30 15:02:09 -0500393 ${match_state}= Create Dictionary os_ping=^1$ os_login=^1$
394 ... os_run_cmd=^1$
395 ${status} ${ret_values}= Run Keyword and Ignore Error Check State
396 ... ${match_state} quiet=0
397
398 Run Keyword If '${status}' == 'FAIL' Run Keywords
Michael Walshc3b28d72017-11-16 10:43:14 -0600399 ... Print Timen The OS is not communicating so no OS FFDC will be done.\n
Michael Walsh63b70c02017-10-30 15:02:09 -0500400 ... AND
401 ... Return From Keyword ${ffdc_file_list}
Gunnar Millscce185d2016-10-17 17:04:15 -0500402
Michael Walshe53e47a2017-06-30 17:03:24 -0500403 ${stdout} ${stderr} ${rc}= OS Execute Command
404 ... . /etc/os-release; echo $ID ignore_err=${0}
405 Set Global Variable ${linux_distro} ${stdout}
Gunnar Millsaca140d2016-10-26 13:05:10 -0500406
Michael Walshe53e47a2017-06-30 17:03:24 -0500407 Rpvars linux_distro
Gunnar Millsaca140d2016-10-26 13:05:10 -0500408
Steven Sombarfc024cb2019-04-22 11:16:16 -0500409 scp.Open Connection
410 ... ${OS_HOST} username=${OS_USERNAME} password=${OS_PASSWORD}
411
Michael Walsh63b70c02017-10-30 15:02:09 -0500412 @{entries}= Get FFDC OS All Distros Index
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500413 :FOR ${index} IN @{entries}
Michael Walsh63b70c02017-10-30 15:02:09 -0500414 \ ${ffdc_file_sub_list}= Log OS All distros FFDC ${index}
415 \ ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list}
416 ... ${ffdc_file_sub_list}
Gunnar Millscce185d2016-10-17 17:04:15 -0500417
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600418 Return From Keyword If
419 ... '${linux_distro}' == '${EMPTY}' or '${linux_distro}' == 'None'
Michael Walsh63b70c02017-10-30 15:02:09 -0500420 ... ${ffdc_file_list}
Gunnar Millscce185d2016-10-17 17:04:15 -0500421
422 @{entries}= Get ffdc os distro index ${linux_distro}
423 :FOR ${index} IN @{entries}
Michael Walsh63b70c02017-10-30 15:02:09 -0500424 \ ${ffdc_file_sub_list}= Log OS SPECIFIC DISTRO FFDC ${index}
425 ... ${linux_distro}
426 \ ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list}
427 ... ${ffdc_file_sub_list}
428
Steven Sombarfc024cb2019-04-22 11:16:16 -0500429 # Delete ffdc files still on OS and close scp.
Steven Sombare70acbe2019-08-20 14:37:47 -0500430 OS Execute Command rm -rf /tmp/OS_* /tmp/sosreport*FFDC* ignore_err=${True}
Steven Sombarfc024cb2019-04-22 11:16:16 -0500431 scp.Close Connection
432
Michael Walsh63b70c02017-10-30 15:02:09 -0500433 [Return] ${ffdc_file_list}
George Keishing7a520222017-02-27 09:44:30 -0600434
435
Steven Sombar96de5732017-08-17 09:04:52 -0500436System Inventory Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500437 [Documentation] Copy systest os_inventory files and return a list of
438 ... generated files..
Steven Sombar96de5732017-08-17 09:04:52 -0500439 # The os_inventory files are the result of running
440 # systest/htx_hardbootme_test. If these files exist
441 # they are copied to the FFDC directory.
442 # Global variable ffdc_dir_path is the path name of the
443 # directory they are copied to.
Michael Walsh63b70c02017-10-30 15:02:09 -0500444
445 @{ffdc_file_list}= Create List
446
447 ${globex}= Set Variable os_inventory_*.json
448
449 @{file_list}= OperatingSystem.List Files In Directory . ${globex}
450
451 Copy Files ${globex} ${ffdc_dir_path}
452
453 : FOR ${file_name} IN @{file_list}
454 \ Append To List ${ffdc_file_list} ${ffdc_dir_path}${file_name}
455
456 Run Keyword and Ignore Error Remove Files ${globex}
457
458 [Return] ${ffdc_file_list}
Steven Sombar96de5732017-08-17 09:04:52 -0500459
460
George Keishing7a520222017-02-27 09:44:30 -0600461SCP Coredump Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500462 [Documentation] Copy core dump files from BMC to local system and return a
463 ... list of generated file names.
464
465 @{ffdc_file_list}= Create List
Michael Walshe53e47a2017-06-30 17:03:24 -0500466
George Keishing7a520222017-02-27 09:44:30 -0600467 # Check if core dump exist in the /tmp
Michael Walshe53e47a2017-06-30 17:03:24 -0500468 ${core_files} ${stderr} ${rc}= BMC Execute Command ls /tmp/core_*
Michael Walsh63b70c02017-10-30 15:02:09 -0500469 ... ignore_err=${1}
470 Run Keyword If '${rc}' != '${0}' Return From Keyword ${ffdc_file_list}
471
472 @{core_list}= Split String ${core_files}
George Keishing7a520222017-02-27 09:44:30 -0600473 # Copy the core files
Michael Walsh6bad8412017-06-27 14:09:36 -0500474 Run Key U Open Connection for SCP
George Keishing7a520222017-02-27 09:44:30 -0600475 :FOR ${index} IN @{core_list}
Michael Walsh63b70c02017-10-30 15:02:09 -0500476 \ ${ffdc_file_path}= Catenate ${LOG_PREFIX}${index.lstrip("/tmp/")}
477 \ ${status}= Run Keyword and Return Status
478 ... scp.Get File ${index} ${ffdc_file_path}
479 \ Run Keyword If '${status}' == '${False}' Continue For Loop
480 \ Append To List ${ffdc_file_list} ${ffdc_file_path}
George Keishing7a520222017-02-27 09:44:30 -0600481 # Remove the file from remote to avoid re-copying on next FFDC call
Michael Walsh63b70c02017-10-30 15:02:09 -0500482 \ BMC Execute Command rm ${index} ignore_err=${1}
Michael Walshe53e47a2017-06-30 17:03:24 -0500483 # I can't find a way to do this: scp.Close Connection
George Keishing7a520222017-02-27 09:44:30 -0600484
Michael Walsh63b70c02017-10-30 15:02:09 -0500485 [Return] ${ffdc_file_list}
486
Steven Sombar2da175f2019-02-04 10:32:58 -0600487
Sweta Potthuria82efd62017-10-18 05:34:59 -0500488SCP Dump Files
489 [Documentation] Copy all dump files from BMC to local system.
490
491 # Check if dumps exist
492 ${ffdc_file_list}= Scp Dumps ${FFDC_DIR_PATH} ${FFDC_PREFIX}
493
494 [Return] ${ffdc_file_list}
495
Steven Sombar2da175f2019-02-04 10:32:58 -0600496
Sweta Potthuria82efd62017-10-18 05:34:59 -0500497Collect Dump Log
498 [Documentation] Collect dumps from dump entry.
499 [Arguments] ${log_prefix_path}=${LOG_PREFIX}
500
Steven Sombaraaaab222018-12-19 13:16:23 -0600501 ${data}= Read Properties ${DUMP_ENTRY_URI}enumerate quiet=${1}
Sweta Potthuria82efd62017-10-18 05:34:59 -0500502
503 # Grab the list of entries from dump/entry/
504 # The data shown below is the result of the "Get Dictionary Keys".
505 # Example:
506 # /xyz/openbmc_project/dump/entry/1
507 # /xyz/openbmc_project/dump/entry/2
508
509 ${dump_list}= Get Dictionary Keys ${data}
George Keishingeb7b4bc2017-03-13 12:04:28 -0500510
Michael Walsh02c70602018-10-04 15:24:30 -0500511
Steven Sombar2da175f2019-02-04 10:32:58 -0600512Enumerate Redfish Resources
513 [Documentation] Enumerate /redfish/v1 resources and properties to
514 ... a file. Return a list which contains the file name.
515 [Arguments] ${log_prefix_path}=${LOG_PREFIX}
516
517 # Description of argument(s):
518 # log_prefix_path The location specifying where to create FFDC file(s).
519
520 # Login is needed to fetch Redfish information.
521 # If login fails, return from keyword.
George Keishing97c93942019-03-04 12:45:07 -0600522 ${status}= Run Keyword And Return Status Redfish.Login
Steven Sombar2da175f2019-02-04 10:32:58 -0600523 Return From Keyword If ${status} == ${False}
524
525 # Get the Redfish resources and properties.
George Keishingf2613b72019-02-13 12:45:59 -0600526 ${json_data}= redfish_utils.Enumerate Request /redfish/v1
Steven Sombar2da175f2019-02-04 10:32:58 -0600527 # Typical output:
528 # {
529 # "@odata.id": "/redfish/v1",
530 # "@odata.type": "#ServiceRoot.v1_1_1.ServiceRoot",
531 # "AccountService": {
532 # "@odata.id": "/redfish/v1/AccountService"
533 # },
534 # "Chassis": {
535 # "@odata.id": "/redfish/v1/Chassis"
536 # },
537 # "Id": "RootService",
538 # "JsonSchemas": {
539 # "@odata.id": "/redfish/v1/JsonSchemas"
540 # },
541 # ..etc...
542 # }
543
544 @{ffdc_file_list}= Create List
545 ${logpath}= Catenate SEPARATOR= ${log_prefix_path}
546 ... redfish_resource_properties.txt
547 Create File ${logpath}
548 Write Data To File "${\n}${json_data}${\n}" ${logpath}
549
550 Append To List ${ffdc_file_list} ${logpath}
551
552 [Return] ${ffdc_file_list}
553
554
George Keishingeb7b4bc2017-03-13 12:04:28 -0500555Collect eSEL Log
Michael Walsh02c70602018-10-04 15:24:30 -0500556 [Documentation] Create raw and formatted eSEL files.
Sridevi Rameshb180c9f2017-08-06 10:27:41 -0500557 [Arguments] ${log_prefix_path}=${LOG_PREFIX}
558
Michael Walsh02c70602018-10-04 15:24:30 -0500559 # NOTE: If no eSEL.pl program can be located, then no formatted eSEL file
560 # will be generated.
561
562 # Description of argument(s):
563 # log_prefix_path The path prefix to be used in creating
564 # eSEL file path names. For example, if
565 # log_prefix_path is
566 # "/tmp/user1/dummy.181001.120000.", then
567 # this keyword will create
568 # /tmp/user1/dummy.181001.120000.esel (raw)
569 # and
570 # /tmp/user1/dummy.181001.120000.esel.txt
571 # (formatted).
572
Michael Walsh63b70c02017-10-30 15:02:09 -0500573 @{ffdc_file_list}= Create List
574
Michael Walsh02c70602018-10-04 15:24:30 -0500575 ${esels}= Get Esels
576 ${num_esels}= Evaluate len(${esels})
577 Rprint Vars num_esels
578 Return From Keyword If ${num_esels} == ${0} ${ffdc_file_list}
George Keishingeb7b4bc2017-03-13 12:04:28 -0500579
Sridevi Rameshb180c9f2017-08-06 10:27:41 -0500580 ${logpath}= Catenate SEPARATOR= ${log_prefix_path} esel
George Keishingeb7b4bc2017-03-13 12:04:28 -0500581 Create File ${logpath}
George Keishingeb7b4bc2017-03-13 12:04:28 -0500582
Michael Walsh02c70602018-10-04 15:24:30 -0500583 :FOR ${esel} IN @{esels}
584 \ Write Data To File "${esel}"${\n} ${logpath}
George Keishingb1ca69b2017-03-31 14:36:42 -0500585
Michael Walsh63b70c02017-10-30 15:02:09 -0500586 Append To List ${ffdc_file_list} ${logpath}
Michael Walsh02c70602018-10-04 15:24:30 -0500587
588 ${rc} ${output}= Shell Cmd which eSEL.pl show_err=0
589 Return From Keyword If ${rc} != ${0} ${ffdc_file_list}
590
591 Convert eSEL To Elog Format ${logpath}
Michael Walsh0192e9b2017-11-14 15:14:15 -0600592 Append To List ${ffdc_file_list} ${logpath}.txt
Michael Walsh63b70c02017-10-30 15:02:09 -0500593
594 [Return] ${ffdc_file_list}
595
George Keishingb1ca69b2017-03-31 14:36:42 -0500596
George Keishingb1ca69b2017-03-31 14:36:42 -0500597Convert eSEL To Elog Format
598 [Documentation] Execute parser tool on the eSEL data file to generate
Michael Walsha30dac72017-04-04 17:56:23 -0500599 ... formatted error log.
George Keishingb1ca69b2017-03-31 14:36:42 -0500600 [Arguments] ${esel_file_path}
Michael Walsh02c70602018-10-04 15:24:30 -0500601 # Description of argument(s):
George Keishing7c6342b2018-11-02 14:27:13 -0500602 # esel_file_path The path to the file containing raw eSEL
Michael Walsh02c70602018-10-04 15:24:30 -0500603 # data (e.g.
604 # "/tmp/user1/dummy.181001.120000.esel").
George Keishingb1ca69b2017-03-31 14:36:42 -0500605
Michael Walsha30dac72017-04-04 17:56:23 -0500606 # Note: The only way to get eSEL.pl to put the output in a particular
607 # directory is to cd to that directory.
608 ${cmd_buf}= Catenate cd $(dirname ${esel_file_path}) ; eSEL.pl -l
609 ... ${esel_file_path} -p decode_obmc_data
Michael Walsh02c70602018-10-04 15:24:30 -0500610 Qprint Issuing ${cmd_buf}
Michael Walsha30dac72017-04-04 17:56:23 -0500611 Run ${cmd_buf}
Michael Walsh02c70602018-10-04 15:24:30 -0500612 # The .binary file, which is generated by eSEL.pl, is of no use to us.
613 Remove File ${esel_file_path}.binary