blob: 42b3a35d9bd3300c991deab1608feae0bd0f476b [file] [log] [blame]
George Keishing69e6f712016-09-12 06:30:09 -05001*** Settings ***
2Documentation Methods to execute commands on BMC and collect
3... data to a list of FFDC files
4
Michael Walsh265ae412018-11-16 15:32:03 -06005Resource openbmc_ffdc_utils.robot
6Resource rest_client.robot
7Resource utils.robot
8Resource list_utils.robot
9Resource logging_utils.robot
Steven Sombar2da175f2019-02-04 10:32:58 -060010Resource bmc_redfish_resource.robot
Michael Walsh265ae412018-11-16 15:32:03 -060011Library SSHLibrary
12Library OperatingSystem
13Library Collections
14Library String
15Library gen_print.py
16Library gen_cmd.py
17Library gen_robot_keyword.py
18Library dump_utils.py
19Library logging_utils.py
Gunnar Mills7e2cda22016-10-11 15:37:34 -050020
Michael Walshde5e9372018-11-15 11:28:23 -060021*** Variables ***
22
Michael Walsh265ae412018-11-16 15:32:03 -060023${FFDC_CMD_TIMEOUT} 240
Michael Walshde5e9372018-11-15 11:28:23 -060024
George Keishing69e6f712016-09-12 06:30:09 -050025*** Keywords ***
26
George Keishing69e6f712016-09-12 06:30:09 -050027# Method : Call FFDC Methods #
28# Execute the user define keywords from the FFDC List #
29# Unlike any other keywords this will call into the #
30# list of keywords defined in the FFDC list at one go #
George Keishing69e6f712016-09-12 06:30:09 -050031
32Call FFDC Methods
Michael Walshe844e9a2017-04-20 16:51:10 -050033 [Documentation] Call into FFDC Keyword index list.
34 [Arguments] ${ffdc_function_list}=${EMPTY}
George Keishing69e6f712016-09-12 06:30:09 -050035
Michael Walshe844e9a2017-04-20 16:51:10 -050036 # Description of argument(s):
37 # ffdc_function_list A colon-delimited list naming the kinds of FFDC that
Michael Walsh63b70c02017-10-30 15:02:09 -050038 # are to be collected
Michael Walshe844e9a2017-04-20 16:51:10 -050039 # (e.g. "FFDC Generic Report:BMC Specific Files").
40 # Acceptable values can be found in the description
41 # field of FFDC_METHOD_CALL in
42 # lib/openbmc_ffdc_list.py. Those values can be
43 # obtained via a call to 'Get FFDC Method Desc' (also
44 # from lib/openbmc_ffdc_list.py).
45
46 @{entries}= Get FFDC Method Index
Michael Walsh63b70c02017-10-30 15:02:09 -050047 # Example entries:
48 # entries:
49 # entries[0]: BMC LOGS
50
51 @{ffdc_file_list}= Create List
Michael Walshe844e9a2017-04-20 16:51:10 -050052 :FOR ${index} IN @{entries}
Michael Walsh63b70c02017-10-30 15:02:09 -050053 \ ${ffdc_file_sub_list}= Method Call Keyword List ${index}
54 ... ${ffdc_function_list}
55 \ ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list}
56 ... ${ffdc_file_sub_list}
57
Michael Walsh6bad8412017-06-27 14:09:36 -050058 Run Key U SSHLibrary.Close All Connections
George Keishing69e6f712016-09-12 06:30:09 -050059
Michael Walsh63b70c02017-10-30 15:02:09 -050060 [Return] ${ffdc_file_list}
61
Steven Sombar2da175f2019-02-04 10:32:58 -060062
George Keishing69e6f712016-09-12 06:30:09 -050063Method Call Keyword List
Michael Walsh63b70c02017-10-30 15:02:09 -050064 [Documentation] Process FFDC request and return a list of generated files.
Michael Walshe844e9a2017-04-20 16:51:10 -050065 [Arguments] ${index} ${ffdc_function_list}=${EMPTY}
George Keishing69e6f712016-09-12 06:30:09 -050066
Michael Walshe844e9a2017-04-20 16:51:10 -050067 # Description of argument(s):
68 # index The index into the FFDC_METHOD_CALL dictionary (e.g.
69 # 'BMC LOGS').
70 # ffdc_function_list See ffdc_function_list description in
71 # "Call FFDC Methods" (above).
72
Michael Walsh63b70c02017-10-30 15:02:09 -050073 @{method_list}= Get FFDC Method Call ${index}
74 # Example method_list:
75 # method_list:
76 # method_list[0]:
77 # method_list[0][0]: FFDC Generic Report
78 # method_list[0][1]: BMC FFDC Manifest
79 # method_list[1]:
80 # method_list[1][0]: Get Request FFDC
81 # method_list[1][1]: BMC FFDC Get Requests
82 # (etc.)
Michael Walshe844e9a2017-04-20 16:51:10 -050083
84 # If function list is empty assign default (i.e. a list of all allowable
85 # values). In either case, convert ffdc_function_list from a string to
86 # a list.
87 @{ffdc_function_list}=
88 ... Run Keyword If '${ffdc_function_list}' == '${EMPTY}'
89 ... Get FFDC Method Desc ${index}
90 ... ELSE
91 ... Split String ${ffdc_function_list} separator=:
92
Michael Walsh63b70c02017-10-30 15:02:09 -050093 @{ffdc_file_list}= Create List
George Keishing69e6f712016-09-12 06:30:09 -050094 :FOR ${method} IN @{method_list}
Michael Walsh63b70c02017-10-30 15:02:09 -050095 \ ${ffdc_file_sub_list}= Execute Keyword Method ${method[0]}
96 ... ${method[1]} @{ffdc_function_list}
97 \ ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list}
98 ... ${ffdc_file_sub_list}
99
100 [Return] ${ffdc_file_list}
101
George Keishing69e6f712016-09-12 06:30:09 -0500102
103Execute Keyword Method
Michael Walshe844e9a2017-04-20 16:51:10 -0500104 [Documentation] Call into BMC method keywords. Don't let one
105 ... failure skip the remaining. Get whatever data
106 ... it could gather at worse case scenario.
107 [Arguments] ${description} ${keyword_name} @{ffdc_function_list}
George Keishing69e6f712016-09-12 06:30:09 -0500108
Michael Walshe844e9a2017-04-20 16:51:10 -0500109 # Description of argument(s):
110 # description The description of the FFDC to be collected. This
111 # would be any value returned by
112 # 'Get FFDC Method Desc' (e.g. "FFDC Generic Report").
113 # keyword_name The name of the keyword to call to collect the FFDC
114 # data (again, see FFDC_METHOD_CALL).
115 # ffdc_function_list See ffdc_function_list description in
116 # "Call FFDC Methods" (above). The only difference is
117 # in this case, it should be a list rather than a
118 # colon-delimited value.
George Keishing69e6f712016-09-12 06:30:09 -0500119
Michael Walsh63b70c02017-10-30 15:02:09 -0500120 @{ffdc_file_list}= Create List
Michael Walshe844e9a2017-04-20 16:51:10 -0500121
Michael Walsh63b70c02017-10-30 15:02:09 -0500122 ${index}= Get Index From List ${ffdc_function_list} ${description}
123 Run Keyword If '${index}' == '${-1}' Return from Keyword
124 ... ${ffdc_file_list}
125
126 ${status} ${ffdc_file_list}= Run Key ${keyword_name} ignore=1
127 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500128
George Keishing69e6f712016-09-12 06:30:09 -0500129# Method : BMC FFDC Manifest #
130# Execute command on BMC and write to ffdc_report.txt #
George Keishing69e6f712016-09-12 06:30:09 -0500131
132BMC FFDC Manifest
Michael Walsh63b70c02017-10-30 15:02:09 -0500133 [Documentation] Run the ssh commands from FFDC_BMC_CMD and return a list
134 ... of generated files.
George Keishing69e6f712016-09-12 06:30:09 -0500135
Michael Walsh63b70c02017-10-30 15:02:09 -0500136 @{ffdc_file_list}= Create List ${FFDC_FILE_PATH}
137 @{entries}= Get FFDC Cmd Index
138 :FOR ${index} IN @{entries}
139 \ Iterate BMC Command List Pairs ${index}
140
141 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500142
143
144Iterate BMC Command List Pairs
145 [Documentation] Feed in key pair list from dictionary to execute
146 [Arguments] ${key_index}
147
148 @{cmd_list}= Get ffdc bmc cmd ${key_index}
149 Set Suite Variable ${ENTRY_INDEX} ${key_index}
150 :FOR ${cmd} IN @{cmd_list}
151 \ Execute Command and Write FFDC ${cmd[0]} ${cmd[1]}
152
153
154Execute Command and Write FFDC
Michael Walsh63b70c02017-10-30 15:02:09 -0500155 [Documentation] Run a command on the BMC or OS, write the output to the
156 ... specified file and return a list of generated files.
157 [Arguments] ${key_index} ${cmd} ${logpath}=${FFDC_FILE_PATH}
158 ... ${target}=BMC
George Keishing69e6f712016-09-12 06:30:09 -0500159
Michael Walsh63b70c02017-10-30 15:02:09 -0500160 Run Keyword If '${logpath}' == '${FFDC_FILE_PATH}'
161 ... Write Cmd Output to FFDC File ${key_index} ${cmd}
162
163 @{ffdc_file_list}= Create List ${log_path}
George Keishing69e6f712016-09-12 06:30:09 -0500164
Michael Walshe53e47a2017-06-30 17:03:24 -0500165 ${cmd_buf}= Catenate ${target} Execute Command \ ${cmd} \ ignore_err=${1}
Michael Walsh265ae412018-11-16 15:32:03 -0600166 ... \ time_out=${FFDC_CMD_TIMEOUT}
Michael Walshe53e47a2017-06-30 17:03:24 -0500167 ${status} ${ret_values}= Run Key ${cmd_buf} ignore=${1}
Michael Walshde5e9372018-11-15 11:28:23 -0600168 # If the command times out, status will be 'FAIL'.
169 Return From Keyword If '${status}' == 'FAIL' ${ffdc_file_list}
Michael Walshe53e47a2017-06-30 17:03:24 -0500170
171 ${stdout}= Set Variable @{ret_values}[0]
172 ${stderr}= Set Variable @{ret_values}[1]
George Keishing69e6f712016-09-12 06:30:09 -0500173
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600174 # Write stdout on success and stderr/stdout to the file on failure.
175 Run Keyword If $stderr == '${EMPTY}'
Sweta Potthuri15e6d2f2017-02-28 03:10:02 -0600176 ... Write Data To File ${stdout}${\n} ${logpath}
177 ... ELSE Write Data To File
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600178 ... ERROR output:${\n}${stderr}${\n}Output:${\n}${stdout}${\n}
179 ... ${logpath}
George Keishing69e6f712016-09-12 06:30:09 -0500180
Michael Walsh63b70c02017-10-30 15:02:09 -0500181 [Return] ${ffdc_file_list}
182
George Keishing69e6f712016-09-12 06:30:09 -0500183
George Keishing69e6f712016-09-12 06:30:09 -0500184# Method : BMC FFDC Files #
185# Execute command on BMC and write to individual file #
186# based on the file name pre-defined in the list #
George Keishing69e6f712016-09-12 06:30:09 -0500187
188BMC FFDC Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500189 [Documentation] Run the commands from FFDC_BMC_FILE and return a list of
190 ... generated files.
191
192 @{entries}= Get FFDC File Index
193 # Example of entries:
194 # entries:
195 # entries[0]: BMC FILES
196
Steven Sombara31f4c82019-08-01 16:00:23 -0500197 scp.Open Connection
198 ... ${OPENBMC_HOST} username=${OPENBMC_USERNAME} password=${OPENBMC_PASSWORD}
199
Michael Walsh63b70c02017-10-30 15:02:09 -0500200 @{ffdc_file_list}= Create List
201 :FOR ${index} IN @{entries}
202 \ ${ffdc_file_sub_list}= Create File and Write Data ${index}
203 \ ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list}
204 ... ${ffdc_file_sub_list}
205
Steven Sombara31f4c82019-08-01 16:00:23 -0500206 BMC Execute Command rm -rf /tmp/BMC_* /tmp/SOL.txt
207 scp.Close Connection
208
Michael Walsh63b70c02017-10-30 15:02:09 -0500209 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500210
211
212Create File and Write Data
Michael Walsh63b70c02017-10-30 15:02:09 -0500213 [Documentation] Run commands from FFDC_BMC_FILE to create FFDC files and
214 ... return a list of generated files.
215 [Arguments] ${key_index}
George Keishing69e6f712016-09-12 06:30:09 -0500216
Michael Walsh63b70c02017-10-30 15:02:09 -0500217 # Description of argument(s):
218 # key_index The index into the FFDC_BMC_FILE dictionary.
219
220 @{ffdc_file_list}= Create List
221 @{cmd_list}= Get FFDC BMC File ${key_index}
George Keishing69e6f712016-09-12 06:30:09 -0500222 :FOR ${cmd} IN @{cmd_list}
Michael Walsh63b70c02017-10-30 15:02:09 -0500223 \ ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}.txt
224 \ ${ffdc_file_sub_list}= Execute Command and Write FFDC ${cmd[0]}
225 ... ${cmd[1]} ${logpath}
Steven Sombara31f4c82019-08-01 16:00:23 -0500226 \ Run Key U scp.Get File \ /tmp/${cmd[0]}.txt \ ${LOG_PREFIX}${cmd[0]}.txt
Michael Walsh63b70c02017-10-30 15:02:09 -0500227 \ ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list}
228 ... ${ffdc_file_sub_list}
Sivas SRR0d657e82017-03-13 01:00:34 -0500229
Michael Walsh63b70c02017-10-30 15:02:09 -0500230 [Return] ${ffdc_file_list}
George Keishing69e6f712016-09-12 06:30:09 -0500231
232
George Keishing69e6f712016-09-12 06:30:09 -0500233# Method : Log Test Case Status #
234# Creates test result history footprint for reference #
George Keishing69e6f712016-09-12 06:30:09 -0500235
236Log Test Case Status
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600237 [Documentation] Test case execution result history.
238 ... Create once and append to this file
239 ... logs/test_history.txt
240 ... Format Date:Test suite:Test case:Status
241 ... 20160909214053719992:Test Warmreset:Test WarmReset via REST:FAIL
Michael Walshfa9f70f2017-04-21 16:00:18 -0500242
243 ${FFDC_DIR_PATH_STYLE}= Get Variable Value ${FFDC_DIR_PATH_STYLE}
244 ... ${EMPTY}
245 ${FFDC_DIR_PATH}= Get Variable Value ${FFDC_DIR_PATH} ${EMPTY}
246
247 Run Keyword If '${FFDC_DIR_PATH}' == '${EMPTY}' Set FFDC Defaults
248
249 Run Keyword If '${FFDC_DIR_PATH_STYLE}' == '${1}' Run Keywords
250 ... Set Global Variable ${FFDC_LOG_PATH} ${FFDC_DIR_PATH} AND
251 ... Set Global Variable ${TEST_HISTORY} ${FFDC_DIR_PATH}test_history.txt
252
George Keishing69e6f712016-09-12 06:30:09 -0500253 Create Directory ${FFDC_LOG_PATH}
254
255 ${exist}= Run Keyword and Return Status
256 ... OperatingSystem.File Should Exist ${TEST_HISTORY}
257
258 Run Keyword If '${exist}' == '${False}'
259 ... Create File ${TEST_HISTORY}
260
Michael Walshfa9f70f2017-04-21 16:00:18 -0500261 Rpvars TEST_HISTORY
262
George Keishing69e6f712016-09-12 06:30:09 -0500263 ${cur_time}= Get Current Time Stamp
264
265 Append To File ${TEST_HISTORY}
266 ... ${cur_time}:${SUITE_NAME}:${TEST_NAME}:${TEST_STATUS}${\n}
267
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500268
269Log FFDC Get Requests
Michael Walsh63b70c02017-10-30 15:02:09 -0500270 [Documentation] Run the get requests associated with the key and return a
271 ... list of generated files.
272 [Arguments] ${key_index}
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500273
Michael Walsh63b70c02017-10-30 15:02:09 -0500274 # Note: Output will be in JSON pretty_print format.
275
276 # Description of argument(s):
277 # key_index The key to the FFDC_GET_REQUEST dictionary that contains the
278 # get requests that are to be run.
279
280 @{ffdc_file_list}= Create List
281 @{cmd_list}= Get FFDC Get Request ${key_index}
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500282 :FOR ${cmd} IN @{cmd_list}
Michael Walsh63b70c02017-10-30 15:02:09 -0500283 \ ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}.txt
284 \ ${resp}= OpenBMC Get Request ${cmd[1]} quiet=${1}
285 \ ${status}= Run Keyword and Return Status
286 ... Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
287 \ Run Keyword If '${status}' == '${False}' Continue For Loop
288 \ ${jsondata}= to json ${resp.content} pretty_print=True
289 \ Write Data To File ${\n}${jsondata}${\n} ${logpath}
290 \ Append To List ${ffdc_file_list} ${logpath}
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500291
Michael Walsh63b70c02017-10-30 15:02:09 -0500292 [Return] ${ffdc_file_list}
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500293
294BMC FFDC Get Requests
Michael Walsh63b70c02017-10-30 15:02:09 -0500295 [Documentation] Iterate over get request list and return a list of
296 ... generated files.
297
298 @{ffdc_file_list}= Create List
299
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500300 @{entries}= Get ffdc get request index
Michael Walsh63b70c02017-10-30 15:02:09 -0500301 # Example of entries:
302 # entries:
303 # entries[0]: GET REQUESTS
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500304 :FOR ${index} IN @{entries}
Michael Walsh63b70c02017-10-30 15:02:09 -0500305 \ ${ffdc_file_sub_list}= Log FFDC Get Requests ${index}
306 \ ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list}
307 ... ${ffdc_file_sub_list}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500308
Michael Walsh63b70c02017-10-30 15:02:09 -0500309 [Return] ${ffdc_file_list}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500310
Michael Walsh63b70c02017-10-30 15:02:09 -0500311Log OS All distros FFDC
312 [Documentation] Run commands from FFDC_OS_ALL_DISTROS_FILE to create FFDC
313 ... files and return a list of generated files.
314 [Arguments] ${key_index}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500315
Michael Walsh63b70c02017-10-30 15:02:09 -0500316 # Description of argument(s):
317 # key_index The index into the FFDC_OS_ALL_DISTROS_FILE dictionary.
318
319 @{ffdc_file_list}= Create List
320
321 @{cmd_list}= Get FFDC OS All Distros Call ${key_index}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500322 :FOR ${cmd} IN @{cmd_list}
Michael Walsh63b70c02017-10-30 15:02:09 -0500323 \ ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}.txt
324 \ ${ffdc_file_sub_list}= Execute Command and Write FFDC ${cmd[0]}
Steven Sombarfc024cb2019-04-22 11:16:16 -0500325 ... ${cmd[1]} ${logpath} target=OS
326 \ # scp it to the LOG_PREFIX ffdc directory.
Steven Sombara31f4c82019-08-01 16:00:23 -0500327 \ Run Key U scp.Get File \ /tmp/${cmd[0]}.txt \ ${LOG_PREFIX}${cmd[0]}.txt
Michael Walsh63b70c02017-10-30 15:02:09 -0500328 \ ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list}
329 ... ${ffdc_file_sub_list}
330
331 [Return] ${ffdc_file_list}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500332
333
Gunnar Millscce185d2016-10-17 17:04:15 -0500334Log OS SPECIFIC DISTRO FFDC
Michael Walsh63b70c02017-10-30 15:02:09 -0500335 [Documentation] Run commands from the FFDC_OS_<distro>_FILE to create FFDC
336 ... files and return a list of generated files.
337 [Arguments] ${key_index} ${linux_distro}
Gunnar Millscce185d2016-10-17 17:04:15 -0500338
Michael Walsh63b70c02017-10-30 15:02:09 -0500339 # Description of argument(s):
340 # key_index The index into the FFDC_OS_<distro>_FILE dictionary.
341 # linux_distro Your OS's linux distro (e.g. "UBUNTU", "RHEL", etc).
342
343 @{ffdc_file_list}= Create List
344
345 @{cmd_list}= Get FFDC OS Distro Call ${key_index} ${linux_distro}
Gunnar Millscce185d2016-10-17 17:04:15 -0500346 :FOR ${cmd} IN @{cmd_list}
Michael Walsh63b70c02017-10-30 15:02:09 -0500347 \ ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}.txt
348 \ ${ffdc_file_sub_list}= Execute Command and Write FFDC ${cmd[0]}
349 ... ${cmd[1]} ${logpath} target=OS
Steven Sombara31f4c82019-08-01 16:00:23 -0500350 \ Run Key U scp.Get File \ /tmp/${cmd[0]}.txt \ ${LOG_PREFIX}${cmd[0]}.txt
Michael Walsh63b70c02017-10-30 15:02:09 -0500351 \ ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list}
352 ... ${ffdc_file_sub_list}
353
354 [Return] ${ffdc_file_list}
Gunnar Millscce185d2016-10-17 17:04:15 -0500355
356
George Keishingca8c4c62016-10-14 10:08:40 -0500357OS FFDC Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500358 [Documentation] Run the commands from FFDC_OS_ALL_DISTROS_FILE to create
359 ... FFDC files and return a list of generated files.
Gunnar Millsaca140d2016-10-26 13:05:10 -0500360 [Arguments] ${OS_HOST}=${OS_HOST} ${OS_USERNAME}=${OS_USERNAME}
Michael Walsh63b70c02017-10-30 15:02:09 -0500361 ... ${OS_PASSWORD}=${OS_PASSWORD}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500362
Michael Walsh63b70c02017-10-30 15:02:09 -0500363 @{ffdc_file_list}= Create List
Gunnar Millsaca140d2016-10-26 13:05:10 -0500364
Michael Walsh63b70c02017-10-30 15:02:09 -0500365 Run Keyword If '${OS_HOST}' == '${EMPTY}' Run Keywords
366 ... Print Timen No OS Host provided so no OS FFDC will be done. AND
367 ... Return From Keyword ${ffdc_file_list}
Gunnar Millsaca140d2016-10-26 13:05:10 -0500368
Michael Walsh63b70c02017-10-30 15:02:09 -0500369 ${match_state}= Create Dictionary os_ping=^1$ os_login=^1$
370 ... os_run_cmd=^1$
371 ${status} ${ret_values}= Run Keyword and Ignore Error Check State
372 ... ${match_state} quiet=0
373
374 Run Keyword If '${status}' == 'FAIL' Run Keywords
Michael Walshc3b28d72017-11-16 10:43:14 -0600375 ... Print Timen The OS is not communicating so no OS FFDC will be done.\n
Michael Walsh63b70c02017-10-30 15:02:09 -0500376 ... AND
377 ... Return From Keyword ${ffdc_file_list}
Gunnar Millscce185d2016-10-17 17:04:15 -0500378
Michael Walshe53e47a2017-06-30 17:03:24 -0500379 ${stdout} ${stderr} ${rc}= OS Execute Command
380 ... . /etc/os-release; echo $ID ignore_err=${0}
381 Set Global Variable ${linux_distro} ${stdout}
Gunnar Millsaca140d2016-10-26 13:05:10 -0500382
Michael Walshe53e47a2017-06-30 17:03:24 -0500383 Rpvars linux_distro
Gunnar Millsaca140d2016-10-26 13:05:10 -0500384
Steven Sombarfc024cb2019-04-22 11:16:16 -0500385 scp.Open Connection
386 ... ${OS_HOST} username=${OS_USERNAME} password=${OS_PASSWORD}
387
Michael Walsh63b70c02017-10-30 15:02:09 -0500388 @{entries}= Get FFDC OS All Distros Index
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500389 :FOR ${index} IN @{entries}
Michael Walsh63b70c02017-10-30 15:02:09 -0500390 \ ${ffdc_file_sub_list}= Log OS All distros FFDC ${index}
391 \ ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list}
392 ... ${ffdc_file_sub_list}
Gunnar Millscce185d2016-10-17 17:04:15 -0500393
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600394 Return From Keyword If
395 ... '${linux_distro}' == '${EMPTY}' or '${linux_distro}' == 'None'
Michael Walsh63b70c02017-10-30 15:02:09 -0500396 ... ${ffdc_file_list}
Gunnar Millscce185d2016-10-17 17:04:15 -0500397
398 @{entries}= Get ffdc os distro index ${linux_distro}
399 :FOR ${index} IN @{entries}
Michael Walsh63b70c02017-10-30 15:02:09 -0500400 \ ${ffdc_file_sub_list}= Log OS SPECIFIC DISTRO FFDC ${index}
401 ... ${linux_distro}
402 \ ${ffdc_file_list}= Smart Combine Lists ${ffdc_file_list}
403 ... ${ffdc_file_sub_list}
404
Steven Sombarfc024cb2019-04-22 11:16:16 -0500405 # Delete ffdc files still on OS and close scp.
406 OS Execute Command rm -rf /tmp/OS_*
407 scp.Close Connection
408
Michael Walsh63b70c02017-10-30 15:02:09 -0500409 [Return] ${ffdc_file_list}
George Keishing7a520222017-02-27 09:44:30 -0600410
411
Steven Sombar96de5732017-08-17 09:04:52 -0500412System Inventory Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500413 [Documentation] Copy systest os_inventory files and return a list of
414 ... generated files..
Steven Sombar96de5732017-08-17 09:04:52 -0500415 # The os_inventory files are the result of running
416 # systest/htx_hardbootme_test. If these files exist
417 # they are copied to the FFDC directory.
418 # Global variable ffdc_dir_path is the path name of the
419 # directory they are copied to.
Michael Walsh63b70c02017-10-30 15:02:09 -0500420
421 @{ffdc_file_list}= Create List
422
423 ${globex}= Set Variable os_inventory_*.json
424
425 @{file_list}= OperatingSystem.List Files In Directory . ${globex}
426
427 Copy Files ${globex} ${ffdc_dir_path}
428
429 : FOR ${file_name} IN @{file_list}
430 \ Append To List ${ffdc_file_list} ${ffdc_dir_path}${file_name}
431
432 Run Keyword and Ignore Error Remove Files ${globex}
433
434 [Return] ${ffdc_file_list}
Steven Sombar96de5732017-08-17 09:04:52 -0500435
436
George Keishing7a520222017-02-27 09:44:30 -0600437SCP Coredump Files
Michael Walsh63b70c02017-10-30 15:02:09 -0500438 [Documentation] Copy core dump files from BMC to local system and return a
439 ... list of generated file names.
440
441 @{ffdc_file_list}= Create List
Michael Walshe53e47a2017-06-30 17:03:24 -0500442
George Keishing7a520222017-02-27 09:44:30 -0600443 # Check if core dump exist in the /tmp
Michael Walshe53e47a2017-06-30 17:03:24 -0500444 ${core_files} ${stderr} ${rc}= BMC Execute Command ls /tmp/core_*
Michael Walsh63b70c02017-10-30 15:02:09 -0500445 ... ignore_err=${1}
446 Run Keyword If '${rc}' != '${0}' Return From Keyword ${ffdc_file_list}
447
448 @{core_list}= Split String ${core_files}
George Keishing7a520222017-02-27 09:44:30 -0600449 # Copy the core files
Michael Walsh6bad8412017-06-27 14:09:36 -0500450 Run Key U Open Connection for SCP
George Keishing7a520222017-02-27 09:44:30 -0600451 :FOR ${index} IN @{core_list}
Michael Walsh63b70c02017-10-30 15:02:09 -0500452 \ ${ffdc_file_path}= Catenate ${LOG_PREFIX}${index.lstrip("/tmp/")}
453 \ ${status}= Run Keyword and Return Status
454 ... scp.Get File ${index} ${ffdc_file_path}
455 \ Run Keyword If '${status}' == '${False}' Continue For Loop
456 \ Append To List ${ffdc_file_list} ${ffdc_file_path}
George Keishing7a520222017-02-27 09:44:30 -0600457 # Remove the file from remote to avoid re-copying on next FFDC call
Michael Walsh63b70c02017-10-30 15:02:09 -0500458 \ BMC Execute Command rm ${index} ignore_err=${1}
Michael Walshe53e47a2017-06-30 17:03:24 -0500459 # I can't find a way to do this: scp.Close Connection
George Keishing7a520222017-02-27 09:44:30 -0600460
Michael Walsh63b70c02017-10-30 15:02:09 -0500461 [Return] ${ffdc_file_list}
462
Steven Sombar2da175f2019-02-04 10:32:58 -0600463
Sweta Potthuria82efd62017-10-18 05:34:59 -0500464SCP Dump Files
465 [Documentation] Copy all dump files from BMC to local system.
466
467 # Check if dumps exist
468 ${ffdc_file_list}= Scp Dumps ${FFDC_DIR_PATH} ${FFDC_PREFIX}
469
470 [Return] ${ffdc_file_list}
471
Steven Sombar2da175f2019-02-04 10:32:58 -0600472
Sweta Potthuria82efd62017-10-18 05:34:59 -0500473Collect Dump Log
474 [Documentation] Collect dumps from dump entry.
475 [Arguments] ${log_prefix_path}=${LOG_PREFIX}
476
Steven Sombaraaaab222018-12-19 13:16:23 -0600477 ${data}= Read Properties ${DUMP_ENTRY_URI}enumerate quiet=${1}
Sweta Potthuria82efd62017-10-18 05:34:59 -0500478
479 # Grab the list of entries from dump/entry/
480 # The data shown below is the result of the "Get Dictionary Keys".
481 # Example:
482 # /xyz/openbmc_project/dump/entry/1
483 # /xyz/openbmc_project/dump/entry/2
484
485 ${dump_list}= Get Dictionary Keys ${data}
George Keishingeb7b4bc2017-03-13 12:04:28 -0500486
Michael Walsh02c70602018-10-04 15:24:30 -0500487
Steven Sombar2da175f2019-02-04 10:32:58 -0600488Enumerate Redfish Resources
489 [Documentation] Enumerate /redfish/v1 resources and properties to
490 ... a file. Return a list which contains the file name.
491 [Arguments] ${log_prefix_path}=${LOG_PREFIX}
492
493 # Description of argument(s):
494 # log_prefix_path The location specifying where to create FFDC file(s).
495
496 # Login is needed to fetch Redfish information.
497 # If login fails, return from keyword.
George Keishing97c93942019-03-04 12:45:07 -0600498 ${status}= Run Keyword And Return Status Redfish.Login
Steven Sombar2da175f2019-02-04 10:32:58 -0600499 Return From Keyword If ${status} == ${False}
500
501 # Get the Redfish resources and properties.
George Keishingf2613b72019-02-13 12:45:59 -0600502 ${json_data}= redfish_utils.Enumerate Request /redfish/v1
Steven Sombar2da175f2019-02-04 10:32:58 -0600503 # Typical output:
504 # {
505 # "@odata.id": "/redfish/v1",
506 # "@odata.type": "#ServiceRoot.v1_1_1.ServiceRoot",
507 # "AccountService": {
508 # "@odata.id": "/redfish/v1/AccountService"
509 # },
510 # "Chassis": {
511 # "@odata.id": "/redfish/v1/Chassis"
512 # },
513 # "Id": "RootService",
514 # "JsonSchemas": {
515 # "@odata.id": "/redfish/v1/JsonSchemas"
516 # },
517 # ..etc...
518 # }
519
520 @{ffdc_file_list}= Create List
521 ${logpath}= Catenate SEPARATOR= ${log_prefix_path}
522 ... redfish_resource_properties.txt
523 Create File ${logpath}
524 Write Data To File "${\n}${json_data}${\n}" ${logpath}
525
526 Append To List ${ffdc_file_list} ${logpath}
527
528 [Return] ${ffdc_file_list}
529
530
George Keishingeb7b4bc2017-03-13 12:04:28 -0500531Collect eSEL Log
Michael Walsh02c70602018-10-04 15:24:30 -0500532 [Documentation] Create raw and formatted eSEL files.
Sridevi Rameshb180c9f2017-08-06 10:27:41 -0500533 [Arguments] ${log_prefix_path}=${LOG_PREFIX}
534
Michael Walsh02c70602018-10-04 15:24:30 -0500535 # NOTE: If no eSEL.pl program can be located, then no formatted eSEL file
536 # will be generated.
537
538 # Description of argument(s):
539 # log_prefix_path The path prefix to be used in creating
540 # eSEL file path names. For example, if
541 # log_prefix_path is
542 # "/tmp/user1/dummy.181001.120000.", then
543 # this keyword will create
544 # /tmp/user1/dummy.181001.120000.esel (raw)
545 # and
546 # /tmp/user1/dummy.181001.120000.esel.txt
547 # (formatted).
548
Michael Walsh63b70c02017-10-30 15:02:09 -0500549 @{ffdc_file_list}= Create List
550
Michael Walsh02c70602018-10-04 15:24:30 -0500551 ${esels}= Get Esels
552 ${num_esels}= Evaluate len(${esels})
553 Rprint Vars num_esels
554 Return From Keyword If ${num_esels} == ${0} ${ffdc_file_list}
George Keishingeb7b4bc2017-03-13 12:04:28 -0500555
Sridevi Rameshb180c9f2017-08-06 10:27:41 -0500556 ${logpath}= Catenate SEPARATOR= ${log_prefix_path} esel
George Keishingeb7b4bc2017-03-13 12:04:28 -0500557 Create File ${logpath}
George Keishingeb7b4bc2017-03-13 12:04:28 -0500558
Michael Walsh02c70602018-10-04 15:24:30 -0500559 :FOR ${esel} IN @{esels}
560 \ Write Data To File "${esel}"${\n} ${logpath}
George Keishingb1ca69b2017-03-31 14:36:42 -0500561
Michael Walsh63b70c02017-10-30 15:02:09 -0500562 Append To List ${ffdc_file_list} ${logpath}
Michael Walsh02c70602018-10-04 15:24:30 -0500563
564 ${rc} ${output}= Shell Cmd which eSEL.pl show_err=0
565 Return From Keyword If ${rc} != ${0} ${ffdc_file_list}
566
567 Convert eSEL To Elog Format ${logpath}
Michael Walsh0192e9b2017-11-14 15:14:15 -0600568 Append To List ${ffdc_file_list} ${logpath}.txt
Michael Walsh63b70c02017-10-30 15:02:09 -0500569
570 [Return] ${ffdc_file_list}
571
George Keishingb1ca69b2017-03-31 14:36:42 -0500572
George Keishingb1ca69b2017-03-31 14:36:42 -0500573Convert eSEL To Elog Format
574 [Documentation] Execute parser tool on the eSEL data file to generate
Michael Walsha30dac72017-04-04 17:56:23 -0500575 ... formatted error log.
George Keishingb1ca69b2017-03-31 14:36:42 -0500576 [Arguments] ${esel_file_path}
Michael Walsh02c70602018-10-04 15:24:30 -0500577 # Description of argument(s):
George Keishing7c6342b2018-11-02 14:27:13 -0500578 # esel_file_path The path to the file containing raw eSEL
Michael Walsh02c70602018-10-04 15:24:30 -0500579 # data (e.g.
580 # "/tmp/user1/dummy.181001.120000.esel").
George Keishingb1ca69b2017-03-31 14:36:42 -0500581
Michael Walsha30dac72017-04-04 17:56:23 -0500582 # Note: The only way to get eSEL.pl to put the output in a particular
583 # directory is to cd to that directory.
584 ${cmd_buf}= Catenate cd $(dirname ${esel_file_path}) ; eSEL.pl -l
585 ... ${esel_file_path} -p decode_obmc_data
Michael Walsh02c70602018-10-04 15:24:30 -0500586 Qprint Issuing ${cmd_buf}
Michael Walsha30dac72017-04-04 17:56:23 -0500587 Run ${cmd_buf}
Michael Walsh02c70602018-10-04 15:24:30 -0500588 # The .binary file, which is generated by eSEL.pl, is of no use to us.
589 Remove File ${esel_file_path}.binary