blob: a451f3d8e8361794eb4d7e1356b71df6edab155c [file] [log] [blame]
George Keishing69e6f712016-09-12 06:30:09 -05001*** Settings ***
2Documentation Methods to execute commands on BMC and collect
3... data to a list of FFDC files
4
5Resource openbmc_ffdc_utils.robot
George Keishing8e47f532016-10-05 08:41:45 -05006Resource rest_client.robot
Gunnar Millsaca140d2016-10-26 13:05:10 -05007Resource utils.robot
George Keishingab7a9072017-02-24 13:03:24 -06008Library SSHLibrary
Sivas SRR0d657e82017-03-13 01:00:34 -05009Library OperatingSystem
George Keishing16333ec2017-03-21 06:18:29 -050010Library Collections
Michael Walshe844e9a2017-04-20 16:51:10 -050011Library String
Michael Walshe53e47a2017-06-30 17:03:24 -050012Library gen_print.py
Michael Walshe844e9a2017-04-20 16:51:10 -050013Library gen_robot_keyword.py
Gunnar Mills7e2cda22016-10-11 15:37:34 -050014
George Keishing69e6f712016-09-12 06:30:09 -050015*** Keywords ***
16
17################################################################
18# Method : Call FFDC Methods #
19# Execute the user define keywords from the FFDC List #
20# Unlike any other keywords this will call into the #
21# list of keywords defined in the FFDC list at one go #
22################################################################
23
24Call FFDC Methods
Michael Walshe844e9a2017-04-20 16:51:10 -050025 [Documentation] Call into FFDC Keyword index list.
26 [Arguments] ${ffdc_function_list}=${EMPTY}
George Keishing69e6f712016-09-12 06:30:09 -050027
Michael Walshe844e9a2017-04-20 16:51:10 -050028 # Description of argument(s):
29 # ffdc_function_list A colon-delimited list naming the kinds of FFDC that
30 # is to be collected
31 # (e.g. "FFDC Generic Report:BMC Specific Files").
32 # Acceptable values can be found in the description
33 # field of FFDC_METHOD_CALL in
34 # lib/openbmc_ffdc_list.py. Those values can be
35 # obtained via a call to 'Get FFDC Method Desc' (also
36 # from lib/openbmc_ffdc_list.py).
37
38 @{entries}= Get FFDC Method Index
39 :FOR ${index} IN @{entries}
40 \ Method Call Keyword List ${index} ${ffdc_function_list}
Michael Walsh6bad8412017-06-27 14:09:36 -050041 Run Key U SSHLibrary.Close All Connections
George Keishing69e6f712016-09-12 06:30:09 -050042
George Keishing69e6f712016-09-12 06:30:09 -050043Method Call Keyword List
Michael Walshe844e9a2017-04-20 16:51:10 -050044 [Documentation] Iterate the list through keyword index.
45 [Arguments] ${index} ${ffdc_function_list}=${EMPTY}
George Keishing69e6f712016-09-12 06:30:09 -050046
Michael Walshe844e9a2017-04-20 16:51:10 -050047 # Description of argument(s):
48 # index The index into the FFDC_METHOD_CALL dictionary (e.g.
49 # 'BMC LOGS').
50 # ffdc_function_list See ffdc_function_list description in
51 # "Call FFDC Methods" (above).
52
53 @{method_list}= Get ffdc method call ${index}
54
55 # If function list is empty assign default (i.e. a list of all allowable
56 # values). In either case, convert ffdc_function_list from a string to
57 # a list.
58 @{ffdc_function_list}=
59 ... Run Keyword If '${ffdc_function_list}' == '${EMPTY}'
60 ... Get FFDC Method Desc ${index}
61 ... ELSE
62 ... Split String ${ffdc_function_list} separator=:
63
George Keishing69e6f712016-09-12 06:30:09 -050064 :FOR ${method} IN @{method_list}
Michael Walshe844e9a2017-04-20 16:51:10 -050065 \ Execute Keyword Method ${method[0]} ${method[1]}
66 ... @{ffdc_function_list}
George Keishing69e6f712016-09-12 06:30:09 -050067
68Execute Keyword Method
Michael Walshe844e9a2017-04-20 16:51:10 -050069 [Documentation] Call into BMC method keywords. Don't let one
70 ... failure skip the remaining. Get whatever data
71 ... it could gather at worse case scenario.
72 [Arguments] ${description} ${keyword_name} @{ffdc_function_list}
George Keishing69e6f712016-09-12 06:30:09 -050073
Michael Walshe844e9a2017-04-20 16:51:10 -050074 # Description of argument(s):
75 # description The description of the FFDC to be collected. This
76 # would be any value returned by
77 # 'Get FFDC Method Desc' (e.g. "FFDC Generic Report").
78 # keyword_name The name of the keyword to call to collect the FFDC
79 # data (again, see FFDC_METHOD_CALL).
80 # ffdc_function_list See ffdc_function_list description in
81 # "Call FFDC Methods" (above). The only difference is
82 # in this case, it should be a list rather than a
83 # colon-delimited value.
George Keishing69e6f712016-09-12 06:30:09 -050084
Michael Walshe844e9a2017-04-20 16:51:10 -050085 ${status} ${ret_values}= Run Keyword And Ignore Error
86 ... List Should Contain Value ${ffdc_function_list} ${description}
87 Run Keyword If '${status}' != 'PASS' Return from Keyword
88
89 Run Key ${keyword_name} ignore=1
George Keishing69e6f712016-09-12 06:30:09 -050090
91################################################################
92# Method : BMC FFDC Manifest #
93# Execute command on BMC and write to ffdc_report.txt #
94################################################################
95
96BMC FFDC Manifest
97 [Documentation] Get the commands index for the FFDC_BMC_CMD,
98 ... login to BMC and execute commands.
George Keishing69e6f712016-09-12 06:30:09 -050099
100 @{entries}= Get ffdc cmd index
101 :FOR ${index} IN @{entries}
102 \ Iterate BMC Command List Pairs ${index}
103
104
105Iterate BMC Command List Pairs
106 [Documentation] Feed in key pair list from dictionary to execute
107 [Arguments] ${key_index}
108
109 @{cmd_list}= Get ffdc bmc cmd ${key_index}
110 Set Suite Variable ${ENTRY_INDEX} ${key_index}
111 :FOR ${cmd} IN @{cmd_list}
112 \ Execute Command and Write FFDC ${cmd[0]} ${cmd[1]}
113
114
115Execute Command and Write FFDC
Gunnar Millscce185d2016-10-17 17:04:15 -0500116 [Documentation] Execute command on BMC or OS and write to ffdc
George Keishing69e6f712016-09-12 06:30:09 -0500117 ... By default to ffdc_report.txt file else to
118 ... specified file path.
119 [Arguments] ${key_index}
120 ... ${cmd}
121 ... ${logpath}=${FFDC_FILE_PATH}
Michael Walshe53e47a2017-06-30 17:03:24 -0500122 ... ${target}=BMC
George Keishing69e6f712016-09-12 06:30:09 -0500123
124 Run Keyword If '${logpath}' == '${FFDC_FILE_PATH}'
125 ... Write Cmd Output to FFDC File ${key_index} ${cmd}
126
Michael Walshe53e47a2017-06-30 17:03:24 -0500127 ${cmd_buf}= Catenate ${target} Execute Command \ ${cmd} \ ignore_err=${1}
128 ${status} ${ret_values}= Run Key ${cmd_buf} ignore=${1}
129
130 ${stdout}= Set Variable @{ret_values}[0]
131 ${stderr}= Set Variable @{ret_values}[1]
George Keishing69e6f712016-09-12 06:30:09 -0500132
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600133 # Write stdout on success and stderr/stdout to the file on failure.
134 Run Keyword If $stderr == '${EMPTY}'
Sweta Potthuri15e6d2f2017-02-28 03:10:02 -0600135 ... Write Data To File ${stdout}${\n} ${logpath}
136 ... ELSE Write Data To File
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600137 ... ERROR output:${\n}${stderr}${\n}Output:${\n}${stdout}${\n}
138 ... ${logpath}
George Keishing69e6f712016-09-12 06:30:09 -0500139
140
141################################################################
142# Method : BMC FFDC Files #
143# Execute command on BMC and write to individual file #
144# based on the file name pre-defined in the list #
145################################################################
146
147BMC FFDC Files
148 [Documentation] Get the command list and iterate
George Keishing69e6f712016-09-12 06:30:09 -0500149 @{entries}= Get ffdc file index
150 :FOR ${index} IN @{entries}
151 \ Create File and Write Data ${index}
152
153
154Create File and Write Data
155 [Documentation] Create files to current FFDC log directory,
156 ... executes command and write to corresponding
157 ... file name in the current FFDC directory.
158 [Arguments] ${key_index}
159
160 @{cmd_list}= Get ffdc bmc file ${key_index}
161 :FOR ${cmd} IN @{cmd_list}
Gunnar Mills50070fa2016-11-14 11:17:30 -0600162 \ ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}.txt
George Keishing69e6f712016-09-12 06:30:09 -0500163 \ Execute Command and Write FFDC ${cmd[0]} ${cmd[1]} ${logpath}
Sivas SRR0d657e82017-03-13 01:00:34 -0500164
George Keishing69e6f712016-09-12 06:30:09 -0500165
166
167################################################################
168# Method : Log Test Case Status #
169# Creates test result history footprint for reference #
170################################################################
171
172Log Test Case Status
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600173 [Documentation] Test case execution result history.
174 ... Create once and append to this file
175 ... logs/test_history.txt
176 ... Format Date:Test suite:Test case:Status
177 ... 20160909214053719992:Test Warmreset:Test WarmReset via REST:FAIL
Michael Walshfa9f70f2017-04-21 16:00:18 -0500178
179 ${FFDC_DIR_PATH_STYLE}= Get Variable Value ${FFDC_DIR_PATH_STYLE}
180 ... ${EMPTY}
181 ${FFDC_DIR_PATH}= Get Variable Value ${FFDC_DIR_PATH} ${EMPTY}
182
183 Run Keyword If '${FFDC_DIR_PATH}' == '${EMPTY}' Set FFDC Defaults
184
185 Run Keyword If '${FFDC_DIR_PATH_STYLE}' == '${1}' Run Keywords
186 ... Set Global Variable ${FFDC_LOG_PATH} ${FFDC_DIR_PATH} AND
187 ... Set Global Variable ${TEST_HISTORY} ${FFDC_DIR_PATH}test_history.txt
188
George Keishing69e6f712016-09-12 06:30:09 -0500189 Create Directory ${FFDC_LOG_PATH}
190
191 ${exist}= Run Keyword and Return Status
192 ... OperatingSystem.File Should Exist ${TEST_HISTORY}
193
194 Run Keyword If '${exist}' == '${False}'
195 ... Create File ${TEST_HISTORY}
196
Michael Walshfa9f70f2017-04-21 16:00:18 -0500197 Rpvars TEST_HISTORY
198
George Keishing69e6f712016-09-12 06:30:09 -0500199 ${cur_time}= Get Current Time Stamp
200
201 Append To File ${TEST_HISTORY}
202 ... ${cur_time}:${SUITE_NAME}:${TEST_NAME}:${TEST_STATUS}${\n}
203
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500204
205Log FFDC Get Requests
206 [Documentation] Create file in current FFDC log directory.
207 ... Do openbmc get request and write to
208 ... corresponding file name.
George Keishing8e47f532016-10-05 08:41:45 -0500209 ... JSON pretty print for logging to file.
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500210 [Arguments] ${key_index}
211
212 @{cmd_list}= Get ffdc get request ${key_index}
213 :FOR ${cmd} IN @{cmd_list}
Gunnar Mills50070fa2016-11-14 11:17:30 -0600214 \ ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}.txt
George Keishingd71e9122016-11-30 11:15:21 -0600215 \ ${resp}= OpenBMC Get Request ${cmd[1]} quiet=${1}
George Keishing8e47f532016-10-05 08:41:45 -0500216 \ ${status}= Run Keyword and Return Status
217 ... Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
218 \ Run Keyword If '${status}' == '${False}' Continue For Loop
219 \ ${jsondata}= to json ${resp.content} pretty_print=True
Sweta Potthuri15e6d2f2017-02-28 03:10:02 -0600220 \ Write Data To File ${\n}${jsondata}${\n} ${logpath}
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500221
222
223BMC FFDC Get Requests
224 [Documentation] Get the command list and iterate
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500225 @{entries}= Get ffdc get request index
226 :FOR ${index} IN @{entries}
227 \ Log FFDC Get Requests ${index}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500228
229
Gunnar Millscce185d2016-10-17 17:04:15 -0500230Log OS ALL DISTROS FFDC
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500231 [Documentation] Create file in current FFDC log directory.
232 ... Executes OS command and write to
233 ... corresponding file name.
234 [Arguments] ${key_index}
235
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500236 @{cmd_list}= get ffdc os all distros call ${key_index}
237 :FOR ${cmd} IN @{cmd_list}
Gunnar Mills50070fa2016-11-14 11:17:30 -0600238 \ ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}.txt
Michael Walshe53e47a2017-06-30 17:03:24 -0500239 \ Execute Command and Write FFDC ${cmd[0]} ${cmd[1]} ${logpath}
240 \ ... target=OS
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500241
242
Gunnar Millscce185d2016-10-17 17:04:15 -0500243Log OS SPECIFIC DISTRO FFDC
244 [Documentation] Create file in current FFDC log directory.
245 ... Executes OS command and write to
246 ... corresponding file name.
247 [Arguments] ${key_index} ${linux_distro}
248
249 @{cmd_list}= get ffdc os distro call ${key_index} ${linux_distro}
250 :FOR ${cmd} IN @{cmd_list}
Gunnar Mills50070fa2016-11-14 11:17:30 -0600251 \ ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}.txt
Michael Walshe53e47a2017-06-30 17:03:24 -0500252 \ Execute Command and Write FFDC ${cmd[0]} ${cmd[1]} ${logpath}
253 \ ... target=OS
Gunnar Millscce185d2016-10-17 17:04:15 -0500254
255
George Keishingca8c4c62016-10-14 10:08:40 -0500256OS FFDC Files
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500257 [Documentation] Get the command list and iterate
Gunnar Millsaca140d2016-10-26 13:05:10 -0500258 [Arguments] ${OS_HOST}=${OS_HOST} ${OS_USERNAME}=${OS_USERNAME}
259 ... ${OS_PASSWORD}=${OS_PASSWORD}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500260
261 Return From Keyword If '${OS_HOST}' == '${EMPTY}'
262 ... No OS Host Provided
Gunnar Millsaca140d2016-10-26 13:05:10 -0500263
264 # If can't ping, return
265 ${rc}= Run Keyword and Return Status Ping Host ${OS_HOST}
266 Return From Keyword If '${rc}' == '${False}'
Michael Walshe53e47a2017-06-30 17:03:24 -0500267 ... Could not ping OS.
Gunnar Millsaca140d2016-10-26 13:05:10 -0500268
Michael Walshe53e47a2017-06-30 17:03:24 -0500269 ${stdout} ${stderr} ${rc}= OS Execute Command uptime ignore_err=${1}
270 Return From Keyword If '${rc}' != '${0}' Could not connect to OS.
Gunnar Millscce185d2016-10-17 17:04:15 -0500271
Michael Walshe53e47a2017-06-30 17:03:24 -0500272 ${stdout} ${stderr} ${rc}= OS Execute Command
273 ... . /etc/os-release; echo $ID ignore_err=${0}
274 Set Global Variable ${linux_distro} ${stdout}
Gunnar Millsaca140d2016-10-26 13:05:10 -0500275
Michael Walshe53e47a2017-06-30 17:03:24 -0500276 Rpvars linux_distro
Gunnar Millsaca140d2016-10-26 13:05:10 -0500277
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500278 @{entries}= Get ffdc os all distros index
279 :FOR ${index} IN @{entries}
Gunnar Millscce185d2016-10-17 17:04:15 -0500280 \ Log OS ALL DISTROS FFDC ${index}
281
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600282 Return From Keyword If
283 ... '${linux_distro}' == '${EMPTY}' or '${linux_distro}' == 'None'
Michael Walshe53e47a2017-06-30 17:03:24 -0500284 ... Could not determine Linux Distribution.
Gunnar Millscce185d2016-10-17 17:04:15 -0500285
286 @{entries}= Get ffdc os distro index ${linux_distro}
287 :FOR ${index} IN @{entries}
288 \ Log OS SPECIFIC DISTRO FFDC ${index} ${linux_distro}
George Keishing7a520222017-02-27 09:44:30 -0600289
290
Steven Sombar96de5732017-08-17 09:04:52 -0500291System Inventory Files
292 [Documentation] Copy systest os_inventory files.
293 # The os_inventory files are the result of running
294 # systest/htx_hardbootme_test. If these files exist
295 # they are copied to the FFDC directory.
296 # Global variable ffdc_dir_path is the path name of the
297 # directory they are copied to.
298 Copy Files os_inventory_*.json ${ffdc_dir_path}
299 Remove Files os_inventory_*.json
300
301
George Keishing7a520222017-02-27 09:44:30 -0600302##############################################################################
303SCP Coredump Files
304 [Documentation] Copy core dump file from BMC to local system.
Michael Walshe53e47a2017-06-30 17:03:24 -0500305
George Keishing7a520222017-02-27 09:44:30 -0600306 # Check if core dump exist in the /tmp
Michael Walshe53e47a2017-06-30 17:03:24 -0500307 ${core_files} ${stderr} ${rc}= BMC Execute Command ls /tmp/core_*
George Keishing7a520222017-02-27 09:44:30 -0600308 @{core_list} = Split String ${core_files}
309 # Copy the core files
Michael Walsh6bad8412017-06-27 14:09:36 -0500310 Run Key U Open Connection for SCP
George Keishing7a520222017-02-27 09:44:30 -0600311 :FOR ${index} IN @{core_list}
312 \ scp.Get File ${index} ${LOG_PREFIX}${index.lstrip("/tmp/")}
313 # Remove the file from remote to avoid re-copying on next FFDC call
Michael Walshe53e47a2017-06-30 17:03:24 -0500314 \ BMC Execute Command rm ${index}
315 # I can't find a way to do this: scp.Close Connection
George Keishing7a520222017-02-27 09:44:30 -0600316
George Keishingeb7b4bc2017-03-13 12:04:28 -0500317
318##############################################################################
319Collect eSEL Log
George Keishingb1ca69b2017-03-31 14:36:42 -0500320 [Documentation] Collect eSEL log from logging entry and convert eSEL data
321 ... to elog formated string text file.
Sridevi Rameshb180c9f2017-08-06 10:27:41 -0500322 [Arguments] ${log_prefix_path}=${LOG_PREFIX}
323
George Keishingeb7b4bc2017-03-13 12:04:28 -0500324 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}/enumerate quiet=${1}
325 ${status}= Run Keyword And Return Status
326 ... Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
327 Return From Keyword If '${status}' == '${False}'
328
329 ${content}= To Json ${resp.content}
330 # Grab the list of entries from logging/entry/
331 # The data shown below is the result of the "Get Dictionary Keys".
332 # Example:
333 # /xyz/openbmc_project/logging/entry/1
334 # /xyz/openbmc_project/logging/entry/2
335 ${esel_list}= Get Dictionary Keys ${content['data']}
336
Sridevi Rameshb180c9f2017-08-06 10:27:41 -0500337 ${logpath}= Catenate SEPARATOR= ${log_prefix_path} esel
George Keishingeb7b4bc2017-03-13 12:04:28 -0500338 Create File ${logpath}
339 # Fetch data from /xyz/openbmc_project/logging/entry/1/attr/AdditionalData
340 # "ESEL=00 00 df 00 00 00 00 20 00 04 12 35 6f aa 00 00 "
341 # Sample eSEL entry:
342 # "/xyz/openbmc_project/logging/entry/1": {
343 # "Timestamp": 1487744317025,
344 # "AdditionalData": [
345 # "ESEL=00 00 df 00 00 00 00 20 00 04 12 35 6f aa 00 00 "
346 # ],
347 # "Message": "org.open_power.Error.Host.Event.Event",
348 # "Id": 1,
349 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Emergency"
350 # }
351
352 :FOR ${entry_path} IN @{esel_list}
George Keishing28da9482017-07-27 13:38:54 -0500353 # Skip reading attribute if entry URI is a callout.
354 # Example: /xyz/openbmc_project/logging/entry/1/callout
355 \ Continue For Loop If '${entry_path.rsplit('/', 1)[1]}' == 'callout'
George Keishingeb7b4bc2017-03-13 12:04:28 -0500356 \ ${esel_data}= Read Attribute ${entry_path} AdditionalData quiet=${1}
George Keishing6db1d362017-09-22 12:36:56 -0500357 \ ${status}= Run Keyword And Return Status
358 ... Should Contain Match ${esel_data} ESEL*
359 \ Continue For Loop If ${status} == ${False}
George Keishing28da9482017-07-27 13:38:54 -0500360 \ ${index}= Get Esel Index ${esel_data}
361 \ Write Data To File "${esel_data[${index}]}" ${logpath}
George Keishingeb7b4bc2017-03-13 12:04:28 -0500362 \ Write Data To File ${\n} ${logpath}
363
George Keishingb1ca69b2017-03-31 14:36:42 -0500364 ${out}= Run which eSEL.pl
365 ${status}= Run Keyword And Return Status
366 ... Should Contain ${out} eSEL.pl
367 Return From Keyword If '${status}' == '${False}'
368
369 Convert eSEL To Elog Format ${logpath}
370
371
372##############################################################################
373Convert eSEL To Elog Format
374 [Documentation] Execute parser tool on the eSEL data file to generate
Michael Walsha30dac72017-04-04 17:56:23 -0500375 ... formatted error log.
George Keishingb1ca69b2017-03-31 14:36:42 -0500376 [Arguments] ${esel_file_path}
377 # Desription of arguments:
Michael Walsha30dac72017-04-04 17:56:23 -0500378 # esel_file_path Absolute path of the eSEL data (e.g.
379 # /tmp/w55.170404.154820.esel).
George Keishingb1ca69b2017-03-31 14:36:42 -0500380
Michael Walsha30dac72017-04-04 17:56:23 -0500381 # Note: The only way to get eSEL.pl to put the output in a particular
382 # directory is to cd to that directory.
383 ${cmd_buf}= Catenate cd $(dirname ${esel_file_path}) ; eSEL.pl -l
384 ... ${esel_file_path} -p decode_obmc_data
385 Run ${cmd_buf}
George Keishingb1ca69b2017-03-31 14:36:42 -0500386
George Keishingeb7b4bc2017-03-13 12:04:28 -0500387##############################################################################