blob: 2cd8207fdc980e9ffaa158339d529485c64c804d [file] [log] [blame]
George Keishing69e6f712016-09-12 06:30:09 -05001*** Settings ***
2Documentation Methods to execute commands on BMC and collect
3... data to a list of FFDC files
4
5Resource openbmc_ffdc_utils.robot
George Keishing8e47f532016-10-05 08:41:45 -05006Resource rest_client.robot
Gunnar Millsaca140d2016-10-26 13:05:10 -05007Resource utils.robot
George Keishingab7a9072017-02-24 13:03:24 -06008Library SSHLibrary
Sivas SRR0d657e82017-03-13 01:00:34 -05009Library OperatingSystem
George Keishing16333ec2017-03-21 06:18:29 -050010Library Collections
Gunnar Mills7e2cda22016-10-11 15:37:34 -050011
George Keishing69e6f712016-09-12 06:30:09 -050012*** Keywords ***
13
14################################################################
15# Method : Call FFDC Methods #
16# Execute the user define keywords from the FFDC List #
17# Unlike any other keywords this will call into the #
18# list of keywords defined in the FFDC list at one go #
19################################################################
20
21Call FFDC Methods
22 [Documentation] Calls into FFDC Keyword index list
23
24 @{entries}= Get ffdc method index
25 :FOR ${index} IN @{entries}
26 \ Method Call Keyword List ${index}
George Keishingab7a9072017-02-24 13:03:24 -060027 SSHLibrary.Close All Connections
George Keishing69e6f712016-09-12 06:30:09 -050028
29
30Method Call Keyword List
31 [Documentation] Iterate the list through keyword index
32 [Arguments] ${index}
33
34 @{method_list}= Get ffdc method call ${index}
35 :FOR ${method} IN @{method_list}
36 \ Execute Keyword Method ${method[1]}
37
38
39Execute Keyword Method
40 [Documentation] Calls into BMC method keywords. Don't let one
41 ... failure skips the remaining. Get whatever data
42 ... it could gather at worse case scenario.
43 [Arguments] ${keyword_name}
44
45 Run Keyword And Continue On Failure ${keyword_name}
46
47
48################################################################
49# Method : BMC FFDC Manifest #
50# Execute command on BMC and write to ffdc_report.txt #
51################################################################
52
53BMC FFDC Manifest
54 [Documentation] Get the commands index for the FFDC_BMC_CMD,
55 ... login to BMC and execute commands.
56 Open Connection And Log In
57
58 @{entries}= Get ffdc cmd index
59 :FOR ${index} IN @{entries}
60 \ Iterate BMC Command List Pairs ${index}
61
62
63Iterate BMC Command List Pairs
64 [Documentation] Feed in key pair list from dictionary to execute
65 [Arguments] ${key_index}
66
67 @{cmd_list}= Get ffdc bmc cmd ${key_index}
68 Set Suite Variable ${ENTRY_INDEX} ${key_index}
69 :FOR ${cmd} IN @{cmd_list}
70 \ Execute Command and Write FFDC ${cmd[0]} ${cmd[1]}
71
72
73Execute Command and Write FFDC
Gunnar Millscce185d2016-10-17 17:04:15 -050074 [Documentation] Execute command on BMC or OS and write to ffdc
George Keishing69e6f712016-09-12 06:30:09 -050075 ... By default to ffdc_report.txt file else to
76 ... specified file path.
77 [Arguments] ${key_index}
78 ... ${cmd}
79 ... ${logpath}=${FFDC_FILE_PATH}
80
81 Run Keyword If '${logpath}' == '${FFDC_FILE_PATH}'
82 ... Write Cmd Output to FFDC File ${key_index} ${cmd}
83
84 ${stdout} ${stderr}=
85 ... Execute Command ${cmd} return_stderr=True
86
Jeff Whitlock7d805b22017-01-20 14:04:33 -060087 # Write stdout on success and stderr/stdout to the file on failure.
88 Run Keyword If $stderr == '${EMPTY}'
Sweta Potthuri15e6d2f2017-02-28 03:10:02 -060089 ... Write Data To File ${stdout}${\n} ${logpath}
90 ... ELSE Write Data To File
Jeff Whitlock7d805b22017-01-20 14:04:33 -060091 ... ERROR output:${\n}${stderr}${\n}Output:${\n}${stdout}${\n}
92 ... ${logpath}
George Keishing69e6f712016-09-12 06:30:09 -050093
94
95################################################################
96# Method : BMC FFDC Files #
97# Execute command on BMC and write to individual file #
98# based on the file name pre-defined in the list #
99################################################################
100
101BMC FFDC Files
102 [Documentation] Get the command list and iterate
103 Open Connection And Log In
104 @{entries}= Get ffdc file index
105 :FOR ${index} IN @{entries}
106 \ Create File and Write Data ${index}
107
108
109Create File and Write Data
110 [Documentation] Create files to current FFDC log directory,
111 ... executes command and write to corresponding
112 ... file name in the current FFDC directory.
113 [Arguments] ${key_index}
114
Sivas SRR0d657e82017-03-13 01:00:34 -0500115 # To build IP address in searchable form eg: dummy\.domain\.com
116 ${OPENBMC_HOST_REGEX}= Run echo ${OPENBMC_HOST} | sed 's/\(\.\)/\\\1/g'
George Keishing69e6f712016-09-12 06:30:09 -0500117 @{cmd_list}= Get ffdc bmc file ${key_index}
118 :FOR ${cmd} IN @{cmd_list}
Gunnar Mills50070fa2016-11-14 11:17:30 -0600119 \ ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}.txt
George Keishing69e6f712016-09-12 06:30:09 -0500120 \ Execute Command and Write FFDC ${cmd[0]} ${cmd[1]} ${logpath}
Sivas SRR0d657e82017-03-13 01:00:34 -0500121 # Rename OPENBMC_HOST IP address from given file to DUMMYIP
122 \ Run sed -i 's/'${OPENBMC_HOST_REGEX}'/DUMMYIP/g' ${logpath}
123
George Keishing69e6f712016-09-12 06:30:09 -0500124
125
126################################################################
127# Method : Log Test Case Status #
128# Creates test result history footprint for reference #
129################################################################
130
131Log Test Case Status
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600132 [Documentation] Test case execution result history.
133 ... Create once and append to this file
134 ... logs/test_history.txt
135 ... Format Date:Test suite:Test case:Status
136 ... 20160909214053719992:Test Warmreset:Test WarmReset via REST:FAIL
George Keishing69e6f712016-09-12 06:30:09 -0500137 Create Directory ${FFDC_LOG_PATH}
138
139 ${exist}= Run Keyword and Return Status
140 ... OperatingSystem.File Should Exist ${TEST_HISTORY}
141
142 Run Keyword If '${exist}' == '${False}'
143 ... Create File ${TEST_HISTORY}
144
145 ${cur_time}= Get Current Time Stamp
146
147 Append To File ${TEST_HISTORY}
148 ... ${cur_time}:${SUITE_NAME}:${TEST_NAME}:${TEST_STATUS}${\n}
149
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500150
151Log FFDC Get Requests
152 [Documentation] Create file in current FFDC log directory.
153 ... Do openbmc get request and write to
154 ... corresponding file name.
George Keishing8e47f532016-10-05 08:41:45 -0500155 ... JSON pretty print for logging to file.
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500156 [Arguments] ${key_index}
157
158 @{cmd_list}= Get ffdc get request ${key_index}
159 :FOR ${cmd} IN @{cmd_list}
Gunnar Mills50070fa2016-11-14 11:17:30 -0600160 \ ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}.txt
George Keishingd71e9122016-11-30 11:15:21 -0600161 \ ${resp}= OpenBMC Get Request ${cmd[1]} quiet=${1}
George Keishing8e47f532016-10-05 08:41:45 -0500162 \ ${status}= Run Keyword and Return Status
163 ... Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
164 \ Run Keyword If '${status}' == '${False}' Continue For Loop
165 \ ${jsondata}= to json ${resp.content} pretty_print=True
Sweta Potthuri15e6d2f2017-02-28 03:10:02 -0600166 \ Write Data To File ${\n}${jsondata}${\n} ${logpath}
Gunnar Millsa812e0f2016-09-29 20:30:03 -0500167
168
169BMC FFDC Get Requests
170 [Documentation] Get the command list and iterate
171 Open Connection And Log In
172 @{entries}= Get ffdc get request index
173 :FOR ${index} IN @{entries}
174 \ Log FFDC Get Requests ${index}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500175
176
Gunnar Millscce185d2016-10-17 17:04:15 -0500177Log OS ALL DISTROS FFDC
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500178 [Documentation] Create file in current FFDC log directory.
179 ... Executes OS command and write to
180 ... corresponding file name.
181 [Arguments] ${key_index}
182
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500183 @{cmd_list}= get ffdc os all distros call ${key_index}
184 :FOR ${cmd} IN @{cmd_list}
Gunnar Mills50070fa2016-11-14 11:17:30 -0600185 \ ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}.txt
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500186 \ Execute Command and Write FFDC ${cmd[0]} ${cmd[1]} ${logpath}
187
188
Gunnar Millscce185d2016-10-17 17:04:15 -0500189Log OS SPECIFIC DISTRO FFDC
190 [Documentation] Create file in current FFDC log directory.
191 ... Executes OS command and write to
192 ... corresponding file name.
193 [Arguments] ${key_index} ${linux_distro}
194
195 @{cmd_list}= get ffdc os distro call ${key_index} ${linux_distro}
196 :FOR ${cmd} IN @{cmd_list}
Gunnar Mills50070fa2016-11-14 11:17:30 -0600197 \ ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}.txt
Gunnar Millscce185d2016-10-17 17:04:15 -0500198 \ Execute Command and Write FFDC ${cmd[0]} ${cmd[1]} ${logpath}
199
200
201
George Keishingca8c4c62016-10-14 10:08:40 -0500202OS FFDC Files
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500203 [Documentation] Get the command list and iterate
Gunnar Millsaca140d2016-10-26 13:05:10 -0500204 [Arguments] ${OS_HOST}=${OS_HOST} ${OS_USERNAME}=${OS_USERNAME}
205 ... ${OS_PASSWORD}=${OS_PASSWORD}
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500206
207 Return From Keyword If '${OS_HOST}' == '${EMPTY}'
208 ... No OS Host Provided
Gunnar Millsaca140d2016-10-26 13:05:10 -0500209
210 # If can't ping, return
211 ${rc}= Run Keyword and Return Status Ping Host ${OS_HOST}
212 Return From Keyword If '${rc}' == '${False}'
213 ... Could not ping OS
214
Gunnar Millscce185d2016-10-17 17:04:15 -0500215 Open Connection And Log In host=${OS_HOST} username=${OS_USERNAME}
216 ... password=${OS_PASSWORD}
217
Gunnar Millsaca140d2016-10-26 13:05:10 -0500218 ${output} ${stderr} ${rc}= Execute Command uptime return_stderr=True
219 ... return_rc=True
220
221 # If the return code returned by "Execute Command" is non-zero, return
222 Return From Keyword If '${rc}' != '${0}'
223 ... Could not connect to OS
224
Gunnar Mills7e2cda22016-10-11 15:37:34 -0500225 @{entries}= Get ffdc os all distros index
226 :FOR ${index} IN @{entries}
Gunnar Millscce185d2016-10-17 17:04:15 -0500227 \ Log OS ALL DISTROS FFDC ${index}
228
229 ${linux_distro}= Execute Command
230 ... . /etc/os-release; echo $ID
231 ... return_stdout=True return_stderr=False return_rc=False
232
Jeff Whitlock7d805b22017-01-20 14:04:33 -0600233 Return From Keyword If
234 ... '${linux_distro}' == '${EMPTY}' or '${linux_distro}' == 'None'
235 ... Could not determine Linux Distribution
Gunnar Millscce185d2016-10-17 17:04:15 -0500236
237 @{entries}= Get ffdc os distro index ${linux_distro}
238 :FOR ${index} IN @{entries}
239 \ Log OS SPECIFIC DISTRO FFDC ${index} ${linux_distro}
George Keishing7a520222017-02-27 09:44:30 -0600240
241
242##############################################################################
243SCP Coredump Files
244 [Documentation] Copy core dump file from BMC to local system.
245 # Check if core dump exist in the /tmp
246 Open Connection And Log In
Michael Walsh87b1ce32017-03-03 15:21:42 -0600247 ${core_files}= Execute Command ls /tmp/core_*
George Keishing7a520222017-02-27 09:44:30 -0600248 @{core_list} = Split String ${core_files}
249 # Copy the core files
250 Open Connection for SCP
251 :FOR ${index} IN @{core_list}
252 \ scp.Get File ${index} ${LOG_PREFIX}${index.lstrip("/tmp/")}
253 # Remove the file from remote to avoid re-copying on next FFDC call
254 \ Execute Command On BMC rm ${index}
255
George Keishingeb7b4bc2017-03-13 12:04:28 -0500256
257##############################################################################
258Collect eSEL Log
George Keishingb1ca69b2017-03-31 14:36:42 -0500259 [Documentation] Collect eSEL log from logging entry and convert eSEL data
260 ... to elog formated string text file.
George Keishingeb7b4bc2017-03-13 12:04:28 -0500261 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}/enumerate quiet=${1}
262 ${status}= Run Keyword And Return Status
263 ... Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
264 Return From Keyword If '${status}' == '${False}'
265
266 ${content}= To Json ${resp.content}
267 # Grab the list of entries from logging/entry/
268 # The data shown below is the result of the "Get Dictionary Keys".
269 # Example:
270 # /xyz/openbmc_project/logging/entry/1
271 # /xyz/openbmc_project/logging/entry/2
272 ${esel_list}= Get Dictionary Keys ${content['data']}
273
Sweta Potthuri92e7b132017-03-17 06:02:32 -0500274 ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} esel
George Keishingeb7b4bc2017-03-13 12:04:28 -0500275 Create File ${logpath}
276 # Fetch data from /xyz/openbmc_project/logging/entry/1/attr/AdditionalData
277 # "ESEL=00 00 df 00 00 00 00 20 00 04 12 35 6f aa 00 00 "
278 # Sample eSEL entry:
279 # "/xyz/openbmc_project/logging/entry/1": {
280 # "Timestamp": 1487744317025,
281 # "AdditionalData": [
282 # "ESEL=00 00 df 00 00 00 00 20 00 04 12 35 6f aa 00 00 "
283 # ],
284 # "Message": "org.open_power.Error.Host.Event.Event",
285 # "Id": 1,
286 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Emergency"
287 # }
288
289 :FOR ${entry_path} IN @{esel_list}
290 \ ${esel_data}= Read Attribute ${entry_path} AdditionalData quiet=${1}
George Keishing16333ec2017-03-21 06:18:29 -0500291 \ ${length}= Get Length ${esel_data}
292 # Skip writting to file if eSEL AdditionalData is empty
293 \ Continue For Loop If ${length} == ${0}
George Keishingeb7b4bc2017-03-13 12:04:28 -0500294 \ Write Data To File "${esel_data[0]}" ${logpath}
295 \ Write Data To File ${\n} ${logpath}
296
George Keishingb1ca69b2017-03-31 14:36:42 -0500297 ${out}= Run which eSEL.pl
298 ${status}= Run Keyword And Return Status
299 ... Should Contain ${out} eSEL.pl
300 Return From Keyword If '${status}' == '${False}'
301
302 Convert eSEL To Elog Format ${logpath}
303
304
305##############################################################################
306Convert eSEL To Elog Format
307 [Documentation] Execute parser tool on the eSEL data file to generate
Michael Walsha30dac72017-04-04 17:56:23 -0500308 ... formatted error log.
George Keishingb1ca69b2017-03-31 14:36:42 -0500309 [Arguments] ${esel_file_path}
310 # Desription of arguments:
Michael Walsha30dac72017-04-04 17:56:23 -0500311 # esel_file_path Absolute path of the eSEL data (e.g.
312 # /tmp/w55.170404.154820.esel).
George Keishingb1ca69b2017-03-31 14:36:42 -0500313
Michael Walsha30dac72017-04-04 17:56:23 -0500314 # Note: The only way to get eSEL.pl to put the output in a particular
315 # directory is to cd to that directory.
316 ${cmd_buf}= Catenate cd $(dirname ${esel_file_path}) ; eSEL.pl -l
317 ... ${esel_file_path} -p decode_obmc_data
318 Run ${cmd_buf}
George Keishingb1ca69b2017-03-31 14:36:42 -0500319
George Keishingeb7b4bc2017-03-13 12:04:28 -0500320##############################################################################