blob: ee0deba3559a54a2608b612d0a1f0027c3f7a813 [file] [log] [blame]
Michael Tritz935c18e2017-03-08 11:45:41 -06001*** Settings ***
2Documentation Test Error logging.
3
4Resource ../lib/connection_client.robot
5Resource ../lib/openbmc_ffdc.robot
6Resource ../lib/utils.robot
7Resource ../lib/state_manager.robot
Rahul Maheshwari92972e72017-06-22 00:59:59 -05008Resource ../lib/ipmi_client.robot
George Keishing1d82dd12017-07-24 01:16:23 -05009Resource ../lib/boot_utils.robot
Michael Tritz935c18e2017-03-08 11:45:41 -060010
George Keishing95c371a2017-09-14 15:42:33 -050011Test Setup Test Setup Execution
George Keishing845d9dc2017-06-16 14:49:29 -050012Test Teardown Post Test Case Execution
George Keishing184a89a2017-06-27 07:12:16 -050013Suite Teardown Delete Error Logs And Verify
Michael Tritz935c18e2017-03-08 11:45:41 -060014
George Keishing1d82dd12017-07-24 01:16:23 -050015*** Variables ***
16
17${stack_mode} skip
18
Michael Tritz935c18e2017-03-08 11:45:41 -060019*** Test Cases ***
20
George Keishing1d82dd12017-07-24 01:16:23 -050021Error Log Check After BMC Reboot
22 [Documentation] Check error log after BMC rebooted.
23 [Tags] Error_Log_Check_At_BMC_Ready
24 # 1. Power off.
25 # 2. Delete error logs.
26 # 3. Reboot BMC.
27 # 4. Check if eror log exists.
28
George Keishing3f4a3eb2017-09-26 10:33:16 -050029 Smart Power Off
George Keishing1d82dd12017-07-24 01:16:23 -050030 Delete Error Logs And Verify
George Keishing3f4a3eb2017-09-26 10:33:16 -050031 OBMC Reboot (off) stack_mode=normal
George Keishing1d82dd12017-07-24 01:16:23 -050032 Error Logs Should Not Exist
33
34
35Error Log Check After Host Poweron
36 [Documentation] Check error log after host has booted.
37 [Tags] Error_Log_Check_At_Host_Booted
38 # 1. Delete error logs
39 # 1. Power on.
40 # 3. Check if eror log exists.
41
42 Delete Error Logs And Verify
43 REST Power On
44 Error Logs Should Not Exist
45
46
Michael Tritz935c18e2017-03-08 11:45:41 -060047Create Test Error And Verify
48 [Documentation] Create error logs and verify via REST.
49 [Tags] Create_Test_Error_And_Verify
50
51 Create Test Error Log
52 Verify Test Error Log
53
54
55Test Error Persistency On Restart
George Keishing184a89a2017-06-27 07:12:16 -050056 [Documentation] Restart logging service and verify error logs.
Michael Tritz935c18e2017-03-08 11:45:41 -060057 [Tags] Test_Error_Persistency_On_Restart
58
59 Create Test Error Log
60 Verify Test Error Log
61 Execute Command On BMC
62 ... systemctl restart xyz.openbmc_project.Logging.service
63 Sleep 10s reason=Wait for logging service to restart properly.
George Keishing184a89a2017-06-27 07:12:16 -050064 Verify Test Error Log
Michael Tritz935c18e2017-03-08 11:45:41 -060065
66
George Keishing742b0602017-07-28 14:20:53 -050067Test Error Entry Numbering Reset On Restart
68 [Documentation] Restarts logging service and verify error logs entry start
69 ... from entry "Id" 1.
70 # 1. Create error log.
71 # 2. Verify error log.
72 # 3. Delete error log.
73 # 4. Restart logging service.
74 # 5. Create error log.
75 # 6. Verify new error log entry starts with Id entry 1.
76
77 [Tags] Test_Error_Entry_Numbering_Reset_On_Restart
78 # Example Error logs:
79 # "/xyz/openbmc_project/logging/entry/1": {
80 # "AdditionalData": [
81 # "STRING=FOO"
82 # ],
83 # "Id": 1, <--- Entry value should be 1.
84 # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
85 # "Resolved": 0,
86 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
87 # "Timestamp": 1490818990051,
88 # "associations": []
89 # },
90
91 Create Test Error Log
92 Verify Test Error Log
George Keishingcc71c3c2017-09-20 09:38:03 -050093 Delete All Error Logs
George Keishing742b0602017-07-28 14:20:53 -050094 Execute Command On BMC
95 ... systemctl restart xyz.openbmc_project.Logging.service
96 Sleep 10s reason=Wait for logging service to restart properly.
97 Create Test Error Log
98 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
99 ${entry_id}= Read Attribute ${elog_entry[0]} Id
100 Should Be Equal ${entry_id} ${1}
101
102
Michael Tritz935c18e2017-03-08 11:45:41 -0600103Test Error Persistency On Reboot
George Keishing184a89a2017-06-27 07:12:16 -0500104 [Documentation] Reboot BMC and verify error logs.
Michael Tritz935c18e2017-03-08 11:45:41 -0600105 [Tags] Test_Error_Persistency_On_Reboot
106
107 Create Test Error Log
108 Verify Test Error Log
109 Initiate BMC Reboot
110 Wait Until Keyword Succeeds 10 min 10 sec
111 ... Is BMC Ready
George Keishing184a89a2017-06-27 07:12:16 -0500112 Verify Test Error Log
Michael Tritz935c18e2017-03-08 11:45:41 -0600113
114
George Keishing04bc4812017-03-30 03:33:31 -0500115Create Test Error And Verify Resolved Field
116 [Documentation] Create error log and verify "Resolved"
117 ... field is 0.
118 [Tags] Create_Test_Error_And_Verify_Resolved_Field
119
120 # Example Error log:
121 # "/xyz/openbmc_project/logging/entry/1": {
122 # "AdditionalData": [
123 # "STRING=FOO"
124 # ],
125 # "Id": 1,
126 # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
127 # "Resolved": 0,
128 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
129 # "Timestamp": 1490817164983,
130 # "associations": []
131 # },
132
133 # It's work in progress, but it's mnfg need. To mark an error as
134 # resolved, without deleting the error, mfg will set this bool
135 # property.
136 # In this test context we are making sure "Resolved" field is "0"
137 # by default.
138
George Keishingcc71c3c2017-09-20 09:38:03 -0500139 Delete All Error Logs
George Keishing04bc4812017-03-30 03:33:31 -0500140 Create Test Error Log
George Keishing184a89a2017-06-27 07:12:16 -0500141 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
142 ${resolved}= Read Attribute ${elog_entry[0]} Resolved
George Keishing04bc4812017-03-30 03:33:31 -0500143 Should Be True ${resolved} == 0
144
145
146Create Test Errors And Verify Time Stamp
147 [Documentation] Create error logs and verify time stamp.
148 [Tags] Create_Test_Error_And_Verify_Time_Stamp
149
150 # Example Error logs:
151 # "/xyz/openbmc_project/logging/entry/1": {
152 # "AdditionalData": [
153 # "STRING=FOO"
154 # ],
155 # "Id": 1,
156 # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
157 # "Resolved": 0,
158 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
159 # "Timestamp": 1490818990051, <--- Time stamp
160 # "associations": []
161 # },
162 # "/xyz/openbmc_project/logging/entry/2": {
163 # "AdditionalData": [
164 # "STRING=FOO"
165 # ],
166 # "Id": 2,
167 # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
168 # "Resolved": 0,
169 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
170 # "Timestamp": 1490818992116, <---- Time stamp
171 # "associations": []
172 # },
173
George Keishingcc71c3c2017-09-20 09:38:03 -0500174 Delete All Error Logs
George Keishing04bc4812017-03-30 03:33:31 -0500175 Create Test Error Log
176 Create Test Error Log
177 # The error log generated is associated with the epoc time and unique
178 # for every error and in increasing time stamp.
George Keishing184a89a2017-06-27 07:12:16 -0500179 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
180 ${time_stamp1}= Read Attribute ${elog_entry[0]} Timestamp
181 ${time_stamp2}= Read Attribute ${elog_entry[1]} Timestamp
George Keishing6eb10b82017-06-06 10:02:09 -0500182 Should Be True ${time_stamp2} > ${time_stamp1}
George Keishing04bc4812017-03-30 03:33:31 -0500183
Sweta Potthuri9ce8eeb2017-05-08 04:20:11 -0500184Create Test Error Log And Delete
185 [Documentation] Create an error log and delete it.
186 [Tags] Create_Test_Error_Log_And_Delete
187
188 Delete Error Logs And Verify
189 Create Test Error Log
190 Delete Error Logs And Verify
191
192Create Multiple Test Error Logs And Delete All
193 [Documentation] Create multiple error logs and delete all.
194 [Tags] Create_Multiple_Test_Error_Logs_And_Delete_All
195
196 Delete Error Logs And Verify
197 Create Test Error Log
198 Create Test Error Log
199 Create Test Error Log
200 Delete Error Logs And Verify
201
202Create Two Test Error Logs And Delete One
203 [Documentation] Create two error logs and delete the first entry.
204 [Tags] Create_Two_Test_Error_Logs_And_Delete_One
205
206 Delete Error Logs And Verify
207 Create Test Error Log
George Keishing184a89a2017-06-27 07:12:16 -0500208 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
Sweta Potthuri9ce8eeb2017-05-08 04:20:11 -0500209 Create Test Error Log
George Keishing184a89a2017-06-27 07:12:16 -0500210 Delete Error log Entry ${elog_entry[0]}
211 ${resp}= OpenBMC Get Request ${elog_entry[0]}
Sweta Potthuri9ce8eeb2017-05-08 04:20:11 -0500212 Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND}
George Keishing04bc4812017-03-30 03:33:31 -0500213
Rahul Maheshwari92972e72017-06-22 00:59:59 -0500214
215Verify IPMI SEL Version
216 [Documentation] Verify IPMI SEL's version info.
217 [Tags] Verify_IPMI_SEL_Version
218
219 ${version_info}= Get IPMI SEL Setting Version
220 ${setting_status}= Fetch From Left ${version_info} (
221 ${setting_status}= Evaluate $setting_status.replace(' ','')
222
223 Should Be True ${setting_status} >= 1.5
Rahul Maheshwarib44b6ec2017-07-05 00:23:03 -0500224 Should Contain ${version_info} v2 compliant case_insensitive=True
Rahul Maheshwari92972e72017-06-22 00:59:59 -0500225
226
George Keishing845d9dc2017-06-16 14:49:29 -0500227Verify Watchdog Timedout Error
228 [Documentation] Trigger watchdog timed out and verify errorlog generated.
229 [Tags] Verify_Watchdog_Timedout_Error
230
George Keishingd397e6a2017-08-11 11:19:24 -0500231 REST Power Off
232 REST Power On
233
George Keishing845d9dc2017-06-16 14:49:29 -0500234 # Clear errors if there are any.
George Keishingcc71c3c2017-09-20 09:38:03 -0500235 Delete All Error Logs
George Keishing845d9dc2017-06-16 14:49:29 -0500236
George Keishing845d9dc2017-06-16 14:49:29 -0500237 Trigger Host Watchdog Error
238
239 Verify Watchdog Errorlog Content
240
241
Rahul Maheshwarib44b6ec2017-07-05 00:23:03 -0500242Verify IPMI SEL Delete
243 [Documentation] Verify IPMI SEL delete operation.
244 [Tags] Verify_IPMI_SEL_Delete
245
246 Delete Error Logs And Verify
247 Create Test Error Log
248
249 ${sel_list}= Run IPMI Standard Command sel list
250 # Example of SEL List:
251 # 4 | 04/21/2017 | 10:51:16 | System Event #0x01 | Undetermined system hardware failure | Asserted
252
253 ${sel_entry}= Fetch from Left ${sel_list} |
254 ${sel_entry}= Evaluate $sel_entry.replace(' ','')
255 ${sel_entry}= Convert To Integer 0x${sel_entry}
256
257 ${sel_delete}= Run IPMI Standard Command sel delete ${sel_entry}
258 Should Be Equal As Strings ${sel_delete} Deleted entry ${sel_entry}
259 ... case_insensitive=True
260
261 ${sel_list}= Run IPMI Standard Command sel list
262 Should Be Equal As Strings ${sel_list} SEL has no entries
263 ... case_insensitive=True
264
265
266Verify Empty SEL
267 [Documentation] Verify empty SEL list.
268 [Tags] Verify_Empty_SEL
269
270 Delete Error Logs And Verify
271
272 ${resp}= Run IPMI Standard Command sel list
273 Should Contain ${resp} SEL has no entries case_insensitive=True
274
275
276Delete Non Existing SEL Entry
277 [Documentation] Delete non existing SEL entry.
278 [Tags] Delete_Non_Existing_SEL_Entry
279
280 Delete Error Logs And Verify
281 ${sel_delete}= Run Keyword And Expect Error *
282 ... Run IPMI Standard Command sel delete 100
283 Should Contain ${sel_delete} Unable to delete entry
284 ... case_insensitive=True
285
286
287Delete Invalid SEL Entry
288 [Documentation] Delete invalid SEL entry.
289 [Tags] Delete_Invalid_SEL_Entry
290
291 ${sel_delete}= Run Keyword And Expect Error *
292 ... Run IPMI Standard Command sel delete abc
293 Should Contain ${sel_delete} Given SEL ID 'abc' is invalid
294 ... case_insensitive=True
295
296
Rahul Maheshwarie89b5092017-06-23 00:24:15 -0500297Verify IPMI SEL Entries
298 [Documentation] Verify IPMI SEL's entries info.
299 [Tags] Verify_IPMI_SEL_Entries
300
301 Delete Error Logs And Verify
302
303 # Generate error logs of random count.
304 ${count}= Evaluate random.randint(1, 5) modules=random
305 Repeat Keyword ${count} Create Test Error Log
306
307 ${sel_entries_count}= Get IPMI SEL Setting Entries
308 Should Be Equal As Strings ${sel_entries_count} ${count}
309
310
311Verify IPMI SEL Last Add Time
312 [Documentation] Verify IPMI SEL's last added timestamp.
313 [Tags] Verify_IPMI_SEL_Last_Add_Time
314
315 Create Test Error Log
316 ${sel_time}= Run IPMI Standard Command sel time get
317 ${sel_time}= Convert Date ${sel_time}
318 ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True
319
320 ${sel_last_add_time}= Get IPMI SEL Setting Last Add Time
321 ${sel_last_add_time}= Convert Date ${sel_last_add_time}
322 ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True
323
324 ${time-diff}=
325 ... Subtract Date From Date ${sel_last_add_time} ${sel_time}
326
327 # Verify if the delay in current time check and last add SEL time
328 # is less or equals to 2 seconds.
329 Should Be True ${time-diff} <= 2
330
Sweta Potthuri96d82492017-08-23 09:03:38 -0500331Verify Error Logs Capping
332 [Documentation] Verify error logs capping.
333 [Tags] Verify_Error_Logs_Capping
334
335 Delete Error Logs And Verify
336 ${cmd}= Set Variable
George Keishing95c371a2017-09-14 15:42:33 -0500337 ... for i in {1..101}; do /tmp/tarball/bin/logging-test -c AutoTestSimple;done
Sweta Potthuri96d82492017-08-23 09:03:38 -0500338 Execute Command On BMC ${cmd}
339 ${count}= Count Error Entries
340 Run Keyword If ${count} > 100
341 ... Fail Error logs created exceeded max capacity 100.
Rahul Maheshwarib44b6ec2017-07-05 00:23:03 -0500342
Michael Tritz935c18e2017-03-08 11:45:41 -0600343*** Keywords ***
344
Rahul Maheshwari92972e72017-06-22 00:59:59 -0500345Get IPMI SEL Setting
346 [Documentation] Returns status for given IPMI SEL setting.
347 [Arguments] ${setting}
348 # Description of argument(s):
349 # setting SEL setting which needs to be read(e.g. "Last Add Time").
350
351 ${resp}= Run IPMI Standard Command sel info
352
353 ${setting_line}= Get Lines Containing String ${resp} ${setting}
354 ... case-insensitive
Rahul Maheshwarib44b6ec2017-07-05 00:23:03 -0500355 ${setting_status}= Fetch From Right ${setting_line} :${SPACE}
Rahul Maheshwari92972e72017-06-22 00:59:59 -0500356
357 [Return] ${setting_status}
358
359
George Keishing845d9dc2017-06-16 14:49:29 -0500360Verify Watchdog Errorlog Content
361 [Documentation] Verify watchdog errorlog content.
362 # Example:
363 # "/xyz/openbmc_project/logging/entry/1":
364 # {
365 # "AdditionalData": [],
366 # "Id": 1,
367 # "Message": "org.open_power.Host.Error.WatchdogTimedOut",
368 # "Resolved": 0,
369 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Informational",
370 # "Timestamp": 1492715244828,
371 # "associations": []
372 # },
373
George Keishing184a89a2017-06-27 07:12:16 -0500374 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
375 ${elog}= Read Properties ${elog_entry[0]}
George Keishing845d9dc2017-06-16 14:49:29 -0500376 Should Be Equal As Strings
377 ... ${elog["Message"]} org.open_power.Host.Error.WatchdogTimedOut
378 Should Not Be Equal As Strings
379 ... ${elog["Severity"]} xyz.openbmc_project.Logging.Entry.Level.Informational
380
381
George Keishing95c371a2017-09-14 15:42:33 -0500382Logging Test Binary Exist
Michael Tritz935c18e2017-03-08 11:45:41 -0600383 [Documentation] Verify existence of prerequisite logging-test.
384
385 Open Connection And Log In
George Keishing95c371a2017-09-14 15:42:33 -0500386 ${out} ${stderr}= Execute Command
387 ... which /tmp/tarball/bin/logging-test return_stderr=True
Michael Tritz935c18e2017-03-08 11:45:41 -0600388 Should Be Empty ${stderr}
389 Should Contain ${out} logging-test
390
391Clear Existing Error Logs
392 [Documentation] If error log isn't empty, reboot the BMC to clear the log.
393
394 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${1}
395 Return From Keyword If ${resp.status_code} == ${HTTP_NOT_FOUND}
396 Initiate BMC Reboot
397 Wait Until Keyword Succeeds 10 min 10 sec
398 ... Is BMC Ready
399 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${1}
400 Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND}
401
402Create Test Error Log
403 [Documentation] Generate test error log.
404
405 # Test error log entry example:
406 # "/xyz/openbmc_project/logging/entry/1": {
407 # "AdditionalData": [
408 # "STRING=FOO"
409 # ],
410 # "Id": 1,
411 # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
412 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
413 # "Timestamp": 1487743963328,
414 # "associations": []
415 # }
416
George Keishing95c371a2017-09-14 15:42:33 -0500417 Execute Command On BMC /tmp/tarball/bin/logging-test -c AutoTestSimple
Michael Tritz935c18e2017-03-08 11:45:41 -0600418
Sweta Potthuri96d82492017-08-23 09:03:38 -0500419Count Error Entries
420 [Documentation] Count Error entries.
421
422 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}
423 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
424 ${jsondata}= To JSON ${resp.content}
425 ${count}= Get Length ${jsondata["data"]}
426 [Return] ${count}
427
Michael Tritz935c18e2017-03-08 11:45:41 -0600428Verify Test Error Log
429 [Documentation] Verify test error log entries.
George Keishing184a89a2017-06-27 07:12:16 -0500430 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
431 ${entry_id}= Read Attribute ${elog_entry[0]} Message
Sweta Potthuri9ce8eeb2017-05-08 04:20:11 -0500432 Should Be Equal ${entry_id}
Michael Tritz935c18e2017-03-08 11:45:41 -0600433 ... example.xyz.openbmc_project.Example.Elog.AutoTestSimple
George Keishing184a89a2017-06-27 07:12:16 -0500434 ${entry_id}= Read Attribute ${elog_entry[0]} Severity
Sweta Potthuri9ce8eeb2017-05-08 04:20:11 -0500435 Should Be Equal ${entry_id}
Michael Tritz935c18e2017-03-08 11:45:41 -0600436 ... xyz.openbmc_project.Logging.Entry.Level.Error
George Keishing04bc4812017-03-30 03:33:31 -0500437
Sweta Potthuri9ce8eeb2017-05-08 04:20:11 -0500438Delete Error Logs And Verify
439 [Documentation] Delete all error logs and verify.
440
George Keishingcc71c3c2017-09-20 09:38:03 -0500441 Delete All Error Logs
George Keishingd397e6a2017-08-11 11:19:24 -0500442 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}/list quiet=${1}
Sweta Potthuri9ce8eeb2017-05-08 04:20:11 -0500443 Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND}
George Keishing845d9dc2017-06-16 14:49:29 -0500444
George Keishing95c371a2017-09-14 15:42:33 -0500445
446Test Setup Execution
447 [Documentation] Do test case setup tasks.
448
449 ${status}= Run Keyword And Return Status Logging Test Binary Exist
450 Run Keyword If ${status} == ${False} Install Tarball
451 Delete Error Logs And Verify
452
453
454Install Tarball
455 [Documentation] Install tarball on BMC.
456
457 Run Keyword If '${DEBUG_TARBALL_PATH}' == '${EMPTY}' Return From Keyword
458 BMC Execute Command rm -rf /tmp/tarball
459 Install Debug Tarball On BMC ${DEBUG_TARBALL_PATH}
460
461
George Keishing845d9dc2017-06-16 14:49:29 -0500462Post Test Case Execution
463 [Documentation] Do the post test teardown.
464 # 1. Capture FFDC on test failure.
465 # 2. Delete error logs.
466 # 3. Close all open SSH connections.
467 # 4. Clear all REST sessions.
468
469 FFDC On Test Case Fail
George Keishingcc71c3c2017-09-20 09:38:03 -0500470 Delete All Error Logs
George Keishing845d9dc2017-06-16 14:49:29 -0500471 Close All Connections