blob: c80624b3bf7f686fbba16201886ecfdcb42c0fdd [file] [log] [blame]
Michael Tritz935c18e2017-03-08 11:45:41 -06001*** Settings ***
2Documentation Test Error logging.
3
4Resource ../lib/connection_client.robot
5Resource ../lib/openbmc_ffdc.robot
6Resource ../lib/utils.robot
7Resource ../lib/state_manager.robot
Rahul Maheshwari92972e72017-06-22 00:59:59 -05008Resource ../lib/ipmi_client.robot
George Keishing1d82dd12017-07-24 01:16:23 -05009Resource ../lib/boot_utils.robot
Michael Tritz935c18e2017-03-08 11:45:41 -060010
George Keishing95c371a2017-09-14 15:42:33 -050011Test Setup Test Setup Execution
George Keishing845d9dc2017-06-16 14:49:29 -050012Test Teardown Post Test Case Execution
George Keishing184a89a2017-06-27 07:12:16 -050013Suite Teardown Delete Error Logs And Verify
Michael Tritz935c18e2017-03-08 11:45:41 -060014
George Keishing1d82dd12017-07-24 01:16:23 -050015*** Variables ***
16
17${stack_mode} skip
18
Michael Tritz935c18e2017-03-08 11:45:41 -060019*** Test Cases ***
20
George Keishing1d82dd12017-07-24 01:16:23 -050021Error Log Check After BMC Reboot
22 [Documentation] Check error log after BMC rebooted.
23 [Tags] Error_Log_Check_At_BMC_Ready
24 # 1. Power off.
25 # 2. Delete error logs.
26 # 3. Reboot BMC.
27 # 4. Check if eror log exists.
28
George Keishing3f4a3eb2017-09-26 10:33:16 -050029 Smart Power Off
George Keishing1d82dd12017-07-24 01:16:23 -050030 Delete Error Logs And Verify
George Keishing3f4a3eb2017-09-26 10:33:16 -050031 OBMC Reboot (off) stack_mode=normal
George Keishing1d82dd12017-07-24 01:16:23 -050032 Error Logs Should Not Exist
33
34
35Error Log Check After Host Poweron
36 [Documentation] Check error log after host has booted.
37 [Tags] Error_Log_Check_At_Host_Booted
38 # 1. Delete error logs
39 # 1. Power on.
40 # 3. Check if eror log exists.
41
42 Delete Error Logs And Verify
43 REST Power On
44 Error Logs Should Not Exist
45
46
Michael Tritz935c18e2017-03-08 11:45:41 -060047Create Test Error And Verify
48 [Documentation] Create error logs and verify via REST.
49 [Tags] Create_Test_Error_And_Verify
50
51 Create Test Error Log
52 Verify Test Error Log
53
54
55Test Error Persistency On Restart
George Keishing184a89a2017-06-27 07:12:16 -050056 [Documentation] Restart logging service and verify error logs.
Michael Tritz935c18e2017-03-08 11:45:41 -060057 [Tags] Test_Error_Persistency_On_Restart
58
59 Create Test Error Log
60 Verify Test Error Log
61 Execute Command On BMC
62 ... systemctl restart xyz.openbmc_project.Logging.service
63 Sleep 10s reason=Wait for logging service to restart properly.
George Keishing184a89a2017-06-27 07:12:16 -050064 Verify Test Error Log
Michael Tritz935c18e2017-03-08 11:45:41 -060065
66
George Keishing742b0602017-07-28 14:20:53 -050067Test Error Entry Numbering Reset On Restart
68 [Documentation] Restarts logging service and verify error logs entry start
69 ... from entry "Id" 1.
70 # 1. Create error log.
71 # 2. Verify error log.
72 # 3. Delete error log.
73 # 4. Restart logging service.
74 # 5. Create error log.
75 # 6. Verify new error log entry starts with Id entry 1.
76
77 [Tags] Test_Error_Entry_Numbering_Reset_On_Restart
78 # Example Error logs:
79 # "/xyz/openbmc_project/logging/entry/1": {
80 # "AdditionalData": [
81 # "STRING=FOO"
82 # ],
83 # "Id": 1, <--- Entry value should be 1.
84 # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
85 # "Resolved": 0,
86 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
87 # "Timestamp": 1490818990051,
88 # "associations": []
89 # },
90
91 Create Test Error Log
92 Verify Test Error Log
George Keishingcc71c3c2017-09-20 09:38:03 -050093 Delete All Error Logs
George Keishing742b0602017-07-28 14:20:53 -050094 Execute Command On BMC
95 ... systemctl restart xyz.openbmc_project.Logging.service
96 Sleep 10s reason=Wait for logging service to restart properly.
97 Create Test Error Log
98 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
99 ${entry_id}= Read Attribute ${elog_entry[0]} Id
100 Should Be Equal ${entry_id} ${1}
101
102
Michael Tritz935c18e2017-03-08 11:45:41 -0600103Test Error Persistency On Reboot
George Keishing184a89a2017-06-27 07:12:16 -0500104 [Documentation] Reboot BMC and verify error logs.
Michael Tritz935c18e2017-03-08 11:45:41 -0600105 [Tags] Test_Error_Persistency_On_Reboot
106
107 Create Test Error Log
108 Verify Test Error Log
109 Initiate BMC Reboot
110 Wait Until Keyword Succeeds 10 min 10 sec
111 ... Is BMC Ready
George Keishing184a89a2017-06-27 07:12:16 -0500112 Verify Test Error Log
Michael Tritz935c18e2017-03-08 11:45:41 -0600113
114
George Keishing04bc4812017-03-30 03:33:31 -0500115Create Test Error And Verify Resolved Field
116 [Documentation] Create error log and verify "Resolved"
117 ... field is 0.
118 [Tags] Create_Test_Error_And_Verify_Resolved_Field
119
120 # Example Error log:
121 # "/xyz/openbmc_project/logging/entry/1": {
122 # "AdditionalData": [
123 # "STRING=FOO"
124 # ],
125 # "Id": 1,
126 # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
127 # "Resolved": 0,
128 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
129 # "Timestamp": 1490817164983,
130 # "associations": []
131 # },
132
133 # It's work in progress, but it's mnfg need. To mark an error as
134 # resolved, without deleting the error, mfg will set this bool
135 # property.
136 # In this test context we are making sure "Resolved" field is "0"
137 # by default.
138
George Keishingcc71c3c2017-09-20 09:38:03 -0500139 Delete All Error Logs
George Keishing04bc4812017-03-30 03:33:31 -0500140 Create Test Error Log
George Keishing184a89a2017-06-27 07:12:16 -0500141 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
142 ${resolved}= Read Attribute ${elog_entry[0]} Resolved
George Keishing04bc4812017-03-30 03:33:31 -0500143 Should Be True ${resolved} == 0
144
145
146Create Test Errors And Verify Time Stamp
147 [Documentation] Create error logs and verify time stamp.
148 [Tags] Create_Test_Error_And_Verify_Time_Stamp
149
150 # Example Error logs:
151 # "/xyz/openbmc_project/logging/entry/1": {
152 # "AdditionalData": [
153 # "STRING=FOO"
154 # ],
155 # "Id": 1,
156 # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
157 # "Resolved": 0,
158 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
159 # "Timestamp": 1490818990051, <--- Time stamp
160 # "associations": []
161 # },
162 # "/xyz/openbmc_project/logging/entry/2": {
163 # "AdditionalData": [
164 # "STRING=FOO"
165 # ],
166 # "Id": 2,
167 # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
168 # "Resolved": 0,
169 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
170 # "Timestamp": 1490818992116, <---- Time stamp
171 # "associations": []
172 # },
173
George Keishingcc71c3c2017-09-20 09:38:03 -0500174 Delete All Error Logs
George Keishing04bc4812017-03-30 03:33:31 -0500175 Create Test Error Log
176 Create Test Error Log
177 # The error log generated is associated with the epoc time and unique
178 # for every error and in increasing time stamp.
George Keishing184a89a2017-06-27 07:12:16 -0500179 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
180 ${time_stamp1}= Read Attribute ${elog_entry[0]} Timestamp
181 ${time_stamp2}= Read Attribute ${elog_entry[1]} Timestamp
George Keishing6eb10b82017-06-06 10:02:09 -0500182 Should Be True ${time_stamp2} > ${time_stamp1}
George Keishing04bc4812017-03-30 03:33:31 -0500183
Sweta Potthuri9ce8eeb2017-05-08 04:20:11 -0500184Create Test Error Log And Delete
185 [Documentation] Create an error log and delete it.
186 [Tags] Create_Test_Error_Log_And_Delete
187
188 Delete Error Logs And Verify
189 Create Test Error Log
190 Delete Error Logs And Verify
191
192Create Multiple Test Error Logs And Delete All
193 [Documentation] Create multiple error logs and delete all.
194 [Tags] Create_Multiple_Test_Error_Logs_And_Delete_All
195
196 Delete Error Logs And Verify
197 Create Test Error Log
198 Create Test Error Log
199 Create Test Error Log
200 Delete Error Logs And Verify
201
202Create Two Test Error Logs And Delete One
203 [Documentation] Create two error logs and delete the first entry.
204 [Tags] Create_Two_Test_Error_Logs_And_Delete_One
205
206 Delete Error Logs And Verify
207 Create Test Error Log
George Keishing184a89a2017-06-27 07:12:16 -0500208 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
Sweta Potthuri9ce8eeb2017-05-08 04:20:11 -0500209 Create Test Error Log
George Keishing184a89a2017-06-27 07:12:16 -0500210 Delete Error log Entry ${elog_entry[0]}
211 ${resp}= OpenBMC Get Request ${elog_entry[0]}
Sweta Potthuri9ce8eeb2017-05-08 04:20:11 -0500212 Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND}
George Keishing04bc4812017-03-30 03:33:31 -0500213
Rahul Maheshwari92972e72017-06-22 00:59:59 -0500214
215Verify IPMI SEL Version
216 [Documentation] Verify IPMI SEL's version info.
217 [Tags] Verify_IPMI_SEL_Version
218
219 ${version_info}= Get IPMI SEL Setting Version
220 ${setting_status}= Fetch From Left ${version_info} (
221 ${setting_status}= Evaluate $setting_status.replace(' ','')
222
223 Should Be True ${setting_status} >= 1.5
Rahul Maheshwarib44b6ec2017-07-05 00:23:03 -0500224 Should Contain ${version_info} v2 compliant case_insensitive=True
Rahul Maheshwari92972e72017-06-22 00:59:59 -0500225
226
George Keishing845d9dc2017-06-16 14:49:29 -0500227Verify Watchdog Timedout Error
228 [Documentation] Trigger watchdog timed out and verify errorlog generated.
229 [Tags] Verify_Watchdog_Timedout_Error
230
George Keishingd397e6a2017-08-11 11:19:24 -0500231 REST Power Off
232 REST Power On
233
George Keishing845d9dc2017-06-16 14:49:29 -0500234 # Clear errors if there are any.
George Keishingcc71c3c2017-09-20 09:38:03 -0500235 Delete All Error Logs
George Keishing845d9dc2017-06-16 14:49:29 -0500236
George Keishing845d9dc2017-06-16 14:49:29 -0500237 Trigger Host Watchdog Error
238
George Keishing45921ea2017-11-23 04:32:56 -0600239 # Logging took time to generate the timedout error.
240 Wait Until Keyword Succeeds 2 min 30 sec
241 ... Verify Watchdog Errorlog Content
George Keishing845d9dc2017-06-16 14:49:29 -0500242
243
Rahul Maheshwarib44b6ec2017-07-05 00:23:03 -0500244Verify IPMI SEL Delete
245 [Documentation] Verify IPMI SEL delete operation.
246 [Tags] Verify_IPMI_SEL_Delete
247
248 Delete Error Logs And Verify
249 Create Test Error Log
250
251 ${sel_list}= Run IPMI Standard Command sel list
252 # Example of SEL List:
253 # 4 | 04/21/2017 | 10:51:16 | System Event #0x01 | Undetermined system hardware failure | Asserted
254
255 ${sel_entry}= Fetch from Left ${sel_list} |
256 ${sel_entry}= Evaluate $sel_entry.replace(' ','')
257 ${sel_entry}= Convert To Integer 0x${sel_entry}
258
259 ${sel_delete}= Run IPMI Standard Command sel delete ${sel_entry}
260 Should Be Equal As Strings ${sel_delete} Deleted entry ${sel_entry}
261 ... case_insensitive=True
262
263 ${sel_list}= Run IPMI Standard Command sel list
264 Should Be Equal As Strings ${sel_list} SEL has no entries
265 ... case_insensitive=True
266
267
268Verify Empty SEL
269 [Documentation] Verify empty SEL list.
270 [Tags] Verify_Empty_SEL
271
272 Delete Error Logs And Verify
273
274 ${resp}= Run IPMI Standard Command sel list
275 Should Contain ${resp} SEL has no entries case_insensitive=True
276
277
278Delete Non Existing SEL Entry
279 [Documentation] Delete non existing SEL entry.
280 [Tags] Delete_Non_Existing_SEL_Entry
281
282 Delete Error Logs And Verify
283 ${sel_delete}= Run Keyword And Expect Error *
284 ... Run IPMI Standard Command sel delete 100
285 Should Contain ${sel_delete} Unable to delete entry
286 ... case_insensitive=True
287
288
289Delete Invalid SEL Entry
290 [Documentation] Delete invalid SEL entry.
291 [Tags] Delete_Invalid_SEL_Entry
292
293 ${sel_delete}= Run Keyword And Expect Error *
294 ... Run IPMI Standard Command sel delete abc
295 Should Contain ${sel_delete} Given SEL ID 'abc' is invalid
296 ... case_insensitive=True
297
298
Rahul Maheshwarie89b5092017-06-23 00:24:15 -0500299Verify IPMI SEL Entries
300 [Documentation] Verify IPMI SEL's entries info.
301 [Tags] Verify_IPMI_SEL_Entries
302
303 Delete Error Logs And Verify
304
305 # Generate error logs of random count.
306 ${count}= Evaluate random.randint(1, 5) modules=random
307 Repeat Keyword ${count} Create Test Error Log
308
309 ${sel_entries_count}= Get IPMI SEL Setting Entries
310 Should Be Equal As Strings ${sel_entries_count} ${count}
311
312
313Verify IPMI SEL Last Add Time
314 [Documentation] Verify IPMI SEL's last added timestamp.
315 [Tags] Verify_IPMI_SEL_Last_Add_Time
316
317 Create Test Error Log
318 ${sel_time}= Run IPMI Standard Command sel time get
319 ${sel_time}= Convert Date ${sel_time}
320 ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True
321
322 ${sel_last_add_time}= Get IPMI SEL Setting Last Add Time
323 ${sel_last_add_time}= Convert Date ${sel_last_add_time}
324 ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True
325
326 ${time-diff}=
327 ... Subtract Date From Date ${sel_last_add_time} ${sel_time}
328
329 # Verify if the delay in current time check and last add SEL time
330 # is less or equals to 2 seconds.
331 Should Be True ${time-diff} <= 2
332
Sweta Potthuri96d82492017-08-23 09:03:38 -0500333Verify Error Logs Capping
334 [Documentation] Verify error logs capping.
335 [Tags] Verify_Error_Logs_Capping
336
337 Delete Error Logs And Verify
338 ${cmd}= Set Variable
George Keishing95c371a2017-09-14 15:42:33 -0500339 ... for i in {1..101}; do /tmp/tarball/bin/logging-test -c AutoTestSimple;done
Sweta Potthuri96d82492017-08-23 09:03:38 -0500340 Execute Command On BMC ${cmd}
341 ${count}= Count Error Entries
342 Run Keyword If ${count} > 100
343 ... Fail Error logs created exceeded max capacity 100.
Rahul Maheshwarib44b6ec2017-07-05 00:23:03 -0500344
Michael Tritz935c18e2017-03-08 11:45:41 -0600345*** Keywords ***
346
Rahul Maheshwari92972e72017-06-22 00:59:59 -0500347Get IPMI SEL Setting
348 [Documentation] Returns status for given IPMI SEL setting.
349 [Arguments] ${setting}
350 # Description of argument(s):
351 # setting SEL setting which needs to be read(e.g. "Last Add Time").
352
353 ${resp}= Run IPMI Standard Command sel info
354
355 ${setting_line}= Get Lines Containing String ${resp} ${setting}
356 ... case-insensitive
Rahul Maheshwarib44b6ec2017-07-05 00:23:03 -0500357 ${setting_status}= Fetch From Right ${setting_line} :${SPACE}
Rahul Maheshwari92972e72017-06-22 00:59:59 -0500358
359 [Return] ${setting_status}
360
361
George Keishing845d9dc2017-06-16 14:49:29 -0500362Verify Watchdog Errorlog Content
363 [Documentation] Verify watchdog errorlog content.
364 # Example:
365 # "/xyz/openbmc_project/logging/entry/1":
366 # {
367 # "AdditionalData": [],
368 # "Id": 1,
George Keishing27405ee2017-11-18 04:45:04 -0600369 # "Message": "org.open_power.Host.Boot.Error.WatchdogTimedOut",
George Keishing845d9dc2017-06-16 14:49:29 -0500370 # "Resolved": 0,
George Keishing27405ee2017-11-18 04:45:04 -0600371 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
George Keishing845d9dc2017-06-16 14:49:29 -0500372 # "Timestamp": 1492715244828,
373 # "associations": []
374 # },
375
George Keishing184a89a2017-06-27 07:12:16 -0500376 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
377 ${elog}= Read Properties ${elog_entry[0]}
George Keishing845d9dc2017-06-16 14:49:29 -0500378 Should Be Equal As Strings
George Keishing27405ee2017-11-18 04:45:04 -0600379 ... ${elog["Message"]} org.open_power.Host.Boot.Error.WatchdogTimedOut
380 Should Be Equal As Strings
381 ... ${elog["Severity"]} xyz.openbmc_project.Logging.Entry.Level.Error
George Keishing845d9dc2017-06-16 14:49:29 -0500382
383
George Keishing95c371a2017-09-14 15:42:33 -0500384Logging Test Binary Exist
Michael Tritz935c18e2017-03-08 11:45:41 -0600385 [Documentation] Verify existence of prerequisite logging-test.
386
387 Open Connection And Log In
George Keishing95c371a2017-09-14 15:42:33 -0500388 ${out} ${stderr}= Execute Command
389 ... which /tmp/tarball/bin/logging-test return_stderr=True
Michael Tritz935c18e2017-03-08 11:45:41 -0600390 Should Be Empty ${stderr}
391 Should Contain ${out} logging-test
392
393Clear Existing Error Logs
394 [Documentation] If error log isn't empty, reboot the BMC to clear the log.
395
396 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${1}
397 Return From Keyword If ${resp.status_code} == ${HTTP_NOT_FOUND}
398 Initiate BMC Reboot
399 Wait Until Keyword Succeeds 10 min 10 sec
400 ... Is BMC Ready
401 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${1}
402 Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND}
403
404Create Test Error Log
405 [Documentation] Generate test error log.
406
407 # Test error log entry example:
408 # "/xyz/openbmc_project/logging/entry/1": {
409 # "AdditionalData": [
410 # "STRING=FOO"
411 # ],
412 # "Id": 1,
413 # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
414 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
415 # "Timestamp": 1487743963328,
416 # "associations": []
417 # }
418
George Keishing95c371a2017-09-14 15:42:33 -0500419 Execute Command On BMC /tmp/tarball/bin/logging-test -c AutoTestSimple
Michael Tritz935c18e2017-03-08 11:45:41 -0600420
Sweta Potthuri96d82492017-08-23 09:03:38 -0500421Count Error Entries
422 [Documentation] Count Error entries.
423
424 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}
425 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
426 ${jsondata}= To JSON ${resp.content}
427 ${count}= Get Length ${jsondata["data"]}
428 [Return] ${count}
429
Michael Tritz935c18e2017-03-08 11:45:41 -0600430Verify Test Error Log
431 [Documentation] Verify test error log entries.
George Keishing184a89a2017-06-27 07:12:16 -0500432 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
433 ${entry_id}= Read Attribute ${elog_entry[0]} Message
Sweta Potthuri9ce8eeb2017-05-08 04:20:11 -0500434 Should Be Equal ${entry_id}
Michael Tritz935c18e2017-03-08 11:45:41 -0600435 ... example.xyz.openbmc_project.Example.Elog.AutoTestSimple
George Keishing184a89a2017-06-27 07:12:16 -0500436 ${entry_id}= Read Attribute ${elog_entry[0]} Severity
Sweta Potthuri9ce8eeb2017-05-08 04:20:11 -0500437 Should Be Equal ${entry_id}
Michael Tritz935c18e2017-03-08 11:45:41 -0600438 ... xyz.openbmc_project.Logging.Entry.Level.Error
George Keishing04bc4812017-03-30 03:33:31 -0500439
Sweta Potthuri9ce8eeb2017-05-08 04:20:11 -0500440Delete Error Logs And Verify
441 [Documentation] Delete all error logs and verify.
442
George Keishingcc71c3c2017-09-20 09:38:03 -0500443 Delete All Error Logs
George Keishingd397e6a2017-08-11 11:19:24 -0500444 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}/list quiet=${1}
Sweta Potthuri9ce8eeb2017-05-08 04:20:11 -0500445 Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND}
George Keishing845d9dc2017-06-16 14:49:29 -0500446
George Keishing95c371a2017-09-14 15:42:33 -0500447
448Test Setup Execution
449 [Documentation] Do test case setup tasks.
450
451 ${status}= Run Keyword And Return Status Logging Test Binary Exist
452 Run Keyword If ${status} == ${False} Install Tarball
453 Delete Error Logs And Verify
454
455
456Install Tarball
457 [Documentation] Install tarball on BMC.
458
459 Run Keyword If '${DEBUG_TARBALL_PATH}' == '${EMPTY}' Return From Keyword
460 BMC Execute Command rm -rf /tmp/tarball
461 Install Debug Tarball On BMC ${DEBUG_TARBALL_PATH}
462
463
George Keishing845d9dc2017-06-16 14:49:29 -0500464Post Test Case Execution
465 [Documentation] Do the post test teardown.
466 # 1. Capture FFDC on test failure.
467 # 2. Delete error logs.
468 # 3. Close all open SSH connections.
469 # 4. Clear all REST sessions.
470
471 FFDC On Test Case Fail
George Keishingcc71c3c2017-09-20 09:38:03 -0500472 Delete All Error Logs
George Keishing845d9dc2017-06-16 14:49:29 -0500473 Close All Connections