blob: db93d2665e917419ab8fcfdeb7369ebc6ad8ddf9 [file] [log] [blame]
George Keishing8cca7fb2019-05-30 04:23:50 -05001*** Settings ***
2Documentation Inventory of hardware resources under systems.
3
4Resource ../../../lib/bmc_redfish_resource.robot
5Resource ../../../lib/bmc_redfish_utils.robot
6Resource ../../../lib/logging_utils.robot
7Resource ../../../lib/openbmc_ffdc.robot
George Keishing8fe60242019-06-03 23:06:12 -05008Resource ../../../lib/ipmi_client.robot
George Keishing759c6f92019-06-03 12:14:42 -05009Library ../../../lib/logging_utils.py
George Keishing8cca7fb2019-05-30 04:23:50 -050010
George Keishing2f369472019-07-23 12:03:10 -050011Suite Setup Suite Setup Execution
12Suite Teardown Suite Teardown Execution
George Keishingb0a491f2019-05-31 02:49:22 -050013Test Setup Test Setup Execution
George Keishing8cca7fb2019-05-30 04:23:50 -050014Test Teardown Test Teardown Execution
15
George Keishing759c6f92019-06-03 12:14:42 -050016** Variables ***
17
18${max_num_event_logs} ${200}
19
George Keishing8cca7fb2019-05-30 04:23:50 -050020*** Test Cases ***
21
22Event Log Check After BMC Reboot
23 [Documentation] Check event log after BMC rebooted.
24 [Tags] Event_Log_Check_After_BMC_Reboot
25
26 Redfish Purge Event Log
27 Event Log Should Not Exist
28
29 Redfish OBMC Reboot (off)
30
31 Redfish.Login
32 Wait Until Keyword Succeeds 1 mins 15 secs Redfish.Get ${EVENT_LOG_URI}Entries
33
34 Event Log Should Not Exist
35
36
George Keishingb0a491f2019-05-31 02:49:22 -050037Event Log Check After Host Poweron
38 [Documentation] Check event log after host has booted.
39 [Tags] Event_Log_Check_After_Host_Poweron
40
41 Redfish Purge Event Log
42 Event Log Should Not Exist
43
44 Redfish Power On
45
46 Redfish.Login
47 Event Log Should Not Exist
48
49
50Create Test Event Log And Verify
51 [Documentation] Create event logs and verify via redfish.
52 [Tags] Create_Test_Event_Log_And_Verify
53
54 Create Test Error Log
55 Event Log Should Exist
56
57
58Test Event Log Persistency On Restart
59 [Documentation] Restart logging service and verify event logs.
60 [Tags] Test_Event_Log_Persistency_On_Restart
61
62 Create Test Error Log
63 Event Log Should Exist
64
65 BMC Execute Command
66 ... systemctl restart xyz.openbmc_project.Logging.service
67 Sleep 10s reason=Wait for logging service to restart properly.
68
69 Event Log Should Exist
70
71
72Test Event Entry Numbering Reset On Restart
73 [Documentation] Restart logging service and verify event logs entry starts
74 ... from entry "Id" 1.
75 [Tags] Test_Event_Entry_Numbering_Reset_On_Restart
76
77 #{
78 # "@odata.context": "/redfish/v1/$metadata#LogEntryCollection.LogEntryCollection",
79 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries",
80 # "@odata.type": "#LogEntryCollection.LogEntryCollection",
81 # "Description": "Collection of System Event Log Entries",
82 # "Members": [
83 # {
84 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry",
85 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/1",
86 # "@odata.type": "#LogEntry.v1_4_0.LogEntry",
87 # "Created": "2019-05-29T13:19:27+00:00",
88 # "EntryType": "Event",
89 # "Id": "1", <----- Event log ID
90 # "Message": "org.open_power.Host.Error.Event",
91 # "Name": "System DBus Event Log Entry",
92 # "Severity": "Critical"
93 # }
94 # ],
95 # "Members@odata.count": 1,
96 # "Name": "System Event Log Entries"
97 #}
98
99 Create Test Error Log
100 Create Test Error Log
101 Event Log Should Exist
102
103 Redfish Purge Event Log
104 Event Log Should Not Exist
105
106 BMC Execute Command
107 ... systemctl restart xyz.openbmc_project.Logging.service
108 Sleep 10s reason=Wait for logging service to restart properly.
109
110 Create Test Error Log
111 ${elogs}= Get Event Logs
112 Should Be Equal ${elogs[0]["Id"]} 1 msg=Event log entry is not 1.
113
114
115Test Event Log Persistency On Reboot
116 [Documentation] Reboot BMC and verify event log.
117 [Tags] Test_Event_Log_Persistency_On_Reboot
118
119 Redfish Purge Event Log
120 Create Test Error Log
121 Event Log Should Exist
122
123 Redfish OBMC Reboot (off)
124
125 Redfish.Login
126 Wait Until Keyword Succeeds 1 mins 15 secs Redfish.Get ${EVENT_LOG_URI}Entries
127
128 Event Log Should Exist
129
130
George Keishingddbc88c2019-06-06 07:29:41 -0500131# TODO: openbmc/openbmc-test-automation#1789
132Create Test Event Log And Verify Resolved Field
133 [Documentation] Create event log and verify "Resolved" field is 0.
134 [Tags] Create_Test_Event_Log_And_Verify_Resolved_Field
135
136 # Example Error log:
137 # "/xyz/openbmc_project/logging/entry/1": {
138 # "AdditionalData": [
139 # "STRING=FOO"
140 # ],
141 # "Id": 1,
142 # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
143 # "Resolved": 0,
144 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
145 # "Timestamp": 1490817164983,
George Keishing58520d02020-02-24 10:55:32 -0600146 # "Associations": []
George Keishingddbc88c2019-06-06 07:29:41 -0500147 # },
148
149 # To mark an error as resolved, without deleting the error, user would
150 # set this bool property.
151 # In this test context we are making sure "Resolved" field is "0"
152 # by default.
153
154 Redfish Purge Event Log
155 Create Test Error Log
156 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
157 ${resolved}= Read Attribute ${elog_entry[0]} Resolved
158 Should Be True ${resolved} == 0
159
160
George Keishing148d08c2019-05-31 12:39:04 -0500161Create Test Event Log And Verify Time Stamp
162 [Documentation] Create event logs and verify time stamp.
163 [Tags] Create_Test_Event_Log_And_Verify_Time_Stamp
164
165 #{
166 # "@odata.context": "/redfish/v1/$metadata#LogEntryCollection.LogEntryCollection",
167 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries",
168 # "@odata.type": "#LogEntryCollection.LogEntryCollection",
169 # "Description": "Collection of System Event Log Entries",
170 # "Members": [
171 # {
172 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry",
173 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/1",
174 # "@odata.type": "#LogEntry.v1_4_0.LogEntry",
175 # "Created": "2019-05-29T13:19:27+00:00", <--- Time stamp
176 # "EntryType": "Event",
177 # "Id": "1",
178 # "Message": "org.open_power.Host.Error.Event",
179 # "Name": "System DBus Event Log Entry",
180 # "Severity": "Critical"
181 # }
182 # ],
183 # "Members@odata.count": 1,
184 # "Name": "System Event Log Entries"
185 #}
186
187 Redfish Purge Event Log
188
189 Create Test Error Log
190 Sleep 2s
191 Create Test Error Log
192
193 ${elog_entry}= Get Event Logs
194
195 # The event log generated is associated with the epoc time and unique
196 # for every error and in increasing time stamp.
197 ${time_stamp1}= Convert Date ${elog_entry[0]["Created"]} epoch
198 ${time_stamp2}= Convert Date ${elog_entry[1]["Created"]} epoch
199
200 Should Be True ${time_stamp2} > ${time_stamp1}
201
202
George Keishingddbc88c2019-06-06 07:29:41 -0500203Verify IPMI SEL Delete
204 [Documentation] Verify IPMI SEL delete operation.
205 [Tags] Verify_IPMI_SEL_Delete
206
207 Redfish Purge Event Log
208 Create Test Error Log
209
210 ${sel_list}= Run IPMI Standard Command sel list
211 Should Not Be Equal As Strings ${sel_list} SEL has no entries
212
213 # Example of SEL List:
214 # 4 | 04/21/2017 | 10:51:16 | System Event #0x01 | Undetermined system hardware failure | Asserted
215
216 ${sel_entry}= Fetch from Left ${sel_list} |
217 ${sel_entry}= Evaluate $sel_entry.replace(' ','')
218 ${sel_entry}= Convert To Integer 0x${sel_entry}
219
220 ${sel_delete}= Run IPMI Standard Command sel delete ${sel_entry}
221 Should Be Equal As Strings ${sel_delete} Deleted entry ${sel_entry}
222 ... case_insensitive=True
223
224 ${sel_list}= Run IPMI Standard Command sel list
225 Should Be Equal As Strings ${sel_list} SEL has no entries
226 ... case_insensitive=True
227
228
George Keishing0b76e422019-06-03 01:58:52 -0500229Delete Non Existing SEL Event Entry
230 [Documentation] Delete non existing SEL event entry.
231 [Tags] Delete_Non_Existing_SEL_Event_Entry
232
233 ${sel_delete}= Run Keyword And Expect Error *
234 ... Run IPMI Standard Command sel delete 100
235 Should Contain ${sel_delete} Unable to delete entry
236 ... case_insensitive=True
237
238
239Delete Invalid SEL Event Entry
240 [Documentation] Delete invalid SEL event entry.
241 [Tags] Delete_Invalid_SEL_Event_Entry
242
243 ${sel_delete}= Run Keyword And Expect Error *
244 ... Run IPMI Standard Command sel delete abc
245 Should Contain ${sel_delete} Given SEL ID 'abc' is invalid
246 ... case_insensitive=True
247
248
249Verify IPMI SEL Event Entries
250 [Documentation] Verify IPMI SEL's entries info.
251 [Tags] Verify_IPMI_SEL_Event_Entries
252
253 # Generate error logs of random count.
254 ${count}= Evaluate random.randint(1, 5) modules=random
255 Repeat Keyword ${count} Create Test Error Log
256
257 ${sel_entries_count}= Get IPMI SEL Setting Entries
258 Should Be Equal As Strings ${sel_entries_count} ${count}
259
260
261Verify IPMI SEL Event Last Add Time
262 [Documentation] Verify IPMI SEL's last added timestamp.
263 [Tags] Verify_IPMI_SEL_Event_Last_Add_Time
264
265 Create Test Error Log
266 ${sel_time}= Run IPMI Standard Command sel time get
267 ${sel_time}= Convert Date ${sel_time}
268 ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True
269
270 ${sel_last_add_time}= Get IPMI SEL Setting Last Add Time
271 ${sel_last_add_time}= Convert Date ${sel_last_add_time}
272 ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True
273
274 ${time_diff}=
275 ... Subtract Date From Date ${sel_last_add_time} ${sel_time}
276
277 # Verify if the delay in current time check and last add SEL time
278 # is less or equals to 2 seconds.
279 Should Be True ${time_diff} <= 2
280
281
George Keishingd35b5312019-05-31 14:29:07 -0500282Create Test Event Log And Delete
283 [Documentation] Create an event log and delete it.
284 [Tags] Create_Test_Event_Log_And_Delete
285
286 Create Test Error Log
287 Redfish Purge Event Log
288 Event Log Should Not Exist
289
290
291Create Multiple Test Event Logs And Delete All
292 [Documentation] Create multiple event logs and delete all.
293 [Tags] Create_Multiple_Test_Event_Logs_And_Delete_All
294
295 Create Test Error Log
296 Create Test Error Log
297 Create Test Error Log
298 Redfish Purge Event Log
299 Event Log Should Not Exist
300
301
George Keishingddbc88c2019-06-06 07:29:41 -0500302# TODO: openbmc/openbmc-test-automation#1789
303Create Two Test Event Logs And Delete One
304 [Documentation] Create two event logs and delete the first entry.
305 [Tags] Create_Two_Test_Eevent_Logs_And_Delete_One
306
307 Redfish Purge Event Log
308 Create Test Error Log
309 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
310 Create Test Error Log
311 Delete Error log Entry ${elog_entry[0]}
312 ${resp}= OpenBMC Get Request ${elog_entry[0]}
313 Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND}
314
315
George Keishingd35b5312019-05-31 14:29:07 -0500316Verify Watchdog Timedout Event
317 [Documentation] Trigger watchdog timed out and verify event log generated.
318 [Tags] Verify_Watchdog_Timedout_Event
319
320 Redfish Power On
321
322 # Clear errors if there are any.
323 Redfish.Login
324 Redfish Purge Event Log
325
326 Trigger Host Watchdog Error
327
328 # Logging takes time to generate the timeout error.
329 Wait Until Keyword Succeeds 2 min 30 sec
330 ... Verify Watchdog EventLog Content
331
George Keishingd35b5312019-05-31 14:29:07 -0500332
333Verify Event Logs Capping
334 [Documentation] Verify event logs capping.
335 [Tags] Verify_Event_Logs_Capping
336
337 Redfish Purge Event Log
338
339 ${cmd}= Catenate for i in {1..201}; do /tmp/tarball/bin/logging-test -c
George Keishingd2d2bd22019-06-12 13:49:41 -0500340 ... AutoTestSimple;sleep 1;done
George Keishingd35b5312019-05-31 14:29:07 -0500341 BMC Execute Command ${cmd}
342
343 ${elogs}= Get Event Logs
344 ${count}= Get Length ${elogs}
345 Run Keyword If ${count} > 200
346 ... Fail Error logs created exceeded max capacity 200.
347
348
George Keishing759c6f92019-06-03 12:14:42 -0500349Test Event Log Wrapping
350 [Documentation] Verify event log entries wraps when 200 max cap is reached.
351 [Tags] Test_Event_Log_Wrapping
352
353 # Restarting logging service in order to clear logs and get the next log
354 # ID set to 1.
355 BMC Execute Command
356 ... systemctl restart xyz.openbmc_project.Logging.service
357 Sleep 10s reason=Wait for logging service to restart properly.
358
359 # Create ${max_num_event_logs} event logs.
360 ${cmd}= Catenate for i in {1..${max_num_event_logs}}; do /tmp/tarball/bin/logging-test -c
George Keishingd2d2bd22019-06-12 13:49:41 -0500361 ... AutoTestSimple;sleep 1;done
George Keishing759c6f92019-06-03 12:14:42 -0500362 BMC Execute Command ${cmd}
363
364 # Verify that event logs with IDs 1 and ${max_num_event_logs} exist.
365 ${event_log}= Get Event Logs
366
367 ${log_entries}= Filter Struct ${event_log} [('Id', '1')]
Michael Walsh39c00512019-07-17 10:54:06 -0500368 Rprint Vars log_entries
George Keishing759c6f92019-06-03 12:14:42 -0500369 Should Be Equal As Strings ${log_entries[0]["Id"]} 1
370
371 ${log_entries}= Filter Struct ${event_log} [('Id', '${max_num_event_logs}')]
Michael Walsh39c00512019-07-17 10:54:06 -0500372 Rprint Vars log_entries
George Keishing759c6f92019-06-03 12:14:42 -0500373 Should Be Equal As Strings ${log_entries[0]["Id"]} ${max_num_event_logs}
374
375 # Create event log and verify the entry ID, ${max_num_event_logs + 1}.
376 ${next_event_log_id}= Set Variable ${max_num_event_logs + 1}
377
378 Create Test Error Log
379
380 ${event_log}= Get Event Logs
381
382 ${log_entries}= Filter Struct ${event_log} [('Id', '${next_event_log_id}')]
Michael Walsh39c00512019-07-17 10:54:06 -0500383 Rprint Vars log_entries
George Keishing759c6f92019-06-03 12:14:42 -0500384 Should Be Equal As Strings ${log_entries[0]["Id"]} ${next_event_log_id}
385
386 # Event log 1 should be wrapped.
387 ${log_entries}= Filter Struct ${event_log} [('Id', '1')]
Michael Walsh39c00512019-07-17 10:54:06 -0500388 Rprint Vars log_entries
George Keishing759c6f92019-06-03 12:14:42 -0500389
390 ${length_log_entries} Get Length ${log_entries}
391 Should Be Equal As Integers ${length_log_entries} 0
392 ... msg=The event log should have wrapped such that entry ID 1 is now purged.
393
394
George Keishing8cca7fb2019-05-30 04:23:50 -0500395*** Keywords ***
396
George Keishing2f369472019-07-23 12:03:10 -0500397Suite Setup Execution
398 [Documentation] Do test case setup tasks.
399
400 Redfish.Login
401
402 Redfish Purge Event Log
403
404 ${status}= Run Keyword And Return Status Logging Test Binary Exist
405 Run Keyword If ${status} == ${False} Install Tarball
406
407
George Keishing8cca7fb2019-05-30 04:23:50 -0500408Suite Teardown Execution
409 [Documentation] Do the post suite teardown.
410
411 Redfish.Logout
412
413
George Keishingb0a491f2019-05-31 02:49:22 -0500414Test Setup Execution
415 [Documentation] Do test case setup tasks.
George Keishing8cca7fb2019-05-30 04:23:50 -0500416
George Keishing0b76e422019-06-03 01:58:52 -0500417 Redfish Purge Event Log
418
George Keishing8cca7fb2019-05-30 04:23:50 -0500419 ${status}= Run Keyword And Return Status Logging Test Binary Exist
420 Run Keyword If ${status} == ${False} Install Tarball
421
422
423Test Teardown Execution
424 [Documentation] Do the post test teardown.
425
426 FFDC On Test Case Fail
George Keishinge5c5ebd2019-06-04 13:52:12 -0500427 Redfish.Login
George Keishingb0a491f2019-05-31 02:49:22 -0500428 Redfish Purge Event Log
George Keishing8cca7fb2019-05-30 04:23:50 -0500429
430
431Event Log Should Not Exist
432 [Documentation] Event log entries should not exist.
433
434 ${elogs}= Get Event Logs
435 Should Be Empty ${elogs} msg=System event log entry is not empty.
George Keishingb0a491f2019-05-31 02:49:22 -0500436
437
438Event Log Should Exist
439 [Documentation] Event log entries should exist.
440
441 ${elogs}= Get Event Logs
442 Should Not Be Empty ${elogs} msg=System event log entry is not empty.
George Keishingd35b5312019-05-31 14:29:07 -0500443
444
445Verify Watchdog EventLog Content
446 [Documentation] Verify watchdog event log content.
447
448 # Example:
449 # {
450 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry",
451 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/31",
452 # "@odata.type": "#LogEntry.v1_4_0.LogEntry",
453 # "Created": "2019-05-31T18:41:33+00:00",
454 # "EntryType": "Event",
455 # "Id": "31",
456 # "Message": "org.open_power.Host.Boot.Error.WatchdogTimedOut",
457 # "Name": "System DBus Event Log Entry",
458 # "Severity": "Critical"
459 # }
460
461 ${elog}= Get Event Logs
462 Should Be Equal As Strings
463 ... ${elog[0]["Message"]} org.open_power.Host.Boot.Error.WatchdogTimedOut
464 ... msg=Watchdog timeout event log was not found.
465 Should Be Equal As Strings
466 ... ${elog[0]["Severity"]} Critical
467 ... msg=Watchdog timeout severity unexpected value.