blob: 342112ea800c686073000d983f7b2ba4c566bf76 [file] [log] [blame]
George Keishing8cca7fb2019-05-30 04:23:50 -05001*** Settings ***
2Documentation Inventory of hardware resources under systems.
3
4Resource ../../../lib/bmc_redfish_resource.robot
5Resource ../../../lib/bmc_redfish_utils.robot
6Resource ../../../lib/logging_utils.robot
7Resource ../../../lib/openbmc_ffdc.robot
George Keishing8fe60242019-06-03 23:06:12 -05008Resource ../../../lib/ipmi_client.robot
George Keishing759c6f92019-06-03 12:14:42 -05009Library ../../../lib/logging_utils.py
George Keishing8cca7fb2019-05-30 04:23:50 -050010
George Keishing2f369472019-07-23 12:03:10 -050011Suite Setup Suite Setup Execution
12Suite Teardown Suite Teardown Execution
George Keishingb0a491f2019-05-31 02:49:22 -050013Test Setup Test Setup Execution
George Keishing8cca7fb2019-05-30 04:23:50 -050014Test Teardown Test Teardown Execution
15
George Keishing759c6f92019-06-03 12:14:42 -050016** Variables ***
17
18${max_num_event_logs} ${200}
19
George Keishing8cca7fb2019-05-30 04:23:50 -050020*** Test Cases ***
21
22Event Log Check After BMC Reboot
23 [Documentation] Check event log after BMC rebooted.
24 [Tags] Event_Log_Check_After_BMC_Reboot
25
26 Redfish Purge Event Log
27 Event Log Should Not Exist
28
29 Redfish OBMC Reboot (off)
30
31 Redfish.Login
32 Wait Until Keyword Succeeds 1 mins 15 secs Redfish.Get ${EVENT_LOG_URI}Entries
33
34 Event Log Should Not Exist
35
36
George Keishingb0a491f2019-05-31 02:49:22 -050037Event Log Check After Host Poweron
38 [Documentation] Check event log after host has booted.
39 [Tags] Event_Log_Check_After_Host_Poweron
40
41 Redfish Purge Event Log
42 Event Log Should Not Exist
43
44 Redfish Power On
45
46 Redfish.Login
47 Event Log Should Not Exist
48
49
50Create Test Event Log And Verify
51 [Documentation] Create event logs and verify via redfish.
52 [Tags] Create_Test_Event_Log_And_Verify
53
54 Create Test Error Log
55 Event Log Should Exist
56
57
Rahul Maheshwari20c08e42021-05-10 01:41:01 -050058Delete Redfish Event Log And Verify
59 [Documentation] Delete Redfish event log and verify via Redfish.
60 [Tags] Delete_Redfish_Event_Log_And_Verify
61
62 Redfish.Login
63 Redfish Purge Event Log
64 Create Test PEL Log
65 ${elog_entry}= Get Event Logs
66
67 Redfish.Delete /redfish/v1/Systems/system/LogServices/EventLog/Entries/${elog_entry[0]["Id"]}
68
69 ${error_entries}= Get Redfish Error Entries
70 Should Be Empty ${error_entries}
71
72
George Keishingb0a491f2019-05-31 02:49:22 -050073Test Event Log Persistency On Restart
74 [Documentation] Restart logging service and verify event logs.
75 [Tags] Test_Event_Log_Persistency_On_Restart
76
77 Create Test Error Log
78 Event Log Should Exist
79
80 BMC Execute Command
81 ... systemctl restart xyz.openbmc_project.Logging.service
82 Sleep 10s reason=Wait for logging service to restart properly.
83
84 Event Log Should Exist
85
86
87Test Event Entry Numbering Reset On Restart
88 [Documentation] Restart logging service and verify event logs entry starts
89 ... from entry "Id" 1.
90 [Tags] Test_Event_Entry_Numbering_Reset_On_Restart
Rahul Maheshwari20560f92021-12-21 05:16:11 -060091 [Setup] Redfish Power Off stack_mode=skip
George Keishingb0a491f2019-05-31 02:49:22 -050092
93 #{
94 # "@odata.context": "/redfish/v1/$metadata#LogEntryCollection.LogEntryCollection",
95 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries",
96 # "@odata.type": "#LogEntryCollection.LogEntryCollection",
97 # "Description": "Collection of System Event Log Entries",
98 # "Members": [
99 # {
100 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry",
101 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/1",
102 # "@odata.type": "#LogEntry.v1_4_0.LogEntry",
103 # "Created": "2019-05-29T13:19:27+00:00",
104 # "EntryType": "Event",
105 # "Id": "1", <----- Event log ID
106 # "Message": "org.open_power.Host.Error.Event",
107 # "Name": "System DBus Event Log Entry",
108 # "Severity": "Critical"
109 # }
110 # ],
111 # "Members@odata.count": 1,
112 # "Name": "System Event Log Entries"
113 #}
114
Rahul Maheshwari20560f92021-12-21 05:16:11 -0600115 Create Test PEL Log
116 Create Test PEL Log
George Keishingb0a491f2019-05-31 02:49:22 -0500117 Event Log Should Exist
118
119 Redfish Purge Event Log
120 Event Log Should Not Exist
121
122 BMC Execute Command
123 ... systemctl restart xyz.openbmc_project.Logging.service
124 Sleep 10s reason=Wait for logging service to restart properly.
125
Rahul Maheshwari20560f92021-12-21 05:16:11 -0600126 Create Test PEL Log
George Keishingb0a491f2019-05-31 02:49:22 -0500127 ${elogs}= Get Event Logs
Rahul Maheshwari20560f92021-12-21 05:16:11 -0600128 Should Be Equal ${elogs[0]["Id"]} 1 msg=Event log entry is not 1
George Keishingb0a491f2019-05-31 02:49:22 -0500129
130
131Test Event Log Persistency On Reboot
132 [Documentation] Reboot BMC and verify event log.
133 [Tags] Test_Event_Log_Persistency_On_Reboot
134
135 Redfish Purge Event Log
136 Create Test Error Log
137 Event Log Should Exist
138
139 Redfish OBMC Reboot (off)
140
141 Redfish.Login
142 Wait Until Keyword Succeeds 1 mins 15 secs Redfish.Get ${EVENT_LOG_URI}Entries
143
144 Event Log Should Exist
145
146
George Keishing148d08c2019-05-31 12:39:04 -0500147Create Test Event Log And Verify Time Stamp
148 [Documentation] Create event logs and verify time stamp.
149 [Tags] Create_Test_Event_Log_And_Verify_Time_Stamp
150
151 #{
152 # "@odata.context": "/redfish/v1/$metadata#LogEntryCollection.LogEntryCollection",
153 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries",
154 # "@odata.type": "#LogEntryCollection.LogEntryCollection",
155 # "Description": "Collection of System Event Log Entries",
156 # "Members": [
157 # {
158 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry",
159 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/1",
160 # "@odata.type": "#LogEntry.v1_4_0.LogEntry",
161 # "Created": "2019-05-29T13:19:27+00:00", <--- Time stamp
162 # "EntryType": "Event",
163 # "Id": "1",
164 # "Message": "org.open_power.Host.Error.Event",
165 # "Name": "System DBus Event Log Entry",
166 # "Severity": "Critical"
167 # }
168 # ],
169 # "Members@odata.count": 1,
170 # "Name": "System Event Log Entries"
171 #}
172
173 Redfish Purge Event Log
174
175 Create Test Error Log
176 Sleep 2s
177 Create Test Error Log
178
179 ${elog_entry}= Get Event Logs
180
181 # The event log generated is associated with the epoc time and unique
182 # for every error and in increasing time stamp.
183 ${time_stamp1}= Convert Date ${elog_entry[0]["Created"]} epoch
184 ${time_stamp2}= Convert Date ${elog_entry[1]["Created"]} epoch
185
186 Should Be True ${time_stamp2} > ${time_stamp1}
187
188
Rahul Maheshwari24034a92021-08-18 02:23:42 -0500189# TODO: openbmc/openbmc-test-automation#1789
Rahul Maheshwari6462f262021-05-10 00:49:46 -0500190Verify Setting Error Log As Resolved
191 [Documentation] Verify modified field of error log is updated when error log is marked resolved.
192 [Tags] Verify_Setting_Error_Log_As_Resolved
193
194 Create Test PEL Log
195 ${elog_entry}= Get Event Logs
196
197 # Wait for 5 seconds after creating error log.
198 Sleep 5s
199
200 # Mark error log as resolved by setting it to true.
201 Redfish.Patch ${EVENT_LOG_URI}Entries/${elog_entry[0]["Id"]} body={'Resolved':True}
202
203 ${elog_entry}= Get Event Logs
204
205 # Example error log with resolve field set to true:
206 # {
207 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/2045",
208 # "@odata.type": "#LogEntry.v1_8_0.LogEntry",
209 # "AdditionalDataURI": "/redfish/v1/Systems/system/LogServices/EventLog/attachment/2045",
210 # "Created": "2021-05-11T04:45:07+00:00",
211 # "EntryType": "Event",
212 # "Id": "2045",
213 # "Message": "xyz.openbmc_project.Host.Error.Event",
214 # "Modified": "2021-05-11T07:24:36+00:00",
215 # "Name": "System Event Log Entry",
216 # "Resolved": true,
217 # "Severity": "OK"
218 # }
219
220 Should Be Equal As Strings ${elog_entry[0]["Resolved"]} True
221
222 # Difference created and modified time of error log should be around 5 seconds.
223 ${creation_time}= Convert Date ${elog_entry[0]["Created"]} epoch
224 ${modification_time}= Convert Date ${elog_entry[0]["Modified"]} epoch
225
226 ${diff}= Subtract Date From Date ${modification_time} ${creation_time}
227 ${diff}= Convert To Number ${diff}
228 Should Be True 4 < ${diff} < 8
229
230
George Keishingddbc88c2019-06-06 07:29:41 -0500231Verify IPMI SEL Delete
232 [Documentation] Verify IPMI SEL delete operation.
233 [Tags] Verify_IPMI_SEL_Delete
234
235 Redfish Purge Event Log
236 Create Test Error Log
237
238 ${sel_list}= Run IPMI Standard Command sel list
239 Should Not Be Equal As Strings ${sel_list} SEL has no entries
240
241 # Example of SEL List:
242 # 4 | 04/21/2017 | 10:51:16 | System Event #0x01 | Undetermined system hardware failure | Asserted
243
244 ${sel_entry}= Fetch from Left ${sel_list} |
245 ${sel_entry}= Evaluate $sel_entry.replace(' ','')
246 ${sel_entry}= Convert To Integer 0x${sel_entry}
247
248 ${sel_delete}= Run IPMI Standard Command sel delete ${sel_entry}
249 Should Be Equal As Strings ${sel_delete} Deleted entry ${sel_entry}
250 ... case_insensitive=True
251
252 ${sel_list}= Run IPMI Standard Command sel list
253 Should Be Equal As Strings ${sel_list} SEL has no entries
254 ... case_insensitive=True
255
256
George Keishing0b76e422019-06-03 01:58:52 -0500257Delete Non Existing SEL Event Entry
258 [Documentation] Delete non existing SEL event entry.
259 [Tags] Delete_Non_Existing_SEL_Event_Entry
260
261 ${sel_delete}= Run Keyword And Expect Error *
262 ... Run IPMI Standard Command sel delete 100
263 Should Contain ${sel_delete} Unable to delete entry
264 ... case_insensitive=True
265
266
267Delete Invalid SEL Event Entry
268 [Documentation] Delete invalid SEL event entry.
269 [Tags] Delete_Invalid_SEL_Event_Entry
270
271 ${sel_delete}= Run Keyword And Expect Error *
272 ... Run IPMI Standard Command sel delete abc
273 Should Contain ${sel_delete} Given SEL ID 'abc' is invalid
274 ... case_insensitive=True
275
276
277Verify IPMI SEL Event Entries
278 [Documentation] Verify IPMI SEL's entries info.
279 [Tags] Verify_IPMI_SEL_Event_Entries
280
281 # Generate error logs of random count.
282 ${count}= Evaluate random.randint(1, 5) modules=random
283 Repeat Keyword ${count} Create Test Error Log
284
285 ${sel_entries_count}= Get IPMI SEL Setting Entries
286 Should Be Equal As Strings ${sel_entries_count} ${count}
287
288
289Verify IPMI SEL Event Last Add Time
290 [Documentation] Verify IPMI SEL's last added timestamp.
291 [Tags] Verify_IPMI_SEL_Event_Last_Add_Time
292
293 Create Test Error Log
294 ${sel_time}= Run IPMI Standard Command sel time get
295 ${sel_time}= Convert Date ${sel_time}
296 ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True
297
298 ${sel_last_add_time}= Get IPMI SEL Setting Last Add Time
299 ${sel_last_add_time}= Convert Date ${sel_last_add_time}
300 ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True
301
302 ${time_diff}=
303 ... Subtract Date From Date ${sel_last_add_time} ${sel_time}
304
305 # Verify if the delay in current time check and last add SEL time
306 # is less or equals to 2 seconds.
307 Should Be True ${time_diff} <= 2
308
309
George Keishingd35b5312019-05-31 14:29:07 -0500310Create Test Event Log And Delete
311 [Documentation] Create an event log and delete it.
312 [Tags] Create_Test_Event_Log_And_Delete
313
314 Create Test Error Log
315 Redfish Purge Event Log
316 Event Log Should Not Exist
317
318
319Create Multiple Test Event Logs And Delete All
320 [Documentation] Create multiple event logs and delete all.
321 [Tags] Create_Multiple_Test_Event_Logs_And_Delete_All
322
323 Create Test Error Log
324 Create Test Error Log
325 Create Test Error Log
326 Redfish Purge Event Log
327 Event Log Should Not Exist
328
329
George Keishingddbc88c2019-06-06 07:29:41 -0500330Create Two Test Event Logs And Delete One
331 [Documentation] Create two event logs and delete the first entry.
George Keishing89f499d2021-04-19 05:25:40 -0500332 [Tags] Create_Two_Test_Event_Logs_And_Delete_One
Rahul Maheshwaric2d33652021-12-21 00:19:58 -0600333 [Setup] Redfish Power Off stack_mode=skip
George Keishingddbc88c2019-06-06 07:29:41 -0500334
335 Redfish Purge Event Log
Rahul Maheshwaric2d33652021-12-21 00:19:58 -0600336 Create Test PEL Log
337 Create Test PEL Log
338 ${error_entries_before}= Get Redfish Error Entries
339 Redfish.Delete /redfish/v1/Systems/system/LogServices/EventLog/Entries/${error_entries_before[0]}
340
341 ${error_entries_after}= Get Redfish Error Entries
342 Should Not Contain ${error_entries_after} ${error_entries_before[0]}
343 Should Contain ${error_entries_after} ${error_entries_before[1]}
George Keishingddbc88c2019-06-06 07:29:41 -0500344
345
George Keishingd35b5312019-05-31 14:29:07 -0500346Verify Watchdog Timedout Event
347 [Documentation] Trigger watchdog timed out and verify event log generated.
348 [Tags] Verify_Watchdog_Timedout_Event
349
350 Redfish Power On
351
352 # Clear errors if there are any.
353 Redfish.Login
354 Redfish Purge Event Log
355
356 Trigger Host Watchdog Error
357
358 # Logging takes time to generate the timeout error.
359 Wait Until Keyword Succeeds 2 min 30 sec
360 ... Verify Watchdog EventLog Content
361
Tim Leed226bbd2022-04-26 15:10:47 +0800362 ${running_states}= Create Dictionary
363 ... bmc=Enabled
364 ... chassis=On
365 ... host=Enabled
366 ... boot_progress=OSBootStarted
367
368 Wait Until Keyword Succeeds 1 min 10 sec Match State ${running_states}
George Keishingd35b5312019-05-31 14:29:07 -0500369
370Verify Event Logs Capping
371 [Documentation] Verify event logs capping.
372 [Tags] Verify_Event_Logs_Capping
373
374 Redfish Purge Event Log
375
376 ${cmd}= Catenate for i in {1..201}; do /tmp/tarball/bin/logging-test -c
George Keishingd2d2bd22019-06-12 13:49:41 -0500377 ... AutoTestSimple;sleep 1;done
George Keishingd35b5312019-05-31 14:29:07 -0500378 BMC Execute Command ${cmd}
379
380 ${elogs}= Get Event Logs
381 ${count}= Get Length ${elogs}
382 Run Keyword If ${count} > 200
383 ... Fail Error logs created exceeded max capacity 200.
384
385
George Keishing759c6f92019-06-03 12:14:42 -0500386Test Event Log Wrapping
387 [Documentation] Verify event log entries wraps when 200 max cap is reached.
388 [Tags] Test_Event_Log_Wrapping
389
390 # Restarting logging service in order to clear logs and get the next log
391 # ID set to 1.
392 BMC Execute Command
393 ... systemctl restart xyz.openbmc_project.Logging.service
394 Sleep 10s reason=Wait for logging service to restart properly.
395
396 # Create ${max_num_event_logs} event logs.
397 ${cmd}= Catenate for i in {1..${max_num_event_logs}}; do /tmp/tarball/bin/logging-test -c
George Keishingd2d2bd22019-06-12 13:49:41 -0500398 ... AutoTestSimple;sleep 1;done
George Keishing759c6f92019-06-03 12:14:42 -0500399 BMC Execute Command ${cmd}
400
401 # Verify that event logs with IDs 1 and ${max_num_event_logs} exist.
402 ${event_log}= Get Event Logs
403
404 ${log_entries}= Filter Struct ${event_log} [('Id', '1')]
Michael Walsh39c00512019-07-17 10:54:06 -0500405 Rprint Vars log_entries
George Keishing759c6f92019-06-03 12:14:42 -0500406 Should Be Equal As Strings ${log_entries[0]["Id"]} 1
407
408 ${log_entries}= Filter Struct ${event_log} [('Id', '${max_num_event_logs}')]
Michael Walsh39c00512019-07-17 10:54:06 -0500409 Rprint Vars log_entries
George Keishing759c6f92019-06-03 12:14:42 -0500410 Should Be Equal As Strings ${log_entries[0]["Id"]} ${max_num_event_logs}
411
412 # Create event log and verify the entry ID, ${max_num_event_logs + 1}.
413 ${next_event_log_id}= Set Variable ${max_num_event_logs + 1}
414
415 Create Test Error Log
416
417 ${event_log}= Get Event Logs
418
419 ${log_entries}= Filter Struct ${event_log} [('Id', '${next_event_log_id}')]
Michael Walsh39c00512019-07-17 10:54:06 -0500420 Rprint Vars log_entries
George Keishing759c6f92019-06-03 12:14:42 -0500421 Should Be Equal As Strings ${log_entries[0]["Id"]} ${next_event_log_id}
422
423 # Event log 1 should be wrapped.
424 ${log_entries}= Filter Struct ${event_log} [('Id', '1')]
Michael Walsh39c00512019-07-17 10:54:06 -0500425 Rprint Vars log_entries
George Keishing759c6f92019-06-03 12:14:42 -0500426
427 ${length_log_entries} Get Length ${log_entries}
428 Should Be Equal As Integers ${length_log_entries} 0
429 ... msg=The event log should have wrapped such that entry ID 1 is now purged.
430
431
George Keishing8cca7fb2019-05-30 04:23:50 -0500432*** Keywords ***
433
George Keishing2f369472019-07-23 12:03:10 -0500434Suite Setup Execution
435 [Documentation] Do test case setup tasks.
436
437 Redfish.Login
438
439 Redfish Purge Event Log
440
441 ${status}= Run Keyword And Return Status Logging Test Binary Exist
442 Run Keyword If ${status} == ${False} Install Tarball
443
444
George Keishing8cca7fb2019-05-30 04:23:50 -0500445Suite Teardown Execution
446 [Documentation] Do the post suite teardown.
447
448 Redfish.Logout
449
450
George Keishingb0a491f2019-05-31 02:49:22 -0500451Test Setup Execution
452 [Documentation] Do test case setup tasks.
George Keishing8cca7fb2019-05-30 04:23:50 -0500453
George Keishing0b76e422019-06-03 01:58:52 -0500454 Redfish Purge Event Log
455
George Keishing8cca7fb2019-05-30 04:23:50 -0500456 ${status}= Run Keyword And Return Status Logging Test Binary Exist
457 Run Keyword If ${status} == ${False} Install Tarball
458
459
460Test Teardown Execution
461 [Documentation] Do the post test teardown.
462
463 FFDC On Test Case Fail
George Keishinge5c5ebd2019-06-04 13:52:12 -0500464 Redfish.Login
George Keishingb0a491f2019-05-31 02:49:22 -0500465 Redfish Purge Event Log
George Keishing8cca7fb2019-05-30 04:23:50 -0500466
467
Rahul Maheshwari20c08e42021-05-10 01:41:01 -0500468Get Redfish Error Entries
469 [Documentation] Return Redfish error ids list.
470 ${error_uris}= redfish_utils.get_member_list /redfish/v1/Systems/system/LogServices/EventLog/Entries
471 ${error_ids}= Create List
472
473 FOR ${error_uri} IN @{error_uris}
474 ${error_id}= Fetch From Right ${error_uri} /
475 Append To List ${error_ids} ${error_id}
476 END
477
478 [Return] ${error_ids}
479
480
George Keishing8cca7fb2019-05-30 04:23:50 -0500481Event Log Should Not Exist
482 [Documentation] Event log entries should not exist.
483
484 ${elogs}= Get Event Logs
485 Should Be Empty ${elogs} msg=System event log entry is not empty.
George Keishingb0a491f2019-05-31 02:49:22 -0500486
487
488Event Log Should Exist
489 [Documentation] Event log entries should exist.
490
491 ${elogs}= Get Event Logs
492 Should Not Be Empty ${elogs} msg=System event log entry is not empty.
George Keishingd35b5312019-05-31 14:29:07 -0500493
494
495Verify Watchdog EventLog Content
496 [Documentation] Verify watchdog event log content.
497
498 # Example:
499 # {
500 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry",
501 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/31",
502 # "@odata.type": "#LogEntry.v1_4_0.LogEntry",
503 # "Created": "2019-05-31T18:41:33+00:00",
504 # "EntryType": "Event",
505 # "Id": "31",
506 # "Message": "org.open_power.Host.Boot.Error.WatchdogTimedOut",
507 # "Name": "System DBus Event Log Entry",
508 # "Severity": "Critical"
509 # }
510
511 ${elog}= Get Event Logs
512 Should Be Equal As Strings
513 ... ${elog[0]["Message"]} org.open_power.Host.Boot.Error.WatchdogTimedOut
514 ... msg=Watchdog timeout event log was not found.
515 Should Be Equal As Strings
516 ... ${elog[0]["Severity"]} Critical
517 ... msg=Watchdog timeout severity unexpected value.