blob: 82cd44df49ff6c1d933d59e1040bf76eca3d6f64 [file] [log] [blame]
George Keishing8cca7fb2019-05-30 04:23:50 -05001*** Settings ***
2Documentation Inventory of hardware resources under systems.
3
4Resource ../../../lib/bmc_redfish_resource.robot
5Resource ../../../lib/bmc_redfish_utils.robot
6Resource ../../../lib/logging_utils.robot
7Resource ../../../lib/openbmc_ffdc.robot
George Keishing8fe60242019-06-03 23:06:12 -05008Resource ../../../lib/ipmi_client.robot
George Keishing759c6f92019-06-03 12:14:42 -05009Library ../../../lib/logging_utils.py
George Keishing8cca7fb2019-05-30 04:23:50 -050010
George Keishing2f369472019-07-23 12:03:10 -050011Suite Setup Suite Setup Execution
12Suite Teardown Suite Teardown Execution
George Keishingb0a491f2019-05-31 02:49:22 -050013Test Setup Test Setup Execution
George Keishing8cca7fb2019-05-30 04:23:50 -050014Test Teardown Test Teardown Execution
15
George Keishing759c6f92019-06-03 12:14:42 -050016** Variables ***
17
18${max_num_event_logs} ${200}
19
George Keishing8cca7fb2019-05-30 04:23:50 -050020*** Test Cases ***
21
22Event Log Check After BMC Reboot
23 [Documentation] Check event log after BMC rebooted.
24 [Tags] Event_Log_Check_After_BMC_Reboot
25
26 Redfish Purge Event Log
27 Event Log Should Not Exist
28
29 Redfish OBMC Reboot (off)
30
31 Redfish.Login
32 Wait Until Keyword Succeeds 1 mins 15 secs Redfish.Get ${EVENT_LOG_URI}Entries
33
34 Event Log Should Not Exist
35
36
George Keishingb0a491f2019-05-31 02:49:22 -050037Event Log Check After Host Poweron
38 [Documentation] Check event log after host has booted.
39 [Tags] Event_Log_Check_After_Host_Poweron
40
41 Redfish Purge Event Log
42 Event Log Should Not Exist
43
44 Redfish Power On
45
46 Redfish.Login
47 Event Log Should Not Exist
48
49
50Create Test Event Log And Verify
51 [Documentation] Create event logs and verify via redfish.
52 [Tags] Create_Test_Event_Log_And_Verify
53
54 Create Test Error Log
55 Event Log Should Exist
56
57
Rahul Maheshwari20c08e42021-05-10 01:41:01 -050058Delete Redfish Event Log And Verify
59 [Documentation] Delete Redfish event log and verify via Redfish.
60 [Tags] Delete_Redfish_Event_Log_And_Verify
61
62 Redfish.Login
63 Redfish Purge Event Log
64 Create Test PEL Log
65 ${elog_entry}= Get Event Logs
66
67 Redfish.Delete /redfish/v1/Systems/system/LogServices/EventLog/Entries/${elog_entry[0]["Id"]}
68
69 ${error_entries}= Get Redfish Error Entries
70 Should Be Empty ${error_entries}
71
72
George Keishingb0a491f2019-05-31 02:49:22 -050073Test Event Log Persistency On Restart
74 [Documentation] Restart logging service and verify event logs.
75 [Tags] Test_Event_Log_Persistency_On_Restart
76
77 Create Test Error Log
78 Event Log Should Exist
79
80 BMC Execute Command
81 ... systemctl restart xyz.openbmc_project.Logging.service
82 Sleep 10s reason=Wait for logging service to restart properly.
83
84 Event Log Should Exist
85
86
87Test Event Entry Numbering Reset On Restart
88 [Documentation] Restart logging service and verify event logs entry starts
89 ... from entry "Id" 1.
90 [Tags] Test_Event_Entry_Numbering_Reset_On_Restart
Rahul Maheshwari20560f92021-12-21 05:16:11 -060091 [Setup] Redfish Power Off stack_mode=skip
George Keishingb0a491f2019-05-31 02:49:22 -050092
93 #{
94 # "@odata.context": "/redfish/v1/$metadata#LogEntryCollection.LogEntryCollection",
95 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries",
96 # "@odata.type": "#LogEntryCollection.LogEntryCollection",
97 # "Description": "Collection of System Event Log Entries",
98 # "Members": [
99 # {
100 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry",
101 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/1",
102 # "@odata.type": "#LogEntry.v1_4_0.LogEntry",
103 # "Created": "2019-05-29T13:19:27+00:00",
104 # "EntryType": "Event",
105 # "Id": "1", <----- Event log ID
106 # "Message": "org.open_power.Host.Error.Event",
107 # "Name": "System DBus Event Log Entry",
108 # "Severity": "Critical"
109 # }
110 # ],
111 # "Members@odata.count": 1,
112 # "Name": "System Event Log Entries"
113 #}
114
Rahul Maheshwari20560f92021-12-21 05:16:11 -0600115 Create Test PEL Log
116 Create Test PEL Log
George Keishingb0a491f2019-05-31 02:49:22 -0500117 Event Log Should Exist
118
119 Redfish Purge Event Log
120 Event Log Should Not Exist
121
122 BMC Execute Command
123 ... systemctl restart xyz.openbmc_project.Logging.service
124 Sleep 10s reason=Wait for logging service to restart properly.
125
Rahul Maheshwari20560f92021-12-21 05:16:11 -0600126 Create Test PEL Log
George Keishingb0a491f2019-05-31 02:49:22 -0500127 ${elogs}= Get Event Logs
Rahul Maheshwari20560f92021-12-21 05:16:11 -0600128 Should Be Equal ${elogs[0]["Id"]} 1 msg=Event log entry is not 1
George Keishingb0a491f2019-05-31 02:49:22 -0500129
130
131Test Event Log Persistency On Reboot
132 [Documentation] Reboot BMC and verify event log.
133 [Tags] Test_Event_Log_Persistency_On_Reboot
134
135 Redfish Purge Event Log
136 Create Test Error Log
137 Event Log Should Exist
138
139 Redfish OBMC Reboot (off)
140
141 Redfish.Login
142 Wait Until Keyword Succeeds 1 mins 15 secs Redfish.Get ${EVENT_LOG_URI}Entries
143
144 Event Log Should Exist
145
146
George Keishing148d08c2019-05-31 12:39:04 -0500147Create Test Event Log And Verify Time Stamp
148 [Documentation] Create event logs and verify time stamp.
149 [Tags] Create_Test_Event_Log_And_Verify_Time_Stamp
150
151 #{
152 # "@odata.context": "/redfish/v1/$metadata#LogEntryCollection.LogEntryCollection",
153 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries",
154 # "@odata.type": "#LogEntryCollection.LogEntryCollection",
155 # "Description": "Collection of System Event Log Entries",
156 # "Members": [
157 # {
158 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry",
159 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/1",
160 # "@odata.type": "#LogEntry.v1_4_0.LogEntry",
161 # "Created": "2019-05-29T13:19:27+00:00", <--- Time stamp
162 # "EntryType": "Event",
163 # "Id": "1",
164 # "Message": "org.open_power.Host.Error.Event",
165 # "Name": "System DBus Event Log Entry",
166 # "Severity": "Critical"
167 # }
168 # ],
169 # "Members@odata.count": 1,
170 # "Name": "System Event Log Entries"
171 #}
172
173 Redfish Purge Event Log
174
175 Create Test Error Log
176 Sleep 2s
177 Create Test Error Log
178
179 ${elog_entry}= Get Event Logs
180
181 # The event log generated is associated with the epoc time and unique
182 # for every error and in increasing time stamp.
183 ${time_stamp1}= Convert Date ${elog_entry[0]["Created"]} epoch
184 ${time_stamp2}= Convert Date ${elog_entry[1]["Created"]} epoch
185
186 Should Be True ${time_stamp2} > ${time_stamp1}
187
188
Rahul Maheshwari24034a92021-08-18 02:23:42 -0500189# TODO: openbmc/openbmc-test-automation#1789
Rahul Maheshwari6462f262021-05-10 00:49:46 -0500190Verify Setting Error Log As Resolved
191 [Documentation] Verify modified field of error log is updated when error log is marked resolved.
192 [Tags] Verify_Setting_Error_Log_As_Resolved
193
194 Create Test PEL Log
195 ${elog_entry}= Get Event Logs
196
197 # Wait for 5 seconds after creating error log.
198 Sleep 5s
199
200 # Mark error log as resolved by setting it to true.
201 Redfish.Patch ${EVENT_LOG_URI}Entries/${elog_entry[0]["Id"]} body={'Resolved':True}
202
203 ${elog_entry}= Get Event Logs
204
205 # Example error log with resolve field set to true:
206 # {
207 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/2045",
208 # "@odata.type": "#LogEntry.v1_8_0.LogEntry",
209 # "AdditionalDataURI": "/redfish/v1/Systems/system/LogServices/EventLog/attachment/2045",
210 # "Created": "2021-05-11T04:45:07+00:00",
211 # "EntryType": "Event",
212 # "Id": "2045",
213 # "Message": "xyz.openbmc_project.Host.Error.Event",
214 # "Modified": "2021-05-11T07:24:36+00:00",
215 # "Name": "System Event Log Entry",
216 # "Resolved": true,
217 # "Severity": "OK"
218 # }
219
220 Should Be Equal As Strings ${elog_entry[0]["Resolved"]} True
221
222 # Difference created and modified time of error log should be around 5 seconds.
223 ${creation_time}= Convert Date ${elog_entry[0]["Created"]} epoch
224 ${modification_time}= Convert Date ${elog_entry[0]["Modified"]} epoch
225
226 ${diff}= Subtract Date From Date ${modification_time} ${creation_time}
227 ${diff}= Convert To Number ${diff}
228 Should Be True 4 < ${diff} < 8
229
230
George Keishingddbc88c2019-06-06 07:29:41 -0500231Verify IPMI SEL Delete
232 [Documentation] Verify IPMI SEL delete operation.
233 [Tags] Verify_IPMI_SEL_Delete
234
235 Redfish Purge Event Log
236 Create Test Error Log
237
238 ${sel_list}= Run IPMI Standard Command sel list
239 Should Not Be Equal As Strings ${sel_list} SEL has no entries
240
241 # Example of SEL List:
242 # 4 | 04/21/2017 | 10:51:16 | System Event #0x01 | Undetermined system hardware failure | Asserted
243
244 ${sel_entry}= Fetch from Left ${sel_list} |
245 ${sel_entry}= Evaluate $sel_entry.replace(' ','')
246 ${sel_entry}= Convert To Integer 0x${sel_entry}
247
248 ${sel_delete}= Run IPMI Standard Command sel delete ${sel_entry}
249 Should Be Equal As Strings ${sel_delete} Deleted entry ${sel_entry}
250 ... case_insensitive=True
251
252 ${sel_list}= Run IPMI Standard Command sel list
253 Should Be Equal As Strings ${sel_list} SEL has no entries
254 ... case_insensitive=True
255
256
George Keishing0b76e422019-06-03 01:58:52 -0500257Delete Non Existing SEL Event Entry
258 [Documentation] Delete non existing SEL event entry.
259 [Tags] Delete_Non_Existing_SEL_Event_Entry
260
261 ${sel_delete}= Run Keyword And Expect Error *
262 ... Run IPMI Standard Command sel delete 100
263 Should Contain ${sel_delete} Unable to delete entry
264 ... case_insensitive=True
265
266
267Delete Invalid SEL Event Entry
268 [Documentation] Delete invalid SEL event entry.
269 [Tags] Delete_Invalid_SEL_Event_Entry
270
271 ${sel_delete}= Run Keyword And Expect Error *
272 ... Run IPMI Standard Command sel delete abc
273 Should Contain ${sel_delete} Given SEL ID 'abc' is invalid
274 ... case_insensitive=True
275
276
277Verify IPMI SEL Event Entries
278 [Documentation] Verify IPMI SEL's entries info.
279 [Tags] Verify_IPMI_SEL_Event_Entries
280
281 # Generate error logs of random count.
282 ${count}= Evaluate random.randint(1, 5) modules=random
283 Repeat Keyword ${count} Create Test Error Log
284
285 ${sel_entries_count}= Get IPMI SEL Setting Entries
286 Should Be Equal As Strings ${sel_entries_count} ${count}
287
288
289Verify IPMI SEL Event Last Add Time
290 [Documentation] Verify IPMI SEL's last added timestamp.
291 [Tags] Verify_IPMI_SEL_Event_Last_Add_Time
292
293 Create Test Error Log
294 ${sel_time}= Run IPMI Standard Command sel time get
295 ${sel_time}= Convert Date ${sel_time}
296 ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True
297
298 ${sel_last_add_time}= Get IPMI SEL Setting Last Add Time
299 ${sel_last_add_time}= Convert Date ${sel_last_add_time}
300 ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True
301
302 ${time_diff}=
303 ... Subtract Date From Date ${sel_last_add_time} ${sel_time}
304
305 # Verify if the delay in current time check and last add SEL time
306 # is less or equals to 2 seconds.
307 Should Be True ${time_diff} <= 2
308
309
George Keishingd35b5312019-05-31 14:29:07 -0500310Create Test Event Log And Delete
311 [Documentation] Create an event log and delete it.
312 [Tags] Create_Test_Event_Log_And_Delete
313
314 Create Test Error Log
315 Redfish Purge Event Log
316 Event Log Should Not Exist
317
318
319Create Multiple Test Event Logs And Delete All
320 [Documentation] Create multiple event logs and delete all.
321 [Tags] Create_Multiple_Test_Event_Logs_And_Delete_All
322
323 Create Test Error Log
324 Create Test Error Log
325 Create Test Error Log
326 Redfish Purge Event Log
327 Event Log Should Not Exist
328
329
George Keishingddbc88c2019-06-06 07:29:41 -0500330Create Two Test Event Logs And Delete One
331 [Documentation] Create two event logs and delete the first entry.
George Keishing89f499d2021-04-19 05:25:40 -0500332 [Tags] Create_Two_Test_Event_Logs_And_Delete_One
Rahul Maheshwaric2d33652021-12-21 00:19:58 -0600333 [Setup] Redfish Power Off stack_mode=skip
George Keishingddbc88c2019-06-06 07:29:41 -0500334
335 Redfish Purge Event Log
Rahul Maheshwaric2d33652021-12-21 00:19:58 -0600336 Create Test PEL Log
337 Create Test PEL Log
338 ${error_entries_before}= Get Redfish Error Entries
339 Redfish.Delete /redfish/v1/Systems/system/LogServices/EventLog/Entries/${error_entries_before[0]}
340
341 ${error_entries_after}= Get Redfish Error Entries
342 Should Not Contain ${error_entries_after} ${error_entries_before[0]}
343 Should Contain ${error_entries_after} ${error_entries_before[1]}
George Keishingddbc88c2019-06-06 07:29:41 -0500344
345
George Keishingd35b5312019-05-31 14:29:07 -0500346Verify Watchdog Timedout Event
347 [Documentation] Trigger watchdog timed out and verify event log generated.
348 [Tags] Verify_Watchdog_Timedout_Event
George Keishing2ab62412022-07-28 11:53:23 -0500349 [Teardown] Run Keywords Test Teardown Execution AND Redfish Power Off stack_mode=skip
George Keishingd35b5312019-05-31 14:29:07 -0500350
George Keishing2ab62412022-07-28 11:53:23 -0500351 Redfish Power Off stack_mode=skip
George Keishingd35b5312019-05-31 14:29:07 -0500352
353 # Clear errors if there are any.
354 Redfish.Login
355 Redfish Purge Event Log
356
George Keishing2ab62412022-07-28 11:53:23 -0500357 # Reference: [Old legacy REST code] Trigger Host Watchdog Error
358 # Currently, no known redfish interface to set to trigger watchdog timer.
359
360 Redfish Initiate Auto Reboot 1000
George Keishingd35b5312019-05-31 14:29:07 -0500361
362 # Logging takes time to generate the timeout error.
George Keishing2ab62412022-07-28 11:53:23 -0500363 Wait Until Keyword Succeeds 3 min 20 sec Verify Watchdog EventLog Content
George Keishingd35b5312019-05-31 14:29:07 -0500364
George Keishingd35b5312019-05-31 14:29:07 -0500365
366Verify Event Logs Capping
367 [Documentation] Verify event logs capping.
368 [Tags] Verify_Event_Logs_Capping
369
370 Redfish Purge Event Log
371
George Keishing033ee182022-07-28 05:24:55 -0500372 ${cmd}= Catenate uptime; for i in {1..201}; do /tmp/tarball/bin/logging-test -c
373 ... AutoTestSimple;sleep 1;done; uptime
George Keishingd35b5312019-05-31 14:29:07 -0500374 BMC Execute Command ${cmd}
375
376 ${elogs}= Get Event Logs
377 ${count}= Get Length ${elogs}
378 Run Keyword If ${count} > 200
379 ... Fail Error logs created exceeded max capacity 200.
380
381
George Keishing759c6f92019-06-03 12:14:42 -0500382Test Event Log Wrapping
383 [Documentation] Verify event log entries wraps when 200 max cap is reached.
384 [Tags] Test_Event_Log_Wrapping
385
386 # Restarting logging service in order to clear logs and get the next log
387 # ID set to 1.
388 BMC Execute Command
389 ... systemctl restart xyz.openbmc_project.Logging.service
390 Sleep 10s reason=Wait for logging service to restart properly.
391
392 # Create ${max_num_event_logs} event logs.
George Keishing033ee182022-07-28 05:24:55 -0500393 ${cmd}= Catenate uptime; for i in {1..${max_num_event_logs}}; do /tmp/tarball/bin/logging-test -c
394 ... AutoTestSimple;sleep 1;done; uptime
George Keishing759c6f92019-06-03 12:14:42 -0500395 BMC Execute Command ${cmd}
396
397 # Verify that event logs with IDs 1 and ${max_num_event_logs} exist.
398 ${event_log}= Get Event Logs
399
400 ${log_entries}= Filter Struct ${event_log} [('Id', '1')]
Michael Walsh39c00512019-07-17 10:54:06 -0500401 Rprint Vars log_entries
George Keishing759c6f92019-06-03 12:14:42 -0500402 Should Be Equal As Strings ${log_entries[0]["Id"]} 1
403
404 ${log_entries}= Filter Struct ${event_log} [('Id', '${max_num_event_logs}')]
Michael Walsh39c00512019-07-17 10:54:06 -0500405 Rprint Vars log_entries
George Keishing759c6f92019-06-03 12:14:42 -0500406 Should Be Equal As Strings ${log_entries[0]["Id"]} ${max_num_event_logs}
407
408 # Create event log and verify the entry ID, ${max_num_event_logs + 1}.
409 ${next_event_log_id}= Set Variable ${max_num_event_logs + 1}
410
411 Create Test Error Log
412
413 ${event_log}= Get Event Logs
414
415 ${log_entries}= Filter Struct ${event_log} [('Id', '${next_event_log_id}')]
Michael Walsh39c00512019-07-17 10:54:06 -0500416 Rprint Vars log_entries
George Keishing759c6f92019-06-03 12:14:42 -0500417 Should Be Equal As Strings ${log_entries[0]["Id"]} ${next_event_log_id}
418
419 # Event log 1 should be wrapped.
420 ${log_entries}= Filter Struct ${event_log} [('Id', '1')]
Michael Walsh39c00512019-07-17 10:54:06 -0500421 Rprint Vars log_entries
George Keishing759c6f92019-06-03 12:14:42 -0500422
423 ${length_log_entries} Get Length ${log_entries}
424 Should Be Equal As Integers ${length_log_entries} 0
425 ... msg=The event log should have wrapped such that entry ID 1 is now purged.
426
427
George Keishing8cca7fb2019-05-30 04:23:50 -0500428*** Keywords ***
429
George Keishing2f369472019-07-23 12:03:10 -0500430Suite Setup Execution
431 [Documentation] Do test case setup tasks.
432
433 Redfish.Login
434
435 Redfish Purge Event Log
436
437 ${status}= Run Keyword And Return Status Logging Test Binary Exist
438 Run Keyword If ${status} == ${False} Install Tarball
439
440
George Keishing8cca7fb2019-05-30 04:23:50 -0500441Suite Teardown Execution
442 [Documentation] Do the post suite teardown.
443
444 Redfish.Logout
445
446
George Keishingb0a491f2019-05-31 02:49:22 -0500447Test Setup Execution
448 [Documentation] Do test case setup tasks.
George Keishing8cca7fb2019-05-30 04:23:50 -0500449
George Keishing0b76e422019-06-03 01:58:52 -0500450 Redfish Purge Event Log
451
George Keishing8cca7fb2019-05-30 04:23:50 -0500452 ${status}= Run Keyword And Return Status Logging Test Binary Exist
453 Run Keyword If ${status} == ${False} Install Tarball
454
455
456Test Teardown Execution
457 [Documentation] Do the post test teardown.
458
459 FFDC On Test Case Fail
George Keishinge5c5ebd2019-06-04 13:52:12 -0500460 Redfish.Login
George Keishingb0a491f2019-05-31 02:49:22 -0500461 Redfish Purge Event Log
George Keishing8cca7fb2019-05-30 04:23:50 -0500462
463
Rahul Maheshwari20c08e42021-05-10 01:41:01 -0500464Get Redfish Error Entries
465 [Documentation] Return Redfish error ids list.
466 ${error_uris}= redfish_utils.get_member_list /redfish/v1/Systems/system/LogServices/EventLog/Entries
467 ${error_ids}= Create List
468
469 FOR ${error_uri} IN @{error_uris}
470 ${error_id}= Fetch From Right ${error_uri} /
471 Append To List ${error_ids} ${error_id}
472 END
473
474 [Return] ${error_ids}
475
476
George Keishing8cca7fb2019-05-30 04:23:50 -0500477Event Log Should Not Exist
478 [Documentation] Event log entries should not exist.
479
480 ${elogs}= Get Event Logs
481 Should Be Empty ${elogs} msg=System event log entry is not empty.
George Keishingb0a491f2019-05-31 02:49:22 -0500482
483
484Event Log Should Exist
485 [Documentation] Event log entries should exist.
486
487 ${elogs}= Get Event Logs
488 Should Not Be Empty ${elogs} msg=System event log entry is not empty.
George Keishingd35b5312019-05-31 14:29:07 -0500489
490
491Verify Watchdog EventLog Content
492 [Documentation] Verify watchdog event log content.
493
494 # Example:
495 # {
496 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry",
497 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/31",
498 # "@odata.type": "#LogEntry.v1_4_0.LogEntry",
499 # "Created": "2019-05-31T18:41:33+00:00",
500 # "EntryType": "Event",
501 # "Id": "31",
502 # "Message": "org.open_power.Host.Boot.Error.WatchdogTimedOut",
503 # "Name": "System DBus Event Log Entry",
504 # "Severity": "Critical"
505 # }
506
George Keishing2ab62412022-07-28 11:53:23 -0500507 ${elog_list}= Get Event Logs
508
509 Rprint Vars elog_list
510
511 FOR ${entry} IN @{elog_list}
512 ${found_match}= Run Keyword And Return Status Is Watchdog Error Found ${entry}
513 Exit For Loop If '${found_match}' == 'True'
514 END
515
516 Run Keyword If '${found_match}' == 'False' Fail msg=No watchdog error logged.
517
518
519Is Watchdog Error Found
520 [Documentation] Check if the give log entry matches specific watchdog error.
521 [Arguments] ${elog}
522
523 # Description of argument(s):
524 # elog Error log entry dictionary data.
525
526 Should Contain Any
527 ... ${elog["Message"]} org.open_power.Host.Boot.Error.WatchdogTimedOut
528 ... CEC Hardware - Hostboot-Service Processor Interface
George Keishingd35b5312019-05-31 14:29:07 -0500529 ... msg=Watchdog timeout event log was not found.
George Keishing2ab62412022-07-28 11:53:23 -0500530
531 Log To Console Matched Found: ${elog}