blob: 8aa676b8f3d329617df2dd62528ba07549cf5248 [file] [log] [blame]
George Keishing8cca7fb2019-05-30 04:23:50 -05001*** Settings ***
2Documentation Inventory of hardware resources under systems.
3
4Resource ../../../lib/bmc_redfish_resource.robot
5Resource ../../../lib/bmc_redfish_utils.robot
6Resource ../../../lib/logging_utils.robot
7Resource ../../../lib/openbmc_ffdc.robot
George Keishing8fe60242019-06-03 23:06:12 -05008Resource ../../../lib/ipmi_client.robot
George Keishing759c6f92019-06-03 12:14:42 -05009Library ../../../lib/logging_utils.py
George Keishing8cca7fb2019-05-30 04:23:50 -050010
George Keishing2f369472019-07-23 12:03:10 -050011Suite Setup Suite Setup Execution
12Suite Teardown Suite Teardown Execution
George Keishingb0a491f2019-05-31 02:49:22 -050013Test Setup Test Setup Execution
George Keishing8cca7fb2019-05-30 04:23:50 -050014Test Teardown Test Teardown Execution
15
George Keishingf699f552023-09-14 19:05:01 +053016Force Tags Event_Logging
17
George Keishing759c6f92019-06-03 12:14:42 -050018** Variables ***
19
20${max_num_event_logs} ${200}
21
George Keishing8cca7fb2019-05-30 04:23:50 -050022*** Test Cases ***
23
24Event Log Check After BMC Reboot
25 [Documentation] Check event log after BMC rebooted.
26 [Tags] Event_Log_Check_After_BMC_Reboot
27
28 Redfish Purge Event Log
29 Event Log Should Not Exist
30
31 Redfish OBMC Reboot (off)
32
33 Redfish.Login
34 Wait Until Keyword Succeeds 1 mins 15 secs Redfish.Get ${EVENT_LOG_URI}Entries
35
36 Event Log Should Not Exist
37
38
George Keishingb0a491f2019-05-31 02:49:22 -050039Event Log Check After Host Poweron
40 [Documentation] Check event log after host has booted.
41 [Tags] Event_Log_Check_After_Host_Poweron
42
43 Redfish Purge Event Log
44 Event Log Should Not Exist
45
46 Redfish Power On
47
48 Redfish.Login
49 Event Log Should Not Exist
50
51
52Create Test Event Log And Verify
53 [Documentation] Create event logs and verify via redfish.
54 [Tags] Create_Test_Event_Log_And_Verify
55
56 Create Test Error Log
57 Event Log Should Exist
58
59
Rahul Maheshwari20c08e42021-05-10 01:41:01 -050060Delete Redfish Event Log And Verify
61 [Documentation] Delete Redfish event log and verify via Redfish.
62 [Tags] Delete_Redfish_Event_Log_And_Verify
63
64 Redfish.Login
65 Redfish Purge Event Log
66 Create Test PEL Log
67 ${elog_entry}= Get Event Logs
68
Yi Huc32434a2024-01-11 17:33:10 -080069 Redfish.Delete /redfish/v1/Systems/${SYSTEM_ID}/LogServices/EventLog/Entries/${elog_entry[0]["Id"]}
Rahul Maheshwari20c08e42021-05-10 01:41:01 -050070
71 ${error_entries}= Get Redfish Error Entries
72 Should Be Empty ${error_entries}
73
74
George Keishingb0a491f2019-05-31 02:49:22 -050075Test Event Log Persistency On Restart
76 [Documentation] Restart logging service and verify event logs.
77 [Tags] Test_Event_Log_Persistency_On_Restart
78
79 Create Test Error Log
80 Event Log Should Exist
81
82 BMC Execute Command
83 ... systemctl restart xyz.openbmc_project.Logging.service
84 Sleep 10s reason=Wait for logging service to restart properly.
85
86 Event Log Should Exist
87
88
89Test Event Entry Numbering Reset On Restart
90 [Documentation] Restart logging service and verify event logs entry starts
91 ... from entry "Id" 1.
92 [Tags] Test_Event_Entry_Numbering_Reset_On_Restart
Rahul Maheshwari20560f92021-12-21 05:16:11 -060093 [Setup] Redfish Power Off stack_mode=skip
George Keishingb0a491f2019-05-31 02:49:22 -050094
95 #{
96 # "@odata.context": "/redfish/v1/$metadata#LogEntryCollection.LogEntryCollection",
97 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries",
98 # "@odata.type": "#LogEntryCollection.LogEntryCollection",
99 # "Description": "Collection of System Event Log Entries",
100 # "Members": [
101 # {
102 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry",
103 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/1",
104 # "@odata.type": "#LogEntry.v1_4_0.LogEntry",
105 # "Created": "2019-05-29T13:19:27+00:00",
106 # "EntryType": "Event",
107 # "Id": "1", <----- Event log ID
108 # "Message": "org.open_power.Host.Error.Event",
109 # "Name": "System DBus Event Log Entry",
110 # "Severity": "Critical"
111 # }
112 # ],
113 # "Members@odata.count": 1,
114 # "Name": "System Event Log Entries"
115 #}
116
Rahul Maheshwari20560f92021-12-21 05:16:11 -0600117 Create Test PEL Log
118 Create Test PEL Log
George Keishingb0a491f2019-05-31 02:49:22 -0500119 Event Log Should Exist
120
121 Redfish Purge Event Log
122 Event Log Should Not Exist
123
124 BMC Execute Command
125 ... systemctl restart xyz.openbmc_project.Logging.service
126 Sleep 10s reason=Wait for logging service to restart properly.
127
Rahul Maheshwari20560f92021-12-21 05:16:11 -0600128 Create Test PEL Log
George Keishingb0a491f2019-05-31 02:49:22 -0500129 ${elogs}= Get Event Logs
manashsarmaadf10ee2024-05-14 07:14:17 -0500130
131 # After issuing Redfish purge event log, there will be one informational error log
132 # in BMC with ID 1. Due to this the newly generated error log would have ID as 2.
133 Should Be Equal ${elogs[0]["Id"]} 2 msg=Event log entry is not 2
George Keishingb0a491f2019-05-31 02:49:22 -0500134
135
136Test Event Log Persistency On Reboot
137 [Documentation] Reboot BMC and verify event log.
138 [Tags] Test_Event_Log_Persistency_On_Reboot
139
140 Redfish Purge Event Log
141 Create Test Error Log
142 Event Log Should Exist
143
144 Redfish OBMC Reboot (off)
145
146 Redfish.Login
147 Wait Until Keyword Succeeds 1 mins 15 secs Redfish.Get ${EVENT_LOG_URI}Entries
148
149 Event Log Should Exist
150
151
George Keishing148d08c2019-05-31 12:39:04 -0500152Create Test Event Log And Verify Time Stamp
153 [Documentation] Create event logs and verify time stamp.
154 [Tags] Create_Test_Event_Log_And_Verify_Time_Stamp
155
156 #{
157 # "@odata.context": "/redfish/v1/$metadata#LogEntryCollection.LogEntryCollection",
158 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries",
159 # "@odata.type": "#LogEntryCollection.LogEntryCollection",
160 # "Description": "Collection of System Event Log Entries",
161 # "Members": [
162 # {
163 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry",
164 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/1",
165 # "@odata.type": "#LogEntry.v1_4_0.LogEntry",
dnirmalaf4372442023-05-09 07:58:32 -0500166 # "Created": "2023-05-10T10:26:02.186+00:00", <--- Time stamp
George Keishing148d08c2019-05-31 12:39:04 -0500167 # "EntryType": "Event",
168 # "Id": "1",
169 # "Message": "org.open_power.Host.Error.Event",
170 # "Name": "System DBus Event Log Entry",
171 # "Severity": "Critical"
172 # }
173 # ],
174 # "Members@odata.count": 1,
175 # "Name": "System Event Log Entries"
176 #}
177
178 Redfish Purge Event Log
179
180 Create Test Error Log
181 Sleep 2s
182 Create Test Error Log
183
184 ${elog_entry}= Get Event Logs
185
186 # The event log generated is associated with the epoc time and unique
187 # for every error and in increasing time stamp.
dnirmalaf4372442023-05-09 07:58:32 -0500188 ${time_stamp1}= Convert Date ${elog_entry[0]["Created"].split('.')[0]} epoch
189 ${time_stamp2}= Convert Date ${elog_entry[1]["Created"].split('.')[0]} epoch
George Keishing148d08c2019-05-31 12:39:04 -0500190
191 Should Be True ${time_stamp2} > ${time_stamp1}
192
193
Rahul Maheshwari24034a92021-08-18 02:23:42 -0500194# TODO: openbmc/openbmc-test-automation#1789
Rahul Maheshwari6462f262021-05-10 00:49:46 -0500195Verify Setting Error Log As Resolved
196 [Documentation] Verify modified field of error log is updated when error log is marked resolved.
197 [Tags] Verify_Setting_Error_Log_As_Resolved
198
199 Create Test PEL Log
200 ${elog_entry}= Get Event Logs
201
202 # Wait for 5 seconds after creating error log.
203 Sleep 5s
204
205 # Mark error log as resolved by setting it to true.
206 Redfish.Patch ${EVENT_LOG_URI}Entries/${elog_entry[0]["Id"]} body={'Resolved':True}
207
208 ${elog_entry}= Get Event Logs
209
210 # Example error log with resolve field set to true:
211 # {
212 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/2045",
213 # "@odata.type": "#LogEntry.v1_8_0.LogEntry",
214 # "AdditionalDataURI": "/redfish/v1/Systems/system/LogServices/EventLog/attachment/2045",
dnirmalaf4372442023-05-09 07:58:32 -0500215 # "Created": "2023-05-10T10:26:02.186+00:00",
Rahul Maheshwari6462f262021-05-10 00:49:46 -0500216 # "EntryType": "Event",
217 # "Id": "2045",
218 # "Message": "xyz.openbmc_project.Host.Error.Event",
dnirmalaf4372442023-05-09 07:58:32 -0500219 # "Modified": "2023-05-10T10:26:02.186+00:00",
Rahul Maheshwari6462f262021-05-10 00:49:46 -0500220 # "Name": "System Event Log Entry",
221 # "Resolved": true,
222 # "Severity": "OK"
223 # }
224
225 Should Be Equal As Strings ${elog_entry[0]["Resolved"]} True
226
227 # Difference created and modified time of error log should be around 5 seconds.
dnirmalaf4372442023-05-09 07:58:32 -0500228 ${creation_time}= Convert Date ${elog_entry[0]["Created"].split('.')[0]} epoch
229 ${modification_time}= Convert Date ${elog_entry[0]["Modified"].split('.')[0]} epoch
Rahul Maheshwari6462f262021-05-10 00:49:46 -0500230
231 ${diff}= Subtract Date From Date ${modification_time} ${creation_time}
232 ${diff}= Convert To Number ${diff}
233 Should Be True 4 < ${diff} < 8
234
235
George Keishingddbc88c2019-06-06 07:29:41 -0500236Verify IPMI SEL Delete
237 [Documentation] Verify IPMI SEL delete operation.
238 [Tags] Verify_IPMI_SEL_Delete
239
240 Redfish Purge Event Log
241 Create Test Error Log
242
243 ${sel_list}= Run IPMI Standard Command sel list
244 Should Not Be Equal As Strings ${sel_list} SEL has no entries
245
246 # Example of SEL List:
247 # 4 | 04/21/2017 | 10:51:16 | System Event #0x01 | Undetermined system hardware failure | Asserted
248
249 ${sel_entry}= Fetch from Left ${sel_list} |
250 ${sel_entry}= Evaluate $sel_entry.replace(' ','')
251 ${sel_entry}= Convert To Integer 0x${sel_entry}
252
253 ${sel_delete}= Run IPMI Standard Command sel delete ${sel_entry}
254 Should Be Equal As Strings ${sel_delete} Deleted entry ${sel_entry}
255 ... case_insensitive=True
256
257 ${sel_list}= Run IPMI Standard Command sel list
258 Should Be Equal As Strings ${sel_list} SEL has no entries
259 ... case_insensitive=True
260
261
George Keishingd35b5312019-05-31 14:29:07 -0500262Create Test Event Log And Delete
263 [Documentation] Create an event log and delete it.
264 [Tags] Create_Test_Event_Log_And_Delete
265
266 Create Test Error Log
267 Redfish Purge Event Log
268 Event Log Should Not Exist
269
270
271Create Multiple Test Event Logs And Delete All
272 [Documentation] Create multiple event logs and delete all.
273 [Tags] Create_Multiple_Test_Event_Logs_And_Delete_All
274
275 Create Test Error Log
276 Create Test Error Log
277 Create Test Error Log
278 Redfish Purge Event Log
279 Event Log Should Not Exist
280
281
George Keishingddbc88c2019-06-06 07:29:41 -0500282Create Two Test Event Logs And Delete One
283 [Documentation] Create two event logs and delete the first entry.
George Keishing89f499d2021-04-19 05:25:40 -0500284 [Tags] Create_Two_Test_Event_Logs_And_Delete_One
Rahul Maheshwaric2d33652021-12-21 00:19:58 -0600285 [Setup] Redfish Power Off stack_mode=skip
George Keishingddbc88c2019-06-06 07:29:41 -0500286
287 Redfish Purge Event Log
Rahul Maheshwaric2d33652021-12-21 00:19:58 -0600288 Create Test PEL Log
289 Create Test PEL Log
290 ${error_entries_before}= Get Redfish Error Entries
Yi Huc32434a2024-01-11 17:33:10 -0800291 Redfish.Delete /redfish/v1/Systems/${SYSTEM_ID}/LogServices/EventLog/Entries/${error_entries_before[0]}
Rahul Maheshwaric2d33652021-12-21 00:19:58 -0600292
293 ${error_entries_after}= Get Redfish Error Entries
294 Should Not Contain ${error_entries_after} ${error_entries_before[0]}
295 Should Contain ${error_entries_after} ${error_entries_before[1]}
George Keishingddbc88c2019-06-06 07:29:41 -0500296
297
George Keishingd35b5312019-05-31 14:29:07 -0500298Verify Watchdog Timedout Event
299 [Documentation] Trigger watchdog timed out and verify event log generated.
300 [Tags] Verify_Watchdog_Timedout_Event
George Keishing2ab62412022-07-28 11:53:23 -0500301 [Teardown] Run Keywords Test Teardown Execution AND Redfish Power Off stack_mode=skip
George Keishingd35b5312019-05-31 14:29:07 -0500302
George Keishing2ab62412022-07-28 11:53:23 -0500303 Redfish Power Off stack_mode=skip
George Keishingd35b5312019-05-31 14:29:07 -0500304
305 # Clear errors if there are any.
306 Redfish.Login
307 Redfish Purge Event Log
308
George Keishing2ab62412022-07-28 11:53:23 -0500309 # Reference: [Old legacy REST code] Trigger Host Watchdog Error
310 # Currently, no known redfish interface to set to trigger watchdog timer.
311
312 Redfish Initiate Auto Reboot 1000
George Keishingd35b5312019-05-31 14:29:07 -0500313
314 # Logging takes time to generate the timeout error.
George Keishing2ab62412022-07-28 11:53:23 -0500315 Wait Until Keyword Succeeds 3 min 20 sec Verify Watchdog EventLog Content
George Keishingd35b5312019-05-31 14:29:07 -0500316
George Keishingd35b5312019-05-31 14:29:07 -0500317
318Verify Event Logs Capping
319 [Documentation] Verify event logs capping.
320 [Tags] Verify_Event_Logs_Capping
321
322 Redfish Purge Event Log
323
George Keishing033ee182022-07-28 05:24:55 -0500324 ${cmd}= Catenate uptime; for i in {1..201}; do /tmp/tarball/bin/logging-test -c
325 ... AutoTestSimple;sleep 1;done; uptime
George Keishingd35b5312019-05-31 14:29:07 -0500326 BMC Execute Command ${cmd}
327
328 ${elogs}= Get Event Logs
329 ${count}= Get Length ${elogs}
330 Run Keyword If ${count} > 200
331 ... Fail Error logs created exceeded max capacity 200.
332
333
George Keishing759c6f92019-06-03 12:14:42 -0500334Test Event Log Wrapping
335 [Documentation] Verify event log entries wraps when 200 max cap is reached.
336 [Tags] Test_Event_Log_Wrapping
337
338 # Restarting logging service in order to clear logs and get the next log
339 # ID set to 1.
340 BMC Execute Command
341 ... systemctl restart xyz.openbmc_project.Logging.service
342 Sleep 10s reason=Wait for logging service to restart properly.
343
344 # Create ${max_num_event_logs} event logs.
George Keishing033ee182022-07-28 05:24:55 -0500345 ${cmd}= Catenate uptime; for i in {1..${max_num_event_logs}}; do /tmp/tarball/bin/logging-test -c
346 ... AutoTestSimple;sleep 1;done; uptime
George Keishing759c6f92019-06-03 12:14:42 -0500347 BMC Execute Command ${cmd}
348
349 # Verify that event logs with IDs 1 and ${max_num_event_logs} exist.
350 ${event_log}= Get Event Logs
351
352 ${log_entries}= Filter Struct ${event_log} [('Id', '1')]
Michael Walsh39c00512019-07-17 10:54:06 -0500353 Rprint Vars log_entries
George Keishing759c6f92019-06-03 12:14:42 -0500354 Should Be Equal As Strings ${log_entries[0]["Id"]} 1
355
356 ${log_entries}= Filter Struct ${event_log} [('Id', '${max_num_event_logs}')]
Michael Walsh39c00512019-07-17 10:54:06 -0500357 Rprint Vars log_entries
George Keishing759c6f92019-06-03 12:14:42 -0500358 Should Be Equal As Strings ${log_entries[0]["Id"]} ${max_num_event_logs}
359
360 # Create event log and verify the entry ID, ${max_num_event_logs + 1}.
361 ${next_event_log_id}= Set Variable ${max_num_event_logs + 1}
362
363 Create Test Error Log
364
365 ${event_log}= Get Event Logs
366
367 ${log_entries}= Filter Struct ${event_log} [('Id', '${next_event_log_id}')]
Michael Walsh39c00512019-07-17 10:54:06 -0500368 Rprint Vars log_entries
George Keishing759c6f92019-06-03 12:14:42 -0500369 Should Be Equal As Strings ${log_entries[0]["Id"]} ${next_event_log_id}
370
371 # Event log 1 should be wrapped.
372 ${log_entries}= Filter Struct ${event_log} [('Id', '1')]
Michael Walsh39c00512019-07-17 10:54:06 -0500373 Rprint Vars log_entries
George Keishing759c6f92019-06-03 12:14:42 -0500374
375 ${length_log_entries} Get Length ${log_entries}
376 Should Be Equal As Integers ${length_log_entries} 0
377 ... msg=The event log should have wrapped such that entry ID 1 is now purged.
378
379
dnirmala2e092d02023-04-18 01:17:32 -0500380Verify Default Value Of Resolved Field Is False For An Error Log Via Redfish
381 [Documentation] Verify the Resolve field status is false for an error log from Redfish.
382 [Tags] Verify_Default_Value_Of_Resolved_Field_Is_False_For_An_Error_Log_Via_Redfish
383
384 Redfish Purge Event Log
385 Create Test Error Log
386
387 # Check resolve field value of created error log.
388 ${elog_entry}= Get Event Logs
389 Should Be Equal ${elog_entry[0]["Resolved"]} ${False}
390
391
George Keishing8cca7fb2019-05-30 04:23:50 -0500392*** Keywords ***
393
George Keishing2f369472019-07-23 12:03:10 -0500394Suite Setup Execution
395 [Documentation] Do test case setup tasks.
396
397 Redfish.Login
398
399 Redfish Purge Event Log
400
401 ${status}= Run Keyword And Return Status Logging Test Binary Exist
402 Run Keyword If ${status} == ${False} Install Tarball
403
404
George Keishing8cca7fb2019-05-30 04:23:50 -0500405Suite Teardown Execution
406 [Documentation] Do the post suite teardown.
407
408 Redfish.Logout
409
410
George Keishingb0a491f2019-05-31 02:49:22 -0500411Test Setup Execution
412 [Documentation] Do test case setup tasks.
George Keishing8cca7fb2019-05-30 04:23:50 -0500413
George Keishing0b76e422019-06-03 01:58:52 -0500414 Redfish Purge Event Log
415
George Keishing8cca7fb2019-05-30 04:23:50 -0500416 ${status}= Run Keyword And Return Status Logging Test Binary Exist
417 Run Keyword If ${status} == ${False} Install Tarball
418
419
420Test Teardown Execution
421 [Documentation] Do the post test teardown.
422
423 FFDC On Test Case Fail
George Keishinge5c5ebd2019-06-04 13:52:12 -0500424 Redfish.Login
George Keishingb0a491f2019-05-31 02:49:22 -0500425 Redfish Purge Event Log
George Keishing8cca7fb2019-05-30 04:23:50 -0500426
427
Rahul Maheshwari20c08e42021-05-10 01:41:01 -0500428Get Redfish Error Entries
429 [Documentation] Return Redfish error ids list.
Yi Huc32434a2024-01-11 17:33:10 -0800430 ${error_uris}= redfish_utils.get_member_list /redfish/v1/Systems/${SYSTEM_ID}/LogServices/EventLog/Entries
Rahul Maheshwari20c08e42021-05-10 01:41:01 -0500431 ${error_ids}= Create List
432
433 FOR ${error_uri} IN @{error_uris}
434 ${error_id}= Fetch From Right ${error_uri} /
435 Append To List ${error_ids} ${error_id}
436 END
437
George Keishing409df052024-01-17 22:36:14 +0530438 RETURN ${error_ids}
Rahul Maheshwari20c08e42021-05-10 01:41:01 -0500439
440
George Keishing8cca7fb2019-05-30 04:23:50 -0500441Event Log Should Not Exist
442 [Documentation] Event log entries should not exist.
443
444 ${elogs}= Get Event Logs
445 Should Be Empty ${elogs} msg=System event log entry is not empty.
George Keishingb0a491f2019-05-31 02:49:22 -0500446
447
448Event Log Should Exist
449 [Documentation] Event log entries should exist.
450
451 ${elogs}= Get Event Logs
452 Should Not Be Empty ${elogs} msg=System event log entry is not empty.
George Keishingd35b5312019-05-31 14:29:07 -0500453
454
455Verify Watchdog EventLog Content
456 [Documentation] Verify watchdog event log content.
457
458 # Example:
459 # {
460 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry",
461 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/31",
462 # "@odata.type": "#LogEntry.v1_4_0.LogEntry",
463 # "Created": "2019-05-31T18:41:33+00:00",
464 # "EntryType": "Event",
465 # "Id": "31",
466 # "Message": "org.open_power.Host.Boot.Error.WatchdogTimedOut",
467 # "Name": "System DBus Event Log Entry",
468 # "Severity": "Critical"
469 # }
470
George Keishing2ab62412022-07-28 11:53:23 -0500471 ${elog_list}= Get Event Logs
472
473 Rprint Vars elog_list
474
475 FOR ${entry} IN @{elog_list}
476 ${found_match}= Run Keyword And Return Status Is Watchdog Error Found ${entry}
477 Exit For Loop If '${found_match}' == 'True'
478 END
479
480 Run Keyword If '${found_match}' == 'False' Fail msg=No watchdog error logged.
481
482
483Is Watchdog Error Found
484 [Documentation] Check if the give log entry matches specific watchdog error.
485 [Arguments] ${elog}
486
487 # Description of argument(s):
488 # elog Error log entry dictionary data.
489
490 Should Contain Any
491 ... ${elog["Message"]} org.open_power.Host.Boot.Error.WatchdogTimedOut
492 ... CEC Hardware - Hostboot-Service Processor Interface
George Keishingd35b5312019-05-31 14:29:07 -0500493 ... msg=Watchdog timeout event log was not found.
George Keishing2ab62412022-07-28 11:53:23 -0500494
495 Log To Console Matched Found: ${elog}