George Keishing | 8cca7fb | 2019-05-30 04:23:50 -0500 | [diff] [blame] | 1 | *** Settings *** |
| 2 | Documentation Inventory of hardware resources under systems. |
| 3 | |
| 4 | Resource ../../../lib/bmc_redfish_resource.robot |
| 5 | Resource ../../../lib/bmc_redfish_utils.robot |
| 6 | Resource ../../../lib/logging_utils.robot |
| 7 | Resource ../../../lib/openbmc_ffdc.robot |
George Keishing | 8fe6024 | 2019-06-03 23:06:12 -0500 | [diff] [blame] | 8 | Resource ../../../lib/ipmi_client.robot |
George Keishing | 8cca7fb | 2019-05-30 04:23:50 -0500 | [diff] [blame] | 9 | |
George Keishing | b0a491f | 2019-05-31 02:49:22 -0500 | [diff] [blame] | 10 | Test Setup Test Setup Execution |
George Keishing | 8cca7fb | 2019-05-30 04:23:50 -0500 | [diff] [blame] | 11 | Test Teardown Test Teardown Execution |
George Keishing | b0a491f | 2019-05-31 02:49:22 -0500 | [diff] [blame] | 12 | Suite Teardown Suite Teardown Execution |
George Keishing | 8cca7fb | 2019-05-30 04:23:50 -0500 | [diff] [blame] | 13 | |
| 14 | *** Test Cases *** |
| 15 | |
| 16 | Event Log Check After BMC Reboot |
| 17 | [Documentation] Check event log after BMC rebooted. |
| 18 | [Tags] Event_Log_Check_After_BMC_Reboot |
| 19 | |
| 20 | Redfish Purge Event Log |
| 21 | Event Log Should Not Exist |
| 22 | |
| 23 | Redfish OBMC Reboot (off) |
| 24 | |
| 25 | Redfish.Login |
| 26 | Wait Until Keyword Succeeds 1 mins 15 secs Redfish.Get ${EVENT_LOG_URI}Entries |
| 27 | |
| 28 | Event Log Should Not Exist |
| 29 | |
| 30 | |
George Keishing | b0a491f | 2019-05-31 02:49:22 -0500 | [diff] [blame] | 31 | Event Log Check After Host Poweron |
| 32 | [Documentation] Check event log after host has booted. |
| 33 | [Tags] Event_Log_Check_After_Host_Poweron |
| 34 | |
| 35 | Redfish Purge Event Log |
| 36 | Event Log Should Not Exist |
| 37 | |
| 38 | Redfish Power On |
| 39 | |
| 40 | Redfish.Login |
| 41 | Event Log Should Not Exist |
| 42 | |
| 43 | |
| 44 | Create Test Event Log And Verify |
| 45 | [Documentation] Create event logs and verify via redfish. |
| 46 | [Tags] Create_Test_Event_Log_And_Verify |
| 47 | |
| 48 | Create Test Error Log |
| 49 | Event Log Should Exist |
| 50 | |
| 51 | |
| 52 | Test Event Log Persistency On Restart |
| 53 | [Documentation] Restart logging service and verify event logs. |
| 54 | [Tags] Test_Event_Log_Persistency_On_Restart |
| 55 | |
| 56 | Create Test Error Log |
| 57 | Event Log Should Exist |
| 58 | |
| 59 | BMC Execute Command |
| 60 | ... systemctl restart xyz.openbmc_project.Logging.service |
| 61 | Sleep 10s reason=Wait for logging service to restart properly. |
| 62 | |
| 63 | Event Log Should Exist |
| 64 | |
| 65 | |
| 66 | Test Event Entry Numbering Reset On Restart |
| 67 | [Documentation] Restart logging service and verify event logs entry starts |
| 68 | ... from entry "Id" 1. |
| 69 | [Tags] Test_Event_Entry_Numbering_Reset_On_Restart |
| 70 | |
| 71 | #{ |
| 72 | # "@odata.context": "/redfish/v1/$metadata#LogEntryCollection.LogEntryCollection", |
| 73 | # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries", |
| 74 | # "@odata.type": "#LogEntryCollection.LogEntryCollection", |
| 75 | # "Description": "Collection of System Event Log Entries", |
| 76 | # "Members": [ |
| 77 | # { |
| 78 | # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry", |
| 79 | # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/1", |
| 80 | # "@odata.type": "#LogEntry.v1_4_0.LogEntry", |
| 81 | # "Created": "2019-05-29T13:19:27+00:00", |
| 82 | # "EntryType": "Event", |
| 83 | # "Id": "1", <----- Event log ID |
| 84 | # "Message": "org.open_power.Host.Error.Event", |
| 85 | # "Name": "System DBus Event Log Entry", |
| 86 | # "Severity": "Critical" |
| 87 | # } |
| 88 | # ], |
| 89 | # "Members@odata.count": 1, |
| 90 | # "Name": "System Event Log Entries" |
| 91 | #} |
| 92 | |
| 93 | Create Test Error Log |
| 94 | Create Test Error Log |
| 95 | Event Log Should Exist |
| 96 | |
| 97 | Redfish Purge Event Log |
| 98 | Event Log Should Not Exist |
| 99 | |
| 100 | BMC Execute Command |
| 101 | ... systemctl restart xyz.openbmc_project.Logging.service |
| 102 | Sleep 10s reason=Wait for logging service to restart properly. |
| 103 | |
| 104 | Create Test Error Log |
| 105 | ${elogs}= Get Event Logs |
| 106 | Should Be Equal ${elogs[0]["Id"]} 1 msg=Event log entry is not 1. |
| 107 | |
| 108 | |
| 109 | Test Event Log Persistency On Reboot |
| 110 | [Documentation] Reboot BMC and verify event log. |
| 111 | [Tags] Test_Event_Log_Persistency_On_Reboot |
| 112 | |
| 113 | Redfish Purge Event Log |
| 114 | Create Test Error Log |
| 115 | Event Log Should Exist |
| 116 | |
| 117 | Redfish OBMC Reboot (off) |
| 118 | |
| 119 | Redfish.Login |
| 120 | Wait Until Keyword Succeeds 1 mins 15 secs Redfish.Get ${EVENT_LOG_URI}Entries |
| 121 | |
| 122 | Event Log Should Exist |
| 123 | |
| 124 | |
George Keishing | 148d08c | 2019-05-31 12:39:04 -0500 | [diff] [blame] | 125 | Create Test Event Log And Verify Time Stamp |
| 126 | [Documentation] Create event logs and verify time stamp. |
| 127 | [Tags] Create_Test_Event_Log_And_Verify_Time_Stamp |
| 128 | |
| 129 | #{ |
| 130 | # "@odata.context": "/redfish/v1/$metadata#LogEntryCollection.LogEntryCollection", |
| 131 | # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries", |
| 132 | # "@odata.type": "#LogEntryCollection.LogEntryCollection", |
| 133 | # "Description": "Collection of System Event Log Entries", |
| 134 | # "Members": [ |
| 135 | # { |
| 136 | # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry", |
| 137 | # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/1", |
| 138 | # "@odata.type": "#LogEntry.v1_4_0.LogEntry", |
| 139 | # "Created": "2019-05-29T13:19:27+00:00", <--- Time stamp |
| 140 | # "EntryType": "Event", |
| 141 | # "Id": "1", |
| 142 | # "Message": "org.open_power.Host.Error.Event", |
| 143 | # "Name": "System DBus Event Log Entry", |
| 144 | # "Severity": "Critical" |
| 145 | # } |
| 146 | # ], |
| 147 | # "Members@odata.count": 1, |
| 148 | # "Name": "System Event Log Entries" |
| 149 | #} |
| 150 | |
| 151 | Redfish Purge Event Log |
| 152 | |
| 153 | Create Test Error Log |
| 154 | Sleep 2s |
| 155 | Create Test Error Log |
| 156 | |
| 157 | ${elog_entry}= Get Event Logs |
| 158 | |
| 159 | # The event log generated is associated with the epoc time and unique |
| 160 | # for every error and in increasing time stamp. |
| 161 | ${time_stamp1}= Convert Date ${elog_entry[0]["Created"]} epoch |
| 162 | ${time_stamp2}= Convert Date ${elog_entry[1]["Created"]} epoch |
| 163 | |
| 164 | Should Be True ${time_stamp2} > ${time_stamp1} |
| 165 | |
| 166 | |
George Keishing | 0b76e42 | 2019-06-03 01:58:52 -0500 | [diff] [blame] | 167 | Delete Non Existing SEL Event Entry |
| 168 | [Documentation] Delete non existing SEL event entry. |
| 169 | [Tags] Delete_Non_Existing_SEL_Event_Entry |
| 170 | |
| 171 | ${sel_delete}= Run Keyword And Expect Error * |
| 172 | ... Run IPMI Standard Command sel delete 100 |
| 173 | Should Contain ${sel_delete} Unable to delete entry |
| 174 | ... case_insensitive=True |
| 175 | |
| 176 | |
| 177 | Delete Invalid SEL Event Entry |
| 178 | [Documentation] Delete invalid SEL event entry. |
| 179 | [Tags] Delete_Invalid_SEL_Event_Entry |
| 180 | |
| 181 | ${sel_delete}= Run Keyword And Expect Error * |
| 182 | ... Run IPMI Standard Command sel delete abc |
| 183 | Should Contain ${sel_delete} Given SEL ID 'abc' is invalid |
| 184 | ... case_insensitive=True |
| 185 | |
| 186 | |
| 187 | Verify IPMI SEL Event Entries |
| 188 | [Documentation] Verify IPMI SEL's entries info. |
| 189 | [Tags] Verify_IPMI_SEL_Event_Entries |
| 190 | |
| 191 | # Generate error logs of random count. |
| 192 | ${count}= Evaluate random.randint(1, 5) modules=random |
| 193 | Repeat Keyword ${count} Create Test Error Log |
| 194 | |
| 195 | ${sel_entries_count}= Get IPMI SEL Setting Entries |
| 196 | Should Be Equal As Strings ${sel_entries_count} ${count} |
| 197 | |
| 198 | |
| 199 | Verify IPMI SEL Event Last Add Time |
| 200 | [Documentation] Verify IPMI SEL's last added timestamp. |
| 201 | [Tags] Verify_IPMI_SEL_Event_Last_Add_Time |
| 202 | |
| 203 | Create Test Error Log |
| 204 | ${sel_time}= Run IPMI Standard Command sel time get |
| 205 | ${sel_time}= Convert Date ${sel_time} |
| 206 | ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True |
| 207 | |
| 208 | ${sel_last_add_time}= Get IPMI SEL Setting Last Add Time |
| 209 | ${sel_last_add_time}= Convert Date ${sel_last_add_time} |
| 210 | ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True |
| 211 | |
| 212 | ${time_diff}= |
| 213 | ... Subtract Date From Date ${sel_last_add_time} ${sel_time} |
| 214 | |
| 215 | # Verify if the delay in current time check and last add SEL time |
| 216 | # is less or equals to 2 seconds. |
| 217 | Should Be True ${time_diff} <= 2 |
| 218 | |
| 219 | |
George Keishing | d35b531 | 2019-05-31 14:29:07 -0500 | [diff] [blame] | 220 | Create Test Event Log And Delete |
| 221 | [Documentation] Create an event log and delete it. |
| 222 | [Tags] Create_Test_Event_Log_And_Delete |
| 223 | |
| 224 | Create Test Error Log |
| 225 | Redfish Purge Event Log |
| 226 | Event Log Should Not Exist |
| 227 | |
| 228 | |
| 229 | Create Multiple Test Event Logs And Delete All |
| 230 | [Documentation] Create multiple event logs and delete all. |
| 231 | [Tags] Create_Multiple_Test_Event_Logs_And_Delete_All |
| 232 | |
| 233 | Create Test Error Log |
| 234 | Create Test Error Log |
| 235 | Create Test Error Log |
| 236 | Redfish Purge Event Log |
| 237 | Event Log Should Not Exist |
| 238 | |
| 239 | |
| 240 | Verify Watchdog Timedout Event |
| 241 | [Documentation] Trigger watchdog timed out and verify event log generated. |
| 242 | [Tags] Verify_Watchdog_Timedout_Event |
| 243 | |
| 244 | Redfish Power On |
| 245 | |
| 246 | # Clear errors if there are any. |
| 247 | Redfish.Login |
| 248 | Redfish Purge Event Log |
| 249 | |
| 250 | Trigger Host Watchdog Error |
| 251 | |
| 252 | # Logging takes time to generate the timeout error. |
| 253 | Wait Until Keyword Succeeds 2 min 30 sec |
| 254 | ... Verify Watchdog EventLog Content |
| 255 | |
| 256 | Redfish Power Off |
| 257 | |
| 258 | |
| 259 | Verify Event Logs Capping |
| 260 | [Documentation] Verify event logs capping. |
| 261 | [Tags] Verify_Event_Logs_Capping |
| 262 | |
| 263 | Redfish Purge Event Log |
| 264 | |
| 265 | ${cmd}= Catenate for i in {1..201}; do /tmp/tarball/bin/logging-test -c |
| 266 | ... AutoTestSimple; done |
| 267 | BMC Execute Command ${cmd} |
| 268 | |
| 269 | ${elogs}= Get Event Logs |
| 270 | ${count}= Get Length ${elogs} |
| 271 | Run Keyword If ${count} > 200 |
| 272 | ... Fail Error logs created exceeded max capacity 200. |
| 273 | |
| 274 | |
George Keishing | 8cca7fb | 2019-05-30 04:23:50 -0500 | [diff] [blame] | 275 | *** Keywords *** |
| 276 | |
| 277 | Suite Teardown Execution |
| 278 | [Documentation] Do the post suite teardown. |
| 279 | |
| 280 | Redfish.Logout |
| 281 | |
| 282 | |
George Keishing | b0a491f | 2019-05-31 02:49:22 -0500 | [diff] [blame] | 283 | Test Setup Execution |
| 284 | [Documentation] Do test case setup tasks. |
George Keishing | 8cca7fb | 2019-05-30 04:23:50 -0500 | [diff] [blame] | 285 | |
| 286 | Redfish.Login |
| 287 | |
George Keishing | 0b76e42 | 2019-06-03 01:58:52 -0500 | [diff] [blame] | 288 | Redfish Purge Event Log |
| 289 | |
George Keishing | 8cca7fb | 2019-05-30 04:23:50 -0500 | [diff] [blame] | 290 | ${status}= Run Keyword And Return Status Logging Test Binary Exist |
| 291 | Run Keyword If ${status} == ${False} Install Tarball |
| 292 | |
| 293 | |
| 294 | Test Teardown Execution |
| 295 | [Documentation] Do the post test teardown. |
| 296 | |
| 297 | FFDC On Test Case Fail |
George Keishing | e5c5ebd | 2019-06-04 13:52:12 -0500 | [diff] [blame] | 298 | Redfish.Login |
George Keishing | b0a491f | 2019-05-31 02:49:22 -0500 | [diff] [blame] | 299 | Redfish Purge Event Log |
George Keishing | 8cca7fb | 2019-05-30 04:23:50 -0500 | [diff] [blame] | 300 | |
| 301 | |
| 302 | Event Log Should Not Exist |
| 303 | [Documentation] Event log entries should not exist. |
| 304 | |
| 305 | ${elogs}= Get Event Logs |
| 306 | Should Be Empty ${elogs} msg=System event log entry is not empty. |
George Keishing | b0a491f | 2019-05-31 02:49:22 -0500 | [diff] [blame] | 307 | |
| 308 | |
| 309 | Event Log Should Exist |
| 310 | [Documentation] Event log entries should exist. |
| 311 | |
| 312 | ${elogs}= Get Event Logs |
| 313 | Should Not Be Empty ${elogs} msg=System event log entry is not empty. |
George Keishing | d35b531 | 2019-05-31 14:29:07 -0500 | [diff] [blame] | 314 | |
| 315 | |
| 316 | Verify Watchdog EventLog Content |
| 317 | [Documentation] Verify watchdog event log content. |
| 318 | |
| 319 | # Example: |
| 320 | # { |
| 321 | # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry", |
| 322 | # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/31", |
| 323 | # "@odata.type": "#LogEntry.v1_4_0.LogEntry", |
| 324 | # "Created": "2019-05-31T18:41:33+00:00", |
| 325 | # "EntryType": "Event", |
| 326 | # "Id": "31", |
| 327 | # "Message": "org.open_power.Host.Boot.Error.WatchdogTimedOut", |
| 328 | # "Name": "System DBus Event Log Entry", |
| 329 | # "Severity": "Critical" |
| 330 | # } |
| 331 | |
| 332 | ${elog}= Get Event Logs |
| 333 | Should Be Equal As Strings |
| 334 | ... ${elog[0]["Message"]} org.open_power.Host.Boot.Error.WatchdogTimedOut |
| 335 | ... msg=Watchdog timeout event log was not found. |
| 336 | Should Be Equal As Strings |
| 337 | ... ${elog[0]["Severity"]} Critical |
| 338 | ... msg=Watchdog timeout severity unexpected value. |