Michael Tritz | 935c18e | 2017-03-08 11:45:41 -0600 | [diff] [blame] | 1 | *** Settings *** |
| 2 | Documentation Test Error logging. |
| 3 | |
| 4 | Resource ../lib/connection_client.robot |
| 5 | Resource ../lib/openbmc_ffdc.robot |
| 6 | Resource ../lib/utils.robot |
| 7 | Resource ../lib/state_manager.robot |
Rahul Maheshwari | 92972e7 | 2017-06-22 00:59:59 -0500 | [diff] [blame] | 8 | Resource ../lib/ipmi_client.robot |
Michael Tritz | 935c18e | 2017-03-08 11:45:41 -0600 | [diff] [blame] | 9 | |
| 10 | Suite Setup Run Keywords Verify logging-test AND |
George Keishing | 184a89a | 2017-06-27 07:12:16 -0500 | [diff] [blame] | 11 | ... Delete Error Logs And Verify |
Michael Tritz | 935c18e | 2017-03-08 11:45:41 -0600 | [diff] [blame] | 12 | Test Setup Open Connection And Log In |
George Keishing | 845d9dc | 2017-06-16 14:49:29 -0500 | [diff] [blame] | 13 | Test Teardown Post Test Case Execution |
George Keishing | 184a89a | 2017-06-27 07:12:16 -0500 | [diff] [blame] | 14 | Suite Teardown Delete Error Logs And Verify |
Michael Tritz | 935c18e | 2017-03-08 11:45:41 -0600 | [diff] [blame] | 15 | |
| 16 | *** Test Cases *** |
| 17 | |
| 18 | Create Test Error And Verify |
| 19 | [Documentation] Create error logs and verify via REST. |
| 20 | [Tags] Create_Test_Error_And_Verify |
| 21 | |
| 22 | Create Test Error Log |
| 23 | Verify Test Error Log |
| 24 | |
| 25 | |
| 26 | Test Error Persistency On Restart |
George Keishing | 184a89a | 2017-06-27 07:12:16 -0500 | [diff] [blame] | 27 | [Documentation] Restart logging service and verify error logs. |
Michael Tritz | 935c18e | 2017-03-08 11:45:41 -0600 | [diff] [blame] | 28 | [Tags] Test_Error_Persistency_On_Restart |
| 29 | |
| 30 | Create Test Error Log |
| 31 | Verify Test Error Log |
| 32 | Execute Command On BMC |
| 33 | ... systemctl restart xyz.openbmc_project.Logging.service |
| 34 | Sleep 10s reason=Wait for logging service to restart properly. |
George Keishing | 184a89a | 2017-06-27 07:12:16 -0500 | [diff] [blame] | 35 | Verify Test Error Log |
Michael Tritz | 935c18e | 2017-03-08 11:45:41 -0600 | [diff] [blame] | 36 | |
| 37 | |
| 38 | Test Error Persistency On Reboot |
George Keishing | 184a89a | 2017-06-27 07:12:16 -0500 | [diff] [blame] | 39 | [Documentation] Reboot BMC and verify error logs. |
Michael Tritz | 935c18e | 2017-03-08 11:45:41 -0600 | [diff] [blame] | 40 | [Tags] Test_Error_Persistency_On_Reboot |
| 41 | |
| 42 | Create Test Error Log |
| 43 | Verify Test Error Log |
| 44 | Initiate BMC Reboot |
| 45 | Wait Until Keyword Succeeds 10 min 10 sec |
| 46 | ... Is BMC Ready |
George Keishing | 184a89a | 2017-06-27 07:12:16 -0500 | [diff] [blame] | 47 | Verify Test Error Log |
Michael Tritz | 935c18e | 2017-03-08 11:45:41 -0600 | [diff] [blame] | 48 | |
| 49 | |
George Keishing | 04bc481 | 2017-03-30 03:33:31 -0500 | [diff] [blame] | 50 | Create Test Error And Verify Resolved Field |
| 51 | [Documentation] Create error log and verify "Resolved" |
| 52 | ... field is 0. |
| 53 | [Tags] Create_Test_Error_And_Verify_Resolved_Field |
| 54 | |
| 55 | # Example Error log: |
| 56 | # "/xyz/openbmc_project/logging/entry/1": { |
| 57 | # "AdditionalData": [ |
| 58 | # "STRING=FOO" |
| 59 | # ], |
| 60 | # "Id": 1, |
| 61 | # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple", |
| 62 | # "Resolved": 0, |
| 63 | # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error", |
| 64 | # "Timestamp": 1490817164983, |
| 65 | # "associations": [] |
| 66 | # }, |
| 67 | |
| 68 | # It's work in progress, but it's mnfg need. To mark an error as |
| 69 | # resolved, without deleting the error, mfg will set this bool |
| 70 | # property. |
| 71 | # In this test context we are making sure "Resolved" field is "0" |
| 72 | # by default. |
| 73 | |
George Keishing | 6eb10b8 | 2017-06-06 10:02:09 -0500 | [diff] [blame] | 74 | Delete Error Logs |
George Keishing | 04bc481 | 2017-03-30 03:33:31 -0500 | [diff] [blame] | 75 | Create Test Error Log |
George Keishing | 184a89a | 2017-06-27 07:12:16 -0500 | [diff] [blame] | 76 | ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY} |
| 77 | ${resolved}= Read Attribute ${elog_entry[0]} Resolved |
George Keishing | 04bc481 | 2017-03-30 03:33:31 -0500 | [diff] [blame] | 78 | Should Be True ${resolved} == 0 |
| 79 | |
| 80 | |
| 81 | Create Test Errors And Verify Time Stamp |
| 82 | [Documentation] Create error logs and verify time stamp. |
| 83 | [Tags] Create_Test_Error_And_Verify_Time_Stamp |
| 84 | |
| 85 | # Example Error logs: |
| 86 | # "/xyz/openbmc_project/logging/entry/1": { |
| 87 | # "AdditionalData": [ |
| 88 | # "STRING=FOO" |
| 89 | # ], |
| 90 | # "Id": 1, |
| 91 | # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple", |
| 92 | # "Resolved": 0, |
| 93 | # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error", |
| 94 | # "Timestamp": 1490818990051, <--- Time stamp |
| 95 | # "associations": [] |
| 96 | # }, |
| 97 | # "/xyz/openbmc_project/logging/entry/2": { |
| 98 | # "AdditionalData": [ |
| 99 | # "STRING=FOO" |
| 100 | # ], |
| 101 | # "Id": 2, |
| 102 | # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple", |
| 103 | # "Resolved": 0, |
| 104 | # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error", |
| 105 | # "Timestamp": 1490818992116, <---- Time stamp |
| 106 | # "associations": [] |
| 107 | # }, |
| 108 | |
George Keishing | 6eb10b8 | 2017-06-06 10:02:09 -0500 | [diff] [blame] | 109 | Delete Error Logs |
George Keishing | 04bc481 | 2017-03-30 03:33:31 -0500 | [diff] [blame] | 110 | Create Test Error Log |
| 111 | Create Test Error Log |
| 112 | # The error log generated is associated with the epoc time and unique |
| 113 | # for every error and in increasing time stamp. |
George Keishing | 184a89a | 2017-06-27 07:12:16 -0500 | [diff] [blame] | 114 | ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY} |
| 115 | ${time_stamp1}= Read Attribute ${elog_entry[0]} Timestamp |
| 116 | ${time_stamp2}= Read Attribute ${elog_entry[1]} Timestamp |
George Keishing | 6eb10b8 | 2017-06-06 10:02:09 -0500 | [diff] [blame] | 117 | Should Be True ${time_stamp2} > ${time_stamp1} |
George Keishing | 04bc481 | 2017-03-30 03:33:31 -0500 | [diff] [blame] | 118 | |
Sweta Potthuri | 9ce8eeb | 2017-05-08 04:20:11 -0500 | [diff] [blame] | 119 | Create Test Error Log And Delete |
| 120 | [Documentation] Create an error log and delete it. |
| 121 | [Tags] Create_Test_Error_Log_And_Delete |
| 122 | |
| 123 | Delete Error Logs And Verify |
| 124 | Create Test Error Log |
| 125 | Delete Error Logs And Verify |
| 126 | |
| 127 | Create Multiple Test Error Logs And Delete All |
| 128 | [Documentation] Create multiple error logs and delete all. |
| 129 | [Tags] Create_Multiple_Test_Error_Logs_And_Delete_All |
| 130 | |
| 131 | Delete Error Logs And Verify |
| 132 | Create Test Error Log |
| 133 | Create Test Error Log |
| 134 | Create Test Error Log |
| 135 | Delete Error Logs And Verify |
| 136 | |
| 137 | Create Two Test Error Logs And Delete One |
| 138 | [Documentation] Create two error logs and delete the first entry. |
| 139 | [Tags] Create_Two_Test_Error_Logs_And_Delete_One |
| 140 | |
| 141 | Delete Error Logs And Verify |
| 142 | Create Test Error Log |
George Keishing | 184a89a | 2017-06-27 07:12:16 -0500 | [diff] [blame] | 143 | ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY} |
Sweta Potthuri | 9ce8eeb | 2017-05-08 04:20:11 -0500 | [diff] [blame] | 144 | Create Test Error Log |
George Keishing | 184a89a | 2017-06-27 07:12:16 -0500 | [diff] [blame] | 145 | Delete Error log Entry ${elog_entry[0]} |
| 146 | ${resp}= OpenBMC Get Request ${elog_entry[0]} |
Sweta Potthuri | 9ce8eeb | 2017-05-08 04:20:11 -0500 | [diff] [blame] | 147 | Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND} |
George Keishing | 04bc481 | 2017-03-30 03:33:31 -0500 | [diff] [blame] | 148 | |
Rahul Maheshwari | 92972e7 | 2017-06-22 00:59:59 -0500 | [diff] [blame] | 149 | |
| 150 | Verify IPMI SEL Version |
| 151 | [Documentation] Verify IPMI SEL's version info. |
| 152 | [Tags] Verify_IPMI_SEL_Version |
| 153 | |
| 154 | ${version_info}= Get IPMI SEL Setting Version |
| 155 | ${setting_status}= Fetch From Left ${version_info} ( |
| 156 | ${setting_status}= Evaluate $setting_status.replace(' ','') |
| 157 | |
| 158 | Should Be True ${setting_status} >= 1.5 |
Rahul Maheshwari | b44b6ec | 2017-07-05 00:23:03 -0500 | [diff] [blame^] | 159 | Should Contain ${version_info} v2 compliant case_insensitive=True |
Rahul Maheshwari | 92972e7 | 2017-06-22 00:59:59 -0500 | [diff] [blame] | 160 | |
| 161 | |
George Keishing | 845d9dc | 2017-06-16 14:49:29 -0500 | [diff] [blame] | 162 | Verify Watchdog Timedout Error |
| 163 | [Documentation] Trigger watchdog timed out and verify errorlog generated. |
| 164 | [Tags] Verify_Watchdog_Timedout_Error |
| 165 | |
| 166 | # Clear errors if there are any. |
| 167 | Delete Error Logs |
| 168 | |
| 169 | Initiate Host Boot |
| 170 | |
| 171 | # Check if the watchdog interface is created. |
| 172 | Wait Until Keyword Succeeds 3 min 10 sec |
| 173 | ... Read Properties /xyz/openbmc_project/watchdog/host0 |
| 174 | |
| 175 | Trigger Host Watchdog Error |
| 176 | |
| 177 | Verify Watchdog Errorlog Content |
| 178 | |
| 179 | |
Rahul Maheshwari | b44b6ec | 2017-07-05 00:23:03 -0500 | [diff] [blame^] | 180 | Verify IPMI SEL Delete |
| 181 | [Documentation] Verify IPMI SEL delete operation. |
| 182 | [Tags] Verify_IPMI_SEL_Delete |
| 183 | |
| 184 | Delete Error Logs And Verify |
| 185 | Create Test Error Log |
| 186 | |
| 187 | ${sel_list}= Run IPMI Standard Command sel list |
| 188 | # Example of SEL List: |
| 189 | # 4 | 04/21/2017 | 10:51:16 | System Event #0x01 | Undetermined system hardware failure | Asserted |
| 190 | |
| 191 | ${sel_entry}= Fetch from Left ${sel_list} | |
| 192 | ${sel_entry}= Evaluate $sel_entry.replace(' ','') |
| 193 | ${sel_entry}= Convert To Integer 0x${sel_entry} |
| 194 | |
| 195 | ${sel_delete}= Run IPMI Standard Command sel delete ${sel_entry} |
| 196 | Should Be Equal As Strings ${sel_delete} Deleted entry ${sel_entry} |
| 197 | ... case_insensitive=True |
| 198 | |
| 199 | ${sel_list}= Run IPMI Standard Command sel list |
| 200 | Should Be Equal As Strings ${sel_list} SEL has no entries |
| 201 | ... case_insensitive=True |
| 202 | |
| 203 | |
| 204 | Verify Empty SEL |
| 205 | [Documentation] Verify empty SEL list. |
| 206 | [Tags] Verify_Empty_SEL |
| 207 | |
| 208 | Delete Error Logs And Verify |
| 209 | |
| 210 | ${resp}= Run IPMI Standard Command sel list |
| 211 | Should Contain ${resp} SEL has no entries case_insensitive=True |
| 212 | |
| 213 | |
| 214 | Delete Non Existing SEL Entry |
| 215 | [Documentation] Delete non existing SEL entry. |
| 216 | [Tags] Delete_Non_Existing_SEL_Entry |
| 217 | |
| 218 | Delete Error Logs And Verify |
| 219 | ${sel_delete}= Run Keyword And Expect Error * |
| 220 | ... Run IPMI Standard Command sel delete 100 |
| 221 | Should Contain ${sel_delete} Unable to delete entry |
| 222 | ... case_insensitive=True |
| 223 | |
| 224 | |
| 225 | Delete Invalid SEL Entry |
| 226 | [Documentation] Delete invalid SEL entry. |
| 227 | [Tags] Delete_Invalid_SEL_Entry |
| 228 | |
| 229 | ${sel_delete}= Run Keyword And Expect Error * |
| 230 | ... Run IPMI Standard Command sel delete abc |
| 231 | Should Contain ${sel_delete} Given SEL ID 'abc' is invalid |
| 232 | ... case_insensitive=True |
| 233 | |
| 234 | |
| 235 | |
Michael Tritz | 935c18e | 2017-03-08 11:45:41 -0600 | [diff] [blame] | 236 | *** Keywords *** |
| 237 | |
Rahul Maheshwari | 92972e7 | 2017-06-22 00:59:59 -0500 | [diff] [blame] | 238 | Get IPMI SEL Setting |
| 239 | [Documentation] Returns status for given IPMI SEL setting. |
| 240 | [Arguments] ${setting} |
| 241 | # Description of argument(s): |
| 242 | # setting SEL setting which needs to be read(e.g. "Last Add Time"). |
| 243 | |
| 244 | ${resp}= Run IPMI Standard Command sel info |
| 245 | |
| 246 | ${setting_line}= Get Lines Containing String ${resp} ${setting} |
| 247 | ... case-insensitive |
Rahul Maheshwari | b44b6ec | 2017-07-05 00:23:03 -0500 | [diff] [blame^] | 248 | ${setting_status}= Fetch From Right ${setting_line} :${SPACE} |
Rahul Maheshwari | 92972e7 | 2017-06-22 00:59:59 -0500 | [diff] [blame] | 249 | |
| 250 | [Return] ${setting_status} |
| 251 | |
| 252 | |
George Keishing | 845d9dc | 2017-06-16 14:49:29 -0500 | [diff] [blame] | 253 | Verify Watchdog Errorlog Content |
| 254 | [Documentation] Verify watchdog errorlog content. |
| 255 | # Example: |
| 256 | # "/xyz/openbmc_project/logging/entry/1": |
| 257 | # { |
| 258 | # "AdditionalData": [], |
| 259 | # "Id": 1, |
| 260 | # "Message": "org.open_power.Host.Error.WatchdogTimedOut", |
| 261 | # "Resolved": 0, |
| 262 | # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Informational", |
| 263 | # "Timestamp": 1492715244828, |
| 264 | # "associations": [] |
| 265 | # }, |
| 266 | |
George Keishing | 184a89a | 2017-06-27 07:12:16 -0500 | [diff] [blame] | 267 | ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY} |
| 268 | ${elog}= Read Properties ${elog_entry[0]} |
George Keishing | 845d9dc | 2017-06-16 14:49:29 -0500 | [diff] [blame] | 269 | Should Be Equal As Strings |
| 270 | ... ${elog["Message"]} org.open_power.Host.Error.WatchdogTimedOut |
| 271 | Should Not Be Equal As Strings |
| 272 | ... ${elog["Severity"]} xyz.openbmc_project.Logging.Entry.Level.Informational |
| 273 | |
| 274 | |
Michael Tritz | 935c18e | 2017-03-08 11:45:41 -0600 | [diff] [blame] | 275 | Verify logging-test |
| 276 | [Documentation] Verify existence of prerequisite logging-test. |
| 277 | |
| 278 | Open Connection And Log In |
| 279 | ${out} ${stderr}= Execute Command which logging-test return_stderr=True |
| 280 | Should Be Empty ${stderr} |
| 281 | Should Contain ${out} logging-test |
| 282 | |
| 283 | Clear Existing Error Logs |
| 284 | [Documentation] If error log isn't empty, reboot the BMC to clear the log. |
| 285 | |
| 286 | ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${1} |
| 287 | Return From Keyword If ${resp.status_code} == ${HTTP_NOT_FOUND} |
| 288 | Initiate BMC Reboot |
| 289 | Wait Until Keyword Succeeds 10 min 10 sec |
| 290 | ... Is BMC Ready |
| 291 | ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${1} |
| 292 | Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND} |
| 293 | |
| 294 | Create Test Error Log |
| 295 | [Documentation] Generate test error log. |
| 296 | |
| 297 | # Test error log entry example: |
| 298 | # "/xyz/openbmc_project/logging/entry/1": { |
| 299 | # "AdditionalData": [ |
| 300 | # "STRING=FOO" |
| 301 | # ], |
| 302 | # "Id": 1, |
| 303 | # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple", |
| 304 | # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error", |
| 305 | # "Timestamp": 1487743963328, |
| 306 | # "associations": [] |
| 307 | # } |
| 308 | |
| 309 | Execute Command On BMC logging-test -c AutoTestSimple |
| 310 | |
| 311 | Verify Test Error Log |
| 312 | [Documentation] Verify test error log entries. |
George Keishing | 184a89a | 2017-06-27 07:12:16 -0500 | [diff] [blame] | 313 | ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY} |
| 314 | ${entry_id}= Read Attribute ${elog_entry[0]} Message |
Sweta Potthuri | 9ce8eeb | 2017-05-08 04:20:11 -0500 | [diff] [blame] | 315 | Should Be Equal ${entry_id} |
Michael Tritz | 935c18e | 2017-03-08 11:45:41 -0600 | [diff] [blame] | 316 | ... example.xyz.openbmc_project.Example.Elog.AutoTestSimple |
George Keishing | 184a89a | 2017-06-27 07:12:16 -0500 | [diff] [blame] | 317 | ${entry_id}= Read Attribute ${elog_entry[0]} Severity |
Sweta Potthuri | 9ce8eeb | 2017-05-08 04:20:11 -0500 | [diff] [blame] | 318 | Should Be Equal ${entry_id} |
Michael Tritz | 935c18e | 2017-03-08 11:45:41 -0600 | [diff] [blame] | 319 | ... xyz.openbmc_project.Logging.Entry.Level.Error |
George Keishing | 04bc481 | 2017-03-30 03:33:31 -0500 | [diff] [blame] | 320 | |
Sweta Potthuri | 9ce8eeb | 2017-05-08 04:20:11 -0500 | [diff] [blame] | 321 | Delete Error Logs And Verify |
| 322 | [Documentation] Delete all error logs and verify. |
| 323 | |
George Keishing | 6eb10b8 | 2017-06-06 10:02:09 -0500 | [diff] [blame] | 324 | Delete Error Logs |
Sweta Potthuri | 9ce8eeb | 2017-05-08 04:20:11 -0500 | [diff] [blame] | 325 | ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}/list |
| 326 | Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND} |
George Keishing | 845d9dc | 2017-06-16 14:49:29 -0500 | [diff] [blame] | 327 | |
| 328 | Post Test Case Execution |
| 329 | [Documentation] Do the post test teardown. |
| 330 | # 1. Capture FFDC on test failure. |
| 331 | # 2. Delete error logs. |
| 332 | # 3. Close all open SSH connections. |
| 333 | # 4. Clear all REST sessions. |
| 334 | |
| 335 | FFDC On Test Case Fail |
| 336 | Delete Error Logs |
| 337 | Close All Connections |
| 338 | Flush REST Sessions |
| 339 | |