| *** Settings *** |
| Documentation Test Error logging. |
| |
| Resource ../lib/connection_client.robot |
| Resource ../lib/openbmc_ffdc.robot |
| Resource ../lib/utils.robot |
| Resource ../lib/state_manager.robot |
| Resource ../lib/ipmi_client.robot |
| Resource ../lib/boot_utils.robot |
| Resource ../lib/logging_utils.robot |
| |
| Test Setup Test Setup Execution |
| Test Teardown Test Teardown Execution |
| Suite Teardown Delete Error Logs And Verify |
| |
| *** Variables *** |
| |
| ${stack_mode} skip |
| |
| *** Test Cases *** |
| |
| Error Log Check After BMC Reboot |
| [Documentation] Check error log after BMC rebooted. |
| [Tags] Error_Log_Check_At_BMC_Ready |
| # 1. Power off. |
| # 2. Delete error logs. |
| # 3. Reboot BMC. |
| # 4. Check if error log exists. |
| |
| Smart Power Off |
| Delete Error Logs And Verify |
| OBMC Reboot (off) stack_mode=normal |
| Error Logs Should Not Exist |
| |
| |
| Error Log Check After Host Poweron |
| [Documentation] Check error log after host has booted. |
| [Tags] Error_Log_Check_At_Host_Booted |
| # 1. Delete error logs |
| # 1. Power on. |
| # 3. Check if error log exists. |
| |
| Delete Error Logs And Verify |
| REST Power On |
| Error Logs Should Not Exist |
| |
| |
| Create Test Error And Verify |
| [Documentation] Create error logs and verify via REST. |
| [Tags] Create_Test_Error_And_Verify |
| |
| Create Test Error Log |
| Verify Test Error Log |
| |
| |
| Test Error Persistency On Restart |
| [Documentation] Restart logging service and verify error logs. |
| [Tags] Test_Error_Persistency_On_Restart |
| |
| Create Test Error Log |
| Verify Test Error Log |
| BMC Execute Command |
| ... systemctl restart xyz.openbmc_project.Logging.service |
| Sleep 10s reason=Wait for logging service to restart properly. |
| Verify Test Error Log |
| |
| |
| Test Error Entry Numbering Reset On Restart |
| [Documentation] Restarts logging service and verify error logs entry start |
| ... from entry "Id" 1. |
| # 1. Create error log. |
| # 2. Verify error log. |
| # 3. Delete error log. |
| # 4. Restart logging service. |
| # 5. Create error log. |
| # 6. Verify new error log entry starts with Id entry 1. |
| |
| [Tags] Test_Error_Entry_Numbering_Reset_On_Restart |
| # Example Error logs: |
| # "/xyz/openbmc_project/logging/entry/1": { |
| # "AdditionalData": [ |
| # "STRING=FOO" |
| # ], |
| # "Id": 1, <--- Entry value should be 1. |
| # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple", |
| # "Resolved": 0, |
| # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error", |
| # "Timestamp": 1490818990051, |
| # "associations": [] |
| # }, |
| |
| Create Test Error Log |
| Verify Test Error Log |
| Delete All Error Logs |
| BMC Execute Command |
| ... systemctl restart xyz.openbmc_project.Logging.service |
| Sleep 10s reason=Wait for logging service to restart properly. |
| Create Test Error Log |
| ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY} |
| ${entry_id}= Read Attribute ${elog_entry[0]} Id |
| Should Be Equal ${entry_id} ${1} |
| |
| |
| Test Error Persistency On Reboot |
| [Documentation] Reboot BMC and verify error logs. |
| [Tags] Test_Error_Persistency_On_Reboot |
| |
| Create Test Error Log |
| Verify Test Error Log |
| Initiate BMC Reboot |
| Wait Until Keyword Succeeds 10 min 10 sec |
| ... Is BMC Ready |
| Verify Test Error Log |
| |
| |
| Create Test Error And Verify Resolved Field |
| [Documentation] Create error log and verify "Resolved" |
| ... field is 0. |
| [Tags] Create_Test_Error_And_Verify_Resolved_Field |
| |
| # Example Error log: |
| # "/xyz/openbmc_project/logging/entry/1": { |
| # "AdditionalData": [ |
| # "STRING=FOO" |
| # ], |
| # "Id": 1, |
| # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple", |
| # "Resolved": 0, |
| # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error", |
| # "Timestamp": 1490817164983, |
| # "associations": [] |
| # }, |
| |
| # It's work in progress, but it's mnfg need. To mark an error as |
| # resolved, without deleting the error, mfg will set this bool |
| # property. |
| # In this test context we are making sure "Resolved" field is "0" |
| # by default. |
| |
| Delete All Error Logs |
| Create Test Error Log |
| ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY} |
| ${resolved}= Read Attribute ${elog_entry[0]} Resolved |
| Should Be True ${resolved} == 0 |
| |
| |
| Create Test Errors And Verify Time Stamp |
| [Documentation] Create error logs and verify time stamp. |
| [Tags] Create_Test_Error_And_Verify_Time_Stamp |
| |
| # Example Error logs: |
| # "/xyz/openbmc_project/logging/entry/1": { |
| # "AdditionalData": [ |
| # "STRING=FOO" |
| # ], |
| # "Id": 1, |
| # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple", |
| # "Resolved": 0, |
| # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error", |
| # "Timestamp": 1490818990051, <--- Time stamp |
| # "associations": [] |
| # }, |
| # "/xyz/openbmc_project/logging/entry/2": { |
| # "AdditionalData": [ |
| # "STRING=FOO" |
| # ], |
| # "Id": 2, |
| # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple", |
| # "Resolved": 0, |
| # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error", |
| # "Timestamp": 1490818992116, <---- Time stamp |
| # "associations": [] |
| # }, |
| |
| Delete All Error Logs |
| Create Test Error Log |
| Create Test Error Log |
| # The error log generated is associated with the epoc time and unique |
| # for every error and in increasing time stamp. |
| ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY} |
| ${time_stamp1}= Read Attribute ${elog_entry[0]} Timestamp |
| ${time_stamp2}= Read Attribute ${elog_entry[1]} Timestamp |
| Should Be True ${time_stamp2} > ${time_stamp1} |
| |
| Create Test Error Log And Delete |
| [Documentation] Create an error log and delete it. |
| [Tags] Create_Test_Error_Log_And_Delete |
| |
| Delete Error Logs And Verify |
| Create Test Error Log |
| Delete Error Logs And Verify |
| |
| Create Multiple Test Error Logs And Delete All |
| [Documentation] Create multiple error logs and delete all. |
| [Tags] Create_Multiple_Test_Error_Logs_And_Delete_All |
| |
| Delete Error Logs And Verify |
| Create Test Error Log |
| Create Test Error Log |
| Create Test Error Log |
| Delete Error Logs And Verify |
| |
| Create Two Test Error Logs And Delete One |
| [Documentation] Create two error logs and delete the first entry. |
| [Tags] Create_Two_Test_Error_Logs_And_Delete_One |
| |
| Delete Error Logs And Verify |
| Create Test Error Log |
| ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY} |
| Create Test Error Log |
| Delete Error log Entry ${elog_entry[0]} |
| ${resp}= OpenBMC Get Request ${elog_entry[0]} |
| Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND} |
| |
| |
| Verify Watchdog Timedout Error |
| [Documentation] Trigger watchdog timed out and verify errorlog generated. |
| [Tags] Verify_Watchdog_Timedout_Error |
| |
| REST Power Off |
| REST Power On |
| |
| # Clear errors if there are any. |
| Delete All Error Logs |
| |
| Trigger Host Watchdog Error |
| |
| # Logging took time to generate the timedout error. |
| Wait Until Keyword Succeeds 2 min 30 sec |
| ... Verify Watchdog Errorlog Content |
| |
| |
| Verify IPMI SEL Delete |
| [Documentation] Verify IPMI SEL delete operation. |
| [Tags] Verify_IPMI_SEL_Delete |
| |
| Delete Error Logs And Verify |
| Create Test Error Log |
| |
| ${sel_list}= Run IPMI Standard Command sel list |
| Should Not Be Equal As Strings ${sel_list} SEL has no entries |
| |
| # Example of SEL List: |
| # 4 | 04/21/2017 | 10:51:16 | System Event #0x01 | Undetermined system hardware failure | Asserted |
| |
| ${sel_entry}= Fetch from Left ${sel_list} | |
| ${sel_entry}= Evaluate $sel_entry.replace(' ','') |
| ${sel_entry}= Convert To Integer 0x${sel_entry} |
| |
| ${sel_delete}= Run IPMI Standard Command sel delete ${sel_entry} |
| Should Be Equal As Strings ${sel_delete} Deleted entry ${sel_entry} |
| ... case_insensitive=True |
| |
| ${sel_list}= Run IPMI Standard Command sel list |
| Should Be Equal As Strings ${sel_list} SEL has no entries |
| ... case_insensitive=True |
| |
| |
| Delete Non Existing SEL Entry |
| [Documentation] Delete non existing SEL entry. |
| [Tags] Delete_Non_Existing_SEL_Entry |
| |
| Delete Error Logs And Verify |
| ${sel_delete}= Run Keyword And Expect Error * |
| ... Run IPMI Standard Command sel delete 100 |
| Should Contain ${sel_delete} Unable to delete entry |
| ... case_insensitive=True |
| |
| |
| Delete Invalid SEL Entry |
| [Documentation] Delete invalid SEL entry. |
| [Tags] Delete_Invalid_SEL_Entry |
| |
| ${sel_delete}= Run Keyword And Expect Error * |
| ... Run IPMI Standard Command sel delete abc |
| Should Contain ${sel_delete} Given SEL ID 'abc' is invalid |
| ... case_insensitive=True |
| |
| |
| Verify IPMI SEL Entries |
| [Documentation] Verify IPMI SEL's entries info. |
| [Tags] Verify_IPMI_SEL_Entries |
| |
| Delete Error Logs And Verify |
| |
| # Generate error logs of random count. |
| ${count}= Evaluate random.randint(1, 5) modules=random |
| Repeat Keyword ${count} Create Test Error Log |
| |
| ${sel_entries_count}= Get IPMI SEL Setting Entries |
| Should Be Equal As Strings ${sel_entries_count} ${count} |
| |
| |
| Verify IPMI SEL Last Add Time |
| [Documentation] Verify IPMI SEL's last added timestamp. |
| [Tags] Verify_IPMI_SEL_Last_Add_Time |
| |
| Create Test Error Log |
| ${sel_time}= Run IPMI Standard Command sel time get |
| ${sel_time}= Convert Date ${sel_time} |
| ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True |
| |
| ${sel_last_add_time}= Get IPMI SEL Setting Last Add Time |
| ${sel_last_add_time}= Convert Date ${sel_last_add_time} |
| ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True |
| |
| ${time-diff}= |
| ... Subtract Date From Date ${sel_last_add_time} ${sel_time} |
| |
| # Verify if the delay in current time check and last add SEL time |
| # is less or equals to 2 seconds. |
| Should Be True ${time-diff} <= 2 |
| |
| Verify Error Logs Capping |
| [Documentation] Verify error logs capping. |
| [Tags] Verify_Error_Logs_Capping |
| |
| Delete Error Logs And Verify |
| ${cmd}= Catenate for i in {1..201}; do /tmp/tarball/bin/logging-test -c |
| ... AutoTestSimple; done |
| BMC Execute Command ${cmd} |
| ${count}= Count Error Entries |
| Run Keyword If ${count} > 200 |
| ... Fail Error logs created exceeded max capacity 200. |
| |
| Test Error Log Rotation |
| [Documentation] Verify creation of 201 error log is replaced by entry id 1. |
| [Tags] Test_Error_Log_Rotation |
| |
| Delete Error Logs And Verify |
| |
| # Restart service. |
| BMC Execute Command |
| ... systemctl restart xyz.openbmc_project.Logging.service |
| Sleep 10s reason=Wait for logging service to restart properly. |
| |
| # Create 200 error logs. |
| ${cmd}= Catenate for i in {1..200}; do /tmp/tarball/bin/logging-test -c |
| ... AutoTestSimple;done |
| BMC Execute Command ${cmd} |
| |
| # Check the response for 200th error log. |
| ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${200} |
| Should Be Equal As Strings ${resp.status_code} ${HTTP_OK} |
| |
| # Check if error log with id 1 exists. |
| ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${1} |
| Should Be Equal As Strings ${resp.status_code} ${HTTP_OK} |
| |
| # Create error log and verify the entry ID is 201 and not 1. |
| Create Test Error Log |
| ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${201} |
| Should Be Equal As Strings ${resp.status_code} ${HTTP_OK} |
| |
| # Error log 1 is not present. |
| ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${1} |
| Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND} |
| |
| *** Keywords *** |
| |
| Test Setup Execution |
| [Documentation] Do test case setup tasks. |
| |
| ${status}= Run Keyword And Return Status Logging Test Binary Exist |
| Run Keyword If ${status} == ${False} Install Tarball |
| Delete Error Logs And Verify |
| |
| |
| Test Teardown Execution |
| [Documentation] Do the post test teardown. |
| # 1. Capture FFDC on test failure. |
| # 2. Delete error logs. |
| # 3. Close all open SSH connections. |
| # 4. Clear all REST sessions. |
| |
| FFDC On Test Case Fail |
| Delete All Error Logs |
| Close All Connections |