tests/test_error_log.robot - openbmc/openbmc-test-automation - Gitiles

 *** Settings ***
 Documentation       Test Error logging.

 Resource            ../lib/connection_client.robot
 Resource            ../lib/openbmc_ffdc.robot
 Resource            ../lib/utils.robot
 Resource            ../lib/state_manager.robot
 Resource            ../lib/ipmi_client.robot
 Resource            ../lib/boot_utils.robot

 Test Setup          Test Setup Execution
 Test Teardown       Post Test Case Execution
 Suite Teardown      Delete Error Logs And Verify

 *** Variables ***

 ${stack_mode}       skip

 *** Test Cases ***

 Error Log Check After BMC Reboot
     [Documentation]  Check error log after BMC rebooted.
     [Tags]  Error_Log_Check_At_BMC_Ready
     # 1. Power off.
     # 2. Delete error logs.
     # 3. Reboot BMC.
     # 4. Check if eror log exists.

     Smart Power Off
     Delete Error Logs And Verify
     OBMC Reboot (off)  stack_mode=normal
     Error Logs Should Not Exist


 Error Log Check After Host Poweron
     [Documentation]  Check error log after host has booted.
     [Tags]  Error_Log_Check_At_Host_Booted
     # 1. Delete error logs
     # 1. Power on.
     # 3. Check if eror log exists.

     Delete Error Logs And Verify
     REST Power On
     Error Logs Should Not Exist


 Create Test Error And Verify
     [Documentation]  Create error logs and verify via REST.
     [Tags]  Create_Test_Error_And_Verify

     Create Test Error Log
     Verify Test Error Log


 Test Error Persistency On Restart
     [Documentation]  Restart logging service and verify error logs.
     [Tags]  Test_Error_Persistency_On_Restart

     Create Test Error Log
     Verify Test Error Log
     Execute Command On BMC
     ...  systemctl restart xyz.openbmc_project.Logging.service
     Sleep  10s  reason=Wait for logging service to restart properly.
     Verify Test Error Log


 Test Error Entry Numbering Reset On Restart
     [Documentation]  Restarts logging service and verify error logs entry start
     ...  from entry "Id" 1.
     # 1. Create error log.
     # 2. Verify error log.
     # 3. Delete error log.
     # 4. Restart logging service.
     # 5. Create error log.
     # 6. Verify new error log entry starts with Id entry 1.

     [Tags]  Test_Error_Entry_Numbering_Reset_On_Restart
     # Example Error logs:
     #  "/xyz/openbmc_project/logging/entry/1": {
     #    "AdditionalData": [
     #        "STRING=FOO"
     #    ],
     #    "Id": 1,   <--- Entry value should be 1.
     #    "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
     #    "Resolved": 0,
     #    "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
     #    "Timestamp": 1490818990051,
     #    "associations": []
     #  },

     Create Test Error Log
     Verify Test Error Log
     Delete All Error Logs
     Execute Command On BMC
     ...  systemctl restart xyz.openbmc_project.Logging.service
     Sleep  10s  reason=Wait for logging service to restart properly.
     Create Test Error Log
     ${elog_entry}=  Get URL List  ${BMC_LOGGING_ENTRY}
     ${entry_id}=  Read Attribute  ${elog_entry[0]}  Id
     Should Be Equal  ${entry_id}  ${1}


 Test Error Persistency On Reboot
     [Documentation]  Reboot BMC and verify error logs.
     [Tags]  Test_Error_Persistency_On_Reboot

     Create Test Error Log
     Verify Test Error Log
     Initiate BMC Reboot
     Wait Until Keyword Succeeds  10 min  10 sec
     ...  Is BMC Ready
     Verify Test Error Log


 Create Test Error And Verify Resolved Field
     [Documentation]  Create error log and verify "Resolved"
     ...              field is 0.
     [Tags]  Create_Test_Error_And_Verify_Resolved_Field

     # Example Error log:
     #  "/xyz/openbmc_project/logging/entry/1": {
     #    "AdditionalData": [
     #        "STRING=FOO"
     #    ],
     #    "Id": 1,
     #    "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
     #    "Resolved": 0,
     #    "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
     #    "Timestamp": 1490817164983,
     #    "associations": []
     # },

     # It's work in progress, but it's mnfg need. To mark an error as
     # resolved, without deleting the error, mfg will set this bool
     # property.
     # In this test context we are making sure "Resolved" field is "0"
     # by default.

     Delete All Error Logs
     Create Test Error Log
     ${elog_entry}=  Get URL List  ${BMC_LOGGING_ENTRY}
     ${resolved}=  Read Attribute  ${elog_entry[0]}  Resolved
     Should Be True  ${resolved} == 0


 Create Test Errors And Verify Time Stamp
     [Documentation]  Create error logs and verify time stamp.
     [Tags]  Create_Test_Error_And_Verify_Time_Stamp

     # Example Error logs:
     #  "/xyz/openbmc_project/logging/entry/1": {
     #    "AdditionalData": [
     #        "STRING=FOO"
     #    ],
     #    "Id": 1,
     #    "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
     #    "Resolved": 0,
     #    "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
     #    "Timestamp": 1490818990051,  <--- Time stamp
     #    "associations": []
     #  },
     #  "/xyz/openbmc_project/logging/entry/2": {
     #    "AdditionalData": [
     #        "STRING=FOO"
     #    ],
     #    "Id": 2,
     #    "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
     #    "Resolved": 0,
     #    "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
     #    "Timestamp": 1490818992116,   <---- Time stamp
     #    "associations": []
     # },

     Delete All Error Logs
     Create Test Error Log
     Create Test Error Log
     # The error log generated is associated with the epoc time and unique
     # for every error and in increasing time stamp.
     ${elog_entry}=  Get URL List  ${BMC_LOGGING_ENTRY}
     ${time_stamp1}=  Read Attribute  ${elog_entry[0]}  Timestamp
     ${time_stamp2}=  Read Attribute  ${elog_entry[1]}  Timestamp
     Should Be True  ${time_stamp2} > ${time_stamp1}

 Create Test Error Log And Delete
     [Documentation]  Create an error log and delete it.
     [Tags]  Create_Test_Error_Log_And_Delete

     Delete Error Logs And Verify
     Create Test Error Log
     Delete Error Logs And Verify

 Create Multiple Test Error Logs And Delete All
     [Documentation]  Create multiple error logs and delete all.
     [Tags]  Create_Multiple_Test_Error_Logs_And_Delete_All

     Delete Error Logs And Verify
     Create Test Error Log
     Create Test Error Log
     Create Test Error Log
     Delete Error Logs And Verify

 Create Two Test Error Logs And Delete One
     [Documentation]  Create two error logs and delete the first entry.
     [Tags]  Create_Two_Test_Error_Logs_And_Delete_One

     Delete Error Logs And Verify
     Create Test Error Log
     ${elog_entry}=  Get URL List  ${BMC_LOGGING_ENTRY}
     Create Test Error Log
     Delete Error log Entry  ${elog_entry[0]}
     ${resp}=  OpenBMC Get Request  ${elog_entry[0]}
     Should Be Equal As Strings  ${resp.status_code}  ${HTTP_NOT_FOUND}


 Verify IPMI SEL Version
     [Documentation]  Verify IPMI SEL's version info.
     [Tags]  Verify_IPMI_SEL_Version

     ${version_info}=  Get IPMI SEL Setting  Version
     ${setting_status}=  Fetch From Left  ${version_info}  (
     ${setting_status}=  Evaluate  $setting_status.replace(' ','')

     Should Be True  ${setting_status} >= 1.5
     Should Contain  ${version_info}  v2 compliant  case_insensitive=True


 Verify Watchdog Timedout Error
     [Documentation]  Trigger watchdog timed out and verify errorlog generated.
     [Tags]  Verify_Watchdog_Timedout_Error

     REST Power Off
     REST Power On

     # Clear errors if there are any.
     Delete All Error Logs

     Trigger Host Watchdog Error

     Verify Watchdog Errorlog Content


 Verify IPMI SEL Delete
     [Documentation]  Verify IPMI SEL delete operation.
     [Tags]  Verify_IPMI_SEL_Delete

     Delete Error Logs And Verify
     Create Test Error Log

     ${sel_list}=  Run IPMI Standard Command  sel list
     # Example of SEL List:
     # 4 | 04/21/2017 | 10:51:16 | System Event #0x01 | Undetermined system hardware failure | Asserted

     ${sel_entry}=  Fetch from Left  ${sel_list}  |
     ${sel_entry}=  Evaluate  $sel_entry.replace(' ','')
     ${sel_entry}=  Convert To Integer  0x${sel_entry}

     ${sel_delete}=  Run IPMI Standard Command  sel delete ${sel_entry}
     Should Be Equal As Strings  ${sel_delete}  Deleted entry ${sel_entry}
     ...  case_insensitive=True

     ${sel_list}=  Run IPMI Standard Command  sel list
     Should Be Equal As Strings  ${sel_list}  SEL has no entries
     ...  case_insensitive=True


 Verify Empty SEL
     [Documentation]  Verify empty SEL list.
     [Tags]  Verify_Empty_SEL

     Delete Error Logs And Verify

     ${resp}=  Run IPMI Standard Command  sel list
     Should Contain  ${resp}  SEL has no entries  case_insensitive=True


 Delete Non Existing SEL Entry
     [Documentation]  Delete non existing SEL entry.
     [Tags]  Delete_Non_Existing_SEL_Entry

     Delete Error Logs And Verify
     ${sel_delete}=  Run Keyword And Expect Error  *
     ...  Run IPMI Standard Command  sel delete 100
     Should Contain  ${sel_delete}  Unable to delete entry
     ...  case_insensitive=True


 Delete Invalid SEL Entry
     [Documentation]  Delete invalid SEL entry.
     [Tags]  Delete_Invalid_SEL_Entry

     ${sel_delete}=  Run Keyword And Expect Error  *
     ...  Run IPMI Standard Command  sel delete abc
     Should Contain  ${sel_delete}  Given SEL ID 'abc' is invalid
     ...  case_insensitive=True


 Verify IPMI SEL Entries
     [Documentation]  Verify IPMI SEL's entries info.
     [Tags]  Verify_IPMI_SEL_Entries

     Delete Error Logs And Verify

     # Generate error logs of random count.
     ${count}=  Evaluate  random.randint(1, 5)  modules=random
     Repeat Keyword  ${count}  Create Test Error Log

     ${sel_entries_count}=  Get IPMI SEL Setting  Entries
     Should Be Equal As Strings  ${sel_entries_count}  ${count}


 Verify IPMI SEL Last Add Time
     [Documentation]  Verify IPMI SEL's last added timestamp.
     [Tags]  Verify_IPMI_SEL_Last_Add_Time

     Create Test Error Log
     ${sel_time}=  Run IPMI Standard Command  sel time get
     ${sel_time}=  Convert Date  ${sel_time}
     ...  date_format=%m/%d/%Y %H:%M:%S  exclude_millis=True

     ${sel_last_add_time}=  Get IPMI SEL Setting  Last Add Time
     ${sel_last_add_time}=  Convert Date  ${sel_last_add_time}
     ...  date_format=%m/%d/%Y %H:%M:%S  exclude_millis=True

     ${time-diff}=
     ...  Subtract Date From Date  ${sel_last_add_time}  ${sel_time}

     # Verify if the delay in current time check and last add SEL time
     # is less or equals to 2 seconds.
     Should Be True  ${time-diff} <= 2

 Verify Error Logs Capping
     [Documentation]  Verify error logs capping.
     [Tags]  Verify_Error_Logs_Capping

     Delete Error Logs And Verify
     ${cmd}=  Set Variable
     ...  for i in {1..101}; do /tmp/tarball/bin/logging-test -c AutoTestSimple;done
     Execute Command On BMC  ${cmd}
     ${count}=  Count Error Entries
     Run Keyword If  ${count} > 100
     ...  Fail  Error logs created exceeded max capacity 100.

 *** Keywords ***

 Get IPMI SEL Setting
     [Documentation]  Returns status for given IPMI SEL setting.
     [Arguments]  ${setting}
     # Description of argument(s):
     # setting  SEL setting which needs to be read(e.g. "Last Add Time").

     ${resp}=  Run IPMI Standard Command  sel info

     ${setting_line}=  Get Lines Containing String  ${resp}  ${setting}
     ...  case-insensitive
     ${setting_status}=  Fetch From Right  ${setting_line}  :${SPACE}

     [Return]  ${setting_status}


 Verify Watchdog Errorlog Content
     [Documentation]  Verify watchdog errorlog content.
     # Example:
     # "/xyz/openbmc_project/logging/entry/1":
     #  {
     #      "AdditionalData": [],
     #      "Id": 1,
     #      "Message": "org.open_power.Host.Error.WatchdogTimedOut",
     #      "Resolved": 0,
     #      "Severity": "xyz.openbmc_project.Logging.Entry.Level.Informational",
     #      "Timestamp": 1492715244828,
     #      "associations": []
     # },

     ${elog_entry}=  Get URL List  ${BMC_LOGGING_ENTRY}
     ${elog}=  Read Properties  ${elog_entry[0]}
     Should Be Equal As Strings
     ...  ${elog["Message"]}  org.open_power.Host.Error.WatchdogTimedOut
     Should Not Be Equal As Strings
     ...  ${elog["Severity"]}  xyz.openbmc_project.Logging.Entry.Level.Informational


 Logging Test Binary Exist
     [Documentation]  Verify existence of prerequisite logging-test.

     Open Connection And Log In
     ${out}  ${stderr}=  Execute Command
     ...  which /tmp/tarball/bin/logging-test  return_stderr=True
     Should Be Empty  ${stderr}
     Should Contain  ${out}  logging-test

 Clear Existing Error Logs
     [Documentation]  If error log isn't empty, reboot the BMC to clear the log.

     ${resp}=  OpenBMC Get Request  ${BMC_LOGGING_ENTRY}${1}
     Return From Keyword If  ${resp.status_code} == ${HTTP_NOT_FOUND}
     Initiate BMC Reboot
     Wait Until Keyword Succeeds  10 min  10 sec
     ...  Is BMC Ready
     ${resp}=  OpenBMC Get Request  ${BMC_LOGGING_ENTRY}${1}
     Should Be Equal As Strings  ${resp.status_code}  ${HTTP_NOT_FOUND}

 Create Test Error Log
     [Documentation]  Generate test error log.

     # Test error log entry example:
     # "/xyz/openbmc_project/logging/entry/1":  {
     #     "AdditionalData": [
     #         "STRING=FOO"
     #     ],
     #     "Id": 1,
     #     "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
     #     "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
     #     "Timestamp": 1487743963328,
     #     "associations": []
     # }

     Execute Command On BMC  /tmp/tarball/bin/logging-test -c AutoTestSimple

 Count Error Entries
     [Documentation]  Count Error entries.

     ${resp}=  OpenBMC Get Request  ${BMC_LOGGING_ENTRY}
     Should Be Equal As Strings  ${resp.status_code}  ${HTTP_OK}
     ${jsondata}=  To JSON  ${resp.content}
     ${count}=  Get Length  ${jsondata["data"]}
     [Return]  ${count}

 Verify Test Error Log
     [Documentation]  Verify test error log entries.
     ${elog_entry}=  Get URL List  ${BMC_LOGGING_ENTRY}
     ${entry_id}=  Read Attribute  ${elog_entry[0]}  Message
     Should Be Equal  ${entry_id}
     ...  example.xyz.openbmc_project.Example.Elog.AutoTestSimple
     ${entry_id}=  Read Attribute  ${elog_entry[0]}  Severity
     Should Be Equal  ${entry_id}
     ...  xyz.openbmc_project.Logging.Entry.Level.Error

 Delete Error Logs And Verify
     [Documentation]  Delete all error logs and verify.

     Delete All Error Logs
     ${resp}=  OpenBMC Get Request  ${BMC_LOGGING_ENTRY}/list  quiet=${1}
     Should Be Equal As Strings  ${resp.status_code}  ${HTTP_NOT_FOUND}


 Test Setup Execution
    [Documentation]  Do test case setup tasks.

    ${status}=  Run Keyword And Return Status  Logging Test Binary Exist
    Run Keyword If  ${status} == ${False}  Install Tarball
    Delete Error Logs And Verify


 Install Tarball
     [Documentation]  Install tarball on BMC.

     Run Keyword If  '${DEBUG_TARBALL_PATH}' == '${EMPTY}'  Return From Keyword
     BMC Execute Command  rm -rf /tmp/tarball
     Install Debug Tarball On BMC  ${DEBUG_TARBALL_PATH}


 Post Test Case Execution
    [Documentation]  Do the post test teardown.
    # 1. Capture FFDC on test failure.
    # 2. Delete error logs.
    # 3. Close all open SSH connections.
    # 4. Clear all REST sessions.

    FFDC On Test Case Fail
    Delete All Error Logs
    Close All Connections
	* Settings *
	Documentation Test Error logging.

	Resource ../lib/connection_client.robot
	Resource ../lib/openbmc_ffdc.robot
	Resource ../lib/utils.robot
	Resource ../lib/state_manager.robot
	Resource ../lib/ipmi_client.robot
	Resource ../lib/boot_utils.robot

	Test Setup Test Setup Execution
	Test Teardown Post Test Case Execution
	Suite Teardown Delete Error Logs And Verify

	* Variables *

	${stack_mode} skip

	* Test Cases *

	Error Log Check After BMC Reboot
	[Documentation] Check error log after BMC rebooted.
	[Tags] Error_Log_Check_At_BMC_Ready
	# 1. Power off.
	# 2. Delete error logs.
	# 3. Reboot BMC.
	# 4. Check if eror log exists.

	Smart Power Off
	Delete Error Logs And Verify
	OBMC Reboot (off) stack_mode=normal
	Error Logs Should Not Exist


	Error Log Check After Host Poweron
	[Documentation] Check error log after host has booted.
	[Tags] Error_Log_Check_At_Host_Booted
	# 1. Delete error logs
	# 1. Power on.
	# 3. Check if eror log exists.

	Delete Error Logs And Verify
	REST Power On
	Error Logs Should Not Exist


	Create Test Error And Verify
	[Documentation] Create error logs and verify via REST.
	[Tags] Create_Test_Error_And_Verify

	Create Test Error Log
	Verify Test Error Log


	Test Error Persistency On Restart
	[Documentation] Restart logging service and verify error logs.
	[Tags] Test_Error_Persistency_On_Restart

	Create Test Error Log
	Verify Test Error Log
	Execute Command On BMC
	... systemctl restart xyz.openbmc_project.Logging.service
	Sleep 10s reason=Wait for logging service to restart properly.
	Verify Test Error Log


	Test Error Entry Numbering Reset On Restart
	[Documentation] Restarts logging service and verify error logs entry start
	... from entry "Id" 1.
	# 1. Create error log.
	# 2. Verify error log.
	# 3. Delete error log.
	# 4. Restart logging service.
	# 5. Create error log.
	# 6. Verify new error log entry starts with Id entry 1.

	[Tags] Test_Error_Entry_Numbering_Reset_On_Restart
	# Example Error logs:
	# "/xyz/openbmc_project/logging/entry/1": {
	# "AdditionalData": [
	# "STRING=FOO"
	# ],
	# "Id": 1, <--- Entry value should be 1.
	# "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
	# "Resolved": 0,
	# "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
	# "Timestamp": 1490818990051,
	# "associations": []
	# },

	Create Test Error Log
	Verify Test Error Log
	Delete All Error Logs
	Execute Command On BMC
	... systemctl restart xyz.openbmc_project.Logging.service
	Sleep 10s reason=Wait for logging service to restart properly.
	Create Test Error Log
	${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
	${entry_id}= Read Attribute ${elog_entry[0]} Id
	Should Be Equal ${entry_id} ${1}


	Test Error Persistency On Reboot
	[Documentation] Reboot BMC and verify error logs.
	[Tags] Test_Error_Persistency_On_Reboot

	Create Test Error Log
	Verify Test Error Log
	Initiate BMC Reboot
	Wait Until Keyword Succeeds 10 min 10 sec
	... Is BMC Ready
	Verify Test Error Log


	Create Test Error And Verify Resolved Field
	[Documentation] Create error log and verify "Resolved"
	... field is 0.
	[Tags] Create_Test_Error_And_Verify_Resolved_Field

	# Example Error log:
	# "/xyz/openbmc_project/logging/entry/1": {
	# "AdditionalData": [
	# "STRING=FOO"
	# ],
	# "Id": 1,
	# "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
	# "Resolved": 0,
	# "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
	# "Timestamp": 1490817164983,
	# "associations": []
	# },

	# It's work in progress, but it's mnfg need. To mark an error as
	# resolved, without deleting the error, mfg will set this bool
	# property.
	# In this test context we are making sure "Resolved" field is "0"
	# by default.

	Delete All Error Logs
	Create Test Error Log
	${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
	${resolved}= Read Attribute ${elog_entry[0]} Resolved
	Should Be True ${resolved} == 0


	Create Test Errors And Verify Time Stamp
	[Documentation] Create error logs and verify time stamp.
	[Tags] Create_Test_Error_And_Verify_Time_Stamp

	# Example Error logs:
	# "/xyz/openbmc_project/logging/entry/1": {
	# "AdditionalData": [
	# "STRING=FOO"
	# ],
	# "Id": 1,
	# "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
	# "Resolved": 0,
	# "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
	# "Timestamp": 1490818990051, <--- Time stamp
	# "associations": []
	# },
	# "/xyz/openbmc_project/logging/entry/2": {
	# "AdditionalData": [
	# "STRING=FOO"
	# ],
	# "Id": 2,
	# "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
	# "Resolved": 0,
	# "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
	# "Timestamp": 1490818992116, <---- Time stamp
	# "associations": []
	# },

	Delete All Error Logs
	Create Test Error Log
	Create Test Error Log
	# The error log generated is associated with the epoc time and unique
	# for every error and in increasing time stamp.
	${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
	${time_stamp1}= Read Attribute ${elog_entry[0]} Timestamp
	${time_stamp2}= Read Attribute ${elog_entry[1]} Timestamp
	Should Be True ${time_stamp2} > ${time_stamp1}

	Create Test Error Log And Delete
	[Documentation] Create an error log and delete it.
	[Tags] Create_Test_Error_Log_And_Delete

	Delete Error Logs And Verify
	Create Test Error Log
	Delete Error Logs And Verify

	Create Multiple Test Error Logs And Delete All
	[Documentation] Create multiple error logs and delete all.
	[Tags] Create_Multiple_Test_Error_Logs_And_Delete_All

	Delete Error Logs And Verify
	Create Test Error Log
	Create Test Error Log
	Create Test Error Log
	Delete Error Logs And Verify

	Create Two Test Error Logs And Delete One
	[Documentation] Create two error logs and delete the first entry.
	[Tags] Create_Two_Test_Error_Logs_And_Delete_One

	Delete Error Logs And Verify
	Create Test Error Log
	${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
	Create Test Error Log
	Delete Error log Entry ${elog_entry[0]}
	${resp}= OpenBMC Get Request ${elog_entry[0]}
	Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND}


	Verify IPMI SEL Version
	[Documentation] Verify IPMI SEL's version info.
	[Tags] Verify_IPMI_SEL_Version

	${version_info}= Get IPMI SEL Setting Version
	${setting_status}= Fetch From Left ${version_info} (
	${setting_status}= Evaluate $setting_status.replace(' ','')

	Should Be True ${setting_status} >= 1.5
	Should Contain ${version_info} v2 compliant case_insensitive=True


	Verify Watchdog Timedout Error
	[Documentation] Trigger watchdog timed out and verify errorlog generated.
	[Tags] Verify_Watchdog_Timedout_Error

	REST Power Off
	REST Power On

	# Clear errors if there are any.
	Delete All Error Logs

	Trigger Host Watchdog Error

	Verify Watchdog Errorlog Content


	Verify IPMI SEL Delete
	[Documentation] Verify IPMI SEL delete operation.
	[Tags] Verify_IPMI_SEL_Delete

	Delete Error Logs And Verify
	Create Test Error Log

	${sel_list}= Run IPMI Standard Command sel list
	# Example of SEL List:
	# 4 \| 04/21/2017 \| 10:51:16 \| System Event #0x01 \| Undetermined system hardware failure \| Asserted

	${sel_entry}= Fetch from Left ${sel_list} \|
	${sel_entry}= Evaluate $sel_entry.replace(' ','')
	${sel_entry}= Convert To Integer 0x${sel_entry}

	${sel_delete}= Run IPMI Standard Command sel delete ${sel_entry}
	Should Be Equal As Strings ${sel_delete} Deleted entry ${sel_entry}
	... case_insensitive=True

	${sel_list}= Run IPMI Standard Command sel list
	Should Be Equal As Strings ${sel_list} SEL has no entries
	... case_insensitive=True


	Verify Empty SEL
	[Documentation] Verify empty SEL list.
	[Tags] Verify_Empty_SEL

	Delete Error Logs And Verify

	${resp}= Run IPMI Standard Command sel list
	Should Contain ${resp} SEL has no entries case_insensitive=True


	Delete Non Existing SEL Entry
	[Documentation] Delete non existing SEL entry.
	[Tags] Delete_Non_Existing_SEL_Entry

	Delete Error Logs And Verify
	${sel_delete}= Run Keyword And Expect Error *
	... Run IPMI Standard Command sel delete 100
	Should Contain ${sel_delete} Unable to delete entry
	... case_insensitive=True


	Delete Invalid SEL Entry
	[Documentation] Delete invalid SEL entry.
	[Tags] Delete_Invalid_SEL_Entry

	${sel_delete}= Run Keyword And Expect Error *
	... Run IPMI Standard Command sel delete abc
	Should Contain ${sel_delete} Given SEL ID 'abc' is invalid
	... case_insensitive=True


	Verify IPMI SEL Entries
	[Documentation] Verify IPMI SEL's entries info.
	[Tags] Verify_IPMI_SEL_Entries

	Delete Error Logs And Verify

	# Generate error logs of random count.
	${count}= Evaluate random.randint(1, 5) modules=random
	Repeat Keyword ${count} Create Test Error Log

	${sel_entries_count}= Get IPMI SEL Setting Entries
	Should Be Equal As Strings ${sel_entries_count} ${count}


	Verify IPMI SEL Last Add Time
	[Documentation] Verify IPMI SEL's last added timestamp.
	[Tags] Verify_IPMI_SEL_Last_Add_Time

	Create Test Error Log
	${sel_time}= Run IPMI Standard Command sel time get
	${sel_time}= Convert Date ${sel_time}
	... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True

	${sel_last_add_time}= Get IPMI SEL Setting Last Add Time
	${sel_last_add_time}= Convert Date ${sel_last_add_time}
	... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True

	${time-diff}=
	... Subtract Date From Date ${sel_last_add_time} ${sel_time}

	# Verify if the delay in current time check and last add SEL time
	# is less or equals to 2 seconds.
	Should Be True ${time-diff} <= 2

	Verify Error Logs Capping
	[Documentation] Verify error logs capping.
	[Tags] Verify_Error_Logs_Capping

	Delete Error Logs And Verify
	${cmd}= Set Variable
	... for i in {1..101}; do /tmp/tarball/bin/logging-test -c AutoTestSimple;done
	Execute Command On BMC ${cmd}
	${count}= Count Error Entries
	Run Keyword If ${count} > 100
	... Fail Error logs created exceeded max capacity 100.

	* Keywords *

	Get IPMI SEL Setting
	[Documentation] Returns status for given IPMI SEL setting.
	[Arguments] ${setting}
	# Description of argument(s):
	# setting SEL setting which needs to be read(e.g. "Last Add Time").

	${resp}= Run IPMI Standard Command sel info

	${setting_line}= Get Lines Containing String ${resp} ${setting}
	... case-insensitive
	${setting_status}= Fetch From Right ${setting_line} :${SPACE}

	[Return] ${setting_status}


	Verify Watchdog Errorlog Content
	[Documentation] Verify watchdog errorlog content.
	# Example:
	# "/xyz/openbmc_project/logging/entry/1":
	# {
	# "AdditionalData": [],
	# "Id": 1,
	# "Message": "org.open_power.Host.Error.WatchdogTimedOut",
	# "Resolved": 0,
	# "Severity": "xyz.openbmc_project.Logging.Entry.Level.Informational",
	# "Timestamp": 1492715244828,
	# "associations": []
	# },

	${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
	${elog}= Read Properties ${elog_entry[0]}
	Should Be Equal As Strings
	... ${elog["Message"]} org.open_power.Host.Error.WatchdogTimedOut
	Should Not Be Equal As Strings
	... ${elog["Severity"]} xyz.openbmc_project.Logging.Entry.Level.Informational


	Logging Test Binary Exist
	[Documentation] Verify existence of prerequisite logging-test.

	Open Connection And Log In
	${out} ${stderr}= Execute Command
	... which /tmp/tarball/bin/logging-test return_stderr=True
	Should Be Empty ${stderr}
	Should Contain ${out} logging-test

	Clear Existing Error Logs
	[Documentation] If error log isn't empty, reboot the BMC to clear the log.

	${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${1}
	Return From Keyword If ${resp.status_code} == ${HTTP_NOT_FOUND}
	Initiate BMC Reboot
	Wait Until Keyword Succeeds 10 min 10 sec
	... Is BMC Ready
	${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${1}
	Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND}

	Create Test Error Log
	[Documentation] Generate test error log.

	# Test error log entry example:
	# "/xyz/openbmc_project/logging/entry/1": {
	# "AdditionalData": [
	# "STRING=FOO"
	# ],
	# "Id": 1,
	# "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
	# "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
	# "Timestamp": 1487743963328,
	# "associations": []
	# }

	Execute Command On BMC /tmp/tarball/bin/logging-test -c AutoTestSimple

	Count Error Entries
	[Documentation] Count Error entries.

	${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}
	Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
	${jsondata}= To JSON ${resp.content}
	${count}= Get Length ${jsondata["data"]}
	[Return] ${count}

	Verify Test Error Log
	[Documentation] Verify test error log entries.
	${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
	${entry_id}= Read Attribute ${elog_entry[0]} Message
	Should Be Equal ${entry_id}
	... example.xyz.openbmc_project.Example.Elog.AutoTestSimple
	${entry_id}= Read Attribute ${elog_entry[0]} Severity
	Should Be Equal ${entry_id}
	... xyz.openbmc_project.Logging.Entry.Level.Error

	Delete Error Logs And Verify
	[Documentation] Delete all error logs and verify.

	Delete All Error Logs
	${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}/list quiet=${1}
	Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND}


	Test Setup Execution
	[Documentation] Do test case setup tasks.

	${status}= Run Keyword And Return Status Logging Test Binary Exist
	Run Keyword If ${status} == ${False} Install Tarball
	Delete Error Logs And Verify


	Install Tarball
	[Documentation] Install tarball on BMC.

	Run Keyword If '${DEBUG_TARBALL_PATH}' == '${EMPTY}' Return From Keyword
	BMC Execute Command rm -rf /tmp/tarball
	Install Debug Tarball On BMC ${DEBUG_TARBALL_PATH}


	Post Test Case Execution
	[Documentation] Do the post test teardown.
	# 1. Capture FFDC on test failure.
	# 2. Delete error logs.
	# 3. Close all open SSH connections.
	# 4. Clear all REST sessions.

	FFDC On Test Case Fail
	Delete All Error Logs
	Close All Connections