Migrate Remote logging test suite to redfish
Changes:
- Update README.md
- Update variable BMC_STOP_MSG string value.
- Update test suite using Redfish interfaces.
Change-Id: Ib6092e135b57a467584657900f7e01e3189167c6
Signed-off-by: George Keishing <gkeishin@in.ibm.com>
diff --git a/redfish/extended/test_remote_logging.robot b/redfish/extended/test_remote_logging.robot
new file mode 100644
index 0000000..758a0a9
--- /dev/null
+++ b/redfish/extended/test_remote_logging.robot
@@ -0,0 +1,392 @@
+*** Settings ***
+Documentation Remote logging test for rsyslog.
+
+# Program arguments:
+# REMOTE_LOG_SERVER_HOST The host name or IP address of the remote
+# logging server.
+# REMOTE_LOG_SERVER_PORT The port number for the remote logging server.
+# REMOTE_USERNAME The username for the remote logging server.
+# REMOTE_PASSWORD The password for the remote logging server.
+
+Library String
+Resource ../../lib/openbmc_ffdc.robot
+Resource ../../lib/boot_utils.robot
+Resource ../../lib/remote_logging_utils.robot
+Resource ../../lib/bmc_redfish_resource.robot
+Resource ../../lib/ipmi_client.robot
+Resource ../../lib/bmc_redfish_resource.robot
+Library ../../lib/gen_misc.py
+
+Suite Setup Suite Setup Execution
+Test Setup Test Setup Execution
+Test Teardown FFDC On Test Case Fail
+
+*** Variables ***
+
+# Strings to check from journald.
+${BMC_STOP_MSG} Stopping Network IPMI daemon
+${BMC_START_MSG} Starting Flush Journal to Persistent Storage
+${BMC_BOOT_MSG} Startup finished in
+${BMC_SYSLOG_REGEX} dropbear|vrm-control.sh
+${RSYSLOG_REGEX} start|exiting on signal 15|there are no active actions configured
+${RSYSLOG_RETRY_REGEX} suspended
+${valid_password} 0penBmc1
+
+*** Test Cases ***
+
+Test BMC Hostname Service And Verify
+ [Documentation] Write to hostname interface and verify via REST and
+ ... 'hostname' command.
+ [Tags] Test_BMC_Hostname_Service_And_Verify
+
+ ${openbmc_host_name} ${openbmc_ip} ${openbmc_short_name}=
+ ... Get Host Name IP host=${OPENBMC_HOST} short_name=1
+
+ ${host_name_dict}= Create Dictionary data=${openbmc_short_name}
+ Write Attribute ${NETWORK_MANAGER}config HostName data=${host_name_dict}
+ ... verify=${TRUE} expected_value=${openbmc_short_name}
+
+ ${hostname} ${stderr} ${rc}= BMC Execute Command hostname
+
+ Should Be Equal As Strings ${hostname} ${openbmc_short_name}
+ ... msg=The hostname interface ${openbmc_short_name} and command value ${hostname} do not match.
+
+ # Override the suite hostname variable if this test is executed.
+ Set Suite Variable ${bmc_hostname} ${openbmc_short_name}
+
+
+Verify REST Logging On BMC Journal When Disabled
+ [Documentation] Enable REST logging and verify from journald.
+ [Tags] Verify_REST_Logging_On_BMC_Journal_When_Disabled
+
+ ${log_dict}= Create Dictionary data=${False}
+ Write Attribute ${BMC_LOGGING_URI}${/}rest_api_logs Enabled data=${log_dict}
+ ... verify=${True} expected_value=${False}
+
+ # If it was enabled prior, this REST footprint will show up.
+ # Takes around 5 seconds for the REST to restart service when policy is changed.
+ Sleep 10s
+
+ ${login_footprint}= Catenate login json:None 200 OK
+ # Example: Just get the message part of the syslog
+ # user:root POST http://xx.xx.xx.xx/login json:None 200 OK
+ ${cmd}= Catenate SEPARATOR= --no-pager | egrep '${login_footprint}'
+ ... | awk -F': ' '{print $2}'
+
+ Start Journal Log filter=${cmd}
+ Initialize OpenBMC
+ Sleep 5s
+ ${bmc_journald}= Stop Journal Log
+
+ Should Be Empty ${bmc_journald}
+ ... msg=${bmc_journald} contains unexpected REST entries.
+
+
+Verify REST Logging On BMC Journal When Enabled
+ [Documentation] Enable REST logging and verify from journald.
+ [Tags] Verify_REST_Logging_On_BMC_Journal_When_Enabled
+
+ ${log_dict}= Create Dictionary data=${True}
+ Write Attribute ${BMC_LOGGING_URI}${/}rest_api_logs Enabled data=${log_dict}
+ ... verify=${True} expected_value=${True}
+
+ Sleep 5s
+
+ Start Journal Log
+ Initialize OpenBMC
+ Log Out OpenBMC
+ ${bmc_journald}= Stop Journal Log
+
+ Should Contain ${bmc_journald} login json:None 200 OK
+ ... msg=${bmc_journald} doesn't contains REST entries.
+
+
+Test Remote API Valid Config Combination
+ [Documentation] Verify valid combination of address and port.
+ [Tags] Test_Remote_API_Valid_Config_Combination
+ [Template] Verify Configure Remote Logging Server
+ # Forego normal test setup:
+ [Setup] No Operation
+
+ # Address Port Expected result
+ ${EMPTY} ${REMOTE_LOG_SERVER_PORT} ${True}
+ ${REMOTE_LOG_SERVER_HOST} ${REMOTE_LOG_SERVER_PORT} ${True}
+ ${REMOTE_LOG_SERVER_HOST} ${0} ${True}
+
+
+Test Remote API Invalid Config Combination
+ [Documentation] Verify invalid combination of address and port.
+ [Tags] Test_Remote_API_Invalid_Config_Combination
+ [Template] Verify Configure Remote Logging Server
+ # Forego normal test setup:
+ [Setup] No Operation
+
+ # Address Port Expected result
+ ${0} ${REMOTE_LOG_SERVER_PORT} ${False}
+ "0" ${REMOTE_LOG_SERVER_PORT} ${False}
+ ${REMOTE_LOG_SERVER_HOST} ${EMPTY} ${False}
+ ${REMOTE_LOG_SERVER_HOST} "0" ${False}
+
+
+Test Remote Logging REST Interface And Verify Config
+ [Documentation] Test remote logging interface and configuration.
+ [Tags] Test_Remote_Logging_REST_Interface_And_Verify_Config
+
+ Verify Rsyslog Config On BMC
+
+
+Test Remote Logging Invalid Port Config And Verify BMC Journald
+ [Documentation] Test remote logging interface and configuration.
+ [Tags] Test_Remote_Logging_Invalid_Port_Config_And_Verify_BMC_Journald
+
+ # Invalid port derived by (REMOTE_LOG_SERVER_PORT + 1) port config setting.
+ ${INVALID_PORT}= Evaluate ${REMOTE_LOG_SERVER_PORT} + ${1}
+ Configure Remote Log Server With Parameters
+ ... remote_host=${REMOTE_LOG_SERVER_HOST} remote_port=${INVALID_PORT}
+
+ Sleep 3s
+ # rsyslogd[1870]: action 'action 0' suspended,
+ # next retry is Fri Sep 14 05:47:39 2018 [v8.29.0 try http://www.rsyslog.com/e/2007 ]
+ ${bmc_journald} ${stderr} ${rc}= BMC Execute Command
+ ... journalctl -b --no-pager | egrep 'rsyslog.*${RSYSLOG_RETRY_REGEX}'
+
+ Should Contain ${bmc_journald} ${RSYSLOG_RETRY_REGEX}
+ ... msg=${bmc_journald} doesn't contain rsyslog retry entries.
+
+
+Verfiy BMC Journald Synced To Remote Logging Server
+ [Documentation] Check that BMC journald is sync to remote rsyslog.
+ [Tags] Verfiy_BMC_Journald_Synced_To_Remote_Logging_Server
+
+ # Restart BMC dump service and get the last entry of the journald.
+ # Example:
+ # systemd[1]: Started Phosphor Dump Manager.
+ BMC Execute Command
+ ... systemctl restart xyz.openbmc_project.Dump.Manager.service
+
+ ${bmc_journald} ${stderr} ${rc}= BMC Execute Command
+ ... journalctl --no-pager | grep 'Started Phosphor Dump Manager'
+
+ # systemd[1]: Started Phosphor Dump Manager.
+ ${cmd}= Catenate SEPARATOR= egrep '${bmc_hostname}.*Started Phosphor Dump Manager' /var/log/syslog
+ ${remote_journald}= Remote Logging Server Execute Command command=${cmd}
+
+ # TODO: rsyslog configuration and time date template to match BMC journald.
+ # Compare the BMC journlad log. Example:
+ # systemd[1]: Started Phosphor Dump Manager.
+ Should Contain ${remote_journald} ${bmc_journald.split('${bmc_hostname}')[1][0]}
+ ... msg= ${bmc_journald} doesn't match remote rsyslog:${remote_journald}.
+
+
+Verify Journald Post BMC Reset
+ [Documentation] Check that BMC journald is sync'ed to remote rsyslog after
+ ... BMC reset.
+ [Tags] Verify_Journald_Post_BMC_Reset
+
+ ${hostname} ${stderr} ${rc}= BMC Execute Command hostname
+ OBMC Reboot (off)
+
+ ${cmd}= Catenate grep ${hostname} /var/log/syslog |
+ ... egrep '${BMC_STOP_MSG}|${BMC_START_MSG}|${BMC_BOOT_MSG}'
+ ${remote_journald}= Remote Logging Server Execute Command command=${cmd}
+
+ # 1. Last reboot message to verify.
+ Should Contain ${remote_journald} ${BMC_STOP_MSG}
+ ... msg=The remote journald doesn't contain the IPMI shutdown message: ${BMC_STOP_MSG}.
+
+ # 2. Earliest booting message on journald.
+ Should Contain ${remote_journald} ${BMC_START_MSG}
+ ... msg=The remote journald doesn't contain the start message: ${BMC_START_MSG}.
+
+ # 3. Unique boot to standby message.
+ # Startup finished in 9.961s (kernel) + 1min 59.039s (userspace) = 2min 9.000s
+ ${bmc_journald} ${stderr} ${rc}= BMC Execute Command
+ ... journalctl -b --no-pager | egrep '${BMC_BOOT_MSG}'
+
+ Should Contain ${remote_journald}
+ ... ${bmc_journald.split('${hostname}')[1]}
+ ... msg=The remote journald doesn't contain the boot message: ${BMC_BOOT_MSG}.
+
+
+Verify BMC Journald Contains No Credential Data
+ [Documentation] Check that BMC journald doesn't log any credential data.
+ [Tags] Verify_BMC_Journald_Contains_No_Credential_Data
+
+ Initialize OpenBMC
+
+ Create Redfish And IPMI Users
+
+ # Time for user manager to sync.
+ Sleep 5 s
+
+ Open Connection And Log In
+ ${bmc_journald} ${stderr} ${rc}= BMC Execute Command
+ ... journalctl -o json-pretty | cat
+
+ Should Not Contain Any ${bmc_journald} ${OPENBMC_PASSWORD} ${REST_PASSWORD}
+ ... msg=Journald logs BMC credentials/password ${OPENBMC_PASSWORD}.
+
+
+Audit BMC SSH Login And Remote Logging
+ [Documentation] Check that the SSH login to BMC is logged and synced to
+ ... remote logging server.
+ [Tags] Audit_BMC_SSH_Login_And_Remote_Logging
+
+ ${login_footprint}= Catenate Started SSH Per-Connection Server
+ # Example: Just get the message part of the syslog
+ # Started SSH Per-Connection Server (xx.xx.xx.xx:51292)
+ ${cmd}= Catenate SEPARATOR= --no-pager | egrep '${login_footprint}'
+ ... | awk -F': ' '{print $2}'
+
+ Start Journal Log filter=${cmd}
+ Open Connection And Log In
+ Sleep 5s
+ ${bmc_journald}= Stop Journal Log
+ @{ssh_entry}= Split To Lines ${bmc_journald}
+
+ ${cmd}= Catenate SEPARATOR= egrep -E '*${bmc_hostname}.*${login_footprint}' /var/log/syslog
+
+ ${remote_journald}= Remote Logging Server Execute Command command=${cmd}
+
+ Should Contain ${remote_journald} ${ssh_entry[0]}
+ ... msg=${remote_journald} don't contain ${bmc_journald} entry.
+
+
+Verify Rsyslog Does Not Log On BMC
+ [Documentation] Check that rsyslog journald doesn't log on BMC.
+ [Tags] Verify_Rsyslog_Does_Not_Log_On_BMC
+
+ # Expected filter rsyslog entries.
+ # Example:
+ # syslogd[3356]:
+ # [origin software="rsyslogd" swVersion="8.29.0" x-pid="3356" x-info="http://www.rsyslog.com"]
+ # exiting on signal 15.
+ # rsyslogd[3364]:
+ # [origin software="rsyslogd" swVersion="8.29.0" x-pid="3364" x-info="http://www.rsyslog.com"] start
+ ${bmc_journald} ${stderr} ${rc}= BMC Execute Command
+ ... journalctl -b --no-pager | egrep 'rsyslog' | egrep -Ev '${RSYSLOG_REGEX}|${RSYSLOG_RETRY_REGEX}'
+ ... ignore_err=${1}
+
+ Should Be Empty ${bmc_journald}
+ ... msg=${bmc_journald} contains unexpected rsyslog entries.
+
+
+Boot Host And Verify Data Is Synced To Remote Server
+ [Documentation] Boot host and verify the power on sequence logs are synced
+ ... to remote logging server.
+ [Tags] Boot_Host_And_Verify_Data_Is_Synced_To_Remote_Server
+
+ # Filter kernel dmesg from the journald log.
+ # Example: xx.xx.xx kernel:
+
+ ${openbmc_host_name} ${openbmc_ip} ${openbmc_short_name}=
+ ... Get Host Name IP host=${OPENBMC_HOST} short_name=1
+
+ ${cmd}= Catenate SEPARATOR= --no-pager | egrep -Ev '${BMC_SYSLOG_REGEX}
+ ... |${openbmc_short_name} kernel' | awk -F': ' '{print $2}'
+
+ # Example: Just get the message part of the syslog
+ # Started OpenPOWER OCC Active Disable.
+ Start Journal Log filter=${cmd}
+
+ # Irrespective of the outcome, the journald should be synced.
+ Run Keyword And Ignore Error Redfish Power On
+ ${bmc_journald}= Stop Journal Log
+
+ ${cmd}= Catenate SEPARATOR= egrep '${bmc_hostname}' /var/log/syslog
+ ${remote_journald}= Remote Logging Server Execute Command command=${cmd}
+
+ @{lines}= Split To Lines ${bmc_journald}
+
+ :FOR ${line} IN @{lines}
+ \ Log To Console \n ${line}
+ \ Should Contain ${remote_journald} ${line}
+ ... mgs=${line} line doesn't contain in ${remote_journald}.
+
+
+*** Keywords ***
+
+Suite Setup Execution
+ [Documentation] Do the suite setup.
+
+ Should Not Be Empty ${REMOTE_LOG_SERVER_HOST}
+ Should Not Be Empty ${REMOTE_LOG_SERVER_PORT}
+ Should Not Be Empty ${REMOTE_USERNAME}
+ Should Not Be Empty ${REMOTE_PASSWORD}
+ Ping Host ${REMOTE_LOG_SERVER_HOST}
+ Remote Logging Server Execute Command true
+ Remote Logging Interface Should Exist
+
+ ${hostname} ${stderr} ${rc}= BMC Execute Command /bin/hostname
+ Set Suite Variable ${bmc_hostname} ${hostname}
+ Configure Remote Log Server With Parameters
+
+
+Test Setup Execution
+ [Documentation] Do the test setup.
+
+ # Retain only the past 1 second log:
+ BMC Execute Command journalctl --vacuum-time=1s
+
+ ${config_status}= Run Keyword And Return Status
+ ... Get Remote Log Server Configured
+
+ Run Keyword If ${config_status}==${FALSE}
+ ... Configure Remote Log Server With Parameters
+
+
+Remote Logging Interface Should Exist
+ [Documentation] Check that the remote logging URI exist.
+
+ ${resp}= OpenBMC Get Request ${REMOTE_LOGGING_URI}
+ Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
+
+
+Verify Configure Remote Logging Server
+ [Documentation] Configure the remote logging REST interface on BMC.
+ [Arguments] ${remote_host} ${remote_port} ${expectation}
+
+ # Description of argument(s):
+ # remote_host The host name or IP address of the remote logging server
+ # (e.g. "xx.xx.xx.xx").
+ # remote_port Remote ryslog server port number (e.g. "514").
+ # expectation Expect boolean True/False.
+
+
+ ${status}= Run Keyword And Return Status
+ ... Configure Remote Log Server With Parameters remote_host=${remote_host} remote_port=${remote_port}
+
+ Should Be Equal ${status} ${expectation}
+ ... msg=Test result ${status} and expectation ${expectation} do not match.
+
+
+Create Redfish And IPMI Users
+ [Documentation] Create a valid Redfish and IPMI local user accounts and
+ ... delete them.
+
+ # Create redfish local valid user.
+ ${redfish_username}= Generate Random String 8 [LETTERS]
+
+ Redfish.Login
+
+ ${payload}= Create Dictionary
+ ... UserName=${redfish_username} Password=${valid_password}
+ ... RoleId=Administrator Enabled=${True}
+ Redfish.Post /redfish/v1/AccountService/Accounts body=&{payload}
+ ... valid_status_codes=[${HTTP_CREATED}]
+
+ # Delete newly created user.
+ Redfish.Delete /redfish/v1/AccountService/Accounts/${redfish_username}
+
+ Redfish.Logout
+
+ # Create IPMI local valid user.
+ ${ipmi_username}= Generate Random String 8 [LETTERS]
+ Set Test Variable ${ipmi_username}
+ ${random_userid}= Evaluate random.randint(2, 15) modules=random
+ ${ipmi_cmd}= Catenate user set name ${random_userid} ${ipmi_username}
+ Run IPMI Standard Command ${ipmi_cmd}
+
+ # Delete IPMI user.
+ Run IPMI Standard Command user set name ${random_userid} ""
+