| *** Settings *** |
| Documentation Remote logging test for rsyslog. |
| |
| # Program arguments: |
| # REMOTE_LOG_SERVER_HOST The host name or IP address of the remote |
| # logging server. |
| # REMOTE_LOG_SERVER_PORT The port number for the remote logging server. |
| # REMOTE_USERNAME The username for the remote logging server. |
| # REMOTE_PASSWORD The password for the remote logging server. |
| |
| Library String |
| Resource ../lib/openbmc_ffdc.robot |
| Resource ../lib/boot_utils.robot |
| Resource ../lib/remote_logging_utils.robot |
| Library ../lib/gen_misc.py |
| |
| Suite Setup Suite Setup Execution |
| Test Setup Test Setup Execution |
| Test Teardown FFDC On Test Case Fail |
| |
| *** Variables *** |
| |
| # Strings to check from journald. |
| ${BMC_STOP_MSG} Stopping Phosphor IPMI BT DBus Bridge |
| ${BMC_START_MSG} Starting Flush Journal to Persistent Storage |
| ${BMC_BOOT_MSG} Startup finished in |
| ${BMC_SYSLOG_REGEX} dropbear|vrm-control.sh |
| ${RSYSLOG_REGEX} start|exiting on signal 15|there are no active actions configured |
| ${RSYSLOG_RETRY_REGEX} suspended, next retry |
| |
| *** Test Cases *** |
| |
| Test BMC Hostname Service And Verify |
| [Documentation] Write to hostname interface and verify via REST and |
| ... 'hostname' command. |
| [Tags] Test_BMC_Hostname_Service_And_Verify |
| |
| ${openbmc_host_name} ${openbmc_ip} ${openbmc_short_name}= |
| ... Get Host Name IP host=${OPENBMC_HOST} short_name=1 |
| |
| ${host_name_dict}= Create Dictionary data=${openbmc_short_name} |
| Write Attribute ${NETWORK_MANAGER}config HostName data=${host_name_dict} |
| ... verify=${TRUE} expected_value=${openbmc_short_name} |
| |
| ${hostname} ${stderr} ${rc}= BMC Execute Command hostname |
| |
| Should Be Equal As Strings ${hostname} ${openbmc_short_name} |
| ... msg=The hostname interface ${openbmc_short_name} and command value ${hostname} do not match. |
| |
| # Override the suite hostname variable if this test is executed. |
| Set Suite Variable ${bmc_hostname} ${openbmc_short_name} |
| |
| |
| Verify REST Logging On BMC Journal When Disabled |
| [Documentation] Enable REST logging and verify from journald. |
| [Tags] Verify_REST_Logging_On_BMC_Journal_When_Disabled |
| |
| ${log_dict}= Create Dictionary data=${False} |
| Write Attribute ${BMC_LOGGING_URI}${/}rest_api_logs Enabled data=${log_dict} |
| ... verify=${True} expected_value=${False} |
| |
| # If it was enabled prior, this REST footprint will show up. |
| # Takes around 5 seconds for the REST to restart service when policy is changed. |
| Sleep 10s |
| |
| ${login_footprint}= Catenate login json:None 200 OK |
| # Example: Just get the message part of the syslog |
| # user:root POST http://xx.xx.xx.xx/login json:None 200 OK |
| ${cmd}= Catenate SEPARATOR= --no-pager | egrep '${login_footprint}' |
| ... | awk -F': ' '{print $2}' |
| |
| Start Journal Log filter=${cmd} |
| Initialize OpenBMC |
| Sleep 5s |
| ${bmc_journald}= Stop Journal Log |
| |
| Should Be Empty ${bmc_journald} |
| ... msg=${bmc_journald} contains unexpected REST entries. |
| |
| |
| Verify REST Logging On BMC Journal When Enabled |
| [Documentation] Enable REST logging and verify from journald. |
| [Tags] Verify_REST_Logging_On_BMC_Journal_When_Enabled |
| |
| ${log_dict}= Create Dictionary data=${True} |
| Write Attribute ${BMC_LOGGING_URI}${/}rest_api_logs Enabled data=${log_dict} |
| ... verify=${True} expected_value=${True} |
| |
| Initialize OpenBMC |
| Log Out OpenBMC |
| |
| ${bmc_journald} ${stderr} ${rc}= BMC Execute Command |
| ... journalctl --no-pager |
| |
| Should Contain ${bmc_journald} login json:None 200 OK |
| ... msg=${bmc_journald} doesn't contains REST entries. |
| |
| |
| Test Remote API Valid Config Combination |
| [Documentation] Verify valid combination of address and port. |
| [Tags] Test_Remote_API_Valid_Config_Combination |
| [Template] Verify Configure Remote Logging Server |
| # Forego normal test setup: |
| [Setup] No Operation |
| |
| # Address Port Expected result |
| ${EMPTY} ${REMOTE_LOG_SERVER_PORT} ${True} |
| ${REMOTE_LOG_SERVER_HOST} ${REMOTE_LOG_SERVER_PORT} ${True} |
| ${REMOTE_LOG_SERVER_HOST} ${0} ${True} |
| |
| |
| Test Remote API Invalid Config Combination |
| [Documentation] Verify invalid combination of address and port. |
| [Tags] Test_Remote_API_Invalid_Config_Combination |
| [Template] Verify Configure Remote Logging Server |
| # Forego normal test setup: |
| [Setup] No Operation |
| |
| # Address Port Expected result |
| ${0} ${REMOTE_LOG_SERVER_PORT} ${False} |
| "0" ${REMOTE_LOG_SERVER_PORT} ${False} |
| ${REMOTE_LOG_SERVER_HOST} ${EMPTY} ${False} |
| ${REMOTE_LOG_SERVER_HOST} "0" ${False} |
| |
| |
| Test Remote Logging REST Interface And Verify Config |
| [Documentation] Test remote logging interface and configuration. |
| [Tags] Test_Remote_Logging_REST_Interface_And_Verify_Config |
| |
| Verify Rsyslog Config On BMC |
| |
| Configure Remote Log Server With Parameters remote_host=${EMPTY} remote_port=0 |
| Verify Rsyslog Config On BMC remote_host=remote-host remote_port=port |
| |
| |
| Test Remote Logging Invalid Port Config And Verify BMC Journald |
| [Documentation] Test remote logging interface and configuration. |
| [Tags] Test_Remote_Logging_Invalid_Port_Config_And_Verify_BMC_Journald |
| |
| # Invalid port derived by (REMOTE_LOG_SERVER_PORT + 1) port config setting. |
| ${INVALID_PORT}= Evaluate ${REMOTE_LOG_SERVER_PORT} + ${1} |
| Configure Remote Log Server With Parameters |
| ... remote_host=${REMOTE_LOG_SERVER_HOST} remote_port=${INVALID_PORT} |
| |
| Sleep 3s |
| # rsyslogd[1870]: action 'action 0' suspended, next retry is Fri Sep 14 05:47:39 2018 [v8.29.0 try http://www.rsyslog.com/e/2007 ] |
| ${bmc_journald} ${stderr} ${rc}= BMC Execute Command |
| ... journalctl -b --no-pager | egrep 'rsyslog.*${RSYSLOG_RETRY_REGEX}' |
| |
| Should Contain ${bmc_journald} ${RSYSLOG_RETRY_REGEX} |
| ... msg=${bmc_journald} doesn't contain rsyslog retry entries. |
| |
| |
| Verfiy BMC Journald Synced To Remote Logging Server |
| [Documentation] Check that BMC journald is sync to remote rsyslog. |
| [Tags] Verfiy_BMC_Journald_Synced_To_Remote_Logging_Server |
| |
| # Restart BMC dump service and get the last entry of the journald. |
| # Example: |
| # systemd[1]: Started Phosphor Dump Manager. |
| BMC Execute Command |
| ... systemctl restart xyz.openbmc_project.Dump.Manager.service |
| |
| ${bmc_journald} ${stderr} ${rc}= BMC Execute Command |
| ... journalctl --no-pager | grep 'Started Phosphor Dump Manager' |
| |
| # systemd[1]: Started Phosphor Dump Manager. |
| ${cmd}= Catenate SEPARATOR= egrep '${bmc_hostname}.*Started Phosphor Dump Manager' /var/log/syslog |
| ${remote_journald}= Remote Logging Server Execute Command command=${cmd} |
| |
| # TODO: rsyslog configuration and time date template to match BMC journald. |
| # Compare the BMC journlad log. Example: |
| # systemd[1]: Started Phosphor Dump Manager. |
| Should Contain ${remote_journald} ${bmc_journald.split('${bmc_hostname}')[1][0]} |
| ... msg= ${bmc_journald} doesn't match remote rsyslog:${remote_journald}. |
| |
| |
| Verify Journald Post BMC Reset |
| [Documentation] Check that BMC journald is sync'ed to remote rsyslog after |
| ... BMC reset. |
| [Tags] Verify_Journald_Post_BMC_Reset |
| |
| ${hostname} ${stderr} ${rc}= BMC Execute Command hostname |
| OBMC Reboot (off) |
| |
| ${cmd}= Catenate grep ${hostname} /var/log/syslog | |
| ... egrep '${BMC_STOP_MSG}|${BMC_START_MSG}|${BMC_BOOT_MSG}' |
| ${remote_journald}= Remote Logging Server Execute Command command=${cmd} |
| |
| # 1. Last reboot message to verify. |
| Should Contain ${remote_journald} ${BMC_STOP_MSG} |
| ... msg=The remote journald doesn't contain the IPMI shutdown message: ${BMC_STOP_MSG}. |
| |
| # 2. Earliest booting message on journald. |
| Should Contain ${remote_journald} ${BMC_START_MSG} |
| ... msg=The remote journald doesn't contain the start message: ${BMC_START_MSG}. |
| |
| # 3. Unique boot to standby message. |
| # Startup finished in 9.961s (kernel) + 1min 59.039s (userspace) = 2min 9.000s |
| ${bmc_journald} ${stderr} ${rc}= BMC Execute Command |
| ... journalctl -b --no-pager | egrep '${BMC_BOOT_MSG}' |
| |
| Should Contain ${remote_journald} |
| ... ${bmc_journald.split('${hostname}')[1]} |
| ... msg=The remote journald doesn't contain the boot message: ${BMC_BOOT_MSG}. |
| |
| |
| Verify BMC Journald Contains No Credential Data |
| [Documentation] Check that BMC journald doesn't log any credential data. |
| [Tags] Verify_BMC_Journald_Contains_No_Credential_Data |
| |
| Initialize OpenBMC |
| Update Root Password ${REST_PASSWORD} |
| |
| # Time for user manager to sync. |
| Sleep 5 s |
| |
| Open Connection And Log In |
| ${bmc_journald} ${stderr} ${rc}= BMC Execute Command |
| ... journalctl -o json-pretty | cat |
| |
| Should Not Contain Any ${bmc_journald} ${OPENBMC_PASSWORD} ${REST_PASSWORD} |
| ... msg=Journald logs BMC credentials/password ${OPENBMC_PASSWORD}. |
| |
| |
| Audit BMC SSH Login And Remote Logging |
| [Documentation] Check that the SSH login to BMC is logged and synced to |
| ... remote logging server. |
| [Tags] Audit_BMC_SSH_Login_And_Remote_Logging |
| |
| ${login_footprint}= Catenate Started SSH Per-Connection Server |
| # Example: Just get the message part of the syslog |
| # Started SSH Per-Connection Server (xx.xx.xx.xx:51292) |
| ${cmd}= Catenate SEPARATOR= --no-pager | egrep '${login_footprint}' |
| ... | awk -F': ' '{print $2}' |
| |
| Start Journal Log filter=${cmd} |
| Open Connection And Log In |
| Sleep 5s |
| ${bmc_journald}= Stop Journal Log |
| @{ssh_entry}= Split To Lines ${bmc_journald} |
| |
| ${cmd}= Catenate SEPARATOR= egrep -E '*${bmc_hostname}.*${login_footprint}' /var/log/syslog |
| |
| ${remote_journald}= Remote Logging Server Execute Command command=${cmd} |
| |
| Should Contain ${remote_journald} ${ssh_entry[0]} |
| ... msg=${remote_journald} don't contain ${bmc_journald} entry. |
| |
| |
| Verify Rsyslog Does Not Log On BMC |
| [Documentation] Check that rsyslog journald doesn't log on BMC. |
| [Tags] Verify_Rsyslog_Does_Not_Log_On_BMC |
| |
| # Expected filter rsyslog entries. |
| # Example: |
| # syslogd[3356]: [origin software="rsyslogd" swVersion="8.29.0" x-pid="3356" x-info="http://www.rsyslog.com"] exiting on signal 15. |
| # rsyslogd[3364]: [origin software="rsyslogd" swVersion="8.29.0" x-pid="3364" x-info="http://www.rsyslog.com"] start |
| ${bmc_journald} ${stderr} ${rc}= BMC Execute Command |
| ... journalctl -b --no-pager | egrep 'rsyslog' | egrep -Ev '${RSYSLOG_REGEX}|${RSYSLOG_RETRY_REGEX}' |
| ... ignore_err=${1} |
| |
| Should Be Empty ${bmc_journald} |
| ... msg=${bmc_journald} contains unexpected rsyslog entries. |
| |
| |
| Boot Host And Verify Data Is Synced To Remote Server |
| [Documentation] Boot host and verify the power on sequence logs are synced |
| ... to remote logging server. |
| [Tags] Boot_Host_And_Verify_Data_Is_Synced_To_Remote_Server |
| |
| # Filter kernel dmesg from the journald log. |
| # Example: xx.xx.xx kernel: |
| |
| ${openbmc_host_name} ${openbmc_ip} ${openbmc_short_name}= |
| ... Get Host Name IP host=${OPENBMC_HOST} short_name=1 |
| |
| ${cmd}= Catenate SEPARATOR= --no-pager | egrep -Ev '${BMC_SYSLOG_REGEX} |
| ... |${openbmc_short_name} kernel' | awk -F': ' '{print $2}' |
| |
| # Example: Just get the message part of the syslog |
| # Started OpenPOWER OCC Active Disable. |
| Start Journal Log filter=${cmd} |
| |
| # Irrespective of the outcome, the journald should be synced. |
| Run Keyword And Ignore Error REST Power On |
| ${bmc_journald}= Stop Journal Log |
| |
| ${cmd}= Catenate SEPARATOR= egrep '${bmc_hostname}' /var/log/syslog |
| ${remote_journald}= Remote Logging Server Execute Command command=${cmd} |
| |
| @{lines}= Split To Lines ${bmc_journald} |
| |
| :FOR ${line} IN @{lines} |
| \ Log To Console \n ${line} |
| \ Should Contain ${remote_journald} ${line} |
| ... mgs=${line} line doesn't contain in ${remote_journald}. |
| |
| |
| *** Keywords *** |
| |
| Suite Setup Execution |
| [Documentation] Do the suite setup. |
| |
| Should Not Be Empty ${REMOTE_LOG_SERVER_HOST} |
| Should Not Be Empty ${REMOTE_LOG_SERVER_PORT} |
| Should Not Be Empty ${REMOTE_USERNAME} |
| Should Not Be Empty ${REMOTE_PASSWORD} |
| Ping Host ${REMOTE_LOG_SERVER_HOST} |
| Remote Logging Server Execute Command true |
| Remote Logging Interface Should Exist |
| |
| ${hostname} ${stderr} ${rc}= BMC Execute Command /bin/hostname |
| Set Suite Variable ${bmc_hostname} ${hostname} |
| Configure Remote Log Server With Parameters |
| |
| |
| Test Setup Execution |
| [Documentation] Do the test setup. |
| |
| # Retain only the past 1 second log: |
| BMC Execute Command journalctl --vacuum-time=1s |
| |
| ${config_status}= Run Keyword And Return Status |
| ... Get Remote Log Server Configured |
| |
| Run Keyword If ${config_status}==${FALSE} |
| ... Configure Remote Log Server With Parameters |
| |
| ${ActiveState}= Get Service Attribute ActiveState rsyslog.service |
| Should Be Equal active ${ActiveState} |
| ... msg=rsyslog logging service not in active state. |
| |
| |
| Remote Logging Interface Should Exist |
| [Documentation] Check that the remote logging URI exist. |
| |
| ${resp}= OpenBMC Get Request ${REMOTE_LOGGING_URI} |
| Should Be Equal As Strings ${resp.status_code} ${HTTP_OK} |
| |
| |
| Verify Configure Remote Logging Server |
| [Documentation] Configure the remote logging REST interface on BMC. |
| [Arguments] ${remote_host} ${remote_port} ${expectation} |
| |
| # Description of argument(s): |
| # remote_host The host name or IP address of the remote logging server |
| # (e.g. "xx.xx.xx.xx"). |
| # remote_port Remote ryslog server port number (e.g. "514"). |
| # expectation Expect boolean True/False. |
| |
| |
| ${status}= Run Keyword And Return Status |
| ... Configure Remote Log Server With Parameters remote_host=${remote_host} remote_port=${remote_port} |
| |
| Should Be Equal ${status} ${expectation} |
| ... msg=Test result ${status} and expectation ${expectation} do not match. |