| *** Settings *** |
| |
| Documentation Utilities for Robot keywords that do not use REST. |
| |
| Resource ../lib/resource.robot |
| Resource ../lib/connection_client.robot |
| Resource ../lib/boot_utils.robot |
| Library String |
| Library DateTime |
| Library Process |
| Library OperatingSystem |
| Library gen_print.py |
| Library gen_robot_print.py |
| Library gen_cmd.py |
| Library gen_robot_valid.py |
| Library gen_robot_keyword.py |
| Library bmc_ssh_utils.py |
| Library utils.py |
| Library var_funcs.py |
| Library SCPLibrary WITH NAME scp |
| |
| *** Variables *** |
| |
| ${pflash_cmd} /usr/sbin/pflash -r /dev/stdout -P VERSION |
| |
| ${dbuscmdBase} |
| ... dbus-send --system --print-reply --dest=${OPENBMC_BASE_DBUS}.settings.Host |
| ${dbuscmdGet} |
| ... ${SETTINGS_URI}host0 org.freedesktop.DBus.Properties.Get |
| ${dbuscmdString}= string:"xyz.openbmc_project.settings.Host" string: |
| |
| # Assign default value to QUIET for programs which may not define it. |
| ${QUIET} ${0} |
| |
| ${bmc_mem_free_cmd}= free | tr -s ' ' | sed '/^Mem/!d' | cut -d" " -f4 |
| ${bmc_mem_total_cmd}= free | tr -s ' ' | sed '/^Mem/!d' | cut -d" " -f2 |
| ${bmc_cpu_usage_cmd}= top -n 1 | grep CPU: | cut -c 7-9 |
| ${HOST_SETTING} ${SETTINGS_URI}host0 |
| |
| # /run/initramfs/ro associate filesystem should be 100% full always |
| ${bmc_file_system_usage_cmd}= df -h | cut -c 52-54 | grep 100 | wc -l |
| ${total_pnor_ro_file_system_cmd}= df -h | grep /media/pnor-ro | wc -l |
| ${total_bmc_ro_file_system_cmd}= df -h | grep /media/rofs | wc -l |
| |
| ${BOOT_TIME} ${0} |
| ${BOOT_COUNT} ${0} |
| ${count} ${0} |
| ${devicetree_base} /sys/firmware/devicetree/base/model |
| |
| # Initialize default debug value to 0. |
| ${DEBUG} ${0} |
| |
| ${probe_cpu_tool_path} ${EXECDIR}/tools/ras/probe_cpus.sh |
| ${scom_addrs_tool_path} ${EXECDIR}/tools/ras/scom_addr_p9.sh |
| ${target_file_path} /root/ |
| |
| ${default_tarball} ${EXECDIR}/obmc-phosphor-debug-tarball-witherspoon.tar.xz |
| |
| # These variables are used to straddle between new and old methods of setting |
| # values. |
| ${bmc_power_policy_method} ${EMPTY} |
| @{valid_power_policy_vars} RESTORE_LAST_STATE ALWAYS_POWER_ON |
| ... ALWAYS_POWER_OFF |
| |
| ${check_performance} ${1} |
| |
| |
| *** Keywords *** |
| |
| Check BMC Performance |
| [Documentation] Check BMC basic CPU Mem File system performance. |
| |
| Return From Keyword If not ${check_performance} |
| |
| Check BMC CPU Performance |
| Check BMC Mem Performance |
| Check BMC File System Performance |
| |
| |
| Verify PNOR Update |
| [Documentation] Verify that the PNOR is not corrupted. |
| # Example: |
| # FFS: Flash header not found. Code: 100 |
| # Error 100 opening ffs ! |
| |
| ${stdout} ${stderr} ${rc}= |
| ... BMC Execute Command /usr/sbin/pflash -h | egrep -q skip |
| ... ignore_err=${1} |
| ${pflash_cmd}= Set Variable If ${rc} == ${0} ${pflash_cmd} --skip=4096 |
| ... ${pflash_cmd} |
| ${pnor_info}= BMC Execute Command ${pflash_cmd} |
| Should Not Contain Any ${pnor_info} Flash header not found Error |
| |
| |
| Get BMC System Model |
| [Documentation] Get the BMC model from the device tree and return it. |
| |
| ${bmc_model} ${stderr} ${rc}= BMC Execute Command |
| ... cat ${devicetree_base} | cut -d " " -f 1 return_stderr=True |
| ... test_mode=0 |
| Should Be Empty ${stderr} |
| Should Not Be Empty ${bmc_model} msg=BMC model is empty. |
| [Return] ${bmc_model} |
| |
| |
| Verify BMC System Model |
| [Documentation] Verify the BMC model with ${OPENBMC_MODEL}. |
| [Arguments] ${bmc_model} |
| |
| # Description of argument(s): |
| # bmc_model System model (e.g. "witherspoon"). |
| |
| ${tmp_bmc_model}= Fetch From Right ${OPENBMC_MODEL} / |
| ${tmp_bmc_model}= Fetch From Left ${tmp_bmc_model} . |
| ${ret}= Run Keyword And Return Status Should Contain ${bmc_model} |
| ... ${tmp_bmc_model} ignore_case=True |
| [Return] ${ret} |
| |
| |
| Wait For Host To Ping |
| [Documentation] Wait for the given host to ping. |
| [Arguments] ${host} ${timeout}=${OPENBMC_REBOOT_TIMEOUT}min |
| ... ${interval}=5 sec ${expected_rc}=${0} |
| |
| # Description of argument(s): |
| # host The host name or IP of the host to ping. |
| # timeout The amount of time after which ping attempts cease. |
| # This should be expressed in Robot Framework's time format |
| # (e.g. "10 seconds"). |
| # interval The amount of time in between attempts to ping. |
| # This should be expressed in Robot Framework's time format |
| # (e.g. "5 seconds"). |
| # expected_rc Expected return code of ping command. |
| |
| Wait Until Keyword Succeeds ${timeout} ${interval} Ping Host ${host} ${expected_rc} |
| |
| |
| Ping Host |
| [Documentation] Ping the given host. |
| [Arguments] ${host} ${expected_rc}=${0} |
| |
| # Description of argument(s): |
| # host The host name or IP of the host to ping. |
| # expected_rc Expected return code of ping command. |
| |
| Should Not Be Empty ${host} msg=No host provided |
| ${rc} ${output}= Run and return RC and Output ping -c 4 ${host} |
| Log RC: ${rc}\nOutput:\n${output} |
| Should be equal ${rc} ${expected_rc} |
| |
| |
| Check OS |
| [Documentation] Attempts to ping the host OS and then checks that the host |
| ... OS is up by running an SSH command. |
| |
| [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME} |
| ... ${os_password}=${OS_PASSWORD} ${quiet}=${QUIET} |
| ... ${print_string}=${EMPTY} |
| [Teardown] SSHLibrary.Close Connection |
| |
| # Description of argument(s): |
| # os_host The DNS name/IP of the OS host associated with our BMC. |
| # os_username The username to be used to sign on to the OS host. |
| # os_password The password to be used to sign on to the OS host. |
| # quiet Indicates whether this keyword should write to console. |
| # print_string A string to be printed before checking the OS. |
| |
| Log To Console ${print_string} no_newline=True |
| |
| # Attempt to ping the OS. Store the return code to check later. |
| ${ping_rc}= Run Keyword and Return Status Ping Host ${os_host} |
| |
| SSHLibrary.Open connection ${os_host} |
| |
| ${status} ${msg}= Run Keyword And Ignore Error SSHLibrary.Login ${os_username} |
| ... ${os_password} |
| ${err_msg1}= Sprint Error ${msg} |
| ${err_msg}= Catenate SEPARATOR= \n ${err_msg1} |
| Run Keyword If '${status}' == 'FAIL' Fail msg=${err_msg} |
| ${output} ${stderr} ${rc}= Execute Command uptime return_stderr=True |
| ... return_rc=True |
| |
| ${temp_msg}= Catenate Could not execute a command on the operating |
| ... system.\n |
| ${err_msg1}= Sprint Error ${temp_msg} |
| ${err_msg}= Catenate SEPARATOR= \n ${err_msg1} |
| |
| # If the return code returned by "Execute Command" is non-zero, this |
| # keyword will fail. |
| Should Be Equal ${rc} ${0} msg=${err_msg} |
| # We will likewise fail if there is any stderr data. |
| Should Be Empty ${stderr} |
| |
| ${temp_msg}= Set Variable Could not ping the operating system.\n |
| ${err_msg1}= Sprint Error ${temp_msg} |
| ${err_msg}= Catenate SEPARATOR= \n ${err_msg1} |
| # We will likewise fail if the OS did not ping, as we could SSH but not |
| # ping |
| Should Be Equal As Strings ${ping_rc} ${TRUE} msg=${err_msg} |
| |
| |
| Wait for OS |
| [Documentation] Waits for the host OS to come up via calls to "Check OS". |
| [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME} |
| ... ${os_password}=${OS_PASSWORD} ${timeout}=${OS_WAIT_TIMEOUT} |
| ... ${quiet}=${0} |
| [Teardown] Printn |
| |
| # Description of argument(s): |
| # os_host The DNS name or IP of the OS host associated with our |
| # BMC. |
| # os_username The username to be used to sign on to the OS host. |
| # os_password The password to be used to sign on to the OS host. |
| # timeout The timeout in seconds indicating how long you're |
| # willing to wait for the OS to respond. |
| # quiet Indicates whether this keyword should write to console. |
| |
| # The interval to be used between calls to "Check OS". |
| ${interval}= Set Variable 5 |
| |
| ${message}= Catenate Checking every ${interval} seconds for up to |
| ... ${timeout} seconds for the operating system to communicate. |
| Qprint Timen ${message} |
| |
| Wait Until Keyword Succeeds ${timeout} sec ${interval} Check OS |
| ... ${os_host} ${os_username} ${os_password} |
| ... print_string=\# |
| |
| Qprintn |
| |
| Qprint Timen The operating system is now communicating. |
| |
| |
| Copy PNOR to BMC |
| [Documentation] Copy the PNOR image to the BMC. |
| Import Library SCPLibrary WITH NAME scp |
| Open Connection for SCP |
| Log Copying ${PNOR_IMAGE_PATH} to /tmp |
| scp.Put File ${PNOR_IMAGE_PATH} /tmp |
| |
| |
| Is OS Starting |
| [Documentation] Check if boot progress is OS starting. |
| ${boot_progress}= Get Boot Progress |
| Should Be Equal ${boot_progress} OSStart |
| |
| |
| Is OS Off |
| [Documentation] Check if boot progress is "Off". |
| ${boot_progress}= Get Boot Progress |
| Should Be Equal ${boot_progress} Off |
| |
| |
| Get Boot Progress To OS Starting State |
| [Documentation] Get the system to a boot progress state of 'FW Progress, |
| ... Starting OS'. |
| |
| ${boot_progress}= Get Boot Progress |
| Run Keyword If '${boot_progress}' == 'OSStart' |
| ... Log Host is already in OS starting state |
| ... ELSE |
| ... Run Keywords Initiate Host PowerOff AND Initiate Host Boot |
| ... AND Wait Until Keyword Succeeds 10 min 10 sec Is OS Starting |
| |
| |
| Check If warmReset is Initiated |
| [Documentation] Ping would be still alive, so try SSH to connect |
| ... if fails the ports are down indicating reboot |
| ... is in progress |
| |
| # Warm reset adds 3 seconds delay before forcing reboot |
| # To minimize race conditions, we wait for 7 seconds |
| Sleep 7s |
| ${alive}= Run Keyword and Return Status |
| ... Open Connection And Log In |
| Return From Keyword If '${alive}' == '${False}' ${False} |
| [Return] ${True} |
| |
| |
| Initialize DBUS cmd |
| [Documentation] Initialize dbus string with property string to extract |
| [Arguments] ${boot_property} |
| |
| # Description of argument(s): |
| # boot_property Property string. |
| |
| ${cmd}= Catenate ${dbuscmdBase} ${dbuscmdGet} ${dbuscmdString} |
| ${cmd}= Catenate ${cmd}${boot_property} |
| Set Global Variable ${dbuscmd} ${cmd} |
| |
| |
| Create OS Console Command String |
| [Documentation] Return a command string to start OS console logging. |
| |
| # First make sure that the ssh_pw program is available. |
| ${cmd}= Catenate which ssh_pw 2>/dev/null || find |
| ... ${EXECDIR} -name 'ssh_pw' |
| |
| Dprint Issuing ${cmd} |
| ${rc} ${output}= Run And Return Rc And Output ${cmd} |
| Rdpvars rc output |
| |
| Should Be Equal As Integers 0 ${rc} msg=Could not find ssh_pw. |
| |
| ${ssh_pw_file_path}= Set Variable ${output} |
| |
| ${cmd}= Catenate ${ssh_pw_file_path} ${OPENBMC_PASSWORD} -p ${HOST_SOL_PORT} |
| ... -o "StrictHostKeyChecking no" ${OPENBMC_USERNAME}@${OPENBMC_HOST} ${OPENBMC_CONSOLE_CLIENT} |
| |
| [Return] ${cmd.strip()} |
| |
| |
| Get SOL Console Pid |
| [Documentation] Get the pid of the active SOL console job. |
| [Arguments] ${expect_running}=${0} ${log_file_path}=${EMPTY} |
| |
| # Description of argument(s): |
| # expect_running If set and if no SOL console job is found, print debug info and fail. |
| # log_file_path Needed to print debug info if expect_running is set and no pid is found. |
| |
| # Find the pid of the active system console logging session (if any). |
| ${search_string}= Create OS Console Command String |
| # At least in some cases, ps output does not show double quotes so we must |
| # replace them in our search string with the regexes to indicate that they |
| # are optional. |
| ${search_string}= Replace String ${search_string} " ["]? |
| ${ps_cmd}= Catenate ps axwwo user,pid,cmd |
| ${cmd_buf}= Catenate echo $(${ps_cmd} | egrep '${search_string}' | |
| ... egrep -v grep | cut -c10-14) |
| Dprint Issuing ${cmd_buf} |
| ${rc} ${os_con_pid}= Run And Return Rc And Output ${cmd_buf} |
| Rdpvars os_con_pid |
| # If rc is not zero it just means that there is no OS Console process |
| # running. |
| |
| Return From Keyword If '${os_con_pid}' != '${EMPTY}' ${os_con_pid} |
| Return From Keyword If '${expect_running}' == '${0}' ${os_con_pid} |
| |
| Cmd Fnc cat ${log_file_path} ; echo ; ${ps_cmd} quiet=${0} |
| ... print_output=${1} show_err=${1} |
| Valid Value os_con_pid |
| |
| |
| Stop SOL Console Logging |
| [Documentation] Stop system console logging and return log output. |
| [Arguments] ${log_file_path}=${EMPTY} |
| ... ${targ_file_path}=${EXECDIR}${/}logs${/} |
| ... ${return_data}=${1} |
| |
| # If there are multiple system console processes, they will all be stopped. |
| # If there is no existing log file this keyword will return an error |
| # message to that effect (and write that message to targ_file_path, if |
| # specified). |
| # NOTE: This keyword will not fail if there is no running system console |
| # process. |
| |
| # Description of arguments: |
| # log_file_path The file path that was used to call "Start SOL |
| # Console Logging". See that keyword (above) for details. |
| # targ_file_path If specified, the file path to which the source |
| # file path (i.e. "log_file_path") should be copied. |
| # return_data If this is set to ${1}, this keyword will return the SOL |
| # data to the caller as a unicode string. |
| |
| ${log_file_path}= Create OS Console File Path ${log_file_path} |
| |
| ${os_con_pid}= Get SOL Console Pid |
| |
| ${cmd_buf}= Catenate kill -9 ${os_con_pid} |
| Run Keyword If '${os_con_pid}' != '${EMPTY}' Dprint Issuing ${cmd_buf} |
| ${rc} ${output}= Run Keyword If '${os_con_pid}' != '${EMPTY}' |
| ... Run And Return Rc And Output ${cmd_buf} |
| Run Keyword If '${os_con_pid}' != '${EMPTY}' Rdpvars rc output |
| |
| Run Keyword If '${targ_file_path}' != '${EMPTY}' |
| ... Run Keyword And Ignore Error |
| ... Copy File ${log_file_path} ${targ_file_path} |
| |
| ${output}= Set Variable ${EMPTY} |
| ${loc_quiet}= Evaluate ${debug}^1 |
| ${rc} ${output}= Run Keyword If '${return_data}' == '${1}' |
| ... Cmd Fnc cat ${log_file_path} 2>/dev/null quiet=${loc_quiet} |
| ... print_output=${0} show_err=${0} |
| |
| [Return] ${output} |
| |
| |
| Start SOL Console Logging |
| [Documentation] Start system console log to file. |
| [Arguments] ${log_file_path}=${EMPTY} ${return_data}=${1} |
| |
| # This keyword will first call "Stop SOL Console Logging". Only then will |
| # it start SOL console logging. The data returned by "Stop SOL Console |
| # Logging" will in turn be returned by this keyword. |
| |
| # Description of arguments: |
| # log_file_path The file path to which system console log data should be |
| # written. Note that this path is taken to be a location |
| # on the machine where this program is running rather than |
| # on the Open BMC system. |
| # return_data If this is set to ${1}, this keyword will return any SOL |
| # data to the caller as a unicode string. |
| |
| ${log_file_path}= Create OS Console File Path ${log_file_path} |
| |
| ${log_output}= Stop SOL Console Logging ${log_file_path} |
| ... return_data=${return_data} |
| |
| # Validate by making sure we can create the file. Problems creating the |
| # file would not be noticed by the subsequent ssh command because we fork |
| # the command. |
| Create File ${log_file_path} |
| ${sub_cmd_buf}= Create OS Console Command String |
| # Routing stderr to stdout so that any startup error text will go to the |
| # output file. |
| ${cmd_buf}= Catenate ${sub_cmd_buf} > ${log_file_path} 2>&1 & |
| Dprint Issuing ${cmd_buf} |
| ${rc} ${output}= Run And Return Rc And Output ${cmd_buf} |
| # Because we are forking this command, we essentially will never get a |
| # non-zero return code or any output. |
| Should Be Equal ${rc} ${0} |
| |
| Wait Until Keyword Succeeds 10 seconds 0 seconds |
| ... Get SOL Console Pid ${1} ${log_file_path} |
| |
| [Return] ${log_output} |
| |
| |
| Get Time Stamp |
| [Documentation] Get the current time stamp data |
| ${cur_time}= Get Current Date result_format=%Y%m%d%H%M%S%f |
| [Return] ${cur_time} |
| |
| |
| Start Journal Log |
| [Documentation] Start capturing journal log to a file in /tmp using |
| ... journalctl command. By default journal log is collected |
| ... at /tmp/journal_log else user input location. |
| ... The File is appended with datetime. |
| [Arguments] ${file_path}=/tmp/journal_log ${filter}=${EMPTY} |
| |
| # Description of arguments: |
| # file_path The file path of the journal file. |
| |
| ${cur_time}= Get Time Stamp |
| Set Global Variable ${LOG_TIME} ${cur_time} |
| Open Connection And Log In |
| Start Command |
| ... journalctl -f ${filter} > ${file_path}-${LOG_TIME} |
| Log Journal Log Started: ${file_path}-${LOG_TIME} |
| |
| |
| Stop Journal Log |
| [Documentation] Stop journalctl process if its running. |
| ... By default return log from /tmp/journal_log else |
| ... user input location. |
| [Arguments] ${file_path}=/tmp/journal_log |
| |
| # Description of arguments: |
| # file_path The file path of the journal file. |
| |
| Open Connection And Log In |
| |
| ${rc}= |
| ... Execute Command |
| ... ps | grep journalctl | grep -v grep |
| ... return_stdout=False return_rc=True |
| |
| Return From Keyword If '${rc}' == '${1}' |
| ... No journal log process running |
| |
| ${output} ${stderr}= |
| ... Execute Command killall journalctl |
| ... return_stderr=True |
| Should Be Empty ${stderr} |
| |
| ${journal_log} ${stderr}= |
| ... Execute Command |
| ... cat ${file_path}-${LOG_TIME} |
| ... return_stderr=True |
| Should Be Empty ${stderr} |
| |
| Log ${journal_log} |
| |
| Execute Command rm ${file_path}-${LOG_TIME} |
| |
| [Return] ${journal_log} |
| |
| |
| Mac Address To Hex String |
| [Documentation] Converts MAC address into hex format. |
| ... Example |
| ... Given the following MAC: 00:01:6C:80:02:78 |
| ... This keyword will return: 0x00 0x01 0x6C 0x80 0x02 0x78 |
| ... Description of arguments: |
| ... i_macaddress MAC address in the following format |
| ... 00:01:6C:80:02:78 |
| [Arguments] ${i_macaddress} |
| |
| # Description of arguments: |
| # i_macaddress The MAC address. |
| |
| ${mac_hex}= Catenate 0x${i_macaddress.replace(':', ' 0x')} |
| [Return] ${mac_hex} |
| |
| |
| IP Address To Hex String |
| [Documentation] Converts IP address into hex format. |
| ... Example: |
| ... Given the following IP: 10.3.164.100 |
| ... This keyword will return: 0xa 0x3 0xa4 0xa0 |
| [Arguments] ${i_ipaddress} |
| |
| # Description of arguments: |
| # i_macaddress The IP address in the format 10.10.10.10. |
| |
| @{ip}= Split String ${i_ipaddress} . |
| ${index}= Set Variable ${0} |
| |
| FOR ${item} IN @{ip} |
| ${hex}= Convert To Hex ${item} prefix=0x lowercase=yes |
| Set List Value ${ip} ${index} ${hex} |
| ${index}= Set Variable ${index + 1} |
| END |
| ${ip_hex}= Catenate @{ip} |
| |
| [Return] ${ip_hex} |
| |
| |
| BMC CPU Performance Check |
| [Documentation] Minimal 10% of proc should be free in this instance |
| |
| ${bmc_cpu_usage_output} ${stderr} ${rc}= BMC Execute Command |
| ... ${bmc_cpu_usage_cmd} |
| ${bmc_cpu_usage_output} ${stderr} ${rc}= BMC Execute Command |
| ... ${bmc_cpu_usage_cmd} |
| ${bmc_cpu_percentage}= Fetch From Left ${bmc_cpu_usage_output} % |
| Rprint Vars bmc_cpu_percentage |
| Should be true ${bmc_cpu_percentage} < 90 |
| |
| |
| BMC Mem Performance Check |
| [Documentation] Minimal 10% of memory should be free in this instance |
| |
| ${bmc_mem_free_output} ${stderr} ${rc}= BMC Execute Command |
| ... ${bmc_mem_free_cmd} |
| |
| ${bmc_mem_total_output} ${stderr} ${rc}= BMC Execute Command |
| ... ${bmc_mem_total_cmd} |
| ${bmc_mem_free_output} ${stderr} ${rc}= BMC Execute Command |
| ... ${bmc_mem_free_cmd} |
| |
| ${bmc_mem_total_output} ${stderr} ${rc}= BMC Execute Command |
| ... ${bmc_mem_total_cmd} |
| |
| ${bmc_mem_percentage}= Evaluate ${bmc_mem_free_output}*100 |
| ${bmc_mem_percentage}= Evaluate |
| ... ${bmc_mem_percentage}/${bmc_mem_total_output} |
| Rprint Vars bmc_mem_percentage |
| Should be true ${bmc_mem_percentage} > 10 |
| |
| |
| BMC File System Usage Check |
| [Documentation] Check the file system space. 4 file system should be |
| ... 100% full which is expected |
| # Filesystem Size Used Available Use% Mounted on |
| # /dev/root 14.4M 14.4M 0 100% / |
| # /dev/ubiblock0_0 14.4M 14.4M 0 100% /media/rofs-c9249b0e |
| # /dev/ubiblock8_0 19.6M 19.6M 0 100% /media/pnor-ro-8764baa3 |
| # /dev/ubiblock4_0 14.4M 14.4M 0 100% /media/rofs-407816c |
| # /dev/ubiblock8_4 21.1M 21.1M 0 100% /media/pnor-ro-cecc64c4 |
| ${bmc_fs_usage_output} ${stderr} ${rc}= BMC Execute Command |
| ... ${bmc_file_system_usage_cmd} |
| ${bmc_pnor_fs_usage_output} ${stderr} ${rc}= BMC Execute Command |
| ... ${total_pnor_ro_file_system_cmd} |
| ${bmc_bmc_fs_usage_output} ${stderr} ${rc}= BMC Execute Command |
| ... ${total_bmc_ro_file_system_cmd} |
| ${total_bmc_pnor_image}= Evaluate |
| ... ${bmc_pnor_fs_usage_output}+${bmc_bmc_fs_usage_output} |
| # Considering /dev/root also in total 100% used file system |
| ${total_full_fs}= Evaluate ${total_bmc_pnor_image}+1 |
| Rprint Vars bmc_fs_usage_output bmc_pnor_fs_usage_output bmc_bmc_fs_usage_output |
| ... total_full_fs |
| Should Be True ${bmc_fs_usage_output}==${total_full_fs} |
| |
| |
| Check BMC CPU Performance |
| [Documentation] Minimal 10% of proc should be free in 3 sample |
| FOR ${var} IN RANGE 1 4 |
| BMC CPU Performance Check |
| END |
| |
| Check BMC Mem Performance |
| [Documentation] Minimal 10% of memory should be free |
| |
| FOR ${var} IN RANGE 1 4 |
| BMC Mem Performance Check |
| END |
| |
| Check BMC File System Performance |
| [Documentation] Check for file system usage for 4 times |
| |
| FOR ${var} IN RANGE 1 4 |
| BMC File System Usage check |
| END |
| |
| Get URL List |
| [Documentation] Return list of URLs under given URL. |
| [Arguments] ${openbmc_url} |
| |
| # Description of argument(s): |
| # openbmc_url URL for list operation (e.g. |
| # /xyz/openbmc_project/inventory). |
| |
| ${url_list}= Read Properties ${openbmc_url}list quiet=${1} |
| Sort List ${url_list} |
| |
| [Return] ${url_list} |
| |
| |
| Check Zombie Process |
| [Documentation] Check if any defunct process exist or not on BMC |
| ${count} ${stderr} ${rc}= Execute Command ps -o stat | grep Z | wc -l |
| ... return_stderr=True return_rc=True |
| Should Be True ${count}==0 |
| Should Be Empty ${stderr} |
| |
| |
| Prune Journal Log |
| [Documentation] Prune archived journal logs. |
| [Arguments] ${vacuum_size}=1M |
| |
| # This keyword can be used to prevent the journal |
| # log from filling up the /run filesystem. |
| # This command will retain only the latest logs |
| # of the user specified size. |
| |
| # Description of argument(s): |
| # vacuum_size Size of journal. |
| |
| Open Connection And Log In |
| ${output} ${stderr} ${rc}= |
| ... Execute Command |
| ... journalctl --vacuum-size=${vacuum_size} |
| ... return_stderr=True return_rc=True |
| |
| Should Be Equal ${rc} ${0} msg=${stderr} |
| |
| |
| Get System Power Policy |
| [Documentation] Returns the BMC power policy. |
| |
| # Set the bmc_power_policy_method to either 'Old' or 'New'. |
| Set Power Policy Method |
| ${cmd_buf}= Create List ${bmc_power_policy_method} Get Power Policy |
| # Run the appropriate keyword. |
| ${currentPolicy}= Run Keyword @{cmd_buf} |
| |
| [Return] ${currentPolicy} |
| |
| |
| Set BMC Reset Reference Time |
| [Documentation] Set current boot time as a reference and increment |
| ... boot count. |
| |
| ${cur_btime}= Get BMC Boot Time |
| Run Keyword If ${BOOT_TIME} == ${0} and ${BOOT_COUNT} == ${0} |
| ... Set Global Variable ${BOOT_TIME} ${cur_btime} |
| ... ELSE IF ${cur_btime} > ${BOOT_TIME} |
| ... Run Keywords Set Global Variable ${BOOT_TIME} ${cur_btime} |
| ... AND |
| ... Set Global Variable ${BOOT_COUNT} ${BOOT_COUNT + 1} |
| |
| |
| Get BMC Boot Time |
| [Documentation] Returns boot time from /proc/stat. |
| |
| Open Connection And Log In |
| ${output} ${stderr}= |
| ... Execute Command egrep '^btime ' /proc/stat | cut -f 2 -d ' ' |
| ... return_stderr=True |
| Should Be Empty ${stderr} |
| ${btime}= Convert To Integer ${output} |
| [Return] ${btime} |
| |
| |
| Enable Core Dump On BMC |
| [Documentation] Enable core dump collection. |
| ${core_pattern} ${stderr} ${rc}= BMC Execute Command |
| ... echo '/tmp/core_%e.%p' | tee /proc/sys/kernel/core_pattern |
| Should Be Equal As Strings ${core_pattern} /tmp/core_%e.%p |
| |
| |
| Get Number Of BMC Core Dump Files |
| [Documentation] Returns number of core dump files on BMC. |
| Open Connection And Log In |
| ${num_of_core_dump}= Execute Command |
| ... ls /tmp/core* 2>/dev/null | wc -l |
| [Return] ${num_of_core_dump} |
| |
| |
| Set Core Dump File Size Unlimited |
| [Documentation] Set core dump file size to unlimited. |
| BMC Execute Command ulimit -c unlimited |
| |
| |
| Check For Core Dumps |
| [Documentation] Check for any core dumps exist. |
| ${output}= Get Number Of BMC Core Dump Files |
| Run Keyword If ${output} > 0 |
| ... Log **Warning** BMC core dump files exist level=WARN |
| |
| |
| Configure Initial Settings |
| [Documentation] Restore old IP and route. |
| ... This keyword requires initial settings viz IP address, |
| ... Network Mask, default gateway and serial console IP and port |
| ... information which should be provided in command line. |
| |
| [Arguments] ${host}=${OPENBMC_HOST} ${mask}=${NET_MASK} |
| ... ${gw_ip}=${GW_IP} |
| |
| # Description of arguments: |
| # host IP address of the OS Host. |
| # mask Network mask. |
| # gu_ip Gateway IP address or hostname. |
| |
| # Open telnet connection and ignore the error, in case telnet session is |
| # already opened by the program calling this keyword. |
| Run Keyword And Ignore Error Open Telnet Connection to BMC Serial Console |
| Telnet.write ifconfig eth0 ${host} netmask ${mask} |
| Telnet.write route add default gw ${gw_ip} |
| |
| |
| Install Debug Tarball On BMC |
| [Documentation] Copy the debug tar file to BMC and install. |
| [Arguments] ${tarball_file_path}=${default_tarball} |
| ... ${targ_tarball_dir_path}=/tmp/tarball/ |
| |
| # Description of arguments: |
| # tarball_file_path Path of the debug tarball file. |
| # The tar file is downloaded from the build page |
| # https://jenkins.openbmc.org/job/latest-master/ |
| # obmc-phosphor-debug-tarball-witherspoon.tar.xz |
| # |
| # targ_tarball_dir_path The directory path where the tarball is to be |
| # installed. |
| |
| OperatingSystem.File Should Exist ${tarball_file_path} |
| ... msg=${tarball_file_path} doesn't exist. |
| |
| # Upload the file to BMC. |
| Import Library SCPLibrary WITH NAME scp |
| Open Connection for SCP |
| scp.Put File ${tarball_file_path} /tmp/debug-tarball.tar.xz |
| |
| # Create tarball directory and install. |
| BMC Execute Command mkdir -p ${targ_tarball_dir_path} |
| BMC Execute Command |
| ... tar -xf /tmp/debug-tarball.tar.xz -C ${targ_tarball_dir_path} |
| |
| # Remove the tarball file from BMC. |
| BMC Execute Command rm -f /tmp/debug-tarball.tar.xz |
| |
| |
| Get BMC Boot Count |
| [Documentation] Returns BMC boot count based on boot time. |
| ${cur_btime}= Get BMC Boot Time |
| |
| # Set global variable BOOT_TIME to current boot time if current boot time |
| # is changed. Also increase value of global variable BOOT_COUNT by 1. |
| Run Keyword If ${cur_btime} > ${BOOT_TIME} |
| ... Run Keywords Set Global Variable ${BOOT_TIME} ${cur_btime} |
| ... AND |
| ... Set Global Variable ${BOOT_COUNT} ${BOOT_COUNT + 1} |
| |
| [Return] ${BOOT_COUNT} |
| |
| |
| Set BMC Boot Count |
| [Documentation] Set BMC boot count to given value. |
| [Arguments] ${count} |
| |
| # Description of arguments: |
| # count boot count value. |
| ${cur_btime}= Get BMC Boot Time |
| |
| # Set global variable BOOT_COUNT to given value. |
| Set Global Variable ${BOOT_COUNT} ${count} |
| |
| # Set BOOT_TIME variable to current boot time. |
| Set Global Variable ${BOOT_TIME} ${cur_btime} |
| |
| |
| Delete Error Log Entry |
| [Documentation] Delete error log entry. |
| [Arguments] ${entry_path} ${quiet}=${0} |
| |
| # Description of argument(s): |
| # quiet If enabled, turns off logging to console. |
| # entry_path Delete an error log entry. |
| # Ex. /xyz/openbmc_project/logging/entry/1 |
| |
| # Skip delete if entry URI is a callout. |
| # Examples: |
| # /xyz/openbmc_project/logging/entry/1/callout |
| # /xyz/openbmc_project/logging/entry/1/callouts/0 |
| ${callout_entry}= Run Keyword And Return Status |
| ... Should Match Regexp ${entry_path} /callout[s]?(/|$) |
| Return From Keyword If ${callout_entry} |
| |
| ${data}= Create Dictionary data=@{EMPTY} |
| ${resp}= Openbmc Delete Request ${entry_path} data=${data} quiet=${quiet} |
| Should Be Equal As Strings ${resp.status_code} ${HTTP_OK} |
| |
| |
| Get BMC Version |
| [Documentation] Returns BMC version from /etc/os-release. |
| ... e.g. "v1.99.6-141-ge662190" |
| |
| ${cmd}= Set Variable grep ^VERSION_ID= /etc/os-release | cut -f 2 -d '=' |
| ${output} ${stderr} ${rc}= BMC Execute Command ${cmd} |
| [Return] ${output} |
| |
| |
| Get PNOR Version |
| [Documentation] Returns the PNOR version from the BMC. |
| |
| ${pnor_attrs}= Get PNOR Attributes |
| [Return] ${pnor_attrs['version']} |
| |
| |
| Get PNOR Attributes |
| [Documentation] Return PNOR software attributes as a dictionary. |
| |
| # This keyword parses /var/lib/phosphor-software-manager/pnor/ro/pnor.toc |
| # into key/value pairs. |
| |
| ${outbuf} ${stderr} ${rc}= BMC Execute Command |
| ... cat /var/lib/phosphor-software-manager/pnor/ro/pnor.toc |
| ${pnor_attrs}= Key Value Outbuf To Dict ${outbuf} delim== |
| |
| [Return] ${pnor_attrs} |
| |
| |
| GET BMC PNOR Version |
| [Documentation] Return BMC & PNOR version from openbmc shell. |
| |
| ${bmc_version}= GET BMC Version |
| ${pnor_version}= GET PNOR Version |
| Log ${bmc_version} |
| Rprint Vars bmc_version |
| Log ${pnor_version} |
| Rprint Vars pnor_version |
| |
| [Return] ${bmc_version} ${pnor_version} |
| |
| |
| Redfish Get BMC Version |
| [Documentation] Get BMC version via Redfish. |
| |
| ${output}= Redfish.Get Attribute ${REDFISH_BASE_URI}Managers/${MANAGER_ID} FirmwareVersion |
| [Return] ${output} |
| |
| Redfish Get Host Version |
| [Documentation] Get host version via Redfish. |
| |
| ${output}= Redfish.Get Attribute ${REDFISH_BASE_URI}Systems/${SYSTEM_ID} BiosVersion |
| [Return] ${output} |
| |
| |
| Copy Address Translation Utils To HOST OS |
| [Documentation] Copy address translation utils to host OS. |
| |
| OperatingSystem.File Should Exist ${probe_cpu_tool_path} |
| ... msg=${probe_cpu_tool_path} doesn't exist. |
| OperatingSystem.File Should Exist ${probe_cpu_tool_path} |
| ... msg=${probe_cpu_tool_path} doesn't exist. |
| |
| scp.Open connection ${OS_HOST} username=${OS_USERNAME} |
| ... password=${OS_PASSWORD} |
| scp.Put File ${probe_cpu_tool_path} ${target_file_path} |
| scp.Put File ${scom_addrs_tool_path} ${target_file_path} |
| |
| |
| Verify BMC RTC And UTC Time Drift |
| [Documentation] Verify that the RTC and UTC time difference is less than |
| ... the given time_drift_max. |
| [Arguments] ${time_diff_max}=${10} |
| |
| # Description of argument(s): |
| # time_diff_max The max allowable RTC and UTC time difference in seconds. |
| |
| # Example: |
| # time_dict: |
| # [local_time]: Fri 2017-11-03 152756 UTC |
| # [local_time_seconds]: 1509740876 |
| # [universal_time]: Fri 2017-11-03 152756 UTC |
| # [universal_time_seconds]: 1509740876 |
| # [rtc_time]: Fri 2016-05-20 163403 |
| # [rtc_time_seconds]: 1463780043 |
| # [time_zone]: n/a (UTC, +0000) |
| # [network_time_on]: yes |
| # [ntp_synchronized]: no |
| # [rtc_in_local_tz]: no |
| |
| ${time}= Get BMC Date Time |
| ${time_diff}= Evaluate |
| ... ${time['universal_time_seconds']} - ${time['rtc_time_seconds']} |
| Should Be True ${time_diff} < ${time_diff_max} |
| |
| |
| Validate IP On BMC |
| [Documentation] Validate IP address is present in set of IP addresses. |
| [Arguments] ${ip_address} ${ip_data} |
| |
| # Description of argument(s): |
| # ip_address IP address to check (e.g. xx.xx.xx.xx). |
| # ip_data Set of the IP addresses present. |
| |
| Should Contain Match ${ip_data} ${ip_address}/* |
| ... msg=${ip_address} not found in the list provided. |
| |
| |
| Remove Journald Logs |
| [Documentation] Remove all journald logs and restart service. |
| |
| ${cmd}= Catenate systemctl stop systemd-journald.service && |
| ... rm -rf /var/log/journal && systemctl start systemd-journald.service |
| |
| BMC Execute Command ${cmd} |
| |
| |
| Check For Regex In Journald |
| [Documentation] Parse the journal log and check for regex string. |
| [Arguments] ${regex}=${ERROR_REGEX} ${error_check}=${0} ${boot}=${EMPTY} |
| ... ${filter_string}=${EMPTY} |
| |
| # Description of argument(s): |
| # regex Strings to be filter. |
| # error_check Check for errors. |
| # boot Argument to check current or persistent full boot log |
| # (e.g. "-b"). |
| # filter_string String to be stripped out. |
| |
| |
| ${cmd} = Run Keyword If '${filter_string}' == '${EMPTY}' |
| ... Catenate journalctl --no-pager ${boot} | egrep '${regex}' |
| ... ELSE |
| ... Catenate journalctl --no-pager ${boot} | egrep '${regex}' | sed '/${filter_string}/d' |
| |
| ${journal_log} ${stderr} ${rc}= BMC Execute Command ${cmd} ignore_err=1 |
| |
| Run Keyword If ${error_check} == ${0} |
| ... Should Be Empty ${journal_log} |
| ... ELSE |
| ... Should Not Be Empty ${journal_log} |
| |
| |
| Get Service Attribute |
| [Documentation] Get service attribute policy output. |
| [Arguments] ${option} ${servicename} |
| |
| # Description of argument(s): |
| # option systemctl supported options |
| # servicename Qualified service name |
| ${cmd}= Set Variable |
| ... systemctl -p ${option} show ${servicename} | cut -d = -f2 |
| ${attr} ${stderr} ${rc}= BMC Execute Command ${cmd} |
| [Return] ${attr} |
| |
| |
| Verify Watchdog Enabled |
| [Documentation] Check that watchdog settings are enabled. |
| |
| ${properties}= Read Properties /xyz/openbmc_project/watchdog/host0 |
| Should Be Equal As Strings ${properties["Enabled"]} ${True} |
| Should Not Be Equal As Strings ${properties["TimeRemaining"]} 0 |
| |
| |
| Is BMC Unpingable |
| [Documentation] Check if BMC is unpingable. |
| |
| ${RC} ${output}= Run and return RC and Output ping -c 4 ${OPENBMC_HOST} |
| Log RC: ${RC}\nOutput:\n${output} |
| Should be equal ${RC} ${1} |
| |
| |
| Is Host Unpingable |
| [Documentation] Check if Given IP is unpingable. |
| [Arguments] ${ip} |
| |
| # Description of argument(s): |
| # ip HostOS IP address (e.g. "10.7.7.7"). |
| |
| ${RC} ${output}= Run and return RC and Output ping -c 4 ${ip} |
| Log RC: ${RC}\nOutput:\n${output} |
| Should be equal ${RC} ${1} |
| |
| |
| Redfish BMC Match States |
| [Documentation] Verify the BMC match state. |
| [Arguments] ${match_state} |
| |
| # Description of argument(s): |
| # match_state Match the state of BMC. |
| |
| ${bmc_state}= Redfish Get BMC State |
| Should Be Equal As Strings ${match_state} ${bmc_state} |
| |
| |
| Kernel Panic BMC Reset Operation |
| [Documentation] Create kernel panic to reset BMC. |
| |
| ${kernel_panic_cmd_file}= Set Variable /bin/sh -c "echo c > /proc/sysrq-trigger" |
| ${status}= Run Keyword And Return Status BMC Execute Command ${kernel_panic_cmd_file} time_out=20 |
| |
| [Return] ${status} |
| |
| |
| Get Property Value Of Systemd Service In BMC |
| [Documentation] Get property of systemd service in BMC. |
| [Arguments] ${service_name} ${property_name} |
| |
| # Description of argument(s): |
| # service_name Systemd service name. E.g. bmcweb etc. |
| # property_name Property name. E.g. MainPID etc. |
| |
| # ~# systemctl show --property MainPID --value bmcweb |
| # 1273 |
| |
| ${stdout} ${stderr} ${rc}= BMC Execute Command |
| ... systemctl show --property ${property_name} --value ${service_name} |
| |
| [Return] ${stdout} |