blob: 149a0c9a5c7c43a334c33ca94cffff71af0cf088 [file] [log] [blame]
*** Settings ***
Documentation Stress the system using HTX exerciser.
# Test Parameters:
# OPENBMC_HOST The BMC host name or IP address.
# OS_HOST The OS host name or IP Address.
# OS_USERNAME The OS login userid (usually root).
# OS_PASSWORD The password for the OS login.
# HTX_DURATION Duration of HTX run, for example, 2h, or 30m.
# HTX_LOOP The number of times to loop HTX.
# HTX_INTERVAL The time delay between consecutive checks of HTX
# status, for example, 15m.
# In summary: Run HTX for $HTX_DURATION, looping
# $HTX_LOOP times checking for errors every
# $HTX_INTERVAL. Then allow extra time for OS
# Boot, HTX startup, shutdown.
# HTX_KEEP_RUNNING If set to 1, this indicates that the HTX is to
# continue running after an error was found.
# CHECK_INVENTORY If set to 0 or False, OS inventory checking before
# and after each HTX run will be disabled. This
# parameter is optional. The default value is True.
# PREV_INV_FILE_PATH The file path and name of an initial previous
# inventory snapshot file in JSON format. Inventory
# snapshots taken before and after each HTX run will
# be compared to this file.
# This parameter is optional. If not specified an
# initial inventory snapshot will be taken before
# HTX startup.
# INV_IGNORE_LIST A comma-delimited list of strings that
# indicate what to ignore if there are inventory
# differences, (e.g., processor "size").
# If differences are found during inventory checking
# and those items are in this list, the
# differences will be ignored. This parameter is
# optional. If not specified the default value is
# "size".
Resource ../syslib/utils_os.robot
Resource ../lib/openbmc_ffdc_utils.robot
Resource ../lib/logging_utils.robot
Resource ../lib/code_update_utils.robot
Library ../syslib/utils_keywords.py
Library ../lib/utils_files.py
Library ../lib/logging_utils.py
Library ../syslib/utils_os.py
Suite Setup Run Keyword Start SOL Console Logging
Test Setup Test Setup Execution
Test Teardown Test Teardown Execution
*** Variables ****
${stack_mode} skip
${json_initial_file_path} ${EXECDIR}/os_inventory_initial.json
${json_final_file_path} ${EXECDIR}/os_inventory_final.json
${json_diff_file_path} ${EXECDIR}/os_inventory_diff.json
${CHECK_INVENTORY} True
${INV_IGNORE_LIST} size
${PREV_INV_FILE_PATH} NONE
${rest_keyword} REST
# Error log Severities to ignore when checking for eSELs.
@{ESEL_WHITELIST}
... xyz.openbmc_project.Logging.Entry.Level.Informational
*** Test Cases ***
Hard Bootme Test
[Documentation] Stress the system using HTX exerciser.
[Tags] Hard_Bootme_Test
Printn
Rprint Vars HTX_DURATION HTX_LOOP HTX_INTERVAL CHECK_INVENTORY
... INV_IGNORE_LIST PREV_INV_FILE_PATH
Run Keyword If '${PREV_INV_FILE_PATH}' != 'NONE'
... OperatingSystem.File Should Exist ${PREV_INV_FILE_PATH}
Set Suite Variable ${PREV_INV_FILE_PATH} children=true
Set Suite Variable ${INV_IGNORE_LIST} children=true
# Set up the iteration (loop) counter.
Set Suite Variable ${iteration} ${0} children=true
# Estimate the time required for a single iteration loop.
# HTX_DURATION + 10 minutes for OS boot, HTX startup, shutdown.
${loop_body_seconds}= Add Time To Time ${HTX_DURATION} 10m
Set Suite Variable ${loop_body_seconds} children=true
# And save it in printable (compact) format.
${estimated_loop_time}= Convert Time
... ${loop_body_seconds} result_format=compact
Set Suite Variable ${estimated_loop_time} children=true
# Estimated time remaining = loop_body_seconds * HTX_LOOP + 5m
${est_seconds_left}= Evaluate ${loop_body_seconds}*${HTX_LOOP}+(5*60)
Set Suite Variable ${est_seconds_left} children=true
Repeat Keyword ${HTX_LOOP} times Run HTX Exerciser
*** Keywords ***
Run HTX Exerciser
[Documentation] Run HTX exerciser.
# Test Flow:
# - Power on.
# - Establish SSH connection session.
# - Do inventory collection, compare with
# previous inventory run if applicable.
# - Create HTX mdt profile.
# - Run HTX exerciser.
# - Check HTX status for errors.
# - Do inventory collection, compare with
# previous inventory run.
# - Power off.
Set Suite Variable ${iteration} ${iteration + 1}
${loop_count}= Catenate Starting iteration: ${iteration}
${estimated_time_remaining}= Convert Time
... ${est_seconds_left} result_format=compact
Printn
Rprint Vars loop_count estimated_loop_time estimated_time_remaining
Run Keyword ${rest_keyword} Power On stack_mode=skip
Run Key U Sleep \ 15s
# Post Power off and on, the OS SSH session needs to be established.
Login To OS
Run Keyword If '${CHECK_INVENTORY}' == 'True'
... Do Inventory And Compare ${json_initial_file_path}
... ${PREV_INV_FILE_PATH}
Run Keyword If '${HTX_MDT_PROFILE}' == 'mdt.bu'
... Create Default MDT Profile
Run MDT Profile
Loop HTX Health Check
Shutdown HTX Exerciser
Run Keyword If '${CHECK_INVENTORY}' == 'True'
... Do Inventory And Compare ${json_final_file_path}
... ${PREV_INV_FILE_PATH}
Run Keyword ${rest_keyword} Power Off
# Close all SSH and REST active sessions.
Close All Connections
Flush REST Sessions
Print Timen HTX Test ran for: ${HTX_DURATION}
${loop_count}= Catenate Ending iteration: ${iteration}
${est_seconds_left}= Evaluate ${est_seconds_left}-${loop_body_seconds}
Set Suite Variable ${est_seconds_left} children=true
${estimated_time_remaining}= Convert Time
... ${est_seconds_left} result_format=compact
Rprint Vars loop_count estimated_time_remaining
Do Inventory And Compare
[Documentation] Do inventory and compare.
[Arguments] ${inventory_file_path} ${PREV_INV_FILE_PATH}
# Description of argument(s):
# inventory_file_path The file to receive the inventory snapshot.
# PREV_INV_FILE_PATH The previous inventory to compare with.
Create JSON Inventory File ${inventory_file_path}
Run Keyword If '${PREV_INV_FILE_PATH}' != 'NONE'
... Compare Json Inventory Files ${inventory_file_path}
... ${PREV_INV_FILE_PATH}
${PREV_INV_FILE_PATH}= Set Variable ${inventory_file_path}
Set Suite Variable ${PREV_INV_FILE_PATH} children=true
Compare Json Inventory Files
[Documentation] Compare JSON inventory files.
[Arguments] ${file1} ${file2}
# Description of argument(s):
# file1 A file that has an inventory snapshot in JSON format.
# file2 A file that has an inventory snapshot, to compare with file1.
${diff_rc}= File_Diff ${file1}
... ${file2} ${json_diff_file_path} ${INV_IGNORE_LIST}
Run Keyword If '${diff_rc}' != '${0}'
... Report Inventory Mismatch ${diff_rc} ${json_diff_file_path}
... ELSE Print Timen Inventoy check: No differences found.
Report Inventory Mismatch
[Documentation] Report inventory mismatch.
[Arguments] ${diff_rc} ${json_diff_file_path}
# Description of argument(s):
# diff_rc The failing return code from the difference check.
# json_diff_file_path The file that has the latest inventory snapshot.
Log To Console Significant difference in inventory found, rc=${diff_rc}
Log To Console Differences are listed in file: no_newline=true
Log To Console ${json_diff_file_path}
Log To Console File Contents:
Wait Until Created ${json_diff_file_path}
${file_contents}= OperatingSystem.Get File ${json_diff_file_path}
Log ${file_contents} level=WARN
Fail Significant difference in inventory found, rc=${diff_rc}
Check For ESELs
[Documentation] Terminate if eSELs with Severity field not on WHITELIST.
Print Timen Checking eSEL Error Logs.
${error_logs}= Get Error Logs
${num_error_logs}= Get Length ${error_logs}
Rprint Vars num_error_logs
Return From Keyword If ${num_error_logs} == ${0}
Print Error Logs ${error_logs}
# Get a list of the severities of the error logs.
${error_log_severities}= Nested Get Severity ${error_logs}
# Subtract the WHITELIST from the error_log_severities.
${problem_error_logs}=
... Evaluate list(set($error_log_severities) - set($ESEL_WHITELIST))
${num_error_logs_not_on_whitelist}= Get Length ${problem_error_logs}
Return From Keyword If ${num_error_logs_not_on_whitelist} == ${0}
Rprint Vars ESEL_WHITELIST
Fail msg=Found error logs with Severity not matching ESEL_WHITELIST.
Loop HTX Health Check
[Documentation] Run until HTX exerciser fails.
Repeat Keyword ${HTX_DURATION}
... Run Keywords Check HTX Run Status
... AND Check For ESELs
... AND Sleep ${HTX_INTERVAL}
Test Setup Execution
[Documentation] Do the initial test setup.
${bmc_version} ${stderr} ${rc}= BMC Execute Command
... cat /etc/os-release
Printn
Rprint Vars bmc_version
${pnor_version}= Get Host Software Objects Details
Rprint Vars pnor_version
${is_redfish}= Run Keyword And Return Status Redfish.Login
${rest_keyword}= Set Variable If ${is_redfish} Redfish REST
Rprint Vars rest_keyword
Set Suite Variable ${rest_keyword} children=true
Run Keyword ${rest_keyword} Power On stack_mode=skip
Run Key U Sleep \ 15s
Delete All Error Logs
Tool Exist htxcmdline
${os_release_info}= Get OS Release Info
Rprint Vars os_release_info fmt=1
# Shutdown if HTX is running.
${status}= Is HTX Running
Run Keyword If '${status}' == 'True'
... Shutdown HTX Exerciser
Test Teardown Execution
[Documentation] Do the post-test teardown.
# Keep HTX running if user set HTX_KEEP_RUNNING to 1.
Run Keyword If
... '${TEST_STATUS}' == 'FAIL' and ${HTX_KEEP_RUNNING} == ${0}
... Shutdown HTX Exerciser
${keyword_buf}= Catenate Stop SOL Console Logging
... \ targ_file_path=${EXECDIR}${/}logs${/}SOL.log
Run Key ${keyword_buf}
FFDC On Test Case Fail
Close All Connections