Add fan testing
Add power-on system fan automation tests, and
update the fan utilities. Update the power-off
fan tests due to changes in the utilities.
Add 'Error Logs Should Exist' keyword.
Resolves openbmc/openbmc-test-automation#1174
Resolves openbmc/openbmc-test-automation#999
Resolves openbmc/openbmc-test-automation#616
Change-Id: Ic957516c53513dad16bc4cf188d6dfd20ddc5228
Signed-off-by: Steven Sombar <ssombar@us.ibm.com>
diff --git a/lib/fan_utils.robot b/lib/fan_utils.robot
index 7eb9ad7..9bde67b 100755
--- a/lib/fan_utils.robot
+++ b/lib/fan_utils.robot
@@ -1,12 +1,12 @@
*** Settings ***
-Documentation Utilities for fan tests.
+Documentation Utilities for fan tests.
-Library ../lib/bmc_ssh_utils.py
-Variables ../data/variables.py
+Library ../lib/bmc_ssh_utils.py
+Resource ../lib/openbmc_ffdc_utils.robot
+Variables ../data/variables.py
*** Keywords ***
-
Is Water Cooled
[Documentation] Return 1 if system is water cooled, 0 othersise.
@@ -15,26 +15,87 @@
[Return] ${water_cooled}
-Get Number Of Fans
- [Documentation] Get the number of fans currently present in inventory.
+Get Fan Names
+ [Documentation] Get the names of fans marked present in inventory.
+ [Arguments] ${fan_names}
+ # This keyword populates the fan_names list with the names of
+ # fans present in inventory e.g. fan0, fan2, fan3.
- ${num_fans} Set Variable ${0}
+ # Description of Argument(s):
+ # fan_names The list of fan names to which new fan names are to be
+ # added to. This list is returned to the caller.
+
${fan_uris}= Get Endpoint Paths ${HOST_INVENTORY_URI}/system fan
-
: FOR ${fan_uri} IN @{fan_uris}
- \ ${fan_record}= Read Properties ${fan_uri}
- \ Continue For Loop If ${fan_record['Present']} != 1
- \ ${num_fans}= Set Variable ${num_fans+1}
- [Return] ${num_fans}
+ \ ${fan_properties}= Read Properties ${fan_uri}
+ \ Continue For Loop If ${fan_properties['Present']} != 1
+ \ ${remaining_uri} ${fan_name}= Split Path ${fan_uri}
+ \ Append To List ${fan_names} ${fan_name}
+
+ [Return] ${fan_names}
+
+
+Verify System Error Indication Due To Fans
+ [Documentation] Verify enclosure LEDs are on and there's an error log.
+
+ # Both enclosure LEDs should now be On.
+ Verify Front And Rear LED State On
+
+ # An error log should now exist.
+ Error Logs Should Exist
+
+
+Verify Front And Rear LED State
+ [Documentation] Check state of the front and rear enclsure fault LEDs.
+ [Arguments] ${state}
+ # Both LEDs should be in the specified state. If not fail the test case.
+
+ # Description of Argument(s):
+ # state The state to check for, either 'Off' or 'On'.
+
+ ${front_fault}= Get System LED State front_fault
+ ${rear_fault}= Get System LED State rear_fault
+ Run Keyword If
+ ... '${front_fault}' != '${state}' or '${rear_fault}' != '${state}'
+ ... Fail msg=Expecting both enclosure LEDs to be ${state}.
+
+
+Set Fan State
+ [Documentation] Set the fan state, either functional or non-functional.
+ [Arguments] ${fan_name} ${fan_state}
+
+ # Description of Argument(s):
+ # fan_name The name of the fan, e.g. "fan2".
+ # fan_state The state to set, 1 for functional, 2 for non-functional.
+
+ ${valueDict}= Create Dictionary data=${fan_state}
+ Write Attribute
+ ... ${HOST_INVENTORY_URI}system/chassis/motherboard/${fan_name}
+ ... Functional data=${valueDict}
+
+
+Get Target Speed Of Fans
+ [Documentation] Return the maximum target RPM speed of the system fans.
+
+ ${max_target}= Set Variable 0
+ ${paths}= Get Endpoint Paths ${SENSORS_URI}fan_tach/ 0
+ :FOR ${path} IN @{paths}
+ \ ${response}= OpenBMC Get Request ${path}
+ \ ${json}= To JSON ${response.content}
+ \ ${target_speed}= Set Variable ${json["data"]["Target"]}
+ \ ${max_target}= Run Keyword If ${target_speed} > ${max_target}
+ ... Set Variable ${target_speed} ELSE Set Variable ${max_target}
+ [Return] ${max_target}
Verify Minimum Number Of Fans With Cooling Type
[Documentation] Verify minimum number of fans.
- [Arguments] ${water_cooled}
+ [Arguments] ${num_fans} ${water_cooled}
# Description of argument(s):
+ # num_fans The number of fans present in the system.
# water_cooled The value 1 if the system is water cooled,
- # the value 0 if air cooled.
+ # 0 if air cooled.
# For a water cooled system.
${min_fans_water}= Set Variable 2
@@ -42,10 +103,8 @@
# For an air cooled system.
${min_fans_air}= Set Variable 3
- ${num_fans}= Get Number Of Fans
-
Rprintn
- Rpvars water_cooled num_fans
+ Rpvars num_fans water_cooled
# If water cooled must have at least min_fans_water fans, otherwise
# issue Fatal Error and terminate testing.
@@ -61,10 +120,10 @@
Verify Fan Monitors With State
[Documentation] Verify fan monitor daemons in the system state.
+ [Arguments] ${power_state}
# The number of monitoring daemons is dependent upon the system
# power state. If power is off there should be 0, if power
# is on there should be several.
- [Arguments] ${power_state}
# Description of argument(s):
# power_state Power staet of the system, either "On" or "Off"
diff --git a/lib/openbmc_ffdc_utils.robot b/lib/openbmc_ffdc_utils.robot
index a2efabe..35e83fa 100644
--- a/lib/openbmc_ffdc_utils.robot
+++ b/lib/openbmc_ffdc_utils.robot
@@ -122,3 +122,11 @@
${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}/list quiet=${1}
Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND}
+
+
+Error Logs Should Exist
+ [Documentation] Verify that error logs exist.
+
+ ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}/list quiet=${1}
+ Run Keyword If ${resp.status_code} != ${HTTP_OK} Fail
+ ... msg=Expected BMC error log(s) are not present.
diff --git a/systest/test_fans_os.robot b/systest/test_fans_os.robot
new file mode 100755
index 0000000..ecf0b1d
--- /dev/null
+++ b/systest/test_fans_os.robot
@@ -0,0 +1,195 @@
+*** Settings ***
+
+Documentation Operational checks for fans.
+
+# Test Parameters:
+# OPENBMC_HOST The BMC host name or IP address.
+# OPENBMC_USERNAME The userID to login to the BMC as.
+# OPENBMC_PASSWORD The password for OPENBMC_USERNAME.
+# OS_HOST The OS host name or IP Address.
+# OS_USERNAME The OS login userid (usually root).
+# OS_PASSWORD The password for the OS login.
+#
+# Approximate run time: 8 minutes.
+
+Resource ../syslib/utils_os.robot
+Resource ../lib/logging_utils.robot
+Resource ../lib/utils.robot
+Resource ../lib/fan_utils.robot
+
+Suite Setup Suite Setup Execution
+Test Teardown Test Teardown Execution
+
+
+*** Variables ***
+
+# The fan speed-monitoring daemon takes less than one second to
+# notice a fan failure. Allow system_response_time before checking
+# if there was a measurable response to the daemon, such as an increase
+# in RPMs of the other fans.
+${system_response_time} 30s
+
+# The @{fan_names} list holds the names of the fans in the system.
+@{fan_names}
+
+# Fan state values.
+${fan_functional} ${1}
+${fan_nonfunctional} ${0}
+
+
+*** Test Cases ***
+
+
+Check Number Of Fans With Power On
+ [Documentation] Verify system has the minimum number of fans.
+ [Tags] Check_Number_Of_Fans_With_Power_On
+
+ @{fan_names} Create List
+ # Populate the list with the names of the fans in the system.
+ ${fan_names}= Get Fan Names ${fan_names}
+ Set Suite Variable ${fan_names} children=true
+
+ ${number_of_fans}= Get Length ${fan_names}
+
+ # Determine if system is water cooled.
+ ${water_coooled}= Is Water Cooled
+
+ Verify Minimum Number Of Fans With Cooling Type ${number_of_fans}
+ ... ${water_coooled}
+
+
+Check Number Of Fan Monitors With Power On
+ [Documentation] Verify monitors are present when power on.
+ [Tags] Check_Number_Of_Fan_Monitors_With_Power_On
+
+ Verify Fan Monitors With State On
+
+
+Verify Fan RPM Increase
+ [Documentation] Verify that RPMs of working fans increase when one fan
+ ... is disabled.
+ [Tags] Verify_Fan_RPM_Increase
+ # A non-functional fan should cause an error log and
+ # an enclosure LED will light. The other fans should speed up.
+
+ # Any fan at this speed or greater will be considered to be at maximum RPM.
+ ${max_fan_rpm}= Set Variable 10400
+
+ # Choose a fan to test with, e.g., fan1.
+ ${test_fan_name}= Get From List ${fan_names} 1
+ Rpvars test_fan_name
+
+ ${initial_speed}= Get Target Speed Of Fans
+ Rpvars initial_speed
+
+ # If initial speed is not already at maximum, set expect_increase.
+ # This flag is used later to determine if speed checking is
+ # to be done or not.
+ ${expect_increase}= Run Keyword If
+ ... ${initial_speed} < ${max_fan_rpm}
+ ... Set Variable 1 ELSE Set Variable 0
+
+ Set Fan State ${test_fan_name} ${fan_nonfunctional}
+ Sleep ${system_response_time}
+
+ Verify System Error Indication Due To Fans
+
+ # Verify the error log is for test_fan_name.
+ ${elog_entries}= Get Logging Entry List
+ :FOR ${elog_entry} IN @{elog_entries}
+ \ ${elog_entry_callout}= Set Variable ${elog_entry}/callout
+ \ ${endpoint}= Read Attribute ${elog_entry_callout} endpoints
+ \ ${endpoint_name}= Get From List ${endpoint} 0
+ \ Should Contain ${endpoint_name} ${test_fan_name}
+ ... msg=Error log present but not for ${test_fan_name}.
+
+ ${new_fan_speed}= Get Target Speed Of Fans
+ Rpvars expect_increase initial_speed new_fan_speed
+
+ # Fail if current fan speed did not increase past the initial
+ # speed, but do this check only if not at maximum speed to begin with.
+ Run Keyword If
+ ... ${expect_increase} == 1 and ${new_fan_speed} < ${initial_speed}
+ ... Fail msg=Remaining fans did not increase speed with loss of one fan.
+
+ # Recover the fan.
+ Set Fan State ${test_fan_name} ${fan_functional}
+ Sleep ${system_response_time}
+
+ Delete Error Logs
+ Sleep 2s
+
+ # Enclosure LEDs should go off immediately after deleting the error logs.
+ Verify Front And Rear LED State Off
+
+ ${restored_fan_speed}= Get Target Speed Of Fans
+ Rpvars new_fan_speed restored_fan_speed
+
+ # Fan speed should lower because the fan is now functional again.
+ Run Keyword If
+ ... ${expect_increase} == 1 and ${new_fan_speed} < ${restored_fan_speed}
+ ... Fail msg=Fans did not recover speed with all fans functional again.
+
+
+Verify System Shutdown Due To Fans
+ [Documentation] Shut down when not enough fans.
+ [Tags] Verify_System_Shutdown_Due_To_Fans
+
+ # Set fans to be non-functional.
+ :FOR ${fan_name} IN @{fan_names}
+ \ Set Fan State ${fan_name} ${fan_nonfunctional}
+
+ # System should notice the non-functional fans and power-off the
+ # system. The Wait For PowerOff keyword will time-out and report
+ # an error if power off does not happen within a reasonable time.
+ Wait For PowerOff
+
+ Verify System Error Indication Due To Fans
+
+ # Verify there is an error log because of the shutdown.
+ ${expect}= Catenate
+ ... xyz.openbmc_project.State.Shutdown.Inventory.Error.Fan
+ ${elog_entries}= Get Logging Entry List
+ :FOR ${elog_entry} IN @{elog_entries}
+ \ ${elog_message}= Read Attribute ${elog_entry} Message
+ \ ${found}= Set Variable 1
+ \ Run Keyword If '${elog_message}' == '${expect}' Exit For Loop
+ \ ${found}= Set Variable 0
+ Run Keyword If not ${found} Fail
+ ... msg=No error log for event Shutdown.Inventory.Error.Fan.
+
+
+*** Keywords ***
+
+
+Reset Fans
+ [Documentation] Set the fans to functional state.
+ # Set state of fans to functional by writing 1 to the Functional
+ # attribute of each fan in the @{fan_names} list. If @{fan_names}
+ # is empty nothing is done.
+
+ # Description of Argument(s):
+ # fans Suite Variable which is a list containing the
+ # names of the fans (e.g., fan0 fan2 fan3).
+
+ :FOR ${fan_name} IN @{fan_names}
+ \ Set Fan State ${fan_name} ${fan_functional}
+
+
+Suite Setup Execution
+ [Documentation] Do the pre-test setup.
+
+ REST Power On stack_mode=skip
+ Delete All Error Logs
+ Set System LED State front_fault Off
+ Set System LED State rear_fault Off
+
+
+Test Teardown Execution
+ [Documentation] Do the post-test teardown.
+
+ FFDC On Test Case Fail
+ Reset Fans
+ Delete Error Logs
+ Set System LED State front_fault Off
+ Set System LED State rear_fault Off
diff --git a/tests/test_fans.robot b/tests/test_fans.robot
index 98fc7ba..9b05226 100755
--- a/tests/test_fans.robot
+++ b/tests/test_fans.robot
@@ -22,10 +22,16 @@
[Documentation] Verify minimum number of fans.
[Tags] Fan_Base_Check_Number_Of_Fans
+ @{fan_names} Create List
+ # Populate the list with the names of fans in the system.
+ ${fan_names}= Get Fan Names ${fan_names}
+ ${number_of_fans}= Get Length ${fan_names}
+
# Determine if system is water cooled.
${water_coooled}= Is Water Cooled
- Verify Minimum Number Of Fans With Cooling Type ${water_coooled}
+ Verify Minimum Number Of Fans With Cooling Type ${number_of_fans}
+ ... ${water_coooled}
Fan Base Check Number Of Fan Monitors