Move fan tests into library
In preparation of calling fan tests from within an upcoming
Redfish modile, move fan tests to a library.
Update non-Redfish tests (fan_test_os.robot and fan_test.robot) to
use the libary routines.
Signed-off-by: Steven Sombar <ssombar@us.ibm.com>
Change-Id: Iaa2f4137acf8e0fb0e2cc2371674ce7d781a3ac2
diff --git a/lib/fan_utils.robot b/lib/fan_utils.robot
index aa20e33..b4e69d2 100755
--- a/lib/fan_utils.robot
+++ b/lib/fan_utils.robot
@@ -5,6 +5,16 @@
Resource ../lib/openbmc_ffdc_utils.robot
Variables ../data/variables.py
+*** Variables ***
+
+# Fan state values.
+${fan_functional} ${1}
+${fan_nonfunctional} ${0}
+
+# Criteria for a fan at maximum speed.
+${max_speed}= ${10400}
+
+
*** Keywords ***
Is Water Cooled
@@ -27,10 +37,10 @@
${fan_uris}= Get Endpoint Paths ${HOST_INVENTORY_URI}system fan
: FOR ${fan_uri} IN @{fan_uris}
-
\ ${fan_properties}= Read Properties ${fan_uri}
\ ${fan_present}= Get Variable Value ${fan_properties['Present']} 0
- \ ${fan_functional}= Get Variable Value ${fan_properties['Functional']} 0
+ \ ${fan_functional}= Get Variable Value
+ ... ${fan_properties['Functional']} 0
\ Continue For Loop If ${fan_present} == 0 or ${fan_functional} == 0
\ ${remaining_uri} ${fan_name}= Split Path ${fan_uri}
\ Append To List ${fan_names} ${fan_name}
@@ -214,3 +224,249 @@
# Fail if system is Off and the fan monitors are present.
Run Keyword If '${power_state}' == 'Off' and ${num_fan_daemons} != 0
... Fail msg=Phosphor-fan monitors found at power off.
+
+
+Get Fan Count And Names
+ [Documentation] Return the number of fans and the fan names.
+
+ # The @{fan_names} list holds the names of the fans in the system.
+ @{fan_names} Create List
+ ${fan_names}= Get Fan Names ${fan_names}
+
+ ${number_of_fans}= Get Length ${fan_names}
+
+ Printn
+ Rprint Vars number_of_fans fan_names
+ [Return] ${number_of_fans} ${fan_names}
+
+
+Reset Fans
+ [Documentation] Set the fans to functional state.
+ # Set state of fans to functional by writing 1 to the Functional
+ # attribute of each fan in the @{fan_names} list. If @{fan_names}
+ # is empty nothing is done.
+ [Arguments] ${fan_names}
+
+ # Description of Argument(s):
+ # fan_names A list containing the names of the fans (e.g. fan0
+ # fan2 fan3).
+
+ :FOR ${fan_name} IN @{fan_names}
+ \ Set Fan State ${fan_name} ${fan_functional}
+
+
+Verify Fan Speed
+ [Documentation] Verify fans are running at or near target speed.
+ [Arguments] ${tolerance} ${fan_names}
+
+ # Description of argument(s):
+ # tolerance The speed tolerance criteria.
+ # A tolerance value of .15 means that the fan's speed
+ # should be within 15% of its set target speed.
+ # Fans may be accelerating to meet a new target, so
+ # allow .10 extra.
+ # fan_names A list containing the names of the fans (e.g. fan0 fan1).
+
+ # Compare the fan's speed with its target speed.
+ :FOR ${fan_name} IN @{fan_names}
+ \ ${target_speed} ${fan_speed}= Get Fan Target And Speed ${fan_name}
+ \ Rpvars fan_name target_speed fan_speed
+ \ # Calculate tolerance, which is a % of the target speed.
+ \ ${tolerance_value}= Evaluate ${tolerance}*${target_speed}
+ \ # Calculate upper and lower speed limits.
+ \ ${max_limit}= Evaluate ${target_speed}+${tolerance_value}
+ \ ${min_limit}= Evaluate ${target_speed}-${tolerance_value}
+ \ Run Keyword If
+ ... ${fan_speed} < ${min_limit} or ${fan_speed} > ${max_limit}
+ ... Fail msg=${fan_name} speed of ${fan_speed} is out of range.
+
+
+Verify Direct Fan Control
+ [Documentation] Verify direct control of fans.
+ [Arguments] ${max_fan_target_setting} ${min_speed}
+ ... ${minutes_to_stabilize} ${number_of_fans} ${fan_names}
+
+ # Overview:
+ # Turn off BMC's fan control daemon, then test to confirm
+ # that fans can be controlled manually.
+ # The app that takes data from sysfs and updates dbus is named hwmon.
+ # Verify hwmon functionality by comparing with what's on dbus
+ # (/xyz/openbmc_project/sensors/fan_tach/fan0_0, fan0_1, etc.)
+ # with what's in the BMC's file system at
+ # /sys/class/hwmon/hwmon9/fan*_input.
+
+ # Description of argument(s):
+ # max_fan_target_setting Integer value of maximum fan speed.
+ # min_speed Integer value of minimum speed.
+ # minutes_to_stabilize Time to wait for fan daemons to
+ # stabilize fan operation after
+ # tests (e.g. "4").
+ # number_of_fans The number of fans in the system.
+ # fan_names A list containing the names of the
+ # fans (e.g. fan0 fan1).
+
+ # Login to BMC and disable the fan daemon. Disabling the daemon sets
+ # manual mode.
+ Open Connection And Log In
+ Set Fan Daemon State stop
+
+ # For each fan, set a new target speed and wait for the fan to
+ # accelerate. Then check that the fan is running near that speed.
+ :FOR ${fan_name} IN @{fan_names}
+ \ Set Fan Target Speed ${fan_name} ${max_fan_target_setting}
+ \ Run Key U Sleep \ 60s
+ \ ${target_speed} ${cw_speed} ${ccw_speed}=
+ ... Get Target And Blade Speeds ${fan_name}
+ \ Rpvars fan_name target_speed cw_speed ccw_speed
+ \ Run Keyword If
+ ... ${cw_speed} < ${min_speed} or ${ccw_speed} < ${min_speed}
+ ... Fail msg=${fan_name} failed manual speed test.
+
+ # Check the fan speeds in the BMC file system.
+
+ # Get the location of the fan hwmon.
+ ${controller_path} ${stderr} ${rc}= BMC Execute Command
+ ... grep -ir max31785a /sys/class/hwmon/hwmon* | grep name
+ # E.g., controller_path=/sys/class/hwmon/hwmon10/name:max31785a.
+
+ ${hwmon_path} ${file_name}= Split Path ${controller_path}
+ # E.g., /sys/class/hwmon/hwmon10 or /sys/class/hwmon/hwmon9.
+
+ Rpvars controller_path hwmon_path
+
+ # Run the BMC command which gets fan speeds from the file system.
+ ${cmd}= Catenate cat ${hwmon_path}/fan*_input
+ ${stdout} ${stderr} ${rc}=
+ ... BMC Execute Command ${cmd}
+
+ # Convert output to integer values.
+ ${speeds}= Evaluate map(int, $stdout.split(${\n}))
+ Rpvars speeds
+ # Count the number of speeds > ${min_speed}.
+ ${count}= Set Variable ${0}
+ :FOR ${speed} IN @{speeds}
+ \ ${count}= Run Keyword If ${speed} > ${min_speed}
+ ... Evaluate ${count}+1 ELSE Set Variable ${count}
+ # Because each fan has two rotating fan blades, the count should be
+ # equual to 2*${number_of_fans}. On water-cooled systems some
+ # speeds may be reported by hwmon as 0. That is expected,
+ # and the number_of_fans reported in the system will be less.
+ ${fail_test}= Evaluate (2*${number_of_fans})-${count}
+
+ # Re-enable the fan daemon
+ Set Fan Daemon State restart
+
+ Run Keyword If ${fail_test} > ${0} Fail
+ ... msg=hwmon did not properly report fan speeds.
+
+ # Wait for the daemon to take control and gracefully set fan speeds
+ # back to normal.
+ ${msg}= Catenate Waiting ${minutes_to_stabilize} minutes
+ ... for fan daemon to stabilize fans.
+ Print Timen ${msg}
+ Run Key U Sleep \ ${minutes_to_stabilize}m
+
+
+Verify Fan Speed Increase
+ [Documentation] Verify that the speed of working fans increase when
+ ... one fan is marked as disabled.
+ # A non-functional fan should cause an error log and
+ # an enclosure LED will light. The other fans should speed up.
+ [Arguments] ${fan_names}
+
+ # Description of argument(s):
+ # fan_names A list containing the names of the fans (e.g. fan0 fan1).
+
+ # Allow system_response_time before checking if there was a
+ # response by the system to an applied fault.
+ ${system_response_time}= Set Variable 60s
+
+ # Choose a fan to test with, e.g., fan0.
+ ${test_fan_name}= Get From List ${fan_names} 0
+
+ ${initial_speed}= Get Target Speed Of Fans
+ Rpvars test_fan_name initial_speed
+
+ # If initial speed is not already at maximum, set expect_increase.
+ # This flag is used later to determine if speed checking is
+ # to be done or not.
+ ${expect_increase}= Run Keyword If
+ ... ${initial_speed} < ${max_speed}
+ ... Set Variable 1 ELSE Set Variable 0
+
+ Set Fan State ${test_fan_name} ${fan_nonfunctional}
+
+ # Wait for error to be asserted.
+
+ :FOR ${n} IN RANGE 30
+ \ ${front_fault}= Get System LED State front_fault
+ \ ${rear_fault}= Get System LED State rear_fault
+ \ Run Key U Sleep \ 1s
+ \ Exit For Loop If '${front_fault}' == 'On' and '${rear_fault}' == 'On'
+
+ Verify System Error Indication Due To Fans
+
+ # Verify the error log is for test_fan_name.
+ ${elog_entries}= Get Logging Entry List
+ :FOR ${elog_entry} IN @{elog_entries}
+ \ ${elog_entry_callout}= Set Variable ${elog_entry}/callout
+ \ ${endpoint}= Read Attribute ${elog_entry_callout} endpoints
+ \ ${endpoint_name}= Get From List ${endpoint} 0
+ \ Should Contain ${endpoint_name} ${test_fan_name}
+ ... msg=Error log present but not for ${test_fan_name}.
+
+ Run Key U Sleep \ ${system_response_time}
+
+ # A heavily loaded system may have powered-off.
+ ${host_state}= Get Host State
+ Rpvars host_state
+ Run Keyword If 'Running' != '${host_state}' Pass Execution
+ ... msg=System shutdown so skipping remainder of test.
+
+ ${new_fan_speed}= Get Target Speed Of Fans
+ Rpvars expect_increase initial_speed new_fan_speed
+
+ # Fail if current fan speed did not increase past the initial
+ # speed, but do this check only if not at maximum speed to begin with.
+ Run Keyword If
+ ... ${expect_increase} == 1 and ${new_fan_speed} < ${initial_speed}
+ ... Fail msg=Remaining fans did not increase speed with loss of one fan.
+
+
+
+Verify System Shutdown Due To Fans
+ [Documentation] Shut down when not enough fans.
+ [Arguments] ${fan_names}
+
+ # Description of argument(s):
+ # fan_names A list containing the names of the fans (e.g. fan0 fan1).
+
+ ${wait_after_poweroff}= Set Variable 15s
+
+ # A previous test may have shutdown the system.
+ REST Power On stack_mode=skip
+
+ # Set fans to be non-functional.
+ :FOR ${fan_name} IN @{fan_names}
+ \ Set Fan State ${fan_name} ${fan_nonfunctional}
+
+ # System should notice the non-functional fans and power-off.
+ # The Wait For PowerOff keyword will time-out and report
+ # an error if power off does not happen within a reasonable time.
+ Wait For PowerOff
+
+ Run Key U Sleep \ ${wait_after_poweroff}
+
+ Verify System Error Indication Due To Fans
+
+ # Verify there is an error log because of the shutdown.
+ ${expect}= Catenate
+ ... xyz.openbmc_project.State.Shutdown.Inventory.Error.Fan
+ ${elog_entries}= Get Logging Entry List
+ :FOR ${elog_entry} IN @{elog_entries}
+ \ ${elog_message}= Read Attribute ${elog_entry} Message
+ \ ${found}= Set Variable 1
+ \ Run Keyword If '${elog_message}' == '${expect}' Exit For Loop
+ \ ${found}= Set Variable 0
+ Run Keyword If not ${found} Fail
+ ... msg=No error log for event Shutdown.Inventory.Error.Fan.