blob: 3d04a2cfe9e7530da1fdf6330cb1eaf940ae855a [file] [log] [blame]
Steven Sombar43d6ba32018-02-16 11:25:09 -06001*** Settings ***
2
3Documentation Operational checks for fans.
4
5# Test Parameters:
6# OPENBMC_HOST The BMC host name or IP address.
7# OPENBMC_USERNAME The userID to login to the BMC as.
8# OPENBMC_PASSWORD The password for OPENBMC_USERNAME.
Steven Sombar43d6ba32018-02-16 11:25:09 -06009#
Steven Sombar1508aff2018-04-06 12:53:42 -050010# Approximate run time: 18 minutes.
Steven Sombar43d6ba32018-02-16 11:25:09 -060011
12Resource ../syslib/utils_os.robot
13Resource ../lib/logging_utils.robot
14Resource ../lib/utils.robot
15Resource ../lib/fan_utils.robot
Steven Sombar1508aff2018-04-06 12:53:42 -050016Library ../syslib/utils_keywords.py
17Library OperatingSystem
Steven Sombar43d6ba32018-02-16 11:25:09 -060018
Steven Sombar1508aff2018-04-06 12:53:42 -050019Suite Setup Suite Setup Execution
20Test Teardown Test Teardown Execution
Steven Sombar43d6ba32018-02-16 11:25:09 -060021
22
23*** Variables ***
24
Steven Sombar43d6ba32018-02-16 11:25:09 -060025# Fan state values.
26${fan_functional} ${1}
27${fan_nonfunctional} ${0}
28
Steven Sombar1508aff2018-04-06 12:53:42 -050029# Criteria for a fan to be considered to be at maximum speed.
30${max_speed}= ${10400}
31
Steven Sombar43d6ba32018-02-16 11:25:09 -060032
33*** Test Cases ***
34
Steven Sombar43d6ba32018-02-16 11:25:09 -060035Check Number Of Fans With Power On
36 [Documentation] Verify system has the minimum number of fans.
37 [Tags] Check_Number_Of_Fans_With_Power_On
38
Steven Sombar43d6ba32018-02-16 11:25:09 -060039 # Determine if system is water cooled.
40 ${water_coooled}= Is Water Cooled
41
42 Verify Minimum Number Of Fans With Cooling Type ${number_of_fans}
43 ... ${water_coooled}
44
45
46Check Number Of Fan Monitors With Power On
47 [Documentation] Verify monitors are present when power on.
48 [Tags] Check_Number_Of_Fan_Monitors_With_Power_On
49
50 Verify Fan Monitors With State On
51
52
Steven Sombar1508aff2018-04-06 12:53:42 -050053Check Fan Speed
54 [Documentation] Verify fans are running at or near target speed.
55 [Tags] Check_Fan_Speed
56
57 # Set the speed tolerance criteria.
58 # A tolerance value of .15 means that the fan's speed should be
59 # within 15% of its set target speed. Fans may be accelerating
60 # or decelerating to meet a new target, so allow .10 extra.
61 ${tolerance}= Set Variable .25
62 Rpvars tolerance
63
64 # Compare the fan's speed with its target speed.
65 :FOR ${fan_name} IN @{fan_names}
66 \ ${target_speed} ${fan_speed}= Get Fan Target And Speed ${fan_name}
67 \ Rpvars fan_name target_speed fan_speed
68 \ # Calculate tolerance, which is a % of the target speed.
69 \ ${tolerance_value}= Evaluate ${tolerance}*${target_speed}
70 \ # Calculate upper and lower speed limits.
71 \ ${max_limit}= Evaluate ${target_speed}+${tolerance_value}
72 \ ${min_limit}= Evaluate ${target_speed}-${tolerance_value}
73 \ Run Keyword If
74 ... ${fan_speed} < ${min_limit} or ${fan_speed} > ${max_limit}
75 ... Fail msg=${fan_name} speed of ${fan_speed} is out of range.
76
77
78Check Fan Manual Control
79 [Documentation] Check direct control of fans.
80 [Tags] Check_Fan_Manual_Control
81
82 # Test case overview:
83 # Turn off BMC's fan control daemon, then test to confirm
84 # that fans can be controlled manually.
85 # The app that takes data from sysfs and updates dbus is named hwmon.
86 # Verify hwmon functionality by comparing with what's on dbus
87 # (/xyz/openbmc_project/sensors/fan_tach/fan0_0, fan0_1, etc.)
88 # with what's in the BMC's file system at
89 # /sys/class/hwmon/hwmon9/fan*_input.
90
91 # The maximum target speed that can be set.
92 ${max_fan_target_setting}= Set Variable ${10500}
93
94 # Speed criteria for passing, which is 85% of max_fan_target_setting.
95 ${min_speed}= Set Variable ${8925}
96
97 # Time allowed for the fan daemon to take control and return
98 # fans to normal speed.
99 ${minutes_to_stabilize}= Set Variable 4
100
Gunnar Mills7732c7e2018-08-14 11:54:24 -0500101 # Login to BMC and disable the fan daemon. Disabling the daemon sets
Steven Sombar1508aff2018-04-06 12:53:42 -0500102 # manual mode.
103 Open Connection And Log In
104 Set Fan Daemon State stop
105
106 # For each fan, set a new target speed and wait for the fan to
107 # accelerate. Then check that the fan is running near that
108 # target speed.
109 :FOR ${fan_name} IN @{fan_names}
110 \ Set Fan Target Speed ${fan_name} ${max_fan_target_setting}
111 \ Run Key U Sleep \ 60s
112 \ ${target_speed} ${cw_speed} ${ccw_speed}=
113 ... Get Target And Blade Speeds ${fan_name}
114 \ Rpvars fan_name target_speed cw_speed ccw_speed
115 \ Run Keyword If
116 ... ${cw_speed} < ${min_speed} or ${ccw_speed} < ${min_speed}
117 ... Fail msg=${fan_name} failed manual speed test.
118
119 # Check the fan speeds in the BMC file system.
120
121 # Get the location of the fan hwmon.
122 ${controller_path} ${stderr} ${rc}= BMC Execute Command
123 ... grep -ir max31785a /sys/class/hwmon/hwmon* | grep name
124 # E.g., controller_path=/sys/class/hwmon/hwmon10/name:max31785a.
125
126 ${hwmon_path} ${file_name}= Split Path ${controller_path}
127 # E.g., /sys/class/hwmon/hwmon10 or /sys/class/hwmon/hwmon9.
128
129 Rpvars controller_path hwmon_path
130
131 # Run the BMC command which gets fan speeds from the file system.
132 ${cmd}= Catenate cat ${hwmon_path}/fan*_input
133 ${stdout} ${stderr} ${rc}=
134 ... BMC Execute Command ${cmd}
135
Steven Sombar1508aff2018-04-06 12:53:42 -0500136 # Convert output to integer values.
137 ${speeds}= Evaluate map(int, $stdout.split(${\n}))
138 Rpvars speeds
139 # Count the number of speeds > ${min_speed}.
140 ${count}= Set Variable ${0}
141 :FOR ${speed} IN @{speeds}
142 \ ${count}= Run Keyword If ${speed} > ${min_speed}
143 ... Evaluate ${count}+1 ELSE Set Variable ${count}
144 # Because each fan has two rotating fan blades, the count should be
145 # equual to 2*${number_of_fans}. On water-cooled systems some
146 # speeds may be reported by hwmon as 0. That is expected,
147 # and the number_of_fans reported in the system will be less.
148 ${fail_test}= Evaluate (2*${number_of_fans})-${count}
149
150 # Re-enable the fan daemon
151 Set Fan Daemon State restart
152
Steven Sombar84689182018-09-07 13:46:06 -0500153 Run Keyword If ${fail_test} > ${0} Fail
Steven Sombar1508aff2018-04-06 12:53:42 -0500154 ... msg=hwmon did not properly report fan speeds.
155
156 # Wait for the daemon to take control and gracefully set fan speeds
157 # back to normal.
158 ${msg}= Catenate Waiting ${minutes_to_stabilize} minutes
159 ... for fan daemon to stabilize fans.
Michael Walshc108e422019-03-28 12:27:18 -0500160 Print Timen ${msg}
Steven Sombar1508aff2018-04-06 12:53:42 -0500161 Run Key U Sleep \ ${minutes_to_stabilize}m
162
163
164Verify Fan Speed Increase
165 [Documentation] Verify that the speed of working fans increase when
166 ... one fan is disabled.
167 [Tags] Verify_Fan_Speed_Increase
Steven Sombar43d6ba32018-02-16 11:25:09 -0600168 # A non-functional fan should cause an error log and
169 # an enclosure LED will light. The other fans should speed up.
170
Steven Sombar1508aff2018-04-06 12:53:42 -0500171 # Allow system_response_time before checking if there was a
172 # response by the system to an applied fault.
173 ${system_response_time}= Set Variable 60s
Steven Sombar43d6ba32018-02-16 11:25:09 -0600174
Steven Sombar1508aff2018-04-06 12:53:42 -0500175 # Choose a fan to test with, e.g., fan0.
176 ${test_fan_name}= Get From List ${fan_names} 0
Steven Sombar43d6ba32018-02-16 11:25:09 -0600177
178 ${initial_speed}= Get Target Speed Of Fans
Steven Sombar1508aff2018-04-06 12:53:42 -0500179 Rpvars test_fan_name initial_speed
Steven Sombar43d6ba32018-02-16 11:25:09 -0600180
181 # If initial speed is not already at maximum, set expect_increase.
182 # This flag is used later to determine if speed checking is
183 # to be done or not.
184 ${expect_increase}= Run Keyword If
Steven Sombar1508aff2018-04-06 12:53:42 -0500185 ... ${initial_speed} < ${max_speed}
Steven Sombar43d6ba32018-02-16 11:25:09 -0600186 ... Set Variable 1 ELSE Set Variable 0
187
188 Set Fan State ${test_fan_name} ${fan_nonfunctional}
Steven Sombar1508aff2018-04-06 12:53:42 -0500189
190 # Wait for error to be asserted.
191
192 :FOR ${n} IN RANGE 30
193 \ ${front_fault}= Get System LED State front_fault
194 \ ${rear_fault}= Get System LED State rear_fault
195 \ Run Key U Sleep \ 1s
196 \ Exit For Loop If '${front_fault}' == 'On' and '${rear_fault}' == 'On'
Steven Sombar43d6ba32018-02-16 11:25:09 -0600197
198 Verify System Error Indication Due To Fans
199
200 # Verify the error log is for test_fan_name.
201 ${elog_entries}= Get Logging Entry List
202 :FOR ${elog_entry} IN @{elog_entries}
203 \ ${elog_entry_callout}= Set Variable ${elog_entry}/callout
204 \ ${endpoint}= Read Attribute ${elog_entry_callout} endpoints
205 \ ${endpoint_name}= Get From List ${endpoint} 0
206 \ Should Contain ${endpoint_name} ${test_fan_name}
207 ... msg=Error log present but not for ${test_fan_name}.
208
Steven Sombar1508aff2018-04-06 12:53:42 -0500209 Run Key U Sleep \ ${system_response_time}
210
211 # A heavily loaded system may have powered-off.
212 ${host_state}= Get Host State
213 Rpvars host_state
214 Run Keyword If 'Running' != '${host_state}' Pass Execution
215 ... msg=System shutdown so skipping remainder of test.
216
Steven Sombar43d6ba32018-02-16 11:25:09 -0600217 ${new_fan_speed}= Get Target Speed Of Fans
218 Rpvars expect_increase initial_speed new_fan_speed
219
220 # Fail if current fan speed did not increase past the initial
221 # speed, but do this check only if not at maximum speed to begin with.
222 Run Keyword If
223 ... ${expect_increase} == 1 and ${new_fan_speed} < ${initial_speed}
224 ... Fail msg=Remaining fans did not increase speed with loss of one fan.
225
Steven Sombar43d6ba32018-02-16 11:25:09 -0600226
227Verify System Shutdown Due To Fans
228 [Documentation] Shut down when not enough fans.
229 [Tags] Verify_System_Shutdown_Due_To_Fans
230
Steven Sombar1508aff2018-04-06 12:53:42 -0500231 ${wait_after_poweroff}= Set Variable 15s
232
233 # The previous test may have shutdown the system.
234 REST Power On stack_mode=skip
235
Steven Sombar43d6ba32018-02-16 11:25:09 -0600236 # Set fans to be non-functional.
237 :FOR ${fan_name} IN @{fan_names}
238 \ Set Fan State ${fan_name} ${fan_nonfunctional}
239
240 # System should notice the non-functional fans and power-off the
241 # system. The Wait For PowerOff keyword will time-out and report
242 # an error if power off does not happen within a reasonable time.
243 Wait For PowerOff
244
Steven Sombar1508aff2018-04-06 12:53:42 -0500245 Run Key U Sleep \ ${wait_after_poweroff}
246
Steven Sombar43d6ba32018-02-16 11:25:09 -0600247 Verify System Error Indication Due To Fans
248
249 # Verify there is an error log because of the shutdown.
250 ${expect}= Catenate
251 ... xyz.openbmc_project.State.Shutdown.Inventory.Error.Fan
252 ${elog_entries}= Get Logging Entry List
253 :FOR ${elog_entry} IN @{elog_entries}
254 \ ${elog_message}= Read Attribute ${elog_entry} Message
255 \ ${found}= Set Variable 1
256 \ Run Keyword If '${elog_message}' == '${expect}' Exit For Loop
257 \ ${found}= Set Variable 0
258 Run Keyword If not ${found} Fail
259 ... msg=No error log for event Shutdown.Inventory.Error.Fan.
260
261
262*** Keywords ***
263
Steven Sombar43d6ba32018-02-16 11:25:09 -0600264Reset Fans
265 [Documentation] Set the fans to functional state.
266 # Set state of fans to functional by writing 1 to the Functional
267 # attribute of each fan in the @{fan_names} list. If @{fan_names}
268 # is empty nothing is done.
269
270 # Description of Argument(s):
271 # fans Suite Variable which is a list containing the
272 # names of the fans (e.g., fan0 fan2 fan3).
273
274 :FOR ${fan_name} IN @{fan_names}
275 \ Set Fan State ${fan_name} ${fan_functional}
276
277
278Suite Setup Execution
279 [Documentation] Do the pre-test setup.
280
281 REST Power On stack_mode=skip
Steven Sombar1508aff2018-04-06 12:53:42 -0500282
283 # The @{fan_names} list holds the names of the fans in the system.
284 @{fan_names} Create List
285 ${fan_names}= Get Fan Names ${fan_names}
286 Set Suite Variable ${fan_names} children=true
287
288 ${number_of_fans}= Get Length ${fan_names}
289 Set Suite Variable ${number_of_fans} children=true
290
291 Reset Fans
292 Run Key U Sleep \ 15s
George Keishing32fe4e12018-07-13 05:06:47 -0500293 Delete All Error Logs
Steven Sombar43d6ba32018-02-16 11:25:09 -0600294 Set System LED State front_fault Off
295 Set System LED State rear_fault Off
296
297
298Test Teardown Execution
299 [Documentation] Do the post-test teardown.
300
301 FFDC On Test Case Fail
302 Reset Fans
Steven Sombar1508aff2018-04-06 12:53:42 -0500303 Run Key U Sleep \ 15s
George Keishing32fe4e12018-07-13 05:06:47 -0500304 Delete All Error Logs
Steven Sombar43d6ba32018-02-16 11:25:09 -0600305 Set System LED State front_fault Off
306 Set System LED State rear_fault Off