blob: f180a17acb3ae59f3d9f73e49362649bab07f5a9 [file] [log] [blame]
George Keishing8b082fd2017-05-03 13:42:39 -05001*** Settings ***
George Keishing8b082fd2017-05-03 13:42:39 -05002
Steven Sombar130a04f2017-07-16 10:02:37 -05003Documentation Stress the system using HTX exerciser.
4
5# Test Parameters:
6# OPENBMC_HOST The BMC host name or IP address.
7# OS_HOST The OS host name or IP Address.
8# OS_USERNAME The OS login userid (usually root).
9# OS_PASSWORD The password for the OS login.
Steven Sombar2c97f812017-10-31 09:55:13 -050010# HTX_DURATION Duration of HTX run, for example, 2h, or 30m.
Steven Sombar130a04f2017-07-16 10:02:37 -050011# HTX_LOOP The number of times to loop HTX.
12# HTX_INTERVAL The time delay between consecutive checks of HTX
Steven Sombar2c97f812017-10-31 09:55:13 -050013# status, for example, 15m.
Steven Sombar130a04f2017-07-16 10:02:37 -050014# In summary: Run HTX for $HTX_DURATION, looping
Steven Sombar0278b132018-01-09 14:41:32 -060015# $HTX_LOOP times checking for errors every
16# $HTX_INTERVAL. Then allow extra time for OS
17# Boot, HTX startup, shutdown.
Steven Sombar130a04f2017-07-16 10:02:37 -050018# HTX_KEEP_RUNNING If set to 1, this indicates that the HTX is to
Steven Sombar2c97f812017-10-31 09:55:13 -050019# continue running after an error was found.
Steven Sombara2f166c2017-08-02 14:22:45 -050020# CHECK_INVENTORY If set to 0 or False, OS inventory checking before
21# and after each HTX run will be disabled. This
Steven Sombar2c97f812017-10-31 09:55:13 -050022# parameter is optional. The default value is True.
Steven Sombara2f166c2017-08-02 14:22:45 -050023# PREV_INV_FILE_PATH The file path and name of an initial previous
24# inventory snapshot file in JSON format. Inventory
25# snapshots taken before and after each HTX run will
26# be compared to this file.
Steven Sombar2c97f812017-10-31 09:55:13 -050027# This parameter is optional. If not specified an
Steven Sombara2f166c2017-08-02 14:22:45 -050028# initial inventory snapshot will be taken before
29# HTX startup.
Steven Sombarb6749a62017-10-06 08:34:23 -050030# INV_IGNORE_LIST A comma-delimited list of strings that
Steven Sombar48ad01d2017-08-17 14:17:37 -050031# indicate what to ignore if there are inventory
Steven Sombarb6749a62017-10-06 08:34:23 -050032# differences, (e.g., processor "size").
Steven Sombar48ad01d2017-08-17 14:17:37 -050033# If differences are found during inventory checking
Steven Sombarb6749a62017-10-06 08:34:23 -050034# and those items are in this list, the
Steven Sombar48ad01d2017-08-17 14:17:37 -050035# differences will be ignored. This parameter is
36# optional. If not specified the default value is
Steven Sombarb6749a62017-10-06 08:34:23 -050037# "size".
Steven Sombar130a04f2017-07-16 10:02:37 -050038
39Resource ../syslib/utils_os.robot
George Keishingdc1691d2017-12-07 12:17:46 -060040Resource ../lib/openbmc_ffdc_utils.robot
Steven Sombar3ecb6892018-03-23 11:41:08 -050041Resource ../lib/logging_utils.robot
George Keishing559f8382018-10-28 02:17:49 -050042Resource ../lib/code_update_utils.robot
Steven Sombar130a04f2017-07-16 10:02:37 -050043Library ../syslib/utils_keywords.py
Steven Sombarb6749a62017-10-06 08:34:23 -050044Library ../lib/utils_files.py
Steven Sombar3ecb6892018-03-23 11:41:08 -050045Library ../lib/logging_utils.py
Steven Sombare0be9152018-08-08 08:22:53 -050046Library ../syslib/utils_os.py
George Keishingffbe8ce2017-05-04 12:08:54 -050047
Steven Sombar16dba512017-10-20 07:20:08 -050048Suite Setup Run Keyword Start SOL Console Logging
George Keishingdc1691d2017-12-07 12:17:46 -060049Test Setup Test Setup Execution
50Test Teardown Test Teardown Execution
George Keishing8b082fd2017-05-03 13:42:39 -050051
Steven Sombar6f9d41d2017-08-08 13:35:30 -050052
George Keishing8b082fd2017-05-03 13:42:39 -050053*** Variables ****
54
Steven Sombar130a04f2017-07-16 10:02:37 -050055${stack_mode} skip
Steven Sombar6f9d41d2017-08-08 13:35:30 -050056${json_initial_file_path} ${EXECDIR}/os_inventory_initial.json
57${json_final_file_path} ${EXECDIR}/os_inventory_final.json
58${json_diff_file_path} ${EXECDIR}/os_inventory_diff.json
Steven Sombara2f166c2017-08-02 14:22:45 -050059${CHECK_INVENTORY} True
Steven Sombarb6749a62017-10-06 08:34:23 -050060${INV_IGNORE_LIST} size
Steven Sombar2c97f812017-10-31 09:55:13 -050061${PREV_INV_FILE_PATH} NONE
George Keishingffbe8ce2017-05-04 12:08:54 -050062
Steven Sombar377a2cd2019-04-09 11:42:56 -050063${rest_keyword} REST
64
Steven Sombard6365f12019-04-02 09:38:29 -050065# Error log Severities to ignore when checking for eSELs.
66@{ESEL_WHITELIST}
67... xyz.openbmc_project.Logging.Entry.Level.Informational
68
Steven Sombar6f9d41d2017-08-08 13:35:30 -050069
George Keishing8b082fd2017-05-03 13:42:39 -050070*** Test Cases ***
71
72Hard Bootme Test
73 [Documentation] Stress the system using HTX exerciser.
George Keishingffbe8ce2017-05-04 12:08:54 -050074 [Tags] Hard_Bootme_Test
George Keishing8b082fd2017-05-03 13:42:39 -050075
Michael Walshc108e422019-03-28 12:27:18 -050076 Printn
Steven Sombar2c97f812017-10-31 09:55:13 -050077 Rpvars HTX_DURATION HTX_LOOP HTX_INTERVAL CHECK_INVENTORY
78 ... INV_IGNORE_LIST PREV_INV_FILE_PATH
Steven Sombar48ad01d2017-08-17 14:17:37 -050079
Steven Sombar2c97f812017-10-31 09:55:13 -050080 Run Keyword If '${PREV_INV_FILE_PATH}' != 'NONE'
81 ... OperatingSystem.File Should Exist ${PREV_INV_FILE_PATH}
Steven Sombar48ad01d2017-08-17 14:17:37 -050082
Steven Sombar2c97f812017-10-31 09:55:13 -050083 Set Suite Variable ${PREV_INV_FILE_PATH} children=true
Steven Sombar48ad01d2017-08-17 14:17:37 -050084 Set Suite Variable ${INV_IGNORE_LIST} children=true
Steven Sombar2c97f812017-10-31 09:55:13 -050085
86 # Set up the iteration (loop) counter.
Steven Sombar16dba512017-10-20 07:20:08 -050087 Set Suite Variable ${iteration} ${0} children=true
Steven Sombar130a04f2017-07-16 10:02:37 -050088
Steven Sombar2c97f812017-10-31 09:55:13 -050089 # Estimate the time required for a single iteration loop.
90 # HTX_DURATION + 10 minutes for OS boot, HTX startup, shutdown.
91 ${loop_body_seconds}= Add Time To Time ${HTX_DURATION} 10m
92 Set Suite Variable ${loop_body_seconds} children=true
93 # And save it in printable (compact) format.
94 ${estimated_loop_time}= Convert Time
95 ... ${loop_body_seconds} result_format=compact
96 Set Suite Variable ${estimated_loop_time} children=true
97
98 # Estimated time remaining = loop_body_seconds * HTX_LOOP + 5m
99 ${est_seconds_left}= Evaluate ${loop_body_seconds}*${HTX_LOOP}+(5*60)
100 Set Suite Variable ${est_seconds_left} children=true
101
Steven Sombar130a04f2017-07-16 10:02:37 -0500102 Repeat Keyword ${HTX_LOOP} times Run HTX Exerciser
George Keishingffbe8ce2017-05-04 12:08:54 -0500103
104
105*** Keywords ***
106
Steven Sombar16dba512017-10-20 07:20:08 -0500107
Steven Sombar130a04f2017-07-16 10:02:37 -0500108Run HTX Exerciser
109 [Documentation] Run HTX exerciser.
George Keishingffbe8ce2017-05-04 12:08:54 -0500110 # Test Flow:
Steven Sombar130a04f2017-07-16 10:02:37 -0500111 # - Power on.
112 # - Establish SSH connection session.
113 # - Do inventory collection, compare with
114 # previous inventory run if applicable.
115 # - Create HTX mdt profile.
116 # - Run HTX exerciser.
117 # - Check HTX status for errors.
118 # - Do inventory collection, compare with
119 # previous inventory run.
120 # - Power off.
George Keishingffbe8ce2017-05-04 12:08:54 -0500121
Steven Sombar16dba512017-10-20 07:20:08 -0500122 Set Suite Variable ${iteration} ${iteration + 1}
123 ${loop_count}= Catenate Starting iteration: ${iteration}
Steven Sombar2c97f812017-10-31 09:55:13 -0500124 ${estimated_time_remaining}= Convert Time
125 ... ${est_seconds_left} result_format=compact
Michael Walshc108e422019-03-28 12:27:18 -0500126 Printn
Steven Sombar2c97f812017-10-31 09:55:13 -0500127 Rpvars loop_count estimated_loop_time estimated_time_remaining
Steven Sombar16dba512017-10-20 07:20:08 -0500128
Steven Sombar377a2cd2019-04-09 11:42:56 -0500129 Run Keyword ${rest_keyword} Power On stack_mode=skip
Steven Sombar1ddc7c62018-06-01 11:55:34 -0500130 Run Key U Sleep \ 15s
George Keishingffbe8ce2017-05-04 12:08:54 -0500131
132 # Post Power off and on, the OS SSH session needs to be established.
George Keishing8b082fd2017-05-03 13:42:39 -0500133 Login To OS
134
Steven Sombara2f166c2017-08-02 14:22:45 -0500135 Run Keyword If '${CHECK_INVENTORY}' == 'True'
Steven Sombar130a04f2017-07-16 10:02:37 -0500136 ... Do Inventory And Compare ${json_initial_file_path}
Steven Sombar2c97f812017-10-31 09:55:13 -0500137 ... ${PREV_INV_FILE_PATH}
Steven Sombar130a04f2017-07-16 10:02:37 -0500138
George Keishing13f44e52017-05-12 15:28:12 -0500139 Run Keyword If '${HTX_MDT_PROFILE}' == 'mdt.bu'
140 ... Create Default MDT Profile
George Keishing8b082fd2017-05-03 13:42:39 -0500141
George Keishing13f44e52017-05-12 15:28:12 -0500142 Run MDT Profile
George Keishing8b082fd2017-05-03 13:42:39 -0500143
144 Loop HTX Health Check
145
George Keishingffbe8ce2017-05-04 12:08:54 -0500146 Shutdown HTX Exerciser
147
Steven Sombara2f166c2017-08-02 14:22:45 -0500148 Run Keyword If '${CHECK_INVENTORY}' == 'True'
Steven Sombar130a04f2017-07-16 10:02:37 -0500149 ... Do Inventory And Compare ${json_final_file_path}
Steven Sombar2c97f812017-10-31 09:55:13 -0500150 ... ${PREV_INV_FILE_PATH}
Steven Sombar130a04f2017-07-16 10:02:37 -0500151
Steven Sombar377a2cd2019-04-09 11:42:56 -0500152 Run Keyword ${rest_keyword} Power Off
George Keishingffbe8ce2017-05-04 12:08:54 -0500153
George Keishingae5d9782017-06-12 13:35:59 -0500154 # Close all SSH and REST active sessions.
155 Close All Connections
156 Flush REST Sessions
157
Michael Walshc108e422019-03-28 12:27:18 -0500158 Print Timen HTX Test ran for: ${HTX_DURATION}
George Keishing8b082fd2017-05-03 13:42:39 -0500159
Steven Sombar16dba512017-10-20 07:20:08 -0500160 ${loop_count}= Catenate Ending iteration: ${iteration}
Steven Sombar2c97f812017-10-31 09:55:13 -0500161
162 ${est_seconds_left}= Evaluate ${est_seconds_left}-${loop_body_seconds}
163 Set Suite Variable ${est_seconds_left} children=true
164 ${estimated_time_remaining}= Convert Time
165 ... ${est_seconds_left} result_format=compact
166
167 Rpvars loop_count estimated_time_remaining
Steven Sombar16dba512017-10-20 07:20:08 -0500168
George Keishing13f44e52017-05-12 15:28:12 -0500169
Steven Sombara2f166c2017-08-02 14:22:45 -0500170Do Inventory And Compare
Steven Sombar130a04f2017-07-16 10:02:37 -0500171 [Documentation] Do inventory and compare.
Steven Sombar2c97f812017-10-31 09:55:13 -0500172 [Arguments] ${inventory_file_path} ${PREV_INV_FILE_PATH}
Steven Sombar130a04f2017-07-16 10:02:37 -0500173 # Description of argument(s):
Steven Sombar2c97f812017-10-31 09:55:13 -0500174 # inventory_file_path The file to receive the inventory snapshot.
175 # PREV_INV_FILE_PATH The previous inventory to compare with.
Steven Sombar130a04f2017-07-16 10:02:37 -0500176
177 Create JSON Inventory File ${inventory_file_path}
Steven Sombar2c97f812017-10-31 09:55:13 -0500178 Run Keyword If '${PREV_INV_FILE_PATH}' != 'NONE'
Steven Sombar130a04f2017-07-16 10:02:37 -0500179 ... Compare Json Inventory Files ${inventory_file_path}
Steven Sombar2c97f812017-10-31 09:55:13 -0500180 ... ${PREV_INV_FILE_PATH}
181 ${PREV_INV_FILE_PATH}= Set Variable ${inventory_file_path}
182 Set Suite Variable ${PREV_INV_FILE_PATH} children=true
Steven Sombar130a04f2017-07-16 10:02:37 -0500183
184
185Compare Json Inventory Files
186 [Documentation] Compare JSON inventory files.
187 [Arguments] ${file1} ${file2}
188 # Description of argument(s):
189 # file1 A file that has an inventory snapshot in JSON format.
190 # file2 A file that has an inventory snapshot, to compare with file1.
191
Steven Sombarb6749a62017-10-06 08:34:23 -0500192 ${diff_rc}= File_Diff ${file1}
Steven Sombar48ad01d2017-08-17 14:17:37 -0500193 ... ${file2} ${json_diff_file_path} ${INV_IGNORE_LIST}
Steven Sombar130a04f2017-07-16 10:02:37 -0500194 Run Keyword If '${diff_rc}' != '${0}'
195 ... Report Inventory Mismatch ${diff_rc} ${json_diff_file_path}
Michael Walshc108e422019-03-28 12:27:18 -0500196 ... ELSE Print Timen Inventoy check: No differences found.
Steven Sombar130a04f2017-07-16 10:02:37 -0500197
198
199Report Inventory Mismatch
200 [Documentation] Report inventory mismatch.
201 [Arguments] ${diff_rc} ${json_diff_file_path}
202 # Description of argument(s):
203 # diff_rc The failing return code from the difference check.
204 # json_diff_file_path The file that has the latest inventory snapshot.
205
Steven Sombara2216d22017-08-09 12:18:00 -0500206 Log To Console Significant difference in inventory found, rc=${diff_rc}
207 Log To Console Differences are listed in file: no_newline=true
208 Log To Console ${json_diff_file_path}
209 Log To Console File Contents:
210 Wait Until Created ${json_diff_file_path}
211 ${file_contents}= OperatingSystem.Get File ${json_diff_file_path}
212 Log ${file_contents} level=WARN
213 Fail Significant difference in inventory found, rc=${diff_rc}
Steven Sombar130a04f2017-07-16 10:02:37 -0500214
215
Steven Sombar7cc95a32018-11-05 15:01:06 -0600216Check For ESELs
Steven Sombard6365f12019-04-02 09:38:29 -0500217 [Documentation] Terminate if eSELs with Severity field not on WHITELIST.
218
219 Print Timen Checking eSEL Error Logs.
Steven Sombar7cc95a32018-11-05 15:01:06 -0600220 ${error_logs}= Get Error Logs
Steven Sombard6365f12019-04-02 09:38:29 -0500221
Steven Sombar021cbc42019-03-28 11:49:21 -0500222 ${num_error_logs}= Get Length ${error_logs}
223 Rprint Vars num_error_logs
Steven Sombard6365f12019-04-02 09:38:29 -0500224 Return From Keyword If ${num_error_logs} == ${0}
225
226 Print Error Logs
227
228 # Get a list of the severities of the error logs.
229 ${error_log_severities}= Nested Get Severity ${error_logs}
230 # Subtract the WHITELIST from the error_log_severities.
231 ${problem_error_logs}=
232 ... Evaluate list(set($error_log_severities) - set($ESEL_WHITELIST))
233 ${num_error_logs_not_on_whitelist}= Get Length ${problem_error_logs}
234
235 Return From Keyword If ${num_error_logs_not_on_whitelist} == ${0}
236
237 Rprint Vars ESEL_WHITELIST
238 Fail msg=Found error logs with Severity not matching ESEL_WHITELIST.
Steven Sombar7cc95a32018-11-05 15:01:06 -0600239
240
George Keishing8b082fd2017-05-03 13:42:39 -0500241Loop HTX Health Check
George Keishingffbe8ce2017-05-04 12:08:54 -0500242 [Documentation] Run until HTX exerciser fails.
George Keishing8b082fd2017-05-03 13:42:39 -0500243 Repeat Keyword ${HTX_DURATION}
244 ... Run Keywords Check HTX Run Status
Steven Sombar7cc95a32018-11-05 15:01:06 -0600245 ... AND Check For ESELs
George Keishing8b082fd2017-05-03 13:42:39 -0500246 ... AND Sleep ${HTX_INTERVAL}
247
248
Steven Sombar0278b132018-01-09 14:41:32 -0600249Test Setup Execution
250 [Documentation] Do the initial test setup.
251
Steven Sombarac687c42018-08-09 13:39:36 -0500252 ${bmc_version} ${stderr} ${rc}= BMC Execute Command
253 ... cat /etc/os-release
Michael Walshc108e422019-03-28 12:27:18 -0500254 Printn
Steven Sombarac687c42018-08-09 13:39:36 -0500255 Rpvars bmc_version
Steven Sombare0be9152018-08-08 08:22:53 -0500256
George Keishing559f8382018-10-28 02:17:49 -0500257 ${pnor_version}= Get Host Software Objects Details
Steven Sombare0be9152018-08-08 08:22:53 -0500258 Rpvars pnor_version
259
Steven Sombar377a2cd2019-04-09 11:42:56 -0500260 ${is_redfish}= Run Keyword And Return Status Redfish.Login
261 ${rest_keyword}= Set Variable If ${is_redfish} Redfish REST
262 Rprint Vars rest_keyword
263 Set Suite Variable ${rest_keyword} children=true
264
265 Run Keyword ${rest_keyword} Power On stack_mode=skip
266
Steven Sombar1ddc7c62018-06-01 11:55:34 -0500267 Run Key U Sleep \ 15s
Steven Sombar0278b132018-01-09 14:41:32 -0600268 Delete All Error Logs
269 Tool Exist htxcmdline
270
Steven Sombarac687c42018-08-09 13:39:36 -0500271 ${os_release_info}= Get OS Release Info
272 Rpvars 1 os_release_info
273
Steven Sombar0278b132018-01-09 14:41:32 -0600274 # Shutdown if HTX is running.
Steven Sombar3ecb6892018-03-23 11:41:08 -0500275 ${status}= Is HTX Running
Steven Sombar0278b132018-01-09 14:41:32 -0600276 Run Keyword If '${status}' == 'True'
277 ... Shutdown HTX Exerciser
278
279
George Keishingdc1691d2017-12-07 12:17:46 -0600280Test Teardown Execution
Steven Sombare0be9152018-08-08 08:22:53 -0500281 [Documentation] Do the post-test teardown.
George Keishingffbe8ce2017-05-04 12:08:54 -0500282
George Keishing1a23b692017-05-06 12:19:41 -0500283 # Keep HTX running if user set HTX_KEEP_RUNNING to 1.
Steven Sombar130a04f2017-07-16 10:02:37 -0500284 Run Keyword If
285 ... '${TEST_STATUS}' == 'FAIL' and ${HTX_KEEP_RUNNING} == ${0}
George Keishingffbe8ce2017-05-04 12:08:54 -0500286 ... Shutdown HTX Exerciser
287
George Keishingae5d9782017-06-12 13:35:59 -0500288 ${keyword_buf}= Catenate Stop SOL Console Logging
289 ... \ targ_file_path=${EXECDIR}${/}logs${/}SOL.log
290 Run Key ${keyword_buf}
291
George Keishingffbe8ce2017-05-04 12:08:54 -0500292 FFDC On Test Case Fail
293 Close All Connections