blob: dbd1d221a2e1dd595c71d5bcaf1c875679f5ebce [file] [log] [blame]
George Keishing0fff68a2017-04-27 22:25:49 -05001*** Settings ***
Steven Sombar130a04f2017-07-16 10:02:37 -05002Documentation Keywords for system test.
George Keishing0fff68a2017-04-27 22:25:49 -05003
George Keishing91c852f2017-05-03 02:48:21 -05004Library ../lib/gen_robot_keyword.py
5Resource ../extended/obmc_boot_test_resource.robot
George Keishing0fff68a2017-04-27 22:25:49 -05006Resource ../lib/utils.robot
George Keishing91c852f2017-05-03 02:48:21 -05007Resource ../lib/state_manager.robot
George Keishingcff7aa02017-05-16 08:05:10 -05008Resource ../lib/rest_client.robot
George Keishingbbfec9e2017-05-17 10:47:21 -05009Resource resource.txt
George Keishing4ef29082017-05-13 05:01:23 -050010Library OperatingSystem
11Library DateTime
12
George Keishing0fff68a2017-04-27 22:25:49 -050013*** Variables ***
14
Steven Sombar130a04f2017-07-16 10:02:37 -050015${htx_log_dir_path} ${EXECDIR}${/}logs${/}
George Keishing4ef29082017-05-13 05:01:23 -050016
George Keishing4bbf5202017-05-18 06:55:53 -050017# Error strings to check from dmesg.
Steven Sombar130a04f2017-07-16 10:02:37 -050018${ERROR_REGEX} error|GPU|NVRM|nvidia
George Keishing4bbf5202017-05-18 06:55:53 -050019
20# GPU specific error message from dmesg.
Steven Sombar130a04f2017-07-16 10:02:37 -050021${ERROR_DBE_MSG} (DBE) has been detected on GPU
22
23# Inventory - List of I/O devices to collect for Inventory
24@{I/O} communication disk display generic input multimedia
25... network printer tape
26
27# Inventory Paths of the JSON and YAML files
Steven Sombara2f166c2017-08-02 14:22:45 -050028${json_tmp_file_path} ${EXECDIR}/inventory_temp_file.json
29${yaml_file_path} ${EXECDIR}/inventory_temp_file.yaml
Steven Sombar130a04f2017-07-16 10:02:37 -050030
31
George Keishing4ef29082017-05-13 05:01:23 -050032
George Keishing0fff68a2017-04-27 22:25:49 -050033*** Keywords ***
34
35Execute Command On OS
36 [Documentation] Execute given command on OS and return output.
37 [Arguments] ${command}
George Keishing91c852f2017-05-03 02:48:21 -050038 # Description of argument(s):
39 # command Shell command to be executed on OS.
George Keishing0fff68a2017-04-27 22:25:49 -050040 ${stdout} ${stderr}= Execute Command ${command} return_stderr=True
41 Should Be Empty ${stderr}
42 [Return] ${stdout}
43
George Keishing0fff68a2017-04-27 22:25:49 -050044Login To OS
45 [Documentation] Login to OS Host.
46 [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME}
47 ... ${os_password}=${OS_PASSWORD}
George Keishingcff7aa02017-05-16 08:05:10 -050048 ... ${alias_name}=os_connection
George Keishing91c852f2017-05-03 02:48:21 -050049 # Description of argument(s):
50 # os_host IP address of the OS Host.
51 # os_username OS Host Login user name.
52 # os_password OS Host Login passwrd.
George Keishingcff7aa02017-05-16 08:05:10 -050053 # alias_name Default OS SSH session connection alias name.
54 # TODO: Generalize alias naming using openbmc/openbmc-test-automation#633
George Keishing0fff68a2017-04-27 22:25:49 -050055
George Keishing91c852f2017-05-03 02:48:21 -050056 Ping Host ${os_host}
George Keishing664a0162017-06-05 12:24:24 -050057 SSHLibrary.Open Connection ${os_host} alias=${alias_name}
George Keishing91c852f2017-05-03 02:48:21 -050058 Login ${os_username} ${os_password}
59
60
George Keishing88366b32017-05-12 13:50:48 -050061Tool Exist
62 [Documentation] Check whether given tool is installed on OS.
63 [Arguments] ${tool_name}
64 # Description of argument(s):
65 # tool_name Tool name whose existence is to be checked.
George Keishing91c852f2017-05-03 02:48:21 -050066 Login To OS
George Keishing88366b32017-05-12 13:50:48 -050067 ${output}= Execute Command On OS which ${tool_name}
68 Should Contain ${output} ${tool_name}
69 ... msg=Please install ${tool_name} tool.
George Keishing91c852f2017-05-03 02:48:21 -050070
71
72Boot To OS
73 [Documentation] Boot host OS.
74 Run Key OBMC Boot Test \ REST Power On
75
76
77Power Off Host
78 [Documentation] Power off host.
79 Run Key OBMC Boot Test \ REST Power Off
George Keishing0fff68a2017-04-27 22:25:49 -050080
George Keishing6fec3ab2017-05-05 10:45:08 -050081
82File Exist On OS
83 [Documentation] Check if the given file path exist on OS.
84 [Arguments] ${file_path}
85 # Description of argument(s):
86 # file_path Absolute file path.
87
88 Login To OS
89 ${out}= Execute Command On OS ls ${file_path}
90 Log To Console \n File Exist: ${out}
91
George Keishing8740a0c2017-05-13 07:19:16 -050092
93Is HTX Running
94 [Documentation] Check if the HTX exerciser is currently running.
95
96 ${status}= Execute Command On OS htxcmdline -status
97 Should Not Contain ${status} Daemon state is <IDLE>
George Keishing4ef29082017-05-13 05:01:23 -050098
99
100Write Log Data To File
101 [Documentation] Write log data to the logs directory.
102 [Arguments] ${data}= ${log_file_path}=
103 # Description of argument(s):
104 # data String buffer.
105 # log_file_path The log file path.
106
107 Create File ${log_file_path} ${data}
108
109
110Collect HTX Log Files
111 [Documentation] Collect status and error log files.
112 # Collects the following files:
113 # HTX error log file /tmp/htxerr
114 # HTX status log file /tmp/htxstats
115
116 # Create logs directory and get current datetime.
117 Create Directory ${htx_log_dir_path}
118 ${cur_datetime}= Get Current Date result_format=%Y%m%d%H%M%S%f
119
120 File Exist On OS /tmp/htxerr
121 ${htx_err}= Execute Command On BMC cat /tmp/htxerr
122 Write Log Data To File
123 ... ${htx_err} ${htx_log_dir_path}/${OS_HOST}${cur_datetime}.htxerr
124
125 File Exist On OS /tmp/htxstats
126 ${htx_stats}= Execute Command On BMC cat /tmp/htxstats
127 Write Log Data To File
128 ... ${htx_stats} ${htx_log_dir_path}/${OS_HOST}_${cur_datetime}.htxstats
129
George Keishingcff7aa02017-05-16 08:05:10 -0500130
131REST Upload File To BMC
132 [Documentation] Upload a file via REST to BMC.
133
134 # Generate 32 MB file size
135 Run dd if=/dev/zero of=dummyfile bs=1 count=0 seek=32MB
136 OperatingSystem.File Should Exist dummyfile
137
138 # Get the content of the file and upload to BMC
139 ${image_data}= OperatingSystem.Get Binary File dummyfile
140
141 # Get REST session to BMC
142 Initialize OpenBMC
143
144 # Create the REST payload headers and data
145 ${data}= Create Dictionary data ${image_data}
146 ${headers}= Create Dictionary Content-Type=application/octet-stream
147 ... Accept=application/octet-stream
148 Set To Dictionary ${data} headers ${headers}
149
150 ${resp}= Post Request openbmc /upload/image &{data}
151 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
152
153 # Delete uploaded image file.
154 # TODO: Delete via REST openbmc/openbmc#1550
155 # Take SSH connection to BMC and switch to BMC connection to perform
156 # the task.
157 &{bmc_connection_args}= Create Dictionary alias=bmc_connection
158 Open Connection And Log In &{bmc_connection_args}
159
160 # Currently OS SSH session is active, switch to BMC connection.
161 Switch Connection bmc_connection
162 Execute Command On BMC rm -f /tmp/images/*
163
164 # Switch back to OS SSH connection.
165 Switch Connection os_connection
166
George Keishing4bbf5202017-05-18 06:55:53 -0500167
168Check For Errors On OS Dmesg Log
169 [Documentation] Check if dmesg has nvidia errors logged.
170
171 ${dmesg_log}= Execute Command On OS dmesg | egrep '${ERROR_REGEX}'
172 # To enable multiple string check.
173 Should Not Contain Any ${dmesg_log} ${ERROR_DBE_MSG}
174
175
176Collect NVIDIA Log File
177 [Documentation] Collect ndivia-smi command output.
George Keishing664a0162017-06-05 12:24:24 -0500178 [Arguments] ${suffix}
179 # Description of argument(s):
180 # suffix String name to append.
George Keishing4bbf5202017-05-18 06:55:53 -0500181
182 # Collects the output of ndivia-smi cmd output.
183 # TODO: GPU current temperature threshold check.
184 # openbmc/openbmc-test-automation#637
185 # +-----------------------------------------------------------------------------+
186 # | NVIDIA-SMI 361.89 Driver Version: 361.89 |
187 # |-------------------------------+----------------------+----------------------+
188 # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
189 # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
190 # |===============================+======================+======================|
191 # | 0 Tesla P100-SXM2... On | 0002:01:00.0 Off | 0 |
192 # | N/A 25C P0 35W / 300W | 931MiB / 16280MiB | 0% Default |
193 # +-------------------------------+----------------------+----------------------+
194 # | 1 Tesla P100-SXM2... On | 0003:01:00.0 Off | 0 |
195 # | N/A 26C P0 40W / 300W | 1477MiB / 16280MiB | 0% Default |
196 # +-------------------------------+----------------------+----------------------+
197 # | 2 Tesla P100-SXM2... On | 0006:01:00.0 Off | 0 |
198 # | N/A 25C P0 35W / 300W | 931MiB / 16280MiB | 0% Default |
199 # +-------------------------------+----------------------+----------------------+
200 # | 3 Tesla P100-SXM2... On | 0007:01:00.0 Off | 0 |
201 # | N/A 44C P0 290W / 300W | 965MiB / 16280MiB | 99% Default |
202 # +-------------------------------+----------------------+----------------------+
203 # +-----------------------------------------------------------------------------+
204 # | Processes: GPU Memory |
205 # | GPU PID Type Process name Usage |
206 # |=============================================================================|
207 # | 0 28459 C hxenvidia 929MiB |
208 # | 1 28460 C hxenvidia 1475MiB |
209 # | 2 28461 C hxenvidia 929MiB |
210 # | 3 28462 C hxenvidia 963MiB |
211 # +-----------------------------------------------------------------------------+
212
213 # Create logs directory and get current datetime.
214 Create Directory ${htx_log_dir_path}
215 ${cur_datetime}= Get Current Date result_format=%Y%m%d%H%M%S%f
216
217 ${nvidia_out}= Execute Command On BMC nvidia-smi
218 Write Log Data To File
George Keishing664a0162017-06-05 12:24:24 -0500219 ... ${nvidia_out}
220 ... ${htx_log_dir_path}/${OS_HOST}_${cur_datetime}.nvidia_${suffix}
George Keishingbbfec9e2017-05-17 10:47:21 -0500221
222
George Keishingdc1691d2017-12-07 12:17:46 -0600223Test Setup Execution
George Keishingbbfec9e2017-05-17 10:47:21 -0500224 [Documentation] Do the initial test setup.
225 # 1. Check if HTX tool exist.
226 # 2. Power on
227
228 Boot To OS
George Keishingdc1691d2017-12-07 12:17:46 -0600229 Delete All Error Logs
George Keishing5e2dcd02017-06-09 12:01:28 -0500230 Tool Exist htxcmdline
Steven Sombar2c97f812017-10-31 09:55:13 -0500231 Tool Exist lshw
George Keishingbbfec9e2017-05-17 10:47:21 -0500232
233 # Shutdown if HTX is running.
234 ${status}= Run Keyword And Return Status Is HTX Running
235 Run Keyword If '${status}' == 'True'
236 ... Shutdown HTX Exerciser
237
238
239Create Default MDT Profile
240 [Documentation] Create default mdt.bu profile and run.
241
242 Rprint Timen Create HTX mdt profile.
243
244 ${profile}= Execute Command On OS htxcmdline -createmdt
245 Rprintn ${profile}
246 Should Contain ${profile} mdts are created successfully
247
248
249Run MDT Profile
250 [Documentation] Load user pre-defined MDT profile.
251
252 Rprint Timen Start HTX mdt profile execution.
253 ${htx_run}= Execute Command On OS
254 ... htxcmdline -run -mdt ${HTX_MDT_PROFILE}
255 Rprintn ${htx_run}
256 Should Contain ${htx_run} Activated
257
258
259Check HTX Run Status
260 [Documentation] Get HTX exerciser status and check for error.
261
262 Rprint Timen Check HTX mdt Status and error.
263 ${status}= Execute Command On OS
264 ... htxcmdline -status -mdt ${HTX_MDT_PROFILE}
265 Rprintn ${status}
266
267 ${errlog}= Execute Command On OS htxcmdline -geterrlog
268 Rprintn ${errlog}
269
270 Should Contain ${errlog} file </tmp/htxerr> is empty
271
272
273Shutdown HTX Exerciser
274 [Documentation] Shut down HTX exerciser run.
275
276 Rprint Timen Shutdown HTX Run
277 ${shutdown}= Execute Command On OS
278 ... htxcmdline -shutdown -mdt ${HTX_MDT_PROFILE}
279 Rprintn ${shutdown}
280 Should Contain ${shutdown} shutdown successfully
281
Steven Sombar130a04f2017-07-16 10:02:37 -0500282
283Create JSON Inventory File
284 [Documentation] Create a JSON inventory file, and make a YAML copy.
285 [Arguments] ${json_file_path}
286 # Description of argument:
287 # json_file_path Where the inventory file is wrtten to.
288
289 Login To OS
290 Compile Inventory JSON
291 Run json2yaml ${json_tmp_file_path} ${yaml_file_path}
292 # Format to JSON pretty print to file.
293 Run python -m json.tool ${json_tmp_file_path} > ${json_file_path}
294 OperatingSystem.File Should Exist ${json_file_path}
295
296
297Compile Inventory JSON
298 [Documentation] Compile the Inventory into a JSON file.
299 Create File ${json_tmp_file_path}
300 Write New JSON List ${json_tmp_file_path} Inventory
301 Retrieve HW Info And Write processor ${json_tmp_file_path}
302 Retrieve HW Info And Write memory ${json_tmp_file_path}
303 Retrieve HW Info And Write List ${I/O} ${json_tmp_file_path} I/O last
304 Close New JSON List ${json_tmp_file_path}
305
306Write New JSON List
307 [Documentation] Start a new JSON list element in file.
308 [Arguments] ${json_tmp_file_path} ${json_field_name}
309 # Description of argument(s):
310 # json_tmp_file_path Name of file to write to.
311 # json_field_name Name to give json list element.
312 Append to File ${json_tmp_file_path} { "${json_field_name}" : [
313
314Close New JSON List
315 [Documentation] Close JSON list element in file.
316 [Arguments] ${json_tmp_file_path}
317 # Description of argument(s):
318 # json_tmp_file_path Path of file to write to.
319 Append to File ${json_tmp_file_path} ]}
320
321Retrieve HW Info And Write
322 [Documentation] Retrieve and write info, add a comma if not last item.
323 [Arguments] ${class} ${json_tmp_file_path} ${last}=false
324 # Description of argument(s):
325 # class Device class to retrieve with lshw.
326 # json_tmp_file_path Path of file to write to.
327 # last Is this the last element in the parent JSON?
328 Write New JSON List ${json_tmp_file_path} ${class}
329 ${output} = Retrieve Hardware Info ${class}
330 ${output} = Clean Up String ${output}
331 Run Keyword if ${output.__class__ is not type(None)}
332 ... Append To File ${json_tmp_file_path} ${output}
333 Close New JSON List ${json_tmp_file_path}
334 Run Keyword if '${last}' == 'false'
335 ... Append to File ${json_tmp_file_path} ,
336
337Retrieve HW Info And Write List
338 [Documentation] Does a Retrieve/Write with a list of classes and
339 ... encapsulates them into one large JSON element.
340 [Arguments] ${list} ${json_tmp_file_path} ${json_field_name}
341 ... ${last}=false
342 # Description of argument(s):
343 # list The list of devices classes to retrieve with lshw.
344 # json_tmp_file_path Path of file to write to.
345 # json_field_name Name of the JSON element to encapsulate this list.
346 # last Is this the last element in the parent JSON?
347 Write New JSON List ${json_tmp_file_path} ${json_field_name}
348 : FOR ${class} IN @{list}
349 \ ${tail} Get From List ${list} -1
350 \ Run Keyword if '${tail}' == '${class}'
351 \ ... Retrieve HW Info And Write ${class} ${json_tmp_file_path} true
352 \ ... ELSE Retrieve HW Info And Write ${class} ${json_tmp_file_path}
353 Close New JSON List ${json_tmp_file_path}
354 Run Keyword if '${last}' == 'false'
355 ... Append to File ${json_tmp_file_path} ,
356
357Retrieve Hardware Info
358 [Documentation] Retrieves the lshw output of the device class as JSON.
359 [Arguments] ${class}
360 # Description of argument(s):
361 # class Device class to retrieve with lshw.
362 ${output} = Execute Command On OS lshw -c ${class} -json
363 ${output} = Verify JSON string ${output}
364 [Return] ${output}
365
366Verify JSON String
Gunnar Mills28e403b2017-10-25 16:16:38 -0500367 [Documentation] Ensure the JSON string content is separated by commas.
Steven Sombar130a04f2017-07-16 10:02:37 -0500368 [Arguments] ${unver_string}
369 # Description of argument(s):
370 # unver_string JSON String we will be checking for lshw comma errors.
371 ${unver_string} = Convert to String ${unver_string}
372 ${ver_string} = Replace String Using Regexp ${unver_string} }\\s*{ },{
373 [Return] ${ver_string}
374
375Clean Up String
376 [Documentation] Remove extra whitespace and trailing commas.
377 [Arguments] ${dirty_string}
378 # Description of argument(s):
379 # dirty_string String that will be space stripped and have comma removed.
380 ${clean_string} = Strip String ${dirty_string}
381 ${last_char} = Get Substring ${clean_string} -1
382 ${trimmed_string} = Get Substring ${clean_string} 0 -1
383 ${clean_string} = Set Variable If '${last_char}' == ','
384 ... ${trimmed_string} ${clean_string}
385 [Return] ${clean_string}