blob: b7258a8692973770d5c64f8d7ebfb211ea5d4fa6 [file] [log] [blame]
George Keishing0fff68a2017-04-27 22:25:49 -05001*** Settings ***
Steven Sombar130a04f2017-07-16 10:02:37 -05002Documentation Keywords for system test.
George Keishing0fff68a2017-04-27 22:25:49 -05003
George Keishing91c852f2017-05-03 02:48:21 -05004Library ../lib/gen_robot_keyword.py
George Keishingcff7aa02017-05-16 08:05:10 -05005Library OperatingSystem
George Keishing91c852f2017-05-03 02:48:21 -05006Resource ../extended/obmc_boot_test_resource.robot
George Keishing0fff68a2017-04-27 22:25:49 -05007Resource ../lib/utils.robot
George Keishing91c852f2017-05-03 02:48:21 -05008Resource ../lib/state_manager.robot
George Keishingcff7aa02017-05-16 08:05:10 -05009Resource ../lib/rest_client.robot
George Keishingbbfec9e2017-05-17 10:47:21 -050010Resource resource.txt
George Keishing4ef29082017-05-13 05:01:23 -050011Library OperatingSystem
12Library DateTime
13
George Keishing0fff68a2017-04-27 22:25:49 -050014*** Variables ***
15
Steven Sombar130a04f2017-07-16 10:02:37 -050016${htx_log_dir_path} ${EXECDIR}${/}logs${/}
George Keishing4ef29082017-05-13 05:01:23 -050017
George Keishing4bbf5202017-05-18 06:55:53 -050018# Error strings to check from dmesg.
Steven Sombar130a04f2017-07-16 10:02:37 -050019${ERROR_REGEX} error|GPU|NVRM|nvidia
George Keishing4bbf5202017-05-18 06:55:53 -050020
21# GPU specific error message from dmesg.
Steven Sombar130a04f2017-07-16 10:02:37 -050022${ERROR_DBE_MSG} (DBE) has been detected on GPU
23
24# Inventory - List of I/O devices to collect for Inventory
25@{I/O} communication disk display generic input multimedia
26... network printer tape
27
28# Inventory Paths of the JSON and YAML files
Steven Sombara2f166c2017-08-02 14:22:45 -050029${json_tmp_file_path} ${EXECDIR}/inventory_temp_file.json
30${yaml_file_path} ${EXECDIR}/inventory_temp_file.yaml
Steven Sombar130a04f2017-07-16 10:02:37 -050031
32
George Keishing4ef29082017-05-13 05:01:23 -050033
George Keishing0fff68a2017-04-27 22:25:49 -050034*** Keywords ***
35
36Execute Command On OS
37 [Documentation] Execute given command on OS and return output.
38 [Arguments] ${command}
George Keishing91c852f2017-05-03 02:48:21 -050039 # Description of argument(s):
40 # command Shell command to be executed on OS.
George Keishing0fff68a2017-04-27 22:25:49 -050041 ${stdout} ${stderr}= Execute Command ${command} return_stderr=True
42 Should Be Empty ${stderr}
43 [Return] ${stdout}
44
George Keishing0fff68a2017-04-27 22:25:49 -050045Login To OS
46 [Documentation] Login to OS Host.
47 [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME}
48 ... ${os_password}=${OS_PASSWORD}
George Keishingcff7aa02017-05-16 08:05:10 -050049 ... ${alias_name}=os_connection
George Keishing91c852f2017-05-03 02:48:21 -050050 # Description of argument(s):
51 # os_host IP address of the OS Host.
52 # os_username OS Host Login user name.
53 # os_password OS Host Login passwrd.
George Keishingcff7aa02017-05-16 08:05:10 -050054 # alias_name Default OS SSH session connection alias name.
55 # TODO: Generalize alias naming using openbmc/openbmc-test-automation#633
George Keishing0fff68a2017-04-27 22:25:49 -050056
George Keishing91c852f2017-05-03 02:48:21 -050057 Ping Host ${os_host}
George Keishing664a0162017-06-05 12:24:24 -050058 SSHLibrary.Open Connection ${os_host} alias=${alias_name}
George Keishing91c852f2017-05-03 02:48:21 -050059 Login ${os_username} ${os_password}
60
61
George Keishing88366b32017-05-12 13:50:48 -050062Tool Exist
63 [Documentation] Check whether given tool is installed on OS.
64 [Arguments] ${tool_name}
65 # Description of argument(s):
66 # tool_name Tool name whose existence is to be checked.
George Keishing91c852f2017-05-03 02:48:21 -050067 Login To OS
George Keishing88366b32017-05-12 13:50:48 -050068 ${output}= Execute Command On OS which ${tool_name}
69 Should Contain ${output} ${tool_name}
70 ... msg=Please install ${tool_name} tool.
George Keishing91c852f2017-05-03 02:48:21 -050071
72
73Boot To OS
74 [Documentation] Boot host OS.
75 Run Key OBMC Boot Test \ REST Power On
76
77
78Power Off Host
79 [Documentation] Power off host.
80 Run Key OBMC Boot Test \ REST Power Off
George Keishing0fff68a2017-04-27 22:25:49 -050081
George Keishing6fec3ab2017-05-05 10:45:08 -050082
83File Exist On OS
84 [Documentation] Check if the given file path exist on OS.
85 [Arguments] ${file_path}
86 # Description of argument(s):
87 # file_path Absolute file path.
88
89 Login To OS
90 ${out}= Execute Command On OS ls ${file_path}
91 Log To Console \n File Exist: ${out}
92
George Keishing8740a0c2017-05-13 07:19:16 -050093
94Is HTX Running
95 [Documentation] Check if the HTX exerciser is currently running.
96
97 ${status}= Execute Command On OS htxcmdline -status
98 Should Not Contain ${status} Daemon state is <IDLE>
George Keishing4ef29082017-05-13 05:01:23 -050099
100
101Write Log Data To File
102 [Documentation] Write log data to the logs directory.
103 [Arguments] ${data}= ${log_file_path}=
104 # Description of argument(s):
105 # data String buffer.
106 # log_file_path The log file path.
107
108 Create File ${log_file_path} ${data}
109
110
111Collect HTX Log Files
112 [Documentation] Collect status and error log files.
113 # Collects the following files:
114 # HTX error log file /tmp/htxerr
115 # HTX status log file /tmp/htxstats
116
117 # Create logs directory and get current datetime.
118 Create Directory ${htx_log_dir_path}
119 ${cur_datetime}= Get Current Date result_format=%Y%m%d%H%M%S%f
120
121 File Exist On OS /tmp/htxerr
122 ${htx_err}= Execute Command On BMC cat /tmp/htxerr
123 Write Log Data To File
124 ... ${htx_err} ${htx_log_dir_path}/${OS_HOST}${cur_datetime}.htxerr
125
126 File Exist On OS /tmp/htxstats
127 ${htx_stats}= Execute Command On BMC cat /tmp/htxstats
128 Write Log Data To File
129 ... ${htx_stats} ${htx_log_dir_path}/${OS_HOST}_${cur_datetime}.htxstats
130
George Keishingcff7aa02017-05-16 08:05:10 -0500131
132REST Upload File To BMC
133 [Documentation] Upload a file via REST to BMC.
134
135 # Generate 32 MB file size
136 Run dd if=/dev/zero of=dummyfile bs=1 count=0 seek=32MB
137 OperatingSystem.File Should Exist dummyfile
138
139 # Get the content of the file and upload to BMC
140 ${image_data}= OperatingSystem.Get Binary File dummyfile
141
142 # Get REST session to BMC
143 Initialize OpenBMC
144
145 # Create the REST payload headers and data
146 ${data}= Create Dictionary data ${image_data}
147 ${headers}= Create Dictionary Content-Type=application/octet-stream
148 ... Accept=application/octet-stream
149 Set To Dictionary ${data} headers ${headers}
150
151 ${resp}= Post Request openbmc /upload/image &{data}
152 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
153
154 # Delete uploaded image file.
155 # TODO: Delete via REST openbmc/openbmc#1550
156 # Take SSH connection to BMC and switch to BMC connection to perform
157 # the task.
158 &{bmc_connection_args}= Create Dictionary alias=bmc_connection
159 Open Connection And Log In &{bmc_connection_args}
160
161 # Currently OS SSH session is active, switch to BMC connection.
162 Switch Connection bmc_connection
163 Execute Command On BMC rm -f /tmp/images/*
164
165 # Switch back to OS SSH connection.
166 Switch Connection os_connection
167
George Keishing4bbf5202017-05-18 06:55:53 -0500168
169Check For Errors On OS Dmesg Log
170 [Documentation] Check if dmesg has nvidia errors logged.
171
172 ${dmesg_log}= Execute Command On OS dmesg | egrep '${ERROR_REGEX}'
173 # To enable multiple string check.
174 Should Not Contain Any ${dmesg_log} ${ERROR_DBE_MSG}
175
176
177Collect NVIDIA Log File
178 [Documentation] Collect ndivia-smi command output.
George Keishing664a0162017-06-05 12:24:24 -0500179 [Arguments] ${suffix}
180 # Description of argument(s):
181 # suffix String name to append.
George Keishing4bbf5202017-05-18 06:55:53 -0500182
183 # Collects the output of ndivia-smi cmd output.
184 # TODO: GPU current temperature threshold check.
185 # openbmc/openbmc-test-automation#637
186 # +-----------------------------------------------------------------------------+
187 # | NVIDIA-SMI 361.89 Driver Version: 361.89 |
188 # |-------------------------------+----------------------+----------------------+
189 # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
190 # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
191 # |===============================+======================+======================|
192 # | 0 Tesla P100-SXM2... On | 0002:01:00.0 Off | 0 |
193 # | N/A 25C P0 35W / 300W | 931MiB / 16280MiB | 0% Default |
194 # +-------------------------------+----------------------+----------------------+
195 # | 1 Tesla P100-SXM2... On | 0003:01:00.0 Off | 0 |
196 # | N/A 26C P0 40W / 300W | 1477MiB / 16280MiB | 0% Default |
197 # +-------------------------------+----------------------+----------------------+
198 # | 2 Tesla P100-SXM2... On | 0006:01:00.0 Off | 0 |
199 # | N/A 25C P0 35W / 300W | 931MiB / 16280MiB | 0% Default |
200 # +-------------------------------+----------------------+----------------------+
201 # | 3 Tesla P100-SXM2... On | 0007:01:00.0 Off | 0 |
202 # | N/A 44C P0 290W / 300W | 965MiB / 16280MiB | 99% Default |
203 # +-------------------------------+----------------------+----------------------+
204 # +-----------------------------------------------------------------------------+
205 # | Processes: GPU Memory |
206 # | GPU PID Type Process name Usage |
207 # |=============================================================================|
208 # | 0 28459 C hxenvidia 929MiB |
209 # | 1 28460 C hxenvidia 1475MiB |
210 # | 2 28461 C hxenvidia 929MiB |
211 # | 3 28462 C hxenvidia 963MiB |
212 # +-----------------------------------------------------------------------------+
213
214 # Create logs directory and get current datetime.
215 Create Directory ${htx_log_dir_path}
216 ${cur_datetime}= Get Current Date result_format=%Y%m%d%H%M%S%f
217
218 ${nvidia_out}= Execute Command On BMC nvidia-smi
219 Write Log Data To File
George Keishing664a0162017-06-05 12:24:24 -0500220 ... ${nvidia_out}
221 ... ${htx_log_dir_path}/${OS_HOST}_${cur_datetime}.nvidia_${suffix}
George Keishingbbfec9e2017-05-17 10:47:21 -0500222
223
224Pre Test Case Execution
225 [Documentation] Do the initial test setup.
226 # 1. Check if HTX tool exist.
227 # 2. Power on
228
229 Boot To OS
George Keishing5e2dcd02017-06-09 12:01:28 -0500230 Tool Exist htxcmdline
Steven Sombar2c97f812017-10-31 09:55:13 -0500231 Tool Exist lshw
George Keishingbbfec9e2017-05-17 10:47:21 -0500232
233 # Shutdown if HTX is running.
234 ${status}= Run Keyword And Return Status Is HTX Running
235 Run Keyword If '${status}' == 'True'
236 ... Shutdown HTX Exerciser
237
238
239Create Default MDT Profile
240 [Documentation] Create default mdt.bu profile and run.
241
242 Rprint Timen Create HTX mdt profile.
243
244 ${profile}= Execute Command On OS htxcmdline -createmdt
245 Rprintn ${profile}
246 Should Contain ${profile} mdts are created successfully
247
248
249Run MDT Profile
250 [Documentation] Load user pre-defined MDT profile.
251
252 Rprint Timen Start HTX mdt profile execution.
253 ${htx_run}= Execute Command On OS
254 ... htxcmdline -run -mdt ${HTX_MDT_PROFILE}
255 Rprintn ${htx_run}
256 Should Contain ${htx_run} Activated
257
258
259Check HTX Run Status
260 [Documentation] Get HTX exerciser status and check for error.
261
262 Rprint Timen Check HTX mdt Status and error.
263 ${status}= Execute Command On OS
264 ... htxcmdline -status -mdt ${HTX_MDT_PROFILE}
265 Rprintn ${status}
266
267 ${errlog}= Execute Command On OS htxcmdline -geterrlog
268 Rprintn ${errlog}
269
270 Should Contain ${errlog} file </tmp/htxerr> is empty
271
272
273Shutdown HTX Exerciser
274 [Documentation] Shut down HTX exerciser run.
275
276 Rprint Timen Shutdown HTX Run
277 ${shutdown}= Execute Command On OS
278 ... htxcmdline -shutdown -mdt ${HTX_MDT_PROFILE}
279 Rprintn ${shutdown}
280 Should Contain ${shutdown} shutdown successfully
281
Steven Sombar130a04f2017-07-16 10:02:37 -0500282
283Create JSON Inventory File
284 [Documentation] Create a JSON inventory file, and make a YAML copy.
285 [Arguments] ${json_file_path}
286 # Description of argument:
287 # json_file_path Where the inventory file is wrtten to.
288
289 Login To OS
290 Compile Inventory JSON
291 Run json2yaml ${json_tmp_file_path} ${yaml_file_path}
292 # Format to JSON pretty print to file.
293 Run python -m json.tool ${json_tmp_file_path} > ${json_file_path}
294 OperatingSystem.File Should Exist ${json_file_path}
295
296
297Compile Inventory JSON
298 [Documentation] Compile the Inventory into a JSON file.
299 Create File ${json_tmp_file_path}
300 Write New JSON List ${json_tmp_file_path} Inventory
301 Retrieve HW Info And Write processor ${json_tmp_file_path}
302 Retrieve HW Info And Write memory ${json_tmp_file_path}
303 Retrieve HW Info And Write List ${I/O} ${json_tmp_file_path} I/O last
304 Close New JSON List ${json_tmp_file_path}
305
306Write New JSON List
307 [Documentation] Start a new JSON list element in file.
308 [Arguments] ${json_tmp_file_path} ${json_field_name}
309 # Description of argument(s):
310 # json_tmp_file_path Name of file to write to.
311 # json_field_name Name to give json list element.
312 Append to File ${json_tmp_file_path} { "${json_field_name}" : [
313
314Close New JSON List
315 [Documentation] Close JSON list element in file.
316 [Arguments] ${json_tmp_file_path}
317 # Description of argument(s):
318 # json_tmp_file_path Path of file to write to.
319 Append to File ${json_tmp_file_path} ]}
320
321Retrieve HW Info And Write
322 [Documentation] Retrieve and write info, add a comma if not last item.
323 [Arguments] ${class} ${json_tmp_file_path} ${last}=false
324 # Description of argument(s):
325 # class Device class to retrieve with lshw.
326 # json_tmp_file_path Path of file to write to.
327 # last Is this the last element in the parent JSON?
328 Write New JSON List ${json_tmp_file_path} ${class}
329 ${output} = Retrieve Hardware Info ${class}
330 ${output} = Clean Up String ${output}
331 Run Keyword if ${output.__class__ is not type(None)}
332 ... Append To File ${json_tmp_file_path} ${output}
333 Close New JSON List ${json_tmp_file_path}
334 Run Keyword if '${last}' == 'false'
335 ... Append to File ${json_tmp_file_path} ,
336
337Retrieve HW Info And Write List
338 [Documentation] Does a Retrieve/Write with a list of classes and
339 ... encapsulates them into one large JSON element.
340 [Arguments] ${list} ${json_tmp_file_path} ${json_field_name}
341 ... ${last}=false
342 # Description of argument(s):
343 # list The list of devices classes to retrieve with lshw.
344 # json_tmp_file_path Path of file to write to.
345 # json_field_name Name of the JSON element to encapsulate this list.
346 # last Is this the last element in the parent JSON?
347 Write New JSON List ${json_tmp_file_path} ${json_field_name}
348 : FOR ${class} IN @{list}
349 \ ${tail} Get From List ${list} -1
350 \ Run Keyword if '${tail}' == '${class}'
351 \ ... Retrieve HW Info And Write ${class} ${json_tmp_file_path} true
352 \ ... ELSE Retrieve HW Info And Write ${class} ${json_tmp_file_path}
353 Close New JSON List ${json_tmp_file_path}
354 Run Keyword if '${last}' == 'false'
355 ... Append to File ${json_tmp_file_path} ,
356
357Retrieve Hardware Info
358 [Documentation] Retrieves the lshw output of the device class as JSON.
359 [Arguments] ${class}
360 # Description of argument(s):
361 # class Device class to retrieve with lshw.
362 ${output} = Execute Command On OS lshw -c ${class} -json
363 ${output} = Verify JSON string ${output}
364 [Return] ${output}
365
366Verify JSON String
Gunnar Mills28e403b2017-10-25 16:16:38 -0500367 [Documentation] Ensure the JSON string content is separated by commas.
Steven Sombar130a04f2017-07-16 10:02:37 -0500368 [Arguments] ${unver_string}
369 # Description of argument(s):
370 # unver_string JSON String we will be checking for lshw comma errors.
371 ${unver_string} = Convert to String ${unver_string}
372 ${ver_string} = Replace String Using Regexp ${unver_string} }\\s*{ },{
373 [Return] ${ver_string}
374
375Clean Up String
376 [Documentation] Remove extra whitespace and trailing commas.
377 [Arguments] ${dirty_string}
378 # Description of argument(s):
379 # dirty_string String that will be space stripped and have comma removed.
380 ${clean_string} = Strip String ${dirty_string}
381 ${last_char} = Get Substring ${clean_string} -1
382 ${trimmed_string} = Get Substring ${clean_string} 0 -1
383 ${clean_string} = Set Variable If '${last_char}' == ','
384 ... ${trimmed_string} ${clean_string}
385 [Return] ${clean_string}