blob: e358e9f493dda3cf37dea5b2de3e88835fc80e9d [file] [log] [blame]
George Keishing0fff68a2017-04-27 22:25:49 -05001*** Settings ***
Steven Sombar130a04f2017-07-16 10:02:37 -05002Documentation Keywords for system test.
George Keishing0fff68a2017-04-27 22:25:49 -05003
George Keishing91c852f2017-05-03 02:48:21 -05004Library ../lib/gen_robot_keyword.py
George Keishingcff7aa02017-05-16 08:05:10 -05005Library OperatingSystem
George Keishing91c852f2017-05-03 02:48:21 -05006Resource ../extended/obmc_boot_test_resource.robot
George Keishing0fff68a2017-04-27 22:25:49 -05007Resource ../lib/utils.robot
George Keishing91c852f2017-05-03 02:48:21 -05008Resource ../lib/state_manager.robot
George Keishingcff7aa02017-05-16 08:05:10 -05009Resource ../lib/rest_client.robot
George Keishingbbfec9e2017-05-17 10:47:21 -050010Resource resource.txt
George Keishing4ef29082017-05-13 05:01:23 -050011Library OperatingSystem
12Library DateTime
13
George Keishing0fff68a2017-04-27 22:25:49 -050014*** Variables ***
15
Steven Sombar130a04f2017-07-16 10:02:37 -050016${htx_log_dir_path} ${EXECDIR}${/}logs${/}
George Keishing4ef29082017-05-13 05:01:23 -050017
George Keishing4bbf5202017-05-18 06:55:53 -050018# Error strings to check from dmesg.
Steven Sombar130a04f2017-07-16 10:02:37 -050019${ERROR_REGEX} error|GPU|NVRM|nvidia
George Keishing4bbf5202017-05-18 06:55:53 -050020
21# GPU specific error message from dmesg.
Steven Sombar130a04f2017-07-16 10:02:37 -050022${ERROR_DBE_MSG} (DBE) has been detected on GPU
23
24# Inventory - List of I/O devices to collect for Inventory
25@{I/O} communication disk display generic input multimedia
26... network printer tape
27
28# Inventory Paths of the JSON and YAML files
Steven Sombara2f166c2017-08-02 14:22:45 -050029${json_tmp_file_path} ${EXECDIR}/inventory_temp_file.json
30${yaml_file_path} ${EXECDIR}/inventory_temp_file.yaml
Steven Sombar130a04f2017-07-16 10:02:37 -050031
32
George Keishing4ef29082017-05-13 05:01:23 -050033
George Keishing0fff68a2017-04-27 22:25:49 -050034*** Keywords ***
35
36Execute Command On OS
37 [Documentation] Execute given command on OS and return output.
38 [Arguments] ${command}
George Keishing91c852f2017-05-03 02:48:21 -050039 # Description of argument(s):
40 # command Shell command to be executed on OS.
George Keishing0fff68a2017-04-27 22:25:49 -050041 ${stdout} ${stderr}= Execute Command ${command} return_stderr=True
42 Should Be Empty ${stderr}
43 [Return] ${stdout}
44
George Keishing0fff68a2017-04-27 22:25:49 -050045Login To OS
46 [Documentation] Login to OS Host.
47 [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME}
48 ... ${os_password}=${OS_PASSWORD}
George Keishingcff7aa02017-05-16 08:05:10 -050049 ... ${alias_name}=os_connection
George Keishing91c852f2017-05-03 02:48:21 -050050 # Description of argument(s):
51 # os_host IP address of the OS Host.
52 # os_username OS Host Login user name.
53 # os_password OS Host Login passwrd.
George Keishingcff7aa02017-05-16 08:05:10 -050054 # alias_name Default OS SSH session connection alias name.
55 # TODO: Generalize alias naming using openbmc/openbmc-test-automation#633
George Keishing0fff68a2017-04-27 22:25:49 -050056
George Keishing91c852f2017-05-03 02:48:21 -050057 Ping Host ${os_host}
George Keishing664a0162017-06-05 12:24:24 -050058 SSHLibrary.Open Connection ${os_host} alias=${alias_name}
George Keishing91c852f2017-05-03 02:48:21 -050059 Login ${os_username} ${os_password}
60
61
George Keishing88366b32017-05-12 13:50:48 -050062Tool Exist
63 [Documentation] Check whether given tool is installed on OS.
64 [Arguments] ${tool_name}
65 # Description of argument(s):
66 # tool_name Tool name whose existence is to be checked.
George Keishing91c852f2017-05-03 02:48:21 -050067 Login To OS
George Keishing88366b32017-05-12 13:50:48 -050068 ${output}= Execute Command On OS which ${tool_name}
69 Should Contain ${output} ${tool_name}
70 ... msg=Please install ${tool_name} tool.
George Keishing91c852f2017-05-03 02:48:21 -050071
72
73Boot To OS
74 [Documentation] Boot host OS.
75 Run Key OBMC Boot Test \ REST Power On
76
77
78Power Off Host
79 [Documentation] Power off host.
80 Run Key OBMC Boot Test \ REST Power Off
George Keishing0fff68a2017-04-27 22:25:49 -050081
George Keishing6fec3ab2017-05-05 10:45:08 -050082
83File Exist On OS
84 [Documentation] Check if the given file path exist on OS.
85 [Arguments] ${file_path}
86 # Description of argument(s):
87 # file_path Absolute file path.
88
89 Login To OS
90 ${out}= Execute Command On OS ls ${file_path}
91 Log To Console \n File Exist: ${out}
92
George Keishing8740a0c2017-05-13 07:19:16 -050093
94Is HTX Running
95 [Documentation] Check if the HTX exerciser is currently running.
96
97 ${status}= Execute Command On OS htxcmdline -status
98 Should Not Contain ${status} Daemon state is <IDLE>
George Keishing4ef29082017-05-13 05:01:23 -050099
100
101Write Log Data To File
102 [Documentation] Write log data to the logs directory.
103 [Arguments] ${data}= ${log_file_path}=
104 # Description of argument(s):
105 # data String buffer.
106 # log_file_path The log file path.
107
108 Create File ${log_file_path} ${data}
109
110
111Collect HTX Log Files
112 [Documentation] Collect status and error log files.
113 # Collects the following files:
114 # HTX error log file /tmp/htxerr
115 # HTX status log file /tmp/htxstats
116
117 # Create logs directory and get current datetime.
118 Create Directory ${htx_log_dir_path}
119 ${cur_datetime}= Get Current Date result_format=%Y%m%d%H%M%S%f
120
121 File Exist On OS /tmp/htxerr
122 ${htx_err}= Execute Command On BMC cat /tmp/htxerr
123 Write Log Data To File
124 ... ${htx_err} ${htx_log_dir_path}/${OS_HOST}${cur_datetime}.htxerr
125
126 File Exist On OS /tmp/htxstats
127 ${htx_stats}= Execute Command On BMC cat /tmp/htxstats
128 Write Log Data To File
129 ... ${htx_stats} ${htx_log_dir_path}/${OS_HOST}_${cur_datetime}.htxstats
130
George Keishingcff7aa02017-05-16 08:05:10 -0500131
132REST Upload File To BMC
133 [Documentation] Upload a file via REST to BMC.
134
135 # Generate 32 MB file size
136 Run dd if=/dev/zero of=dummyfile bs=1 count=0 seek=32MB
137 OperatingSystem.File Should Exist dummyfile
138
139 # Get the content of the file and upload to BMC
140 ${image_data}= OperatingSystem.Get Binary File dummyfile
141
142 # Get REST session to BMC
143 Initialize OpenBMC
144
145 # Create the REST payload headers and data
146 ${data}= Create Dictionary data ${image_data}
147 ${headers}= Create Dictionary Content-Type=application/octet-stream
148 ... Accept=application/octet-stream
149 Set To Dictionary ${data} headers ${headers}
150
151 ${resp}= Post Request openbmc /upload/image &{data}
152 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
153
154 # Delete uploaded image file.
155 # TODO: Delete via REST openbmc/openbmc#1550
156 # Take SSH connection to BMC and switch to BMC connection to perform
157 # the task.
158 &{bmc_connection_args}= Create Dictionary alias=bmc_connection
159 Open Connection And Log In &{bmc_connection_args}
160
161 # Currently OS SSH session is active, switch to BMC connection.
162 Switch Connection bmc_connection
163 Execute Command On BMC rm -f /tmp/images/*
164
165 # Switch back to OS SSH connection.
166 Switch Connection os_connection
167
George Keishing4bbf5202017-05-18 06:55:53 -0500168
169Check For Errors On OS Dmesg Log
170 [Documentation] Check if dmesg has nvidia errors logged.
171
172 ${dmesg_log}= Execute Command On OS dmesg | egrep '${ERROR_REGEX}'
173 # To enable multiple string check.
174 Should Not Contain Any ${dmesg_log} ${ERROR_DBE_MSG}
175
176
177Collect NVIDIA Log File
178 [Documentation] Collect ndivia-smi command output.
George Keishing664a0162017-06-05 12:24:24 -0500179 [Arguments] ${suffix}
180 # Description of argument(s):
181 # suffix String name to append.
George Keishing4bbf5202017-05-18 06:55:53 -0500182
183 # Collects the output of ndivia-smi cmd output.
184 # TODO: GPU current temperature threshold check.
185 # openbmc/openbmc-test-automation#637
186 # +-----------------------------------------------------------------------------+
187 # | NVIDIA-SMI 361.89 Driver Version: 361.89 |
188 # |-------------------------------+----------------------+----------------------+
189 # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
190 # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
191 # |===============================+======================+======================|
192 # | 0 Tesla P100-SXM2... On | 0002:01:00.0 Off | 0 |
193 # | N/A 25C P0 35W / 300W | 931MiB / 16280MiB | 0% Default |
194 # +-------------------------------+----------------------+----------------------+
195 # | 1 Tesla P100-SXM2... On | 0003:01:00.0 Off | 0 |
196 # | N/A 26C P0 40W / 300W | 1477MiB / 16280MiB | 0% Default |
197 # +-------------------------------+----------------------+----------------------+
198 # | 2 Tesla P100-SXM2... On | 0006:01:00.0 Off | 0 |
199 # | N/A 25C P0 35W / 300W | 931MiB / 16280MiB | 0% Default |
200 # +-------------------------------+----------------------+----------------------+
201 # | 3 Tesla P100-SXM2... On | 0007:01:00.0 Off | 0 |
202 # | N/A 44C P0 290W / 300W | 965MiB / 16280MiB | 99% Default |
203 # +-------------------------------+----------------------+----------------------+
204 # +-----------------------------------------------------------------------------+
205 # | Processes: GPU Memory |
206 # | GPU PID Type Process name Usage |
207 # |=============================================================================|
208 # | 0 28459 C hxenvidia 929MiB |
209 # | 1 28460 C hxenvidia 1475MiB |
210 # | 2 28461 C hxenvidia 929MiB |
211 # | 3 28462 C hxenvidia 963MiB |
212 # +-----------------------------------------------------------------------------+
213
214 # Create logs directory and get current datetime.
215 Create Directory ${htx_log_dir_path}
216 ${cur_datetime}= Get Current Date result_format=%Y%m%d%H%M%S%f
217
218 ${nvidia_out}= Execute Command On BMC nvidia-smi
219 Write Log Data To File
George Keishing664a0162017-06-05 12:24:24 -0500220 ... ${nvidia_out}
221 ... ${htx_log_dir_path}/${OS_HOST}_${cur_datetime}.nvidia_${suffix}
George Keishingbbfec9e2017-05-17 10:47:21 -0500222
223
224Pre Test Case Execution
225 [Documentation] Do the initial test setup.
226 # 1. Check if HTX tool exist.
227 # 2. Power on
228
229 Boot To OS
George Keishing5e2dcd02017-06-09 12:01:28 -0500230 Tool Exist htxcmdline
George Keishingbbfec9e2017-05-17 10:47:21 -0500231
232 # Shutdown if HTX is running.
233 ${status}= Run Keyword And Return Status Is HTX Running
234 Run Keyword If '${status}' == 'True'
235 ... Shutdown HTX Exerciser
236
237
238Create Default MDT Profile
239 [Documentation] Create default mdt.bu profile and run.
240
241 Rprint Timen Create HTX mdt profile.
242
243 ${profile}= Execute Command On OS htxcmdline -createmdt
244 Rprintn ${profile}
245 Should Contain ${profile} mdts are created successfully
246
247
248Run MDT Profile
249 [Documentation] Load user pre-defined MDT profile.
250
251 Rprint Timen Start HTX mdt profile execution.
252 ${htx_run}= Execute Command On OS
253 ... htxcmdline -run -mdt ${HTX_MDT_PROFILE}
254 Rprintn ${htx_run}
255 Should Contain ${htx_run} Activated
256
257
258Check HTX Run Status
259 [Documentation] Get HTX exerciser status and check for error.
260
261 Rprint Timen Check HTX mdt Status and error.
262 ${status}= Execute Command On OS
263 ... htxcmdline -status -mdt ${HTX_MDT_PROFILE}
264 Rprintn ${status}
265
266 ${errlog}= Execute Command On OS htxcmdline -geterrlog
267 Rprintn ${errlog}
268
269 Should Contain ${errlog} file </tmp/htxerr> is empty
270
271
272Shutdown HTX Exerciser
273 [Documentation] Shut down HTX exerciser run.
274
275 Rprint Timen Shutdown HTX Run
276 ${shutdown}= Execute Command On OS
277 ... htxcmdline -shutdown -mdt ${HTX_MDT_PROFILE}
278 Rprintn ${shutdown}
279 Should Contain ${shutdown} shutdown successfully
280
Steven Sombar130a04f2017-07-16 10:02:37 -0500281
282Create JSON Inventory File
283 [Documentation] Create a JSON inventory file, and make a YAML copy.
284 [Arguments] ${json_file_path}
285 # Description of argument:
286 # json_file_path Where the inventory file is wrtten to.
287
288 Login To OS
289 Compile Inventory JSON
290 Run json2yaml ${json_tmp_file_path} ${yaml_file_path}
291 # Format to JSON pretty print to file.
292 Run python -m json.tool ${json_tmp_file_path} > ${json_file_path}
293 OperatingSystem.File Should Exist ${json_file_path}
294
295
296Compile Inventory JSON
297 [Documentation] Compile the Inventory into a JSON file.
298 Create File ${json_tmp_file_path}
299 Write New JSON List ${json_tmp_file_path} Inventory
300 Retrieve HW Info And Write processor ${json_tmp_file_path}
301 Retrieve HW Info And Write memory ${json_tmp_file_path}
302 Retrieve HW Info And Write List ${I/O} ${json_tmp_file_path} I/O last
303 Close New JSON List ${json_tmp_file_path}
304
305Write New JSON List
306 [Documentation] Start a new JSON list element in file.
307 [Arguments] ${json_tmp_file_path} ${json_field_name}
308 # Description of argument(s):
309 # json_tmp_file_path Name of file to write to.
310 # json_field_name Name to give json list element.
311 Append to File ${json_tmp_file_path} { "${json_field_name}" : [
312
313Close New JSON List
314 [Documentation] Close JSON list element in file.
315 [Arguments] ${json_tmp_file_path}
316 # Description of argument(s):
317 # json_tmp_file_path Path of file to write to.
318 Append to File ${json_tmp_file_path} ]}
319
320Retrieve HW Info And Write
321 [Documentation] Retrieve and write info, add a comma if not last item.
322 [Arguments] ${class} ${json_tmp_file_path} ${last}=false
323 # Description of argument(s):
324 # class Device class to retrieve with lshw.
325 # json_tmp_file_path Path of file to write to.
326 # last Is this the last element in the parent JSON?
327 Write New JSON List ${json_tmp_file_path} ${class}
328 ${output} = Retrieve Hardware Info ${class}
329 ${output} = Clean Up String ${output}
330 Run Keyword if ${output.__class__ is not type(None)}
331 ... Append To File ${json_tmp_file_path} ${output}
332 Close New JSON List ${json_tmp_file_path}
333 Run Keyword if '${last}' == 'false'
334 ... Append to File ${json_tmp_file_path} ,
335
336Retrieve HW Info And Write List
337 [Documentation] Does a Retrieve/Write with a list of classes and
338 ... encapsulates them into one large JSON element.
339 [Arguments] ${list} ${json_tmp_file_path} ${json_field_name}
340 ... ${last}=false
341 # Description of argument(s):
342 # list The list of devices classes to retrieve with lshw.
343 # json_tmp_file_path Path of file to write to.
344 # json_field_name Name of the JSON element to encapsulate this list.
345 # last Is this the last element in the parent JSON?
346 Write New JSON List ${json_tmp_file_path} ${json_field_name}
347 : FOR ${class} IN @{list}
348 \ ${tail} Get From List ${list} -1
349 \ Run Keyword if '${tail}' == '${class}'
350 \ ... Retrieve HW Info And Write ${class} ${json_tmp_file_path} true
351 \ ... ELSE Retrieve HW Info And Write ${class} ${json_tmp_file_path}
352 Close New JSON List ${json_tmp_file_path}
353 Run Keyword if '${last}' == 'false'
354 ... Append to File ${json_tmp_file_path} ,
355
356Retrieve Hardware Info
357 [Documentation] Retrieves the lshw output of the device class as JSON.
358 [Arguments] ${class}
359 # Description of argument(s):
360 # class Device class to retrieve with lshw.
361 ${output} = Execute Command On OS lshw -c ${class} -json
362 ${output} = Verify JSON string ${output}
363 [Return] ${output}
364
365Verify JSON String
Gunnar Mills28e403b2017-10-25 16:16:38 -0500366 [Documentation] Ensure the JSON string content is separated by commas.
Steven Sombar130a04f2017-07-16 10:02:37 -0500367 [Arguments] ${unver_string}
368 # Description of argument(s):
369 # unver_string JSON String we will be checking for lshw comma errors.
370 ${unver_string} = Convert to String ${unver_string}
371 ${ver_string} = Replace String Using Regexp ${unver_string} }\\s*{ },{
372 [Return] ${ver_string}
373
374Clean Up String
375 [Documentation] Remove extra whitespace and trailing commas.
376 [Arguments] ${dirty_string}
377 # Description of argument(s):
378 # dirty_string String that will be space stripped and have comma removed.
379 ${clean_string} = Strip String ${dirty_string}
380 ${last_char} = Get Substring ${clean_string} -1
381 ${trimmed_string} = Get Substring ${clean_string} 0 -1
382 ${clean_string} = Set Variable If '${last_char}' == ','
383 ... ${trimmed_string} ${clean_string}
384 [Return] ${clean_string}