blob: a22279178b7a7ab80ef0ceaaabc16ee668dc10fd [file] [log] [blame]
George Keishing0fff68a2017-04-27 22:25:49 -05001*** Settings ***
2Documentation Keywords for system related test. This is a subset of the
3... utils.robot. This resource file keywords is specifically
4... define for system test use cases.
5
George Keishing91c852f2017-05-03 02:48:21 -05006Library ../lib/gen_robot_keyword.py
George Keishingcff7aa02017-05-16 08:05:10 -05007Library OperatingSystem
George Keishing91c852f2017-05-03 02:48:21 -05008Resource ../extended/obmc_boot_test_resource.robot
George Keishing0fff68a2017-04-27 22:25:49 -05009Resource ../lib/utils.robot
George Keishing91c852f2017-05-03 02:48:21 -050010Resource ../lib/state_manager.robot
George Keishingcff7aa02017-05-16 08:05:10 -050011Resource ../lib/rest_client.robot
George Keishingbbfec9e2017-05-17 10:47:21 -050012Resource resource.txt
George Keishing4ef29082017-05-13 05:01:23 -050013Library OperatingSystem
14Library DateTime
15
George Keishing0fff68a2017-04-27 22:25:49 -050016*** Variables ***
17
George Keishing4ef29082017-05-13 05:01:23 -050018${htx_log_dir_path} ${EXECDIR}${/}logs${/}
19
George Keishing4bbf5202017-05-18 06:55:53 -050020# Error strings to check from dmesg.
21${ERROR_REGEX} error|GPU|NVRM|nvidia
22
23# GPU specific error message from dmesg.
24${ERROR_DBE_MSG} (DBE) has been detected on GPU
George Keishing4ef29082017-05-13 05:01:23 -050025
George Keishing0fff68a2017-04-27 22:25:49 -050026*** Keywords ***
27
28Execute Command On OS
29 [Documentation] Execute given command on OS and return output.
30 [Arguments] ${command}
George Keishing91c852f2017-05-03 02:48:21 -050031 # Description of argument(s):
32 # command Shell command to be executed on OS.
George Keishing0fff68a2017-04-27 22:25:49 -050033 ${stdout} ${stderr}= Execute Command ${command} return_stderr=True
34 Should Be Empty ${stderr}
35 [Return] ${stdout}
36
George Keishing0fff68a2017-04-27 22:25:49 -050037Login To OS
38 [Documentation] Login to OS Host.
39 [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME}
40 ... ${os_password}=${OS_PASSWORD}
George Keishingcff7aa02017-05-16 08:05:10 -050041 ... ${alias_name}=os_connection
George Keishing91c852f2017-05-03 02:48:21 -050042 # Description of argument(s):
43 # os_host IP address of the OS Host.
44 # os_username OS Host Login user name.
45 # os_password OS Host Login passwrd.
George Keishingcff7aa02017-05-16 08:05:10 -050046 # alias_name Default OS SSH session connection alias name.
47 # TODO: Generalize alias naming using openbmc/openbmc-test-automation#633
George Keishing0fff68a2017-04-27 22:25:49 -050048
George Keishing91c852f2017-05-03 02:48:21 -050049 Ping Host ${os_host}
George Keishing664a0162017-06-05 12:24:24 -050050 SSHLibrary.Open Connection ${os_host} alias=${alias_name}
George Keishing91c852f2017-05-03 02:48:21 -050051 Login ${os_username} ${os_password}
52
53
George Keishing88366b32017-05-12 13:50:48 -050054Tool Exist
55 [Documentation] Check whether given tool is installed on OS.
56 [Arguments] ${tool_name}
57 # Description of argument(s):
58 # tool_name Tool name whose existence is to be checked.
George Keishing91c852f2017-05-03 02:48:21 -050059 Login To OS
George Keishing88366b32017-05-12 13:50:48 -050060 ${output}= Execute Command On OS which ${tool_name}
61 Should Contain ${output} ${tool_name}
62 ... msg=Please install ${tool_name} tool.
George Keishing91c852f2017-05-03 02:48:21 -050063
64
65Boot To OS
66 [Documentation] Boot host OS.
67 Run Key OBMC Boot Test \ REST Power On
68
69
70Power Off Host
71 [Documentation] Power off host.
72 Run Key OBMC Boot Test \ REST Power Off
George Keishing0fff68a2017-04-27 22:25:49 -050073
George Keishing6fec3ab2017-05-05 10:45:08 -050074
75File Exist On OS
76 [Documentation] Check if the given file path exist on OS.
77 [Arguments] ${file_path}
78 # Description of argument(s):
79 # file_path Absolute file path.
80
81 Login To OS
82 ${out}= Execute Command On OS ls ${file_path}
83 Log To Console \n File Exist: ${out}
84
George Keishing8740a0c2017-05-13 07:19:16 -050085
86Is HTX Running
87 [Documentation] Check if the HTX exerciser is currently running.
88
89 ${status}= Execute Command On OS htxcmdline -status
90 Should Not Contain ${status} Daemon state is <IDLE>
George Keishing4ef29082017-05-13 05:01:23 -050091
92
93Write Log Data To File
94 [Documentation] Write log data to the logs directory.
95 [Arguments] ${data}= ${log_file_path}=
96 # Description of argument(s):
97 # data String buffer.
98 # log_file_path The log file path.
99
100 Create File ${log_file_path} ${data}
101
102
103Collect HTX Log Files
104 [Documentation] Collect status and error log files.
105 # Collects the following files:
106 # HTX error log file /tmp/htxerr
107 # HTX status log file /tmp/htxstats
108
109 # Create logs directory and get current datetime.
110 Create Directory ${htx_log_dir_path}
111 ${cur_datetime}= Get Current Date result_format=%Y%m%d%H%M%S%f
112
113 File Exist On OS /tmp/htxerr
114 ${htx_err}= Execute Command On BMC cat /tmp/htxerr
115 Write Log Data To File
116 ... ${htx_err} ${htx_log_dir_path}/${OS_HOST}${cur_datetime}.htxerr
117
118 File Exist On OS /tmp/htxstats
119 ${htx_stats}= Execute Command On BMC cat /tmp/htxstats
120 Write Log Data To File
121 ... ${htx_stats} ${htx_log_dir_path}/${OS_HOST}_${cur_datetime}.htxstats
122
George Keishingcff7aa02017-05-16 08:05:10 -0500123
124REST Upload File To BMC
125 [Documentation] Upload a file via REST to BMC.
126
127 # Generate 32 MB file size
128 Run dd if=/dev/zero of=dummyfile bs=1 count=0 seek=32MB
129 OperatingSystem.File Should Exist dummyfile
130
131 # Get the content of the file and upload to BMC
132 ${image_data}= OperatingSystem.Get Binary File dummyfile
133
134 # Get REST session to BMC
135 Initialize OpenBMC
136
137 # Create the REST payload headers and data
138 ${data}= Create Dictionary data ${image_data}
139 ${headers}= Create Dictionary Content-Type=application/octet-stream
140 ... Accept=application/octet-stream
141 Set To Dictionary ${data} headers ${headers}
142
143 ${resp}= Post Request openbmc /upload/image &{data}
144 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
145
146 # Delete uploaded image file.
147 # TODO: Delete via REST openbmc/openbmc#1550
148 # Take SSH connection to BMC and switch to BMC connection to perform
149 # the task.
150 &{bmc_connection_args}= Create Dictionary alias=bmc_connection
151 Open Connection And Log In &{bmc_connection_args}
152
153 # Currently OS SSH session is active, switch to BMC connection.
154 Switch Connection bmc_connection
155 Execute Command On BMC rm -f /tmp/images/*
156
157 # Switch back to OS SSH connection.
158 Switch Connection os_connection
159
George Keishing4bbf5202017-05-18 06:55:53 -0500160
161Check For Errors On OS Dmesg Log
162 [Documentation] Check if dmesg has nvidia errors logged.
163
164 ${dmesg_log}= Execute Command On OS dmesg | egrep '${ERROR_REGEX}'
165 # To enable multiple string check.
166 Should Not Contain Any ${dmesg_log} ${ERROR_DBE_MSG}
167
168
169Collect NVIDIA Log File
170 [Documentation] Collect ndivia-smi command output.
George Keishing664a0162017-06-05 12:24:24 -0500171 [Arguments] ${suffix}
172 # Description of argument(s):
173 # suffix String name to append.
George Keishing4bbf5202017-05-18 06:55:53 -0500174
175 # Collects the output of ndivia-smi cmd output.
176 # TODO: GPU current temperature threshold check.
177 # openbmc/openbmc-test-automation#637
178 # +-----------------------------------------------------------------------------+
179 # | NVIDIA-SMI 361.89 Driver Version: 361.89 |
180 # |-------------------------------+----------------------+----------------------+
181 # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
182 # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
183 # |===============================+======================+======================|
184 # | 0 Tesla P100-SXM2... On | 0002:01:00.0 Off | 0 |
185 # | N/A 25C P0 35W / 300W | 931MiB / 16280MiB | 0% Default |
186 # +-------------------------------+----------------------+----------------------+
187 # | 1 Tesla P100-SXM2... On | 0003:01:00.0 Off | 0 |
188 # | N/A 26C P0 40W / 300W | 1477MiB / 16280MiB | 0% Default |
189 # +-------------------------------+----------------------+----------------------+
190 # | 2 Tesla P100-SXM2... On | 0006:01:00.0 Off | 0 |
191 # | N/A 25C P0 35W / 300W | 931MiB / 16280MiB | 0% Default |
192 # +-------------------------------+----------------------+----------------------+
193 # | 3 Tesla P100-SXM2... On | 0007:01:00.0 Off | 0 |
194 # | N/A 44C P0 290W / 300W | 965MiB / 16280MiB | 99% Default |
195 # +-------------------------------+----------------------+----------------------+
196 # +-----------------------------------------------------------------------------+
197 # | Processes: GPU Memory |
198 # | GPU PID Type Process name Usage |
199 # |=============================================================================|
200 # | 0 28459 C hxenvidia 929MiB |
201 # | 1 28460 C hxenvidia 1475MiB |
202 # | 2 28461 C hxenvidia 929MiB |
203 # | 3 28462 C hxenvidia 963MiB |
204 # +-----------------------------------------------------------------------------+
205
206 # Create logs directory and get current datetime.
207 Create Directory ${htx_log_dir_path}
208 ${cur_datetime}= Get Current Date result_format=%Y%m%d%H%M%S%f
209
210 ${nvidia_out}= Execute Command On BMC nvidia-smi
211 Write Log Data To File
George Keishing664a0162017-06-05 12:24:24 -0500212 ... ${nvidia_out}
213 ... ${htx_log_dir_path}/${OS_HOST}_${cur_datetime}.nvidia_${suffix}
George Keishingbbfec9e2017-05-17 10:47:21 -0500214
215
216Pre Test Case Execution
217 [Documentation] Do the initial test setup.
218 # 1. Check if HTX tool exist.
219 # 2. Power on
220
221 Boot To OS
George Keishing5e2dcd02017-06-09 12:01:28 -0500222 Tool Exist htxcmdline
George Keishingbbfec9e2017-05-17 10:47:21 -0500223
224 # Shutdown if HTX is running.
225 ${status}= Run Keyword And Return Status Is HTX Running
226 Run Keyword If '${status}' == 'True'
227 ... Shutdown HTX Exerciser
228
229
230Create Default MDT Profile
231 [Documentation] Create default mdt.bu profile and run.
232
233 Rprint Timen Create HTX mdt profile.
234
235 ${profile}= Execute Command On OS htxcmdline -createmdt
236 Rprintn ${profile}
237 Should Contain ${profile} mdts are created successfully
238
239
240Run MDT Profile
241 [Documentation] Load user pre-defined MDT profile.
242
243 Rprint Timen Start HTX mdt profile execution.
244 ${htx_run}= Execute Command On OS
245 ... htxcmdline -run -mdt ${HTX_MDT_PROFILE}
246 Rprintn ${htx_run}
247 Should Contain ${htx_run} Activated
248
249
250Check HTX Run Status
251 [Documentation] Get HTX exerciser status and check for error.
252
253 Rprint Timen Check HTX mdt Status and error.
254 ${status}= Execute Command On OS
255 ... htxcmdline -status -mdt ${HTX_MDT_PROFILE}
256 Rprintn ${status}
257
258 ${errlog}= Execute Command On OS htxcmdline -geterrlog
259 Rprintn ${errlog}
260
261 Should Contain ${errlog} file </tmp/htxerr> is empty
262
263
264Shutdown HTX Exerciser
265 [Documentation] Shut down HTX exerciser run.
266
267 Rprint Timen Shutdown HTX Run
268 ${shutdown}= Execute Command On OS
269 ... htxcmdline -shutdown -mdt ${HTX_MDT_PROFILE}
270 Rprintn ${shutdown}
271 Should Contain ${shutdown} shutdown successfully
272