blob: 9e7d36b2fce6b70afd4a473999494c5f01135157 [file] [log] [blame]
George Keishing0fff68a2017-04-27 22:25:49 -05001*** Settings ***
2Documentation Keywords for system related test. This is a subset of the
3... utils.robot. This resource file keywords is specifically
4... define for system test use cases.
5
George Keishing91c852f2017-05-03 02:48:21 -05006Library ../lib/gen_robot_keyword.py
George Keishingcff7aa02017-05-16 08:05:10 -05007Library OperatingSystem
George Keishing91c852f2017-05-03 02:48:21 -05008Resource ../extended/obmc_boot_test_resource.robot
George Keishing0fff68a2017-04-27 22:25:49 -05009Resource ../lib/utils.robot
George Keishing91c852f2017-05-03 02:48:21 -050010Resource ../lib/state_manager.robot
George Keishingcff7aa02017-05-16 08:05:10 -050011Resource ../lib/rest_client.robot
George Keishingbbfec9e2017-05-17 10:47:21 -050012Resource resource.txt
George Keishing4ef29082017-05-13 05:01:23 -050013Library OperatingSystem
14Library DateTime
15
George Keishing0fff68a2017-04-27 22:25:49 -050016*** Variables ***
17
George Keishing4ef29082017-05-13 05:01:23 -050018${htx_log_dir_path} ${EXECDIR}${/}logs${/}
19
George Keishing4bbf5202017-05-18 06:55:53 -050020# Error strings to check from dmesg.
21${ERROR_REGEX} error|GPU|NVRM|nvidia
22
23# GPU specific error message from dmesg.
24${ERROR_DBE_MSG} (DBE) has been detected on GPU
George Keishing4ef29082017-05-13 05:01:23 -050025
George Keishing0fff68a2017-04-27 22:25:49 -050026*** Keywords ***
27
28Execute Command On OS
29 [Documentation] Execute given command on OS and return output.
30 [Arguments] ${command}
George Keishing91c852f2017-05-03 02:48:21 -050031 # Description of argument(s):
32 # command Shell command to be executed on OS.
George Keishing0fff68a2017-04-27 22:25:49 -050033 ${stdout} ${stderr}= Execute Command ${command} return_stderr=True
34 Should Be Empty ${stderr}
35 [Return] ${stdout}
36
George Keishing0fff68a2017-04-27 22:25:49 -050037Login To OS
38 [Documentation] Login to OS Host.
39 [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME}
40 ... ${os_password}=${OS_PASSWORD}
George Keishingcff7aa02017-05-16 08:05:10 -050041 ... ${alias_name}=os_connection
George Keishing91c852f2017-05-03 02:48:21 -050042 # Description of argument(s):
43 # os_host IP address of the OS Host.
44 # os_username OS Host Login user name.
45 # os_password OS Host Login passwrd.
George Keishingcff7aa02017-05-16 08:05:10 -050046 # alias_name Default OS SSH session connection alias name.
47 # TODO: Generalize alias naming using openbmc/openbmc-test-automation#633
George Keishing0fff68a2017-04-27 22:25:49 -050048
George Keishing91c852f2017-05-03 02:48:21 -050049 Ping Host ${os_host}
George Keishingcff7aa02017-05-16 08:05:10 -050050 Open Connection ${os_host} alias=${alias_name}
George Keishing91c852f2017-05-03 02:48:21 -050051 Login ${os_username} ${os_password}
52
53
George Keishing88366b32017-05-12 13:50:48 -050054Tool Exist
55 [Documentation] Check whether given tool is installed on OS.
56 [Arguments] ${tool_name}
57 # Description of argument(s):
58 # tool_name Tool name whose existence is to be checked.
George Keishing91c852f2017-05-03 02:48:21 -050059 Login To OS
George Keishing88366b32017-05-12 13:50:48 -050060 ${output}= Execute Command On OS which ${tool_name}
61 Should Contain ${output} ${tool_name}
62 ... msg=Please install ${tool_name} tool.
George Keishing91c852f2017-05-03 02:48:21 -050063
64
65Boot To OS
66 [Documentation] Boot host OS.
67 Run Key OBMC Boot Test \ REST Power On
68
69
70Power Off Host
71 [Documentation] Power off host.
72 Run Key OBMC Boot Test \ REST Power Off
George Keishing0fff68a2017-04-27 22:25:49 -050073
George Keishing6fec3ab2017-05-05 10:45:08 -050074
75File Exist On OS
76 [Documentation] Check if the given file path exist on OS.
77 [Arguments] ${file_path}
78 # Description of argument(s):
79 # file_path Absolute file path.
80
81 Login To OS
82 ${out}= Execute Command On OS ls ${file_path}
83 Log To Console \n File Exist: ${out}
84
George Keishing8740a0c2017-05-13 07:19:16 -050085
86Is HTX Running
87 [Documentation] Check if the HTX exerciser is currently running.
88
89 ${status}= Execute Command On OS htxcmdline -status
90 Should Not Contain ${status} Daemon state is <IDLE>
George Keishing4ef29082017-05-13 05:01:23 -050091
92
93Write Log Data To File
94 [Documentation] Write log data to the logs directory.
95 [Arguments] ${data}= ${log_file_path}=
96 # Description of argument(s):
97 # data String buffer.
98 # log_file_path The log file path.
99
100 Create File ${log_file_path} ${data}
101
102
103Collect HTX Log Files
104 [Documentation] Collect status and error log files.
105 # Collects the following files:
106 # HTX error log file /tmp/htxerr
107 # HTX status log file /tmp/htxstats
108
109 # Create logs directory and get current datetime.
110 Create Directory ${htx_log_dir_path}
111 ${cur_datetime}= Get Current Date result_format=%Y%m%d%H%M%S%f
112
113 File Exist On OS /tmp/htxerr
114 ${htx_err}= Execute Command On BMC cat /tmp/htxerr
115 Write Log Data To File
116 ... ${htx_err} ${htx_log_dir_path}/${OS_HOST}${cur_datetime}.htxerr
117
118 File Exist On OS /tmp/htxstats
119 ${htx_stats}= Execute Command On BMC cat /tmp/htxstats
120 Write Log Data To File
121 ... ${htx_stats} ${htx_log_dir_path}/${OS_HOST}_${cur_datetime}.htxstats
122
George Keishingcff7aa02017-05-16 08:05:10 -0500123
124REST Upload File To BMC
125 [Documentation] Upload a file via REST to BMC.
126
127 # Generate 32 MB file size
128 Run dd if=/dev/zero of=dummyfile bs=1 count=0 seek=32MB
129 OperatingSystem.File Should Exist dummyfile
130
131 # Get the content of the file and upload to BMC
132 ${image_data}= OperatingSystem.Get Binary File dummyfile
133
134 # Get REST session to BMC
135 Initialize OpenBMC
136
137 # Create the REST payload headers and data
138 ${data}= Create Dictionary data ${image_data}
139 ${headers}= Create Dictionary Content-Type=application/octet-stream
140 ... Accept=application/octet-stream
141 Set To Dictionary ${data} headers ${headers}
142
143 ${resp}= Post Request openbmc /upload/image &{data}
144 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
145
146 # Delete uploaded image file.
147 # TODO: Delete via REST openbmc/openbmc#1550
148 # Take SSH connection to BMC and switch to BMC connection to perform
149 # the task.
150 &{bmc_connection_args}= Create Dictionary alias=bmc_connection
151 Open Connection And Log In &{bmc_connection_args}
152
153 # Currently OS SSH session is active, switch to BMC connection.
154 Switch Connection bmc_connection
155 Execute Command On BMC rm -f /tmp/images/*
156
157 # Switch back to OS SSH connection.
158 Switch Connection os_connection
159
George Keishing4bbf5202017-05-18 06:55:53 -0500160
161Check For Errors On OS Dmesg Log
162 [Documentation] Check if dmesg has nvidia errors logged.
163
164 ${dmesg_log}= Execute Command On OS dmesg | egrep '${ERROR_REGEX}'
165 # To enable multiple string check.
166 Should Not Contain Any ${dmesg_log} ${ERROR_DBE_MSG}
167
168
169Collect NVIDIA Log File
170 [Documentation] Collect ndivia-smi command output.
171
172 # Collects the output of ndivia-smi cmd output.
173 # TODO: GPU current temperature threshold check.
174 # openbmc/openbmc-test-automation#637
175 # +-----------------------------------------------------------------------------+
176 # | NVIDIA-SMI 361.89 Driver Version: 361.89 |
177 # |-------------------------------+----------------------+----------------------+
178 # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
179 # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
180 # |===============================+======================+======================|
181 # | 0 Tesla P100-SXM2... On | 0002:01:00.0 Off | 0 |
182 # | N/A 25C P0 35W / 300W | 931MiB / 16280MiB | 0% Default |
183 # +-------------------------------+----------------------+----------------------+
184 # | 1 Tesla P100-SXM2... On | 0003:01:00.0 Off | 0 |
185 # | N/A 26C P0 40W / 300W | 1477MiB / 16280MiB | 0% Default |
186 # +-------------------------------+----------------------+----------------------+
187 # | 2 Tesla P100-SXM2... On | 0006:01:00.0 Off | 0 |
188 # | N/A 25C P0 35W / 300W | 931MiB / 16280MiB | 0% Default |
189 # +-------------------------------+----------------------+----------------------+
190 # | 3 Tesla P100-SXM2... On | 0007:01:00.0 Off | 0 |
191 # | N/A 44C P0 290W / 300W | 965MiB / 16280MiB | 99% Default |
192 # +-------------------------------+----------------------+----------------------+
193 # +-----------------------------------------------------------------------------+
194 # | Processes: GPU Memory |
195 # | GPU PID Type Process name Usage |
196 # |=============================================================================|
197 # | 0 28459 C hxenvidia 929MiB |
198 # | 1 28460 C hxenvidia 1475MiB |
199 # | 2 28461 C hxenvidia 929MiB |
200 # | 3 28462 C hxenvidia 963MiB |
201 # +-----------------------------------------------------------------------------+
202
203 # Create logs directory and get current datetime.
204 Create Directory ${htx_log_dir_path}
205 ${cur_datetime}= Get Current Date result_format=%Y%m%d%H%M%S%f
206
207 ${nvidia_out}= Execute Command On BMC nvidia-smi
208 Write Log Data To File
209 ... ${nvidia_out} ${htx_log_dir_path}/${OS_HOST}_${cur_datetime}.nvidia
George Keishingbbfec9e2017-05-17 10:47:21 -0500210
211
212Pre Test Case Execution
213 [Documentation] Do the initial test setup.
214 # 1. Check if HTX tool exist.
215 # 2. Power on
216
217 Boot To OS
George Keishing5e2dcd02017-06-09 12:01:28 -0500218 Tool Exist htxcmdline
George Keishingbbfec9e2017-05-17 10:47:21 -0500219
220 # Shutdown if HTX is running.
221 ${status}= Run Keyword And Return Status Is HTX Running
222 Run Keyword If '${status}' == 'True'
223 ... Shutdown HTX Exerciser
224
225
226Create Default MDT Profile
227 [Documentation] Create default mdt.bu profile and run.
228
229 Rprint Timen Create HTX mdt profile.
230
231 ${profile}= Execute Command On OS htxcmdline -createmdt
232 Rprintn ${profile}
233 Should Contain ${profile} mdts are created successfully
234
235
236Run MDT Profile
237 [Documentation] Load user pre-defined MDT profile.
238
239 Rprint Timen Start HTX mdt profile execution.
240 ${htx_run}= Execute Command On OS
241 ... htxcmdline -run -mdt ${HTX_MDT_PROFILE}
242 Rprintn ${htx_run}
243 Should Contain ${htx_run} Activated
244
245
246Check HTX Run Status
247 [Documentation] Get HTX exerciser status and check for error.
248
249 Rprint Timen Check HTX mdt Status and error.
250 ${status}= Execute Command On OS
251 ... htxcmdline -status -mdt ${HTX_MDT_PROFILE}
252 Rprintn ${status}
253
254 ${errlog}= Execute Command On OS htxcmdline -geterrlog
255 Rprintn ${errlog}
256
257 Should Contain ${errlog} file </tmp/htxerr> is empty
258
259
260Shutdown HTX Exerciser
261 [Documentation] Shut down HTX exerciser run.
262
263 Rprint Timen Shutdown HTX Run
264 ${shutdown}= Execute Command On OS
265 ... htxcmdline -shutdown -mdt ${HTX_MDT_PROFILE}
266 Rprintn ${shutdown}
267 Should Contain ${shutdown} shutdown successfully
268