blob: 5841543d19c05c88c1b472da924bdee3b6ddb612 [file] [log] [blame]
Steven Sombarf60cbcf2018-12-07 08:12:18 -06001*** Settings ***
2
3Documentation Utilities for Robot keywords that do not use REST.
4
Sandhya Somashekar839a0c22019-01-31 05:05:43 -06005Resource ../lib/resource.robot
Steven Sombarf60cbcf2018-12-07 08:12:18 -06006Resource ../lib/connection_client.robot
7Resource ../lib/boot_utils.robot
8Library String
9Library DateTime
10Library Process
11Library OperatingSystem
12Library gen_print.py
13Library gen_robot_print.py
14Library gen_cmd.py
Michael Walsh07ed7942019-11-08 14:41:30 -060015Library gen_robot_valid.py
Steven Sombarf60cbcf2018-12-07 08:12:18 -060016Library gen_robot_keyword.py
17Library bmc_ssh_utils.py
18Library utils.py
19Library var_funcs.py
20Library SCPLibrary WITH NAME scp
21
22*** Variables ***
23
24${pflash_cmd} /usr/sbin/pflash -r /dev/stdout -P VERSION
25
26${dbuscmdBase}
27... dbus-send --system --print-reply --dest=${OPENBMC_BASE_DBUS}.settings.Host
28${dbuscmdGet}
29... ${SETTINGS_URI}host0 org.freedesktop.DBus.Properties.Get
30${dbuscmdString}= string:"xyz.openbmc_project.settings.Host" string:
31
32# Assign default value to QUIET for programs which may not define it.
33${QUIET} ${0}
34
35${bmc_mem_free_cmd}= free | tr -s ' ' | sed '/^Mem/!d' | cut -d" " -f4
36${bmc_mem_total_cmd}= free | tr -s ' ' | sed '/^Mem/!d' | cut -d" " -f2
37${bmc_cpu_usage_cmd}= top -n 1 | grep CPU: | cut -c 7-9
38${HOST_SETTING} ${SETTINGS_URI}host0
39
40# /run/initramfs/ro associate filesystem should be 100% full always
41${bmc_file_system_usage_cmd}= df -h | cut -c 52-54 | grep 100 | wc -l
42${total_pnor_ro_file_system_cmd}= df -h | grep /media/pnor-ro | wc -l
43${total_bmc_ro_file_system_cmd}= df -h | grep /media/rofs | wc -l
44
45${BOOT_TIME} ${0}
46${BOOT_COUNT} ${0}
47${count} ${0}
48${devicetree_base} /sys/firmware/devicetree/base/model
49
50# Initialize default debug value to 0.
51${DEBUG} ${0}
52
53${probe_cpu_tool_path} ${EXECDIR}/tools/ras/probe_cpus.sh
54${scom_addrs_tool_path} ${EXECDIR}/tools/ras/scom_addr_p9.sh
55${target_file_path} /root/
56
57${default_tarball} ${EXECDIR}/obmc-phosphor-debug-tarball-witherspoon.tar.xz
58
59# These variables are used to straddle between new and old methods of setting
60# values.
61${bmc_power_policy_method} ${EMPTY}
62@{valid_power_policy_vars} RESTORE_LAST_STATE ALWAYS_POWER_ON
63... ALWAYS_POWER_OFF
64
George Keishingb9f3f772020-01-15 23:23:08 -060065${check_performance} ${1}
66
Steven Sombarf60cbcf2018-12-07 08:12:18 -060067
68*** Keywords ***
69
70Check BMC Performance
71 [Documentation] Check BMC basic CPU Mem File system performance.
72
George Keishingb9f3f772020-01-15 23:23:08 -060073 Return From Keyword If not ${check_performance}
74
Steven Sombarf60cbcf2018-12-07 08:12:18 -060075 Check BMC CPU Performance
76 Check BMC Mem Performance
77 Check BMC File System Performance
78
79
80Verify PNOR Update
81 [Documentation] Verify that the PNOR is not corrupted.
82 # Example:
83 # FFS: Flash header not found. Code: 100
84 # Error 100 opening ffs !
85
86 ${stdout} ${stderr} ${rc}=
87 ... BMC Execute Command /usr/sbin/pflash -h | egrep -q skip
88 ... ignore_err=${1}
89 ${pflash_cmd}= Set Variable If ${rc} == ${0} ${pflash_cmd} --skip=4096
90 ... ${pflash_cmd}
91 ${pnor_info}= BMC Execute Command ${pflash_cmd}
92 Should Not Contain Any ${pnor_info} Flash header not found Error
93
94
95Get BMC System Model
96 [Documentation] Get the BMC model from the device tree and return it.
97
98 ${bmc_model} ${stderr} ${rc}= BMC Execute Command
99 ... cat ${devicetree_base} | cut -d " " -f 1 return_stderr=True
100 ... test_mode=0
101 Should Be Empty ${stderr}
102 Should Not Be Empty ${bmc_model} msg=BMC model is empty.
103 [Return] ${bmc_model}
104
105
106Verify BMC System Model
107 [Documentation] Verify the BMC model with ${OPENBMC_MODEL}.
108 [Arguments] ${bmc_model}
109
110 # Description of argument(s):
111 # bmc_model System model (e.g. "witherspoon").
112
113 ${tmp_bmc_model}= Fetch From Right ${OPENBMC_MODEL} /
114 ${tmp_bmc_model}= Fetch From Left ${tmp_bmc_model} .
115 ${ret}= Run Keyword And Return Status Should Contain ${bmc_model}
116 ... ${tmp_bmc_model} ignore_case=True
117 [Return] ${ret}
118
119
120Wait For Host To Ping
121 [Documentation] Wait for the given host to ping.
122 [Arguments] ${host} ${timeout}=${OPENBMC_REBOOT_TIMEOUT}min
Sushil Singh7d5a4302023-08-20 23:55:48 -0500123 ... ${interval}=5 sec ${expected_rc}=${0}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600124
125 # Description of argument(s):
Sushil Singh7d5a4302023-08-20 23:55:48 -0500126 # host The host name or IP of the host to ping.
127 # timeout The amount of time after which ping attempts cease.
128 # This should be expressed in Robot Framework's time format
129 # (e.g. "10 seconds").
130 # interval The amount of time in between attempts to ping.
131 # This should be expressed in Robot Framework's time format
132 # (e.g. "5 seconds").
133 # expected_rc Expected return code of ping command.
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600134
Sushil Singh7d5a4302023-08-20 23:55:48 -0500135 Wait Until Keyword Succeeds ${timeout} ${interval} Ping Host ${host} ${expected_rc}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600136
137
138Ping Host
139 [Documentation] Ping the given host.
Sushil Singh7d5a4302023-08-20 23:55:48 -0500140 [Arguments] ${host} ${expected_rc}=${0}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600141
142 # Description of argument(s):
Sushil Singh7d5a4302023-08-20 23:55:48 -0500143 # host The host name or IP of the host to ping.
144 # expected_rc Expected return code of ping command.
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600145
146 Should Not Be Empty ${host} msg=No host provided
Sushil Singh7d5a4302023-08-20 23:55:48 -0500147 ${rc} ${output}= Run and return RC and Output ping -c 4 ${host}
148 Log RC: ${rc}\nOutput:\n${output}
149 Should be equal ${rc} ${expected_rc}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600150
151
152Check OS
153 [Documentation] Attempts to ping the host OS and then checks that the host
154 ... OS is up by running an SSH command.
155
156 [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME}
157 ... ${os_password}=${OS_PASSWORD} ${quiet}=${QUIET}
158 ... ${print_string}=${EMPTY}
159 [Teardown] SSHLibrary.Close Connection
160
161 # Description of argument(s):
162 # os_host The DNS name/IP of the OS host associated with our BMC.
163 # os_username The username to be used to sign on to the OS host.
164 # os_password The password to be used to sign on to the OS host.
165 # quiet Indicates whether this keyword should write to console.
166 # print_string A string to be printed before checking the OS.
167
Michael Walshc108e422019-03-28 12:27:18 -0500168 Log To Console ${print_string} no_newline=True
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600169
170 # Attempt to ping the OS. Store the return code to check later.
171 ${ping_rc}= Run Keyword and Return Status Ping Host ${os_host}
172
173 SSHLibrary.Open connection ${os_host}
174
George Keishing30751662019-03-05 06:35:55 -0600175 ${status} ${msg}= Run Keyword And Ignore Error SSHLibrary.Login ${os_username}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600176 ... ${os_password}
177 ${err_msg1}= Sprint Error ${msg}
178 ${err_msg}= Catenate SEPARATOR= \n ${err_msg1}
179 Run Keyword If '${status}' == 'FAIL' Fail msg=${err_msg}
180 ${output} ${stderr} ${rc}= Execute Command uptime return_stderr=True
181 ... return_rc=True
182
183 ${temp_msg}= Catenate Could not execute a command on the operating
184 ... system.\n
185 ${err_msg1}= Sprint Error ${temp_msg}
186 ${err_msg}= Catenate SEPARATOR= \n ${err_msg1}
187
188 # If the return code returned by "Execute Command" is non-zero, this
189 # keyword will fail.
190 Should Be Equal ${rc} ${0} msg=${err_msg}
191 # We will likewise fail if there is any stderr data.
192 Should Be Empty ${stderr}
193
194 ${temp_msg}= Set Variable Could not ping the operating system.\n
195 ${err_msg1}= Sprint Error ${temp_msg}
196 ${err_msg}= Catenate SEPARATOR= \n ${err_msg1}
197 # We will likewise fail if the OS did not ping, as we could SSH but not
198 # ping
199 Should Be Equal As Strings ${ping_rc} ${TRUE} msg=${err_msg}
200
201
202Wait for OS
203 [Documentation] Waits for the host OS to come up via calls to "Check OS".
204 [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME}
205 ... ${os_password}=${OS_PASSWORD} ${timeout}=${OS_WAIT_TIMEOUT}
206 ... ${quiet}=${0}
Michael Walshc108e422019-03-28 12:27:18 -0500207 [Teardown] Printn
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600208
209 # Description of argument(s):
210 # os_host The DNS name or IP of the OS host associated with our
211 # BMC.
212 # os_username The username to be used to sign on to the OS host.
213 # os_password The password to be used to sign on to the OS host.
214 # timeout The timeout in seconds indicating how long you're
215 # willing to wait for the OS to respond.
216 # quiet Indicates whether this keyword should write to console.
217
218 # The interval to be used between calls to "Check OS".
219 ${interval}= Set Variable 5
220
221 ${message}= Catenate Checking every ${interval} seconds for up to
222 ... ${timeout} seconds for the operating system to communicate.
Michael Walshc108e422019-03-28 12:27:18 -0500223 Qprint Timen ${message}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600224
225 Wait Until Keyword Succeeds ${timeout} sec ${interval} Check OS
226 ... ${os_host} ${os_username} ${os_password}
227 ... print_string=\#
228
Michael Walshc108e422019-03-28 12:27:18 -0500229 Qprintn
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600230
Michael Walshc108e422019-03-28 12:27:18 -0500231 Qprint Timen The operating system is now communicating.
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600232
233
234Copy PNOR to BMC
235 [Documentation] Copy the PNOR image to the BMC.
236 Import Library SCPLibrary WITH NAME scp
237 Open Connection for SCP
238 Log Copying ${PNOR_IMAGE_PATH} to /tmp
239 scp.Put File ${PNOR_IMAGE_PATH} /tmp
240
241
242Is OS Starting
243 [Documentation] Check if boot progress is OS starting.
244 ${boot_progress}= Get Boot Progress
245 Should Be Equal ${boot_progress} OSStart
246
247
248Is OS Off
249 [Documentation] Check if boot progress is "Off".
250 ${boot_progress}= Get Boot Progress
251 Should Be Equal ${boot_progress} Off
252
253
254Get Boot Progress To OS Starting State
255 [Documentation] Get the system to a boot progress state of 'FW Progress,
256 ... Starting OS'.
257
258 ${boot_progress}= Get Boot Progress
259 Run Keyword If '${boot_progress}' == 'OSStart'
260 ... Log Host is already in OS starting state
261 ... ELSE
262 ... Run Keywords Initiate Host PowerOff AND Initiate Host Boot
263 ... AND Wait Until Keyword Succeeds 10 min 10 sec Is OS Starting
264
265
266Check If warmReset is Initiated
267 [Documentation] Ping would be still alive, so try SSH to connect
268 ... if fails the ports are down indicating reboot
269 ... is in progress
270
271 # Warm reset adds 3 seconds delay before forcing reboot
272 # To minimize race conditions, we wait for 7 seconds
273 Sleep 7s
274 ${alive}= Run Keyword and Return Status
275 ... Open Connection And Log In
276 Return From Keyword If '${alive}' == '${False}' ${False}
277 [Return] ${True}
278
279
280Initialize DBUS cmd
281 [Documentation] Initialize dbus string with property string to extract
282 [Arguments] ${boot_property}
283
284 # Description of argument(s):
285 # boot_property Property string.
286
287 ${cmd}= Catenate ${dbuscmdBase} ${dbuscmdGet} ${dbuscmdString}
288 ${cmd}= Catenate ${cmd}${boot_property}
289 Set Global Variable ${dbuscmd} ${cmd}
290
291
292Create OS Console Command String
293 [Documentation] Return a command string to start OS console logging.
294
295 # First make sure that the ssh_pw program is available.
296 ${cmd}= Catenate which ssh_pw 2>/dev/null || find
297 ... ${EXECDIR} -name 'ssh_pw'
298
Michael Walshc108e422019-03-28 12:27:18 -0500299 Dprint Issuing ${cmd}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600300 ${rc} ${output}= Run And Return Rc And Output ${cmd}
301 Rdpvars rc output
302
303 Should Be Equal As Integers 0 ${rc} msg=Could not find ssh_pw.
304
305 ${ssh_pw_file_path}= Set Variable ${output}
306
George Keishing088166c2019-12-13 08:11:28 -0600307 ${cmd}= Catenate ${ssh_pw_file_path} ${OPENBMC_PASSWORD} -p ${HOST_SOL_PORT}
Jorge Cisnerosf196b242022-09-09 15:46:15 +0000308 ... -o "StrictHostKeyChecking no" ${OPENBMC_USERNAME}@${OPENBMC_HOST} ${OPENBMC_CONSOLE_CLIENT}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600309
Jorge Cisnerosf196b242022-09-09 15:46:15 +0000310 [Return] ${cmd.strip()}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600311
312
313Get SOL Console Pid
314 [Documentation] Get the pid of the active SOL console job.
Michael Walsh07ed7942019-11-08 14:41:30 -0600315 [Arguments] ${expect_running}=${0} ${log_file_path}=${EMPTY}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600316
317 # Description of argument(s):
Michael Walsh07ed7942019-11-08 14:41:30 -0600318 # expect_running If set and if no SOL console job is found, print debug info and fail.
319 # log_file_path Needed to print debug info if expect_running is set and no pid is found.
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600320
321 # Find the pid of the active system console logging session (if any).
322 ${search_string}= Create OS Console Command String
323 # At least in some cases, ps output does not show double quotes so we must
324 # replace them in our search string with the regexes to indicate that they
325 # are optional.
326 ${search_string}= Replace String ${search_string} " ["]?
327 ${ps_cmd}= Catenate ps axwwo user,pid,cmd
328 ${cmd_buf}= Catenate echo $(${ps_cmd} | egrep '${search_string}' |
329 ... egrep -v grep | cut -c10-14)
Michael Walshc108e422019-03-28 12:27:18 -0500330 Dprint Issuing ${cmd_buf}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600331 ${rc} ${os_con_pid}= Run And Return Rc And Output ${cmd_buf}
332 Rdpvars os_con_pid
333 # If rc is not zero it just means that there is no OS Console process
334 # running.
335
336 Return From Keyword If '${os_con_pid}' != '${EMPTY}' ${os_con_pid}
337 Return From Keyword If '${expect_running}' == '${0}' ${os_con_pid}
338
339 Cmd Fnc cat ${log_file_path} ; echo ; ${ps_cmd} quiet=${0}
340 ... print_output=${1} show_err=${1}
Michael Walsh07ed7942019-11-08 14:41:30 -0600341 Valid Value os_con_pid
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600342
343
344Stop SOL Console Logging
345 [Documentation] Stop system console logging and return log output.
346 [Arguments] ${log_file_path}=${EMPTY}
347 ... ${targ_file_path}=${EXECDIR}${/}logs${/}
348 ... ${return_data}=${1}
349
George Keishinge16f1582022-12-15 07:32:21 -0600350 # If there are multiple system console processes, they will all be stopped.
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600351 # If there is no existing log file this keyword will return an error
352 # message to that effect (and write that message to targ_file_path, if
353 # specified).
354 # NOTE: This keyword will not fail if there is no running system console
355 # process.
356
357 # Description of arguments:
358 # log_file_path The file path that was used to call "Start SOL
359 # Console Logging". See that keyword (above) for details.
360 # targ_file_path If specified, the file path to which the source
361 # file path (i.e. "log_file_path") should be copied.
362 # return_data If this is set to ${1}, this keyword will return the SOL
363 # data to the caller as a unicode string.
364
365 ${log_file_path}= Create OS Console File Path ${log_file_path}
366
367 ${os_con_pid}= Get SOL Console Pid
368
369 ${cmd_buf}= Catenate kill -9 ${os_con_pid}
Michael Walshc108e422019-03-28 12:27:18 -0500370 Run Keyword If '${os_con_pid}' != '${EMPTY}' Dprint Issuing ${cmd_buf}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600371 ${rc} ${output}= Run Keyword If '${os_con_pid}' != '${EMPTY}'
372 ... Run And Return Rc And Output ${cmd_buf}
373 Run Keyword If '${os_con_pid}' != '${EMPTY}' Rdpvars rc output
374
375 Run Keyword If '${targ_file_path}' != '${EMPTY}'
376 ... Run Keyword And Ignore Error
377 ... Copy File ${log_file_path} ${targ_file_path}
378
379 ${output}= Set Variable ${EMPTY}
380 ${loc_quiet}= Evaluate ${debug}^1
381 ${rc} ${output}= Run Keyword If '${return_data}' == '${1}'
382 ... Cmd Fnc cat ${log_file_path} 2>/dev/null quiet=${loc_quiet}
383 ... print_output=${0} show_err=${0}
384
385 [Return] ${output}
386
387
388Start SOL Console Logging
389 [Documentation] Start system console log to file.
390 [Arguments] ${log_file_path}=${EMPTY} ${return_data}=${1}
391
392 # This keyword will first call "Stop SOL Console Logging". Only then will
393 # it start SOL console logging. The data returned by "Stop SOL Console
394 # Logging" will in turn be returned by this keyword.
395
396 # Description of arguments:
397 # log_file_path The file path to which system console log data should be
398 # written. Note that this path is taken to be a location
399 # on the machine where this program is running rather than
400 # on the Open BMC system.
401 # return_data If this is set to ${1}, this keyword will return any SOL
402 # data to the caller as a unicode string.
403
404 ${log_file_path}= Create OS Console File Path ${log_file_path}
405
406 ${log_output}= Stop SOL Console Logging ${log_file_path}
407 ... return_data=${return_data}
408
409 # Validate by making sure we can create the file. Problems creating the
410 # file would not be noticed by the subsequent ssh command because we fork
411 # the command.
412 Create File ${log_file_path}
413 ${sub_cmd_buf}= Create OS Console Command String
414 # Routing stderr to stdout so that any startup error text will go to the
415 # output file.
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600416 ${cmd_buf}= Catenate ${sub_cmd_buf} > ${log_file_path} 2>&1 &
Michael Walshc108e422019-03-28 12:27:18 -0500417 Dprint Issuing ${cmd_buf}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600418 ${rc} ${output}= Run And Return Rc And Output ${cmd_buf}
419 # Because we are forking this command, we essentially will never get a
420 # non-zero return code or any output.
421 Should Be Equal ${rc} ${0}
422
423 Wait Until Keyword Succeeds 10 seconds 0 seconds
Michael Walsh07ed7942019-11-08 14:41:30 -0600424 ... Get SOL Console Pid ${1} ${log_file_path}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600425
426 [Return] ${log_output}
427
428
429Get Time Stamp
430 [Documentation] Get the current time stamp data
431 ${cur_time}= Get Current Date result_format=%Y%m%d%H%M%S%f
432 [Return] ${cur_time}
433
434
435Start Journal Log
436 [Documentation] Start capturing journal log to a file in /tmp using
437 ... journalctl command. By default journal log is collected
438 ... at /tmp/journal_log else user input location.
439 ... The File is appended with datetime.
440 [Arguments] ${file_path}=/tmp/journal_log ${filter}=${EMPTY}
441
442 # Description of arguments:
443 # file_path The file path of the journal file.
444
445 ${cur_time}= Get Time Stamp
446 Set Global Variable ${LOG_TIME} ${cur_time}
447 Open Connection And Log In
448 Start Command
449 ... journalctl -f ${filter} > ${file_path}-${LOG_TIME}
450 Log Journal Log Started: ${file_path}-${LOG_TIME}
451
452
453Stop Journal Log
454 [Documentation] Stop journalctl process if its running.
455 ... By default return log from /tmp/journal_log else
456 ... user input location.
457 [Arguments] ${file_path}=/tmp/journal_log
458
459 # Description of arguments:
460 # file_path The file path of the journal file.
461
462 Open Connection And Log In
463
464 ${rc}=
465 ... Execute Command
466 ... ps | grep journalctl | grep -v grep
467 ... return_stdout=False return_rc=True
468
469 Return From Keyword If '${rc}' == '${1}'
470 ... No journal log process running
471
472 ${output} ${stderr}=
473 ... Execute Command killall journalctl
474 ... return_stderr=True
475 Should Be Empty ${stderr}
476
477 ${journal_log} ${stderr}=
478 ... Execute Command
479 ... cat ${file_path}-${LOG_TIME}
480 ... return_stderr=True
481 Should Be Empty ${stderr}
482
483 Log ${journal_log}
484
485 Execute Command rm ${file_path}-${LOG_TIME}
486
487 [Return] ${journal_log}
488
489
490Mac Address To Hex String
491 [Documentation] Converts MAC address into hex format.
492 ... Example
493 ... Given the following MAC: 00:01:6C:80:02:78
494 ... This keyword will return: 0x00 0x01 0x6C 0x80 0x02 0x78
495 ... Description of arguments:
496 ... i_macaddress MAC address in the following format
497 ... 00:01:6C:80:02:78
498 [Arguments] ${i_macaddress}
499
500 # Description of arguments:
501 # i_macaddress The MAC address.
502
503 ${mac_hex}= Catenate 0x${i_macaddress.replace(':', ' 0x')}
504 [Return] ${mac_hex}
505
506
507IP Address To Hex String
508 [Documentation] Converts IP address into hex format.
509 ... Example:
510 ... Given the following IP: 10.3.164.100
511 ... This keyword will return: 0xa 0x3 0xa4 0xa0
512 [Arguments] ${i_ipaddress}
513
514 # Description of arguments:
515 # i_macaddress The IP address in the format 10.10.10.10.
516
517 @{ip}= Split String ${i_ipaddress} .
518 ${index}= Set Variable ${0}
519
Marissa Garza20ccfc72020-06-19 12:51:10 -0500520 FOR ${item} IN @{ip}
521 ${hex}= Convert To Hex ${item} prefix=0x lowercase=yes
522 Set List Value ${ip} ${index} ${hex}
523 ${index}= Set Variable ${index + 1}
524 END
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600525 ${ip_hex}= Catenate @{ip}
526
527 [Return] ${ip_hex}
528
529
530BMC CPU Performance Check
531 [Documentation] Minimal 10% of proc should be free in this instance
532
533 ${bmc_cpu_usage_output} ${stderr} ${rc}= BMC Execute Command
534 ... ${bmc_cpu_usage_cmd}
535 ${bmc_cpu_usage_output} ${stderr} ${rc}= BMC Execute Command
536 ... ${bmc_cpu_usage_cmd}
537 ${bmc_cpu_percentage}= Fetch From Left ${bmc_cpu_usage_output} %
Sivas SRR14426bc2019-04-10 10:14:46 -0500538 Rprint Vars bmc_cpu_percentage
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600539 Should be true ${bmc_cpu_percentage} < 90
540
541
542BMC Mem Performance Check
543 [Documentation] Minimal 10% of memory should be free in this instance
544
545 ${bmc_mem_free_output} ${stderr} ${rc}= BMC Execute Command
546 ... ${bmc_mem_free_cmd}
547
548 ${bmc_mem_total_output} ${stderr} ${rc}= BMC Execute Command
549 ... ${bmc_mem_total_cmd}
550 ${bmc_mem_free_output} ${stderr} ${rc}= BMC Execute Command
551 ... ${bmc_mem_free_cmd}
552
553 ${bmc_mem_total_output} ${stderr} ${rc}= BMC Execute Command
554 ... ${bmc_mem_total_cmd}
555
556 ${bmc_mem_percentage}= Evaluate ${bmc_mem_free_output}*100
557 ${bmc_mem_percentage}= Evaluate
558 ... ${bmc_mem_percentage}/${bmc_mem_total_output}
Sivas SRR14426bc2019-04-10 10:14:46 -0500559 Rprint Vars bmc_mem_percentage
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600560 Should be true ${bmc_mem_percentage} > 10
561
562
563BMC File System Usage Check
564 [Documentation] Check the file system space. 4 file system should be
565 ... 100% full which is expected
566 # Filesystem Size Used Available Use% Mounted on
567 # /dev/root 14.4M 14.4M 0 100% /
568 # /dev/ubiblock0_0 14.4M 14.4M 0 100% /media/rofs-c9249b0e
569 # /dev/ubiblock8_0 19.6M 19.6M 0 100% /media/pnor-ro-8764baa3
570 # /dev/ubiblock4_0 14.4M 14.4M 0 100% /media/rofs-407816c
571 # /dev/ubiblock8_4 21.1M 21.1M 0 100% /media/pnor-ro-cecc64c4
572 ${bmc_fs_usage_output} ${stderr} ${rc}= BMC Execute Command
573 ... ${bmc_file_system_usage_cmd}
574 ${bmc_pnor_fs_usage_output} ${stderr} ${rc}= BMC Execute Command
575 ... ${total_pnor_ro_file_system_cmd}
576 ${bmc_bmc_fs_usage_output} ${stderr} ${rc}= BMC Execute Command
577 ... ${total_bmc_ro_file_system_cmd}
578 ${total_bmc_pnor_image}= Evaluate
579 ... ${bmc_pnor_fs_usage_output}+${bmc_bmc_fs_usage_output}
580 # Considering /dev/root also in total 100% used file system
581 ${total_full_fs}= Evaluate ${total_bmc_pnor_image}+1
Sivas SRR14426bc2019-04-10 10:14:46 -0500582 Rprint Vars bmc_fs_usage_output bmc_pnor_fs_usage_output bmc_bmc_fs_usage_output
583 ... total_full_fs
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600584 Should Be True ${bmc_fs_usage_output}==${total_full_fs}
585
586
587Check BMC CPU Performance
588 [Documentation] Minimal 10% of proc should be free in 3 sample
Marissa Garza20ccfc72020-06-19 12:51:10 -0500589 FOR ${var} IN RANGE 1 4
590 BMC CPU Performance Check
591 END
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600592
593Check BMC Mem Performance
594 [Documentation] Minimal 10% of memory should be free
595
Marissa Garza20ccfc72020-06-19 12:51:10 -0500596 FOR ${var} IN RANGE 1 4
597 BMC Mem Performance Check
598 END
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600599
600Check BMC File System Performance
601 [Documentation] Check for file system usage for 4 times
602
Marissa Garza20ccfc72020-06-19 12:51:10 -0500603 FOR ${var} IN RANGE 1 4
604 BMC File System Usage check
605 END
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600606
607Get URL List
608 [Documentation] Return list of URLs under given URL.
609 [Arguments] ${openbmc_url}
610
611 # Description of argument(s):
612 # openbmc_url URL for list operation (e.g.
613 # /xyz/openbmc_project/inventory).
614
Steven Sombaraaaab222018-12-19 13:16:23 -0600615 ${url_list}= Read Properties ${openbmc_url}list quiet=${1}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600616 Sort List ${url_list}
617
618 [Return] ${url_list}
619
620
621Check Zombie Process
622 [Documentation] Check if any defunct process exist or not on BMC
623 ${count} ${stderr} ${rc}= Execute Command ps -o stat | grep Z | wc -l
624 ... return_stderr=True return_rc=True
625 Should Be True ${count}==0
626 Should Be Empty ${stderr}
627
628
629Prune Journal Log
630 [Documentation] Prune archived journal logs.
631 [Arguments] ${vacuum_size}=1M
632
633 # This keyword can be used to prevent the journal
634 # log from filling up the /run filesystem.
635 # This command will retain only the latest logs
636 # of the user specified size.
637
638 # Description of argument(s):
639 # vacuum_size Size of journal.
640
641 Open Connection And Log In
642 ${output} ${stderr} ${rc}=
643 ... Execute Command
644 ... journalctl --vacuum-size=${vacuum_size}
645 ... return_stderr=True return_rc=True
646
647 Should Be Equal ${rc} ${0} msg=${stderr}
648
649
650Get System Power Policy
651 [Documentation] Returns the BMC power policy.
652
653 # Set the bmc_power_policy_method to either 'Old' or 'New'.
654 Set Power Policy Method
655 ${cmd_buf}= Create List ${bmc_power_policy_method} Get Power Policy
656 # Run the appropriate keyword.
657 ${currentPolicy}= Run Keyword @{cmd_buf}
658
659 [Return] ${currentPolicy}
660
661
662Set BMC Reset Reference Time
663 [Documentation] Set current boot time as a reference and increment
664 ... boot count.
665
666 ${cur_btime}= Get BMC Boot Time
667 Run Keyword If ${BOOT_TIME} == ${0} and ${BOOT_COUNT} == ${0}
668 ... Set Global Variable ${BOOT_TIME} ${cur_btime}
669 ... ELSE IF ${cur_btime} > ${BOOT_TIME}
670 ... Run Keywords Set Global Variable ${BOOT_TIME} ${cur_btime}
671 ... AND
672 ... Set Global Variable ${BOOT_COUNT} ${BOOT_COUNT + 1}
673
674
675Get BMC Boot Time
676 [Documentation] Returns boot time from /proc/stat.
677
678 Open Connection And Log In
679 ${output} ${stderr}=
680 ... Execute Command egrep '^btime ' /proc/stat | cut -f 2 -d ' '
681 ... return_stderr=True
682 Should Be Empty ${stderr}
683 ${btime}= Convert To Integer ${output}
684 [Return] ${btime}
685
686
687Enable Core Dump On BMC
688 [Documentation] Enable core dump collection.
689 ${core_pattern} ${stderr} ${rc}= BMC Execute Command
690 ... echo '/tmp/core_%e.%p' | tee /proc/sys/kernel/core_pattern
691 Should Be Equal As Strings ${core_pattern} /tmp/core_%e.%p
692
693
694Get Number Of BMC Core Dump Files
695 [Documentation] Returns number of core dump files on BMC.
696 Open Connection And Log In
697 ${num_of_core_dump}= Execute Command
698 ... ls /tmp/core* 2>/dev/null | wc -l
699 [Return] ${num_of_core_dump}
700
701
702Set Core Dump File Size Unlimited
703 [Documentation] Set core dump file size to unlimited.
704 BMC Execute Command ulimit -c unlimited
705
706
707Check For Core Dumps
708 [Documentation] Check for any core dumps exist.
709 ${output}= Get Number Of BMC Core Dump Files
710 Run Keyword If ${output} > 0
711 ... Log **Warning** BMC core dump files exist level=WARN
712
713
714Configure Initial Settings
715 [Documentation] Restore old IP and route.
716 ... This keyword requires initial settings viz IP address,
George Keishing16b3c7b2021-01-28 09:23:37 -0600717 ... Network Mask, default gateway and serial console IP and port
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600718 ... information which should be provided in command line.
719
720 [Arguments] ${host}=${OPENBMC_HOST} ${mask}=${NET_MASK}
721 ... ${gw_ip}=${GW_IP}
722
723 # Description of arguments:
724 # host IP address of the OS Host.
725 # mask Network mask.
726 # gu_ip Gateway IP address or hostname.
727
728 # Open telnet connection and ignore the error, in case telnet session is
729 # already opened by the program calling this keyword.
730 Run Keyword And Ignore Error Open Telnet Connection to BMC Serial Console
731 Telnet.write ifconfig eth0 ${host} netmask ${mask}
732 Telnet.write route add default gw ${gw_ip}
733
734
735Install Debug Tarball On BMC
736 [Documentation] Copy the debug tar file to BMC and install.
737 [Arguments] ${tarball_file_path}=${default_tarball}
738 ... ${targ_tarball_dir_path}=/tmp/tarball/
739
740 # Description of arguments:
741 # tarball_file_path Path of the debug tarball file.
742 # The tar file is downloaded from the build page
George Keishinged616122020-07-02 00:57:15 -0500743 # https://jenkins.openbmc.org/job/latest-master/
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600744 # obmc-phosphor-debug-tarball-witherspoon.tar.xz
745 #
746 # targ_tarball_dir_path The directory path where the tarball is to be
747 # installed.
748
749 OperatingSystem.File Should Exist ${tarball_file_path}
750 ... msg=${tarball_file_path} doesn't exist.
751
752 # Upload the file to BMC.
753 Import Library SCPLibrary WITH NAME scp
754 Open Connection for SCP
755 scp.Put File ${tarball_file_path} /tmp/debug-tarball.tar.xz
756
757 # Create tarball directory and install.
758 BMC Execute Command mkdir -p ${targ_tarball_dir_path}
759 BMC Execute Command
760 ... tar -xf /tmp/debug-tarball.tar.xz -C ${targ_tarball_dir_path}
761
762 # Remove the tarball file from BMC.
763 BMC Execute Command rm -f /tmp/debug-tarball.tar.xz
764
765
766Get BMC Boot Count
767 [Documentation] Returns BMC boot count based on boot time.
768 ${cur_btime}= Get BMC Boot Time
769
770 # Set global variable BOOT_TIME to current boot time if current boot time
771 # is changed. Also increase value of global variable BOOT_COUNT by 1.
772 Run Keyword If ${cur_btime} > ${BOOT_TIME}
773 ... Run Keywords Set Global Variable ${BOOT_TIME} ${cur_btime}
774 ... AND
775 ... Set Global Variable ${BOOT_COUNT} ${BOOT_COUNT + 1}
776
777 [Return] ${BOOT_COUNT}
778
779
780Set BMC Boot Count
781 [Documentation] Set BMC boot count to given value.
782 [Arguments] ${count}
783
784 # Description of arguments:
785 # count boot count value.
786 ${cur_btime}= Get BMC Boot Time
787
788 # Set global variable BOOT_COUNT to given value.
789 Set Global Variable ${BOOT_COUNT} ${count}
790
791 # Set BOOT_TIME variable to current boot time.
Tony Lee073fbc02020-05-12 15:12:16 +0800792 Set Global Variable ${BOOT_TIME} ${cur_btime}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600793
794
795Delete Error Log Entry
796 [Documentation] Delete error log entry.
Michael Sheposcc490b42020-08-26 12:53:01 -0500797 [Arguments] ${entry_path} ${quiet}=${0}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600798
799 # Description of argument(s):
Michael Sheposcc490b42020-08-26 12:53:01 -0500800 # quiet If enabled, turns off logging to console.
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600801 # entry_path Delete an error log entry.
802 # Ex. /xyz/openbmc_project/logging/entry/1
803
804 # Skip delete if entry URI is a callout.
805 # Examples:
806 # /xyz/openbmc_project/logging/entry/1/callout
807 # /xyz/openbmc_project/logging/entry/1/callouts/0
808 ${callout_entry}= Run Keyword And Return Status
809 ... Should Match Regexp ${entry_path} /callout[s]?(/|$)
810 Return From Keyword If ${callout_entry}
811
812 ${data}= Create Dictionary data=@{EMPTY}
Michael Sheposcc490b42020-08-26 12:53:01 -0500813 ${resp}= Openbmc Delete Request ${entry_path} data=${data} quiet=${quiet}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600814 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
815
816
817Get BMC Version
818 [Documentation] Returns BMC version from /etc/os-release.
819 ... e.g. "v1.99.6-141-ge662190"
820
821 ${cmd}= Set Variable grep ^VERSION_ID= /etc/os-release | cut -f 2 -d '='
822 ${output} ${stderr} ${rc}= BMC Execute Command ${cmd}
823 [Return] ${output}
824
825
826Get PNOR Version
827 [Documentation] Returns the PNOR version from the BMC.
828
829 ${pnor_attrs}= Get PNOR Attributes
830 [Return] ${pnor_attrs['version']}
831
832
833Get PNOR Attributes
834 [Documentation] Return PNOR software attributes as a dictionary.
835
836 # This keyword parses /var/lib/phosphor-software-manager/pnor/ro/pnor.toc
837 # into key/value pairs.
838
839 ${outbuf} ${stderr} ${rc}= BMC Execute Command
840 ... cat /var/lib/phosphor-software-manager/pnor/ro/pnor.toc
841 ${pnor_attrs}= Key Value Outbuf To Dict ${outbuf} delim==
842
843 [Return] ${pnor_attrs}
844
845
Sivas SRR17dae1a2019-03-24 23:52:02 -0500846GET BMC PNOR Version
847 [Documentation] Return BMC & PNOR version from openbmc shell.
848
849 ${bmc_version}= GET BMC Version
850 ${pnor_version}= GET PNOR Version
851 Log ${bmc_version}
852 Rprint Vars bmc_version
853 Log ${pnor_version}
854 Rprint Vars pnor_version
855
856 [Return] ${bmc_version} ${pnor_version}
857
858
Sushil Singh5ea86d02019-07-11 02:05:16 -0500859Redfish Get BMC Version
860 [Documentation] Get BMC version via Redfish.
861
ganesanb4d430282023-04-27 14:33:23 +0000862 ${output}= Redfish.Get Attribute ${REDFISH_BASE_URI}Managers/${MANAGER_ID} FirmwareVersion
Sushil Singh5ea86d02019-07-11 02:05:16 -0500863 [Return] ${output}
864
865Redfish Get Host Version
866 [Documentation] Get host version via Redfish.
867
868 ${output}= Redfish.Get Attribute ${REDFISH_BASE_URI}Systems/system BiosVersion
869 [Return] ${output}
870
871
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600872Copy Address Translation Utils To HOST OS
873 [Documentation] Copy address translation utils to host OS.
874
875 OperatingSystem.File Should Exist ${probe_cpu_tool_path}
876 ... msg=${probe_cpu_tool_path} doesn't exist.
877 OperatingSystem.File Should Exist ${probe_cpu_tool_path}
878 ... msg=${probe_cpu_tool_path} doesn't exist.
879
880 scp.Open connection ${OS_HOST} username=${OS_USERNAME}
881 ... password=${OS_PASSWORD}
882 scp.Put File ${probe_cpu_tool_path} ${target_file_path}
883 scp.Put File ${scom_addrs_tool_path} ${target_file_path}
884
885
886Verify BMC RTC And UTC Time Drift
887 [Documentation] Verify that the RTC and UTC time difference is less than
888 ... the given time_drift_max.
889 [Arguments] ${time_diff_max}=${10}
890
891 # Description of argument(s):
892 # time_diff_max The max allowable RTC and UTC time difference in seconds.
893
894 # Example:
895 # time_dict:
896 # [local_time]: Fri 2017-11-03 152756 UTC
897 # [local_time_seconds]: 1509740876
898 # [universal_time]: Fri 2017-11-03 152756 UTC
899 # [universal_time_seconds]: 1509740876
900 # [rtc_time]: Fri 2016-05-20 163403
901 # [rtc_time_seconds]: 1463780043
902 # [time_zone]: n/a (UTC, +0000)
903 # [network_time_on]: yes
904 # [ntp_synchronized]: no
905 # [rtc_in_local_tz]: no
906
907 ${time}= Get BMC Date Time
908 ${time_diff}= Evaluate
909 ... ${time['universal_time_seconds']} - ${time['rtc_time_seconds']}
910 Should Be True ${time_diff} < ${time_diff_max}
911
912
913Validate IP On BMC
914 [Documentation] Validate IP address is present in set of IP addresses.
915 [Arguments] ${ip_address} ${ip_data}
916
917 # Description of argument(s):
918 # ip_address IP address to check (e.g. xx.xx.xx.xx).
919 # ip_data Set of the IP addresses present.
920
921 Should Contain Match ${ip_data} ${ip_address}/*
922 ... msg=${ip_address} not found in the list provided.
923
924
925Remove Journald Logs
926 [Documentation] Remove all journald logs and restart service.
927
928 ${cmd}= Catenate systemctl stop systemd-journald.service &&
929 ... rm -rf /var/log/journal && systemctl start systemd-journald.service
930
931 BMC Execute Command ${cmd}
932
933
934Check For Regex In Journald
935 [Documentation] Parse the journal log and check for regex string.
936 [Arguments] ${regex}=${ERROR_REGEX} ${error_check}=${0} ${boot}=${EMPTY}
George Keishing140351f2022-03-10 08:10:01 -0600937 ... ${filter_string}=${EMPTY}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600938
939 # Description of argument(s):
940 # regex Strings to be filter.
941 # error_check Check for errors.
942 # boot Argument to check current or persistent full boot log
943 # (e.g. "-b").
George Keishing140351f2022-03-10 08:10:01 -0600944 # filter_string String to be stripped out.
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600945
George Keishing140351f2022-03-10 08:10:01 -0600946
947 ${cmd} = Run Keyword If '${filter_string}' == '${EMPTY}'
948 ... Catenate journalctl --no-pager ${boot} | egrep '${regex}'
949 ... ELSE
950 ... Catenate journalctl --no-pager ${boot} | egrep '${regex}' | sed '/${filter_string}/d'
951
952 ${journal_log} ${stderr} ${rc}= BMC Execute Command ${cmd} ignore_err=1
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600953
954 Run Keyword If ${error_check} == ${0}
955 ... Should Be Empty ${journal_log}
956 ... ELSE
957 ... Should Not Be Empty ${journal_log}
958
959
960Get Service Attribute
961 [Documentation] Get service attribute policy output.
962 [Arguments] ${option} ${servicename}
963
964 # Description of argument(s):
965 # option systemctl supported options
966 # servicename Qualified service name
967 ${cmd}= Set Variable
968 ... systemctl -p ${option} show ${servicename} | cut -d = -f2
969 ${attr} ${stderr} ${rc}= BMC Execute Command ${cmd}
970 [Return] ${attr}
Anusha Dathatrieda09492019-02-26 08:30:13 -0600971
972
973Verify Watchdog Enabled
974 [Documentation] Check that watchdog settings are enabled.
975
976 ${properties}= Read Properties /xyz/openbmc_project/watchdog/host0
977 Should Be Equal As Strings ${properties["Enabled"]} ${True}
978 Should Not Be Equal As Strings ${properties["TimeRemaining"]} 0
Ashwini Chandrappa233d9652021-08-04 09:56:30 -0500979
980
981Is BMC Unpingable
982 [Documentation] Check if BMC is unpingable.
983
984 ${RC} ${output}= Run and return RC and Output ping -c 4 ${OPENBMC_HOST}
985 Log RC: ${RC}\nOutput:\n${output}
986 Should be equal ${RC} ${1}
Sushil Singh6d170ad2022-11-28 07:04:50 -0600987
Sushil Singh84c80152023-02-06 02:02:48 -0600988
Nagarjun B011c5f32023-07-17 17:14:59 +0530989Is Host Unpingable
990 [Documentation] Check if Given IP is unpingable.
991 [Arguments] ${ip}
992
993 # Description of argument(s):
994 # ip HostOS IP address (e.g. "10.7.7.7").
995
996 ${RC} ${output}= Run and return RC and Output ping -c 4 ${ip}
997 Log RC: ${RC}\nOutput:\n${output}
998 Should be equal ${RC} ${1}
999
1000
Sushil Singh84c80152023-02-06 02:02:48 -06001001Redfish BMC Match States
1002 [Documentation] Verify the BMC match state.
1003 [Arguments] ${match_state}
1004
1005 # Description of argument(s):
1006 # match_state Match the state of BMC.
1007
1008 ${bmc_state}= Redfish Get BMC State
1009 Should Be Equal As Strings ${match_state} ${bmc_state}
1010
1011
Sushil Singh6d170ad2022-11-28 07:04:50 -06001012Kernel Panic BMC Reset Operation
1013 [Documentation] Create kernel panic to reset BMC.
1014
1015 ${kernel_panic_cmd_file}= Set Variable /bin/sh -c "echo c > /proc/sysrq-trigger"
Sushil Singha83bba02023-01-18 08:43:45 -06001016 ${status}= Run Keyword And Return Status BMC Execute Command ${kernel_panic_cmd_file} time_out=20
Sushil Singh6d170ad2022-11-28 07:04:50 -06001017
1018 [Return] ${status}
1019