blob: 2ec9d8503709c299d6264e315ddb7f3f13ad812b [file] [log] [blame]
Steven Sombarf60cbcf2018-12-07 08:12:18 -06001*** Settings ***
2
3Documentation Utilities for Robot keywords that do not use REST.
4
Sandhya Somashekar839a0c22019-01-31 05:05:43 -06005Resource ../lib/resource.robot
Steven Sombarf60cbcf2018-12-07 08:12:18 -06006Resource ../lib/connection_client.robot
7Resource ../lib/boot_utils.robot
8Library String
9Library DateTime
10Library Process
11Library OperatingSystem
12Library gen_print.py
13Library gen_robot_print.py
14Library gen_cmd.py
Michael Walsh07ed7942019-11-08 14:41:30 -060015Library gen_robot_valid.py
Steven Sombarf60cbcf2018-12-07 08:12:18 -060016Library gen_robot_keyword.py
17Library bmc_ssh_utils.py
18Library utils.py
19Library var_funcs.py
20Library SCPLibrary WITH NAME scp
21
22*** Variables ***
23
24${pflash_cmd} /usr/sbin/pflash -r /dev/stdout -P VERSION
25
26${dbuscmdBase}
27... dbus-send --system --print-reply --dest=${OPENBMC_BASE_DBUS}.settings.Host
28${dbuscmdGet}
29... ${SETTINGS_URI}host0 org.freedesktop.DBus.Properties.Get
30${dbuscmdString}= string:"xyz.openbmc_project.settings.Host" string:
31
32# Assign default value to QUIET for programs which may not define it.
33${QUIET} ${0}
34
35${bmc_mem_free_cmd}= free | tr -s ' ' | sed '/^Mem/!d' | cut -d" " -f4
36${bmc_mem_total_cmd}= free | tr -s ' ' | sed '/^Mem/!d' | cut -d" " -f2
37${bmc_cpu_usage_cmd}= top -n 1 | grep CPU: | cut -c 7-9
38${HOST_SETTING} ${SETTINGS_URI}host0
39
40# /run/initramfs/ro associate filesystem should be 100% full always
41${bmc_file_system_usage_cmd}= df -h | cut -c 52-54 | grep 100 | wc -l
42${total_pnor_ro_file_system_cmd}= df -h | grep /media/pnor-ro | wc -l
43${total_bmc_ro_file_system_cmd}= df -h | grep /media/rofs | wc -l
44
45${BOOT_TIME} ${0}
46${BOOT_COUNT} ${0}
47${count} ${0}
48${devicetree_base} /sys/firmware/devicetree/base/model
49
50# Initialize default debug value to 0.
51${DEBUG} ${0}
52
53${probe_cpu_tool_path} ${EXECDIR}/tools/ras/probe_cpus.sh
54${scom_addrs_tool_path} ${EXECDIR}/tools/ras/scom_addr_p9.sh
55${target_file_path} /root/
56
57${default_tarball} ${EXECDIR}/obmc-phosphor-debug-tarball-witherspoon.tar.xz
58
59# These variables are used to straddle between new and old methods of setting
60# values.
61${bmc_power_policy_method} ${EMPTY}
62@{valid_power_policy_vars} RESTORE_LAST_STATE ALWAYS_POWER_ON
63... ALWAYS_POWER_OFF
64
George Keishingb9f3f772020-01-15 23:23:08 -060065${check_performance} ${1}
66
Steven Sombarf60cbcf2018-12-07 08:12:18 -060067
68*** Keywords ***
69
70Check BMC Performance
71 [Documentation] Check BMC basic CPU Mem File system performance.
72
George Keishingb9f3f772020-01-15 23:23:08 -060073 Return From Keyword If not ${check_performance}
74
Steven Sombarf60cbcf2018-12-07 08:12:18 -060075 Check BMC CPU Performance
76 Check BMC Mem Performance
77 Check BMC File System Performance
78
79
80Verify PNOR Update
81 [Documentation] Verify that the PNOR is not corrupted.
82 # Example:
83 # FFS: Flash header not found. Code: 100
84 # Error 100 opening ffs !
85
86 ${stdout} ${stderr} ${rc}=
87 ... BMC Execute Command /usr/sbin/pflash -h | egrep -q skip
88 ... ignore_err=${1}
89 ${pflash_cmd}= Set Variable If ${rc} == ${0} ${pflash_cmd} --skip=4096
90 ... ${pflash_cmd}
91 ${pnor_info}= BMC Execute Command ${pflash_cmd}
92 Should Not Contain Any ${pnor_info} Flash header not found Error
93
94
95Get BMC System Model
96 [Documentation] Get the BMC model from the device tree and return it.
97
98 ${bmc_model} ${stderr} ${rc}= BMC Execute Command
99 ... cat ${devicetree_base} | cut -d " " -f 1 return_stderr=True
100 ... test_mode=0
101 Should Be Empty ${stderr}
102 Should Not Be Empty ${bmc_model} msg=BMC model is empty.
103 [Return] ${bmc_model}
104
105
106Verify BMC System Model
107 [Documentation] Verify the BMC model with ${OPENBMC_MODEL}.
108 [Arguments] ${bmc_model}
109
110 # Description of argument(s):
111 # bmc_model System model (e.g. "witherspoon").
112
113 ${tmp_bmc_model}= Fetch From Right ${OPENBMC_MODEL} /
114 ${tmp_bmc_model}= Fetch From Left ${tmp_bmc_model} .
115 ${ret}= Run Keyword And Return Status Should Contain ${bmc_model}
116 ... ${tmp_bmc_model} ignore_case=True
117 [Return] ${ret}
118
119
120Wait For Host To Ping
121 [Documentation] Wait for the given host to ping.
122 [Arguments] ${host} ${timeout}=${OPENBMC_REBOOT_TIMEOUT}min
123 ... ${interval}=5 sec
124
125 # Description of argument(s):
126 # host The host name or IP of the host to ping.
127 # timeout The amount of time after which ping attempts cease.
128 # This should be expressed in Robot Framework's time format
129 # (e.g. "10 seconds").
130 # interval The amount of time in between attempts to ping.
131 # This should be expressed in Robot Framework's time format
132 # (e.g. "5 seconds").
133
134 Wait Until Keyword Succeeds ${timeout} ${interval} Ping Host ${host}
135
136
137Ping Host
138 [Documentation] Ping the given host.
139 [Arguments] ${host}
140
141 # Description of argument(s):
142 # host The host name or IP of the host to ping.
143
144 Should Not Be Empty ${host} msg=No host provided
145 ${RC} ${output}= Run and return RC and Output ping -c 4 ${host}
146 Log RC: ${RC}\nOutput:\n${output}
147 Should be equal ${RC} ${0}
148
149
150Check OS
151 [Documentation] Attempts to ping the host OS and then checks that the host
152 ... OS is up by running an SSH command.
153
154 [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME}
155 ... ${os_password}=${OS_PASSWORD} ${quiet}=${QUIET}
156 ... ${print_string}=${EMPTY}
157 [Teardown] SSHLibrary.Close Connection
158
159 # Description of argument(s):
160 # os_host The DNS name/IP of the OS host associated with our BMC.
161 # os_username The username to be used to sign on to the OS host.
162 # os_password The password to be used to sign on to the OS host.
163 # quiet Indicates whether this keyword should write to console.
164 # print_string A string to be printed before checking the OS.
165
Michael Walshc108e422019-03-28 12:27:18 -0500166 Log To Console ${print_string} no_newline=True
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600167
168 # Attempt to ping the OS. Store the return code to check later.
169 ${ping_rc}= Run Keyword and Return Status Ping Host ${os_host}
170
171 SSHLibrary.Open connection ${os_host}
172
George Keishing30751662019-03-05 06:35:55 -0600173 ${status} ${msg}= Run Keyword And Ignore Error SSHLibrary.Login ${os_username}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600174 ... ${os_password}
175 ${err_msg1}= Sprint Error ${msg}
176 ${err_msg}= Catenate SEPARATOR= \n ${err_msg1}
177 Run Keyword If '${status}' == 'FAIL' Fail msg=${err_msg}
178 ${output} ${stderr} ${rc}= Execute Command uptime return_stderr=True
179 ... return_rc=True
180
181 ${temp_msg}= Catenate Could not execute a command on the operating
182 ... system.\n
183 ${err_msg1}= Sprint Error ${temp_msg}
184 ${err_msg}= Catenate SEPARATOR= \n ${err_msg1}
185
186 # If the return code returned by "Execute Command" is non-zero, this
187 # keyword will fail.
188 Should Be Equal ${rc} ${0} msg=${err_msg}
189 # We will likewise fail if there is any stderr data.
190 Should Be Empty ${stderr}
191
192 ${temp_msg}= Set Variable Could not ping the operating system.\n
193 ${err_msg1}= Sprint Error ${temp_msg}
194 ${err_msg}= Catenate SEPARATOR= \n ${err_msg1}
195 # We will likewise fail if the OS did not ping, as we could SSH but not
196 # ping
197 Should Be Equal As Strings ${ping_rc} ${TRUE} msg=${err_msg}
198
199
200Wait for OS
201 [Documentation] Waits for the host OS to come up via calls to "Check OS".
202 [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME}
203 ... ${os_password}=${OS_PASSWORD} ${timeout}=${OS_WAIT_TIMEOUT}
204 ... ${quiet}=${0}
Michael Walshc108e422019-03-28 12:27:18 -0500205 [Teardown] Printn
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600206
207 # Description of argument(s):
208 # os_host The DNS name or IP of the OS host associated with our
209 # BMC.
210 # os_username The username to be used to sign on to the OS host.
211 # os_password The password to be used to sign on to the OS host.
212 # timeout The timeout in seconds indicating how long you're
213 # willing to wait for the OS to respond.
214 # quiet Indicates whether this keyword should write to console.
215
216 # The interval to be used between calls to "Check OS".
217 ${interval}= Set Variable 5
218
219 ${message}= Catenate Checking every ${interval} seconds for up to
220 ... ${timeout} seconds for the operating system to communicate.
Michael Walshc108e422019-03-28 12:27:18 -0500221 Qprint Timen ${message}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600222
223 Wait Until Keyword Succeeds ${timeout} sec ${interval} Check OS
224 ... ${os_host} ${os_username} ${os_password}
225 ... print_string=\#
226
Michael Walshc108e422019-03-28 12:27:18 -0500227 Qprintn
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600228
Michael Walshc108e422019-03-28 12:27:18 -0500229 Qprint Timen The operating system is now communicating.
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600230
231
232Copy PNOR to BMC
233 [Documentation] Copy the PNOR image to the BMC.
234 Import Library SCPLibrary WITH NAME scp
235 Open Connection for SCP
236 Log Copying ${PNOR_IMAGE_PATH} to /tmp
237 scp.Put File ${PNOR_IMAGE_PATH} /tmp
238
239
240Is OS Starting
241 [Documentation] Check if boot progress is OS starting.
242 ${boot_progress}= Get Boot Progress
243 Should Be Equal ${boot_progress} OSStart
244
245
246Is OS Off
247 [Documentation] Check if boot progress is "Off".
248 ${boot_progress}= Get Boot Progress
249 Should Be Equal ${boot_progress} Off
250
251
252Get Boot Progress To OS Starting State
253 [Documentation] Get the system to a boot progress state of 'FW Progress,
254 ... Starting OS'.
255
256 ${boot_progress}= Get Boot Progress
257 Run Keyword If '${boot_progress}' == 'OSStart'
258 ... Log Host is already in OS starting state
259 ... ELSE
260 ... Run Keywords Initiate Host PowerOff AND Initiate Host Boot
261 ... AND Wait Until Keyword Succeeds 10 min 10 sec Is OS Starting
262
263
264Check If warmReset is Initiated
265 [Documentation] Ping would be still alive, so try SSH to connect
266 ... if fails the ports are down indicating reboot
267 ... is in progress
268
269 # Warm reset adds 3 seconds delay before forcing reboot
270 # To minimize race conditions, we wait for 7 seconds
271 Sleep 7s
272 ${alive}= Run Keyword and Return Status
273 ... Open Connection And Log In
274 Return From Keyword If '${alive}' == '${False}' ${False}
275 [Return] ${True}
276
277
278Initialize DBUS cmd
279 [Documentation] Initialize dbus string with property string to extract
280 [Arguments] ${boot_property}
281
282 # Description of argument(s):
283 # boot_property Property string.
284
285 ${cmd}= Catenate ${dbuscmdBase} ${dbuscmdGet} ${dbuscmdString}
286 ${cmd}= Catenate ${cmd}${boot_property}
287 Set Global Variable ${dbuscmd} ${cmd}
288
289
290Create OS Console Command String
291 [Documentation] Return a command string to start OS console logging.
292
293 # First make sure that the ssh_pw program is available.
294 ${cmd}= Catenate which ssh_pw 2>/dev/null || find
295 ... ${EXECDIR} -name 'ssh_pw'
296
Michael Walshc108e422019-03-28 12:27:18 -0500297 Dprint Issuing ${cmd}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600298 ${rc} ${output}= Run And Return Rc And Output ${cmd}
299 Rdpvars rc output
300
301 Should Be Equal As Integers 0 ${rc} msg=Could not find ssh_pw.
302
303 ${ssh_pw_file_path}= Set Variable ${output}
304
George Keishing088166c2019-12-13 08:11:28 -0600305 ${cmd}= Catenate ${ssh_pw_file_path} ${OPENBMC_PASSWORD} -p ${HOST_SOL_PORT}
Jorge Cisnerosf196b242022-09-09 15:46:15 +0000306 ... -o "StrictHostKeyChecking no" ${OPENBMC_USERNAME}@${OPENBMC_HOST} ${OPENBMC_CONSOLE_CLIENT}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600307
Jorge Cisnerosf196b242022-09-09 15:46:15 +0000308 [Return] ${cmd.strip()}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600309
310
311Get SOL Console Pid
312 [Documentation] Get the pid of the active SOL console job.
Michael Walsh07ed7942019-11-08 14:41:30 -0600313 [Arguments] ${expect_running}=${0} ${log_file_path}=${EMPTY}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600314
315 # Description of argument(s):
Michael Walsh07ed7942019-11-08 14:41:30 -0600316 # expect_running If set and if no SOL console job is found, print debug info and fail.
317 # log_file_path Needed to print debug info if expect_running is set and no pid is found.
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600318
319 # Find the pid of the active system console logging session (if any).
320 ${search_string}= Create OS Console Command String
321 # At least in some cases, ps output does not show double quotes so we must
322 # replace them in our search string with the regexes to indicate that they
323 # are optional.
324 ${search_string}= Replace String ${search_string} " ["]?
325 ${ps_cmd}= Catenate ps axwwo user,pid,cmd
326 ${cmd_buf}= Catenate echo $(${ps_cmd} | egrep '${search_string}' |
327 ... egrep -v grep | cut -c10-14)
Michael Walshc108e422019-03-28 12:27:18 -0500328 Dprint Issuing ${cmd_buf}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600329 ${rc} ${os_con_pid}= Run And Return Rc And Output ${cmd_buf}
330 Rdpvars os_con_pid
331 # If rc is not zero it just means that there is no OS Console process
332 # running.
333
334 Return From Keyword If '${os_con_pid}' != '${EMPTY}' ${os_con_pid}
335 Return From Keyword If '${expect_running}' == '${0}' ${os_con_pid}
336
337 Cmd Fnc cat ${log_file_path} ; echo ; ${ps_cmd} quiet=${0}
338 ... print_output=${1} show_err=${1}
Michael Walsh07ed7942019-11-08 14:41:30 -0600339 Valid Value os_con_pid
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600340
341
342Stop SOL Console Logging
343 [Documentation] Stop system console logging and return log output.
344 [Arguments] ${log_file_path}=${EMPTY}
345 ... ${targ_file_path}=${EXECDIR}${/}logs${/}
346 ... ${return_data}=${1}
347
George Keishinge16f1582022-12-15 07:32:21 -0600348 # If there are multiple system console processes, they will all be stopped.
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600349 # If there is no existing log file this keyword will return an error
350 # message to that effect (and write that message to targ_file_path, if
351 # specified).
352 # NOTE: This keyword will not fail if there is no running system console
353 # process.
354
355 # Description of arguments:
356 # log_file_path The file path that was used to call "Start SOL
357 # Console Logging". See that keyword (above) for details.
358 # targ_file_path If specified, the file path to which the source
359 # file path (i.e. "log_file_path") should be copied.
360 # return_data If this is set to ${1}, this keyword will return the SOL
361 # data to the caller as a unicode string.
362
363 ${log_file_path}= Create OS Console File Path ${log_file_path}
364
365 ${os_con_pid}= Get SOL Console Pid
366
367 ${cmd_buf}= Catenate kill -9 ${os_con_pid}
Michael Walshc108e422019-03-28 12:27:18 -0500368 Run Keyword If '${os_con_pid}' != '${EMPTY}' Dprint Issuing ${cmd_buf}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600369 ${rc} ${output}= Run Keyword If '${os_con_pid}' != '${EMPTY}'
370 ... Run And Return Rc And Output ${cmd_buf}
371 Run Keyword If '${os_con_pid}' != '${EMPTY}' Rdpvars rc output
372
373 Run Keyword If '${targ_file_path}' != '${EMPTY}'
374 ... Run Keyword And Ignore Error
375 ... Copy File ${log_file_path} ${targ_file_path}
376
377 ${output}= Set Variable ${EMPTY}
378 ${loc_quiet}= Evaluate ${debug}^1
379 ${rc} ${output}= Run Keyword If '${return_data}' == '${1}'
380 ... Cmd Fnc cat ${log_file_path} 2>/dev/null quiet=${loc_quiet}
381 ... print_output=${0} show_err=${0}
382
383 [Return] ${output}
384
385
386Start SOL Console Logging
387 [Documentation] Start system console log to file.
388 [Arguments] ${log_file_path}=${EMPTY} ${return_data}=${1}
389
390 # This keyword will first call "Stop SOL Console Logging". Only then will
391 # it start SOL console logging. The data returned by "Stop SOL Console
392 # Logging" will in turn be returned by this keyword.
393
394 # Description of arguments:
395 # log_file_path The file path to which system console log data should be
396 # written. Note that this path is taken to be a location
397 # on the machine where this program is running rather than
398 # on the Open BMC system.
399 # return_data If this is set to ${1}, this keyword will return any SOL
400 # data to the caller as a unicode string.
401
402 ${log_file_path}= Create OS Console File Path ${log_file_path}
403
404 ${log_output}= Stop SOL Console Logging ${log_file_path}
405 ... return_data=${return_data}
406
407 # Validate by making sure we can create the file. Problems creating the
408 # file would not be noticed by the subsequent ssh command because we fork
409 # the command.
410 Create File ${log_file_path}
411 ${sub_cmd_buf}= Create OS Console Command String
412 # Routing stderr to stdout so that any startup error text will go to the
413 # output file.
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600414 ${cmd_buf}= Catenate ${sub_cmd_buf} > ${log_file_path} 2>&1 &
Michael Walshc108e422019-03-28 12:27:18 -0500415 Dprint Issuing ${cmd_buf}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600416 ${rc} ${output}= Run And Return Rc And Output ${cmd_buf}
417 # Because we are forking this command, we essentially will never get a
418 # non-zero return code or any output.
419 Should Be Equal ${rc} ${0}
420
421 Wait Until Keyword Succeeds 10 seconds 0 seconds
Michael Walsh07ed7942019-11-08 14:41:30 -0600422 ... Get SOL Console Pid ${1} ${log_file_path}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600423
424 [Return] ${log_output}
425
426
427Get Time Stamp
428 [Documentation] Get the current time stamp data
429 ${cur_time}= Get Current Date result_format=%Y%m%d%H%M%S%f
430 [Return] ${cur_time}
431
432
433Start Journal Log
434 [Documentation] Start capturing journal log to a file in /tmp using
435 ... journalctl command. By default journal log is collected
436 ... at /tmp/journal_log else user input location.
437 ... The File is appended with datetime.
438 [Arguments] ${file_path}=/tmp/journal_log ${filter}=${EMPTY}
439
440 # Description of arguments:
441 # file_path The file path of the journal file.
442
443 ${cur_time}= Get Time Stamp
444 Set Global Variable ${LOG_TIME} ${cur_time}
445 Open Connection And Log In
446 Start Command
447 ... journalctl -f ${filter} > ${file_path}-${LOG_TIME}
448 Log Journal Log Started: ${file_path}-${LOG_TIME}
449
450
451Stop Journal Log
452 [Documentation] Stop journalctl process if its running.
453 ... By default return log from /tmp/journal_log else
454 ... user input location.
455 [Arguments] ${file_path}=/tmp/journal_log
456
457 # Description of arguments:
458 # file_path The file path of the journal file.
459
460 Open Connection And Log In
461
462 ${rc}=
463 ... Execute Command
464 ... ps | grep journalctl | grep -v grep
465 ... return_stdout=False return_rc=True
466
467 Return From Keyword If '${rc}' == '${1}'
468 ... No journal log process running
469
470 ${output} ${stderr}=
471 ... Execute Command killall journalctl
472 ... return_stderr=True
473 Should Be Empty ${stderr}
474
475 ${journal_log} ${stderr}=
476 ... Execute Command
477 ... cat ${file_path}-${LOG_TIME}
478 ... return_stderr=True
479 Should Be Empty ${stderr}
480
481 Log ${journal_log}
482
483 Execute Command rm ${file_path}-${LOG_TIME}
484
485 [Return] ${journal_log}
486
487
488Mac Address To Hex String
489 [Documentation] Converts MAC address into hex format.
490 ... Example
491 ... Given the following MAC: 00:01:6C:80:02:78
492 ... This keyword will return: 0x00 0x01 0x6C 0x80 0x02 0x78
493 ... Description of arguments:
494 ... i_macaddress MAC address in the following format
495 ... 00:01:6C:80:02:78
496 [Arguments] ${i_macaddress}
497
498 # Description of arguments:
499 # i_macaddress The MAC address.
500
501 ${mac_hex}= Catenate 0x${i_macaddress.replace(':', ' 0x')}
502 [Return] ${mac_hex}
503
504
505IP Address To Hex String
506 [Documentation] Converts IP address into hex format.
507 ... Example:
508 ... Given the following IP: 10.3.164.100
509 ... This keyword will return: 0xa 0x3 0xa4 0xa0
510 [Arguments] ${i_ipaddress}
511
512 # Description of arguments:
513 # i_macaddress The IP address in the format 10.10.10.10.
514
515 @{ip}= Split String ${i_ipaddress} .
516 ${index}= Set Variable ${0}
517
Marissa Garza20ccfc72020-06-19 12:51:10 -0500518 FOR ${item} IN @{ip}
519 ${hex}= Convert To Hex ${item} prefix=0x lowercase=yes
520 Set List Value ${ip} ${index} ${hex}
521 ${index}= Set Variable ${index + 1}
522 END
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600523 ${ip_hex}= Catenate @{ip}
524
525 [Return] ${ip_hex}
526
527
528BMC CPU Performance Check
529 [Documentation] Minimal 10% of proc should be free in this instance
530
531 ${bmc_cpu_usage_output} ${stderr} ${rc}= BMC Execute Command
532 ... ${bmc_cpu_usage_cmd}
533 ${bmc_cpu_usage_output} ${stderr} ${rc}= BMC Execute Command
534 ... ${bmc_cpu_usage_cmd}
535 ${bmc_cpu_percentage}= Fetch From Left ${bmc_cpu_usage_output} %
Sivas SRR14426bc2019-04-10 10:14:46 -0500536 Rprint Vars bmc_cpu_percentage
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600537 Should be true ${bmc_cpu_percentage} < 90
538
539
540BMC Mem Performance Check
541 [Documentation] Minimal 10% of memory should be free in this instance
542
543 ${bmc_mem_free_output} ${stderr} ${rc}= BMC Execute Command
544 ... ${bmc_mem_free_cmd}
545
546 ${bmc_mem_total_output} ${stderr} ${rc}= BMC Execute Command
547 ... ${bmc_mem_total_cmd}
548 ${bmc_mem_free_output} ${stderr} ${rc}= BMC Execute Command
549 ... ${bmc_mem_free_cmd}
550
551 ${bmc_mem_total_output} ${stderr} ${rc}= BMC Execute Command
552 ... ${bmc_mem_total_cmd}
553
554 ${bmc_mem_percentage}= Evaluate ${bmc_mem_free_output}*100
555 ${bmc_mem_percentage}= Evaluate
556 ... ${bmc_mem_percentage}/${bmc_mem_total_output}
Sivas SRR14426bc2019-04-10 10:14:46 -0500557 Rprint Vars bmc_mem_percentage
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600558 Should be true ${bmc_mem_percentage} > 10
559
560
561BMC File System Usage Check
562 [Documentation] Check the file system space. 4 file system should be
563 ... 100% full which is expected
564 # Filesystem Size Used Available Use% Mounted on
565 # /dev/root 14.4M 14.4M 0 100% /
566 # /dev/ubiblock0_0 14.4M 14.4M 0 100% /media/rofs-c9249b0e
567 # /dev/ubiblock8_0 19.6M 19.6M 0 100% /media/pnor-ro-8764baa3
568 # /dev/ubiblock4_0 14.4M 14.4M 0 100% /media/rofs-407816c
569 # /dev/ubiblock8_4 21.1M 21.1M 0 100% /media/pnor-ro-cecc64c4
570 ${bmc_fs_usage_output} ${stderr} ${rc}= BMC Execute Command
571 ... ${bmc_file_system_usage_cmd}
572 ${bmc_pnor_fs_usage_output} ${stderr} ${rc}= BMC Execute Command
573 ... ${total_pnor_ro_file_system_cmd}
574 ${bmc_bmc_fs_usage_output} ${stderr} ${rc}= BMC Execute Command
575 ... ${total_bmc_ro_file_system_cmd}
576 ${total_bmc_pnor_image}= Evaluate
577 ... ${bmc_pnor_fs_usage_output}+${bmc_bmc_fs_usage_output}
578 # Considering /dev/root also in total 100% used file system
579 ${total_full_fs}= Evaluate ${total_bmc_pnor_image}+1
Sivas SRR14426bc2019-04-10 10:14:46 -0500580 Rprint Vars bmc_fs_usage_output bmc_pnor_fs_usage_output bmc_bmc_fs_usage_output
581 ... total_full_fs
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600582 Should Be True ${bmc_fs_usage_output}==${total_full_fs}
583
584
585Check BMC CPU Performance
586 [Documentation] Minimal 10% of proc should be free in 3 sample
Marissa Garza20ccfc72020-06-19 12:51:10 -0500587 FOR ${var} IN RANGE 1 4
588 BMC CPU Performance Check
589 END
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600590
591Check BMC Mem Performance
592 [Documentation] Minimal 10% of memory should be free
593
Marissa Garza20ccfc72020-06-19 12:51:10 -0500594 FOR ${var} IN RANGE 1 4
595 BMC Mem Performance Check
596 END
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600597
598Check BMC File System Performance
599 [Documentation] Check for file system usage for 4 times
600
Marissa Garza20ccfc72020-06-19 12:51:10 -0500601 FOR ${var} IN RANGE 1 4
602 BMC File System Usage check
603 END
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600604
605Get URL List
606 [Documentation] Return list of URLs under given URL.
607 [Arguments] ${openbmc_url}
608
609 # Description of argument(s):
610 # openbmc_url URL for list operation (e.g.
611 # /xyz/openbmc_project/inventory).
612
Steven Sombaraaaab222018-12-19 13:16:23 -0600613 ${url_list}= Read Properties ${openbmc_url}list quiet=${1}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600614 Sort List ${url_list}
615
616 [Return] ${url_list}
617
618
619Check Zombie Process
620 [Documentation] Check if any defunct process exist or not on BMC
621 ${count} ${stderr} ${rc}= Execute Command ps -o stat | grep Z | wc -l
622 ... return_stderr=True return_rc=True
623 Should Be True ${count}==0
624 Should Be Empty ${stderr}
625
626
627Prune Journal Log
628 [Documentation] Prune archived journal logs.
629 [Arguments] ${vacuum_size}=1M
630
631 # This keyword can be used to prevent the journal
632 # log from filling up the /run filesystem.
633 # This command will retain only the latest logs
634 # of the user specified size.
635
636 # Description of argument(s):
637 # vacuum_size Size of journal.
638
639 Open Connection And Log In
640 ${output} ${stderr} ${rc}=
641 ... Execute Command
642 ... journalctl --vacuum-size=${vacuum_size}
643 ... return_stderr=True return_rc=True
644
645 Should Be Equal ${rc} ${0} msg=${stderr}
646
647
648Get System Power Policy
649 [Documentation] Returns the BMC power policy.
650
651 # Set the bmc_power_policy_method to either 'Old' or 'New'.
652 Set Power Policy Method
653 ${cmd_buf}= Create List ${bmc_power_policy_method} Get Power Policy
654 # Run the appropriate keyword.
655 ${currentPolicy}= Run Keyword @{cmd_buf}
656
657 [Return] ${currentPolicy}
658
659
660Set BMC Reset Reference Time
661 [Documentation] Set current boot time as a reference and increment
662 ... boot count.
663
664 ${cur_btime}= Get BMC Boot Time
665 Run Keyword If ${BOOT_TIME} == ${0} and ${BOOT_COUNT} == ${0}
666 ... Set Global Variable ${BOOT_TIME} ${cur_btime}
667 ... ELSE IF ${cur_btime} > ${BOOT_TIME}
668 ... Run Keywords Set Global Variable ${BOOT_TIME} ${cur_btime}
669 ... AND
670 ... Set Global Variable ${BOOT_COUNT} ${BOOT_COUNT + 1}
671
672
673Get BMC Boot Time
674 [Documentation] Returns boot time from /proc/stat.
675
676 Open Connection And Log In
677 ${output} ${stderr}=
678 ... Execute Command egrep '^btime ' /proc/stat | cut -f 2 -d ' '
679 ... return_stderr=True
680 Should Be Empty ${stderr}
681 ${btime}= Convert To Integer ${output}
682 [Return] ${btime}
683
684
685Enable Core Dump On BMC
686 [Documentation] Enable core dump collection.
687 ${core_pattern} ${stderr} ${rc}= BMC Execute Command
688 ... echo '/tmp/core_%e.%p' | tee /proc/sys/kernel/core_pattern
689 Should Be Equal As Strings ${core_pattern} /tmp/core_%e.%p
690
691
692Get Number Of BMC Core Dump Files
693 [Documentation] Returns number of core dump files on BMC.
694 Open Connection And Log In
695 ${num_of_core_dump}= Execute Command
696 ... ls /tmp/core* 2>/dev/null | wc -l
697 [Return] ${num_of_core_dump}
698
699
700Set Core Dump File Size Unlimited
701 [Documentation] Set core dump file size to unlimited.
702 BMC Execute Command ulimit -c unlimited
703
704
705Check For Core Dumps
706 [Documentation] Check for any core dumps exist.
707 ${output}= Get Number Of BMC Core Dump Files
708 Run Keyword If ${output} > 0
709 ... Log **Warning** BMC core dump files exist level=WARN
710
711
712Configure Initial Settings
713 [Documentation] Restore old IP and route.
714 ... This keyword requires initial settings viz IP address,
George Keishing16b3c7b2021-01-28 09:23:37 -0600715 ... Network Mask, default gateway and serial console IP and port
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600716 ... information which should be provided in command line.
717
718 [Arguments] ${host}=${OPENBMC_HOST} ${mask}=${NET_MASK}
719 ... ${gw_ip}=${GW_IP}
720
721 # Description of arguments:
722 # host IP address of the OS Host.
723 # mask Network mask.
724 # gu_ip Gateway IP address or hostname.
725
726 # Open telnet connection and ignore the error, in case telnet session is
727 # already opened by the program calling this keyword.
728 Run Keyword And Ignore Error Open Telnet Connection to BMC Serial Console
729 Telnet.write ifconfig eth0 ${host} netmask ${mask}
730 Telnet.write route add default gw ${gw_ip}
731
732
733Install Debug Tarball On BMC
734 [Documentation] Copy the debug tar file to BMC and install.
735 [Arguments] ${tarball_file_path}=${default_tarball}
736 ... ${targ_tarball_dir_path}=/tmp/tarball/
737
738 # Description of arguments:
739 # tarball_file_path Path of the debug tarball file.
740 # The tar file is downloaded from the build page
George Keishinged616122020-07-02 00:57:15 -0500741 # https://jenkins.openbmc.org/job/latest-master/
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600742 # obmc-phosphor-debug-tarball-witherspoon.tar.xz
743 #
744 # targ_tarball_dir_path The directory path where the tarball is to be
745 # installed.
746
747 OperatingSystem.File Should Exist ${tarball_file_path}
748 ... msg=${tarball_file_path} doesn't exist.
749
750 # Upload the file to BMC.
751 Import Library SCPLibrary WITH NAME scp
752 Open Connection for SCP
753 scp.Put File ${tarball_file_path} /tmp/debug-tarball.tar.xz
754
755 # Create tarball directory and install.
756 BMC Execute Command mkdir -p ${targ_tarball_dir_path}
757 BMC Execute Command
758 ... tar -xf /tmp/debug-tarball.tar.xz -C ${targ_tarball_dir_path}
759
760 # Remove the tarball file from BMC.
761 BMC Execute Command rm -f /tmp/debug-tarball.tar.xz
762
763
764Get BMC Boot Count
765 [Documentation] Returns BMC boot count based on boot time.
766 ${cur_btime}= Get BMC Boot Time
767
768 # Set global variable BOOT_TIME to current boot time if current boot time
769 # is changed. Also increase value of global variable BOOT_COUNT by 1.
770 Run Keyword If ${cur_btime} > ${BOOT_TIME}
771 ... Run Keywords Set Global Variable ${BOOT_TIME} ${cur_btime}
772 ... AND
773 ... Set Global Variable ${BOOT_COUNT} ${BOOT_COUNT + 1}
774
775 [Return] ${BOOT_COUNT}
776
777
778Set BMC Boot Count
779 [Documentation] Set BMC boot count to given value.
780 [Arguments] ${count}
781
782 # Description of arguments:
783 # count boot count value.
784 ${cur_btime}= Get BMC Boot Time
785
786 # Set global variable BOOT_COUNT to given value.
787 Set Global Variable ${BOOT_COUNT} ${count}
788
789 # Set BOOT_TIME variable to current boot time.
Tony Lee073fbc02020-05-12 15:12:16 +0800790 Set Global Variable ${BOOT_TIME} ${cur_btime}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600791
792
793Delete Error Log Entry
794 [Documentation] Delete error log entry.
Michael Sheposcc490b42020-08-26 12:53:01 -0500795 [Arguments] ${entry_path} ${quiet}=${0}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600796
797 # Description of argument(s):
Michael Sheposcc490b42020-08-26 12:53:01 -0500798 # quiet If enabled, turns off logging to console.
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600799 # entry_path Delete an error log entry.
800 # Ex. /xyz/openbmc_project/logging/entry/1
801
802 # Skip delete if entry URI is a callout.
803 # Examples:
804 # /xyz/openbmc_project/logging/entry/1/callout
805 # /xyz/openbmc_project/logging/entry/1/callouts/0
806 ${callout_entry}= Run Keyword And Return Status
807 ... Should Match Regexp ${entry_path} /callout[s]?(/|$)
808 Return From Keyword If ${callout_entry}
809
810 ${data}= Create Dictionary data=@{EMPTY}
Michael Sheposcc490b42020-08-26 12:53:01 -0500811 ${resp}= Openbmc Delete Request ${entry_path} data=${data} quiet=${quiet}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600812 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
813
814
815Get BMC Version
816 [Documentation] Returns BMC version from /etc/os-release.
817 ... e.g. "v1.99.6-141-ge662190"
818
819 ${cmd}= Set Variable grep ^VERSION_ID= /etc/os-release | cut -f 2 -d '='
820 ${output} ${stderr} ${rc}= BMC Execute Command ${cmd}
821 [Return] ${output}
822
823
824Get PNOR Version
825 [Documentation] Returns the PNOR version from the BMC.
826
827 ${pnor_attrs}= Get PNOR Attributes
828 [Return] ${pnor_attrs['version']}
829
830
831Get PNOR Attributes
832 [Documentation] Return PNOR software attributes as a dictionary.
833
834 # This keyword parses /var/lib/phosphor-software-manager/pnor/ro/pnor.toc
835 # into key/value pairs.
836
837 ${outbuf} ${stderr} ${rc}= BMC Execute Command
838 ... cat /var/lib/phosphor-software-manager/pnor/ro/pnor.toc
839 ${pnor_attrs}= Key Value Outbuf To Dict ${outbuf} delim==
840
841 [Return] ${pnor_attrs}
842
843
Sivas SRR17dae1a2019-03-24 23:52:02 -0500844GET BMC PNOR Version
845 [Documentation] Return BMC & PNOR version from openbmc shell.
846
847 ${bmc_version}= GET BMC Version
848 ${pnor_version}= GET PNOR Version
849 Log ${bmc_version}
850 Rprint Vars bmc_version
851 Log ${pnor_version}
852 Rprint Vars pnor_version
853
854 [Return] ${bmc_version} ${pnor_version}
855
856
Sushil Singh5ea86d02019-07-11 02:05:16 -0500857Redfish Get BMC Version
858 [Documentation] Get BMC version via Redfish.
859
860 ${output}= Redfish.Get Attribute ${REDFISH_BASE_URI}Managers/bmc FirmwareVersion
861 [Return] ${output}
862
863Redfish Get Host Version
864 [Documentation] Get host version via Redfish.
865
866 ${output}= Redfish.Get Attribute ${REDFISH_BASE_URI}Systems/system BiosVersion
867 [Return] ${output}
868
869
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600870Copy Address Translation Utils To HOST OS
871 [Documentation] Copy address translation utils to host OS.
872
873 OperatingSystem.File Should Exist ${probe_cpu_tool_path}
874 ... msg=${probe_cpu_tool_path} doesn't exist.
875 OperatingSystem.File Should Exist ${probe_cpu_tool_path}
876 ... msg=${probe_cpu_tool_path} doesn't exist.
877
878 scp.Open connection ${OS_HOST} username=${OS_USERNAME}
879 ... password=${OS_PASSWORD}
880 scp.Put File ${probe_cpu_tool_path} ${target_file_path}
881 scp.Put File ${scom_addrs_tool_path} ${target_file_path}
882
883
884Verify BMC RTC And UTC Time Drift
885 [Documentation] Verify that the RTC and UTC time difference is less than
886 ... the given time_drift_max.
887 [Arguments] ${time_diff_max}=${10}
888
889 # Description of argument(s):
890 # time_diff_max The max allowable RTC and UTC time difference in seconds.
891
892 # Example:
893 # time_dict:
894 # [local_time]: Fri 2017-11-03 152756 UTC
895 # [local_time_seconds]: 1509740876
896 # [universal_time]: Fri 2017-11-03 152756 UTC
897 # [universal_time_seconds]: 1509740876
898 # [rtc_time]: Fri 2016-05-20 163403
899 # [rtc_time_seconds]: 1463780043
900 # [time_zone]: n/a (UTC, +0000)
901 # [network_time_on]: yes
902 # [ntp_synchronized]: no
903 # [rtc_in_local_tz]: no
904
905 ${time}= Get BMC Date Time
906 ${time_diff}= Evaluate
907 ... ${time['universal_time_seconds']} - ${time['rtc_time_seconds']}
908 Should Be True ${time_diff} < ${time_diff_max}
909
910
911Validate IP On BMC
912 [Documentation] Validate IP address is present in set of IP addresses.
913 [Arguments] ${ip_address} ${ip_data}
914
915 # Description of argument(s):
916 # ip_address IP address to check (e.g. xx.xx.xx.xx).
917 # ip_data Set of the IP addresses present.
918
919 Should Contain Match ${ip_data} ${ip_address}/*
920 ... msg=${ip_address} not found in the list provided.
921
922
923Remove Journald Logs
924 [Documentation] Remove all journald logs and restart service.
925
926 ${cmd}= Catenate systemctl stop systemd-journald.service &&
927 ... rm -rf /var/log/journal && systemctl start systemd-journald.service
928
929 BMC Execute Command ${cmd}
930
931
932Check For Regex In Journald
933 [Documentation] Parse the journal log and check for regex string.
934 [Arguments] ${regex}=${ERROR_REGEX} ${error_check}=${0} ${boot}=${EMPTY}
George Keishing140351f2022-03-10 08:10:01 -0600935 ... ${filter_string}=${EMPTY}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600936
937 # Description of argument(s):
938 # regex Strings to be filter.
939 # error_check Check for errors.
940 # boot Argument to check current or persistent full boot log
941 # (e.g. "-b").
George Keishing140351f2022-03-10 08:10:01 -0600942 # filter_string String to be stripped out.
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600943
George Keishing140351f2022-03-10 08:10:01 -0600944
945 ${cmd} = Run Keyword If '${filter_string}' == '${EMPTY}'
946 ... Catenate journalctl --no-pager ${boot} | egrep '${regex}'
947 ... ELSE
948 ... Catenate journalctl --no-pager ${boot} | egrep '${regex}' | sed '/${filter_string}/d'
949
950 ${journal_log} ${stderr} ${rc}= BMC Execute Command ${cmd} ignore_err=1
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600951
952 Run Keyword If ${error_check} == ${0}
953 ... Should Be Empty ${journal_log}
954 ... ELSE
955 ... Should Not Be Empty ${journal_log}
956
957
958Get Service Attribute
959 [Documentation] Get service attribute policy output.
960 [Arguments] ${option} ${servicename}
961
962 # Description of argument(s):
963 # option systemctl supported options
964 # servicename Qualified service name
965 ${cmd}= Set Variable
966 ... systemctl -p ${option} show ${servicename} | cut -d = -f2
967 ${attr} ${stderr} ${rc}= BMC Execute Command ${cmd}
968 [Return] ${attr}
Anusha Dathatrieda09492019-02-26 08:30:13 -0600969
970
971Verify Watchdog Enabled
972 [Documentation] Check that watchdog settings are enabled.
973
974 ${properties}= Read Properties /xyz/openbmc_project/watchdog/host0
975 Should Be Equal As Strings ${properties["Enabled"]} ${True}
976 Should Not Be Equal As Strings ${properties["TimeRemaining"]} 0
Ashwini Chandrappa233d9652021-08-04 09:56:30 -0500977
978
979Is BMC Unpingable
980 [Documentation] Check if BMC is unpingable.
981
982 ${RC} ${output}= Run and return RC and Output ping -c 4 ${OPENBMC_HOST}
983 Log RC: ${RC}\nOutput:\n${output}
984 Should be equal ${RC} ${1}
Sushil Singh6d170ad2022-11-28 07:04:50 -0600985
986Kernel Panic BMC Reset Operation
987 [Documentation] Create kernel panic to reset BMC.
988
989 ${kernel_panic_cmd_file}= Set Variable /bin/sh -c "echo c > /proc/sysrq-trigger"
990 ${status}= Run Keyword And Return Status BMC Execute Command ${kernel_panic_cmd_file} time_out=120
991
992 [Return] ${status}
993