blob: a9fa72e34a6bb8c17dcc52bc96549164cf4abeca [file] [log] [blame]
Steven Sombarf60cbcf2018-12-07 08:12:18 -06001*** Settings ***
2
3Documentation Utilities for Robot keywords that do not use REST.
4
Sandhya Somashekar839a0c22019-01-31 05:05:43 -06005Resource ../lib/resource.robot
Steven Sombarf60cbcf2018-12-07 08:12:18 -06006Resource ../lib/connection_client.robot
7Resource ../lib/boot_utils.robot
8Library String
9Library DateTime
10Library Process
11Library OperatingSystem
12Library gen_print.py
13Library gen_robot_print.py
14Library gen_cmd.py
Michael Walsh07ed7942019-11-08 14:41:30 -060015Library gen_robot_valid.py
Steven Sombarf60cbcf2018-12-07 08:12:18 -060016Library gen_robot_keyword.py
17Library bmc_ssh_utils.py
18Library utils.py
19Library var_funcs.py
20Library SCPLibrary WITH NAME scp
21
22*** Variables ***
23
24${pflash_cmd} /usr/sbin/pflash -r /dev/stdout -P VERSION
25
26${dbuscmdBase}
27... dbus-send --system --print-reply --dest=${OPENBMC_BASE_DBUS}.settings.Host
28${dbuscmdGet}
29... ${SETTINGS_URI}host0 org.freedesktop.DBus.Properties.Get
30${dbuscmdString}= string:"xyz.openbmc_project.settings.Host" string:
31
32# Assign default value to QUIET for programs which may not define it.
33${QUIET} ${0}
34
35${bmc_mem_free_cmd}= free | tr -s ' ' | sed '/^Mem/!d' | cut -d" " -f4
36${bmc_mem_total_cmd}= free | tr -s ' ' | sed '/^Mem/!d' | cut -d" " -f2
37${bmc_cpu_usage_cmd}= top -n 1 | grep CPU: | cut -c 7-9
38${HOST_SETTING} ${SETTINGS_URI}host0
39
40# /run/initramfs/ro associate filesystem should be 100% full always
41${bmc_file_system_usage_cmd}= df -h | cut -c 52-54 | grep 100 | wc -l
42${total_pnor_ro_file_system_cmd}= df -h | grep /media/pnor-ro | wc -l
43${total_bmc_ro_file_system_cmd}= df -h | grep /media/rofs | wc -l
44
45${BOOT_TIME} ${0}
46${BOOT_COUNT} ${0}
47${count} ${0}
48${devicetree_base} /sys/firmware/devicetree/base/model
49
50# Initialize default debug value to 0.
51${DEBUG} ${0}
52
53${probe_cpu_tool_path} ${EXECDIR}/tools/ras/probe_cpus.sh
54${scom_addrs_tool_path} ${EXECDIR}/tools/ras/scom_addr_p9.sh
55${target_file_path} /root/
56
57${default_tarball} ${EXECDIR}/obmc-phosphor-debug-tarball-witherspoon.tar.xz
58
59# These variables are used to straddle between new and old methods of setting
60# values.
61${bmc_power_policy_method} ${EMPTY}
62@{valid_power_policy_vars} RESTORE_LAST_STATE ALWAYS_POWER_ON
63... ALWAYS_POWER_OFF
64
65
66*** Keywords ***
67
68Check BMC Performance
69 [Documentation] Check BMC basic CPU Mem File system performance.
70
71 Check BMC CPU Performance
72 Check BMC Mem Performance
73 Check BMC File System Performance
74
75
76Verify PNOR Update
77 [Documentation] Verify that the PNOR is not corrupted.
78 # Example:
79 # FFS: Flash header not found. Code: 100
80 # Error 100 opening ffs !
81
82 ${stdout} ${stderr} ${rc}=
83 ... BMC Execute Command /usr/sbin/pflash -h | egrep -q skip
84 ... ignore_err=${1}
85 ${pflash_cmd}= Set Variable If ${rc} == ${0} ${pflash_cmd} --skip=4096
86 ... ${pflash_cmd}
87 ${pnor_info}= BMC Execute Command ${pflash_cmd}
88 Should Not Contain Any ${pnor_info} Flash header not found Error
89
90
91Get BMC System Model
92 [Documentation] Get the BMC model from the device tree and return it.
93
94 ${bmc_model} ${stderr} ${rc}= BMC Execute Command
95 ... cat ${devicetree_base} | cut -d " " -f 1 return_stderr=True
96 ... test_mode=0
97 Should Be Empty ${stderr}
98 Should Not Be Empty ${bmc_model} msg=BMC model is empty.
99 [Return] ${bmc_model}
100
101
102Verify BMC System Model
103 [Documentation] Verify the BMC model with ${OPENBMC_MODEL}.
104 [Arguments] ${bmc_model}
105
106 # Description of argument(s):
107 # bmc_model System model (e.g. "witherspoon").
108
109 ${tmp_bmc_model}= Fetch From Right ${OPENBMC_MODEL} /
110 ${tmp_bmc_model}= Fetch From Left ${tmp_bmc_model} .
111 ${ret}= Run Keyword And Return Status Should Contain ${bmc_model}
112 ... ${tmp_bmc_model} ignore_case=True
113 [Return] ${ret}
114
115
116Wait For Host To Ping
117 [Documentation] Wait for the given host to ping.
118 [Arguments] ${host} ${timeout}=${OPENBMC_REBOOT_TIMEOUT}min
119 ... ${interval}=5 sec
120
121 # Description of argument(s):
122 # host The host name or IP of the host to ping.
123 # timeout The amount of time after which ping attempts cease.
124 # This should be expressed in Robot Framework's time format
125 # (e.g. "10 seconds").
126 # interval The amount of time in between attempts to ping.
127 # This should be expressed in Robot Framework's time format
128 # (e.g. "5 seconds").
129
130 Wait Until Keyword Succeeds ${timeout} ${interval} Ping Host ${host}
131
132
133Ping Host
134 [Documentation] Ping the given host.
135 [Arguments] ${host}
136
137 # Description of argument(s):
138 # host The host name or IP of the host to ping.
139
140 Should Not Be Empty ${host} msg=No host provided
141 ${RC} ${output}= Run and return RC and Output ping -c 4 ${host}
142 Log RC: ${RC}\nOutput:\n${output}
143 Should be equal ${RC} ${0}
144
145
146Check OS
147 [Documentation] Attempts to ping the host OS and then checks that the host
148 ... OS is up by running an SSH command.
149
150 [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME}
151 ... ${os_password}=${OS_PASSWORD} ${quiet}=${QUIET}
152 ... ${print_string}=${EMPTY}
153 [Teardown] SSHLibrary.Close Connection
154
155 # Description of argument(s):
156 # os_host The DNS name/IP of the OS host associated with our BMC.
157 # os_username The username to be used to sign on to the OS host.
158 # os_password The password to be used to sign on to the OS host.
159 # quiet Indicates whether this keyword should write to console.
160 # print_string A string to be printed before checking the OS.
161
Michael Walshc108e422019-03-28 12:27:18 -0500162 Log To Console ${print_string} no_newline=True
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600163
164 # Attempt to ping the OS. Store the return code to check later.
165 ${ping_rc}= Run Keyword and Return Status Ping Host ${os_host}
166
167 SSHLibrary.Open connection ${os_host}
168
George Keishing30751662019-03-05 06:35:55 -0600169 ${status} ${msg}= Run Keyword And Ignore Error SSHLibrary.Login ${os_username}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600170 ... ${os_password}
171 ${err_msg1}= Sprint Error ${msg}
172 ${err_msg}= Catenate SEPARATOR= \n ${err_msg1}
173 Run Keyword If '${status}' == 'FAIL' Fail msg=${err_msg}
174 ${output} ${stderr} ${rc}= Execute Command uptime return_stderr=True
175 ... return_rc=True
176
177 ${temp_msg}= Catenate Could not execute a command on the operating
178 ... system.\n
179 ${err_msg1}= Sprint Error ${temp_msg}
180 ${err_msg}= Catenate SEPARATOR= \n ${err_msg1}
181
182 # If the return code returned by "Execute Command" is non-zero, this
183 # keyword will fail.
184 Should Be Equal ${rc} ${0} msg=${err_msg}
185 # We will likewise fail if there is any stderr data.
186 Should Be Empty ${stderr}
187
188 ${temp_msg}= Set Variable Could not ping the operating system.\n
189 ${err_msg1}= Sprint Error ${temp_msg}
190 ${err_msg}= Catenate SEPARATOR= \n ${err_msg1}
191 # We will likewise fail if the OS did not ping, as we could SSH but not
192 # ping
193 Should Be Equal As Strings ${ping_rc} ${TRUE} msg=${err_msg}
194
195
196Wait for OS
197 [Documentation] Waits for the host OS to come up via calls to "Check OS".
198 [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME}
199 ... ${os_password}=${OS_PASSWORD} ${timeout}=${OS_WAIT_TIMEOUT}
200 ... ${quiet}=${0}
Michael Walshc108e422019-03-28 12:27:18 -0500201 [Teardown] Printn
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600202
203 # Description of argument(s):
204 # os_host The DNS name or IP of the OS host associated with our
205 # BMC.
206 # os_username The username to be used to sign on to the OS host.
207 # os_password The password to be used to sign on to the OS host.
208 # timeout The timeout in seconds indicating how long you're
209 # willing to wait for the OS to respond.
210 # quiet Indicates whether this keyword should write to console.
211
212 # The interval to be used between calls to "Check OS".
213 ${interval}= Set Variable 5
214
215 ${message}= Catenate Checking every ${interval} seconds for up to
216 ... ${timeout} seconds for the operating system to communicate.
Michael Walshc108e422019-03-28 12:27:18 -0500217 Qprint Timen ${message}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600218
219 Wait Until Keyword Succeeds ${timeout} sec ${interval} Check OS
220 ... ${os_host} ${os_username} ${os_password}
221 ... print_string=\#
222
Michael Walshc108e422019-03-28 12:27:18 -0500223 Qprintn
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600224
Michael Walshc108e422019-03-28 12:27:18 -0500225 Qprint Timen The operating system is now communicating.
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600226
227
228Copy PNOR to BMC
229 [Documentation] Copy the PNOR image to the BMC.
230 Import Library SCPLibrary WITH NAME scp
231 Open Connection for SCP
232 Log Copying ${PNOR_IMAGE_PATH} to /tmp
233 scp.Put File ${PNOR_IMAGE_PATH} /tmp
234
235
236Is OS Starting
237 [Documentation] Check if boot progress is OS starting.
238 ${boot_progress}= Get Boot Progress
239 Should Be Equal ${boot_progress} OSStart
240
241
242Is OS Off
243 [Documentation] Check if boot progress is "Off".
244 ${boot_progress}= Get Boot Progress
245 Should Be Equal ${boot_progress} Off
246
247
248Get Boot Progress To OS Starting State
249 [Documentation] Get the system to a boot progress state of 'FW Progress,
250 ... Starting OS'.
251
252 ${boot_progress}= Get Boot Progress
253 Run Keyword If '${boot_progress}' == 'OSStart'
254 ... Log Host is already in OS starting state
255 ... ELSE
256 ... Run Keywords Initiate Host PowerOff AND Initiate Host Boot
257 ... AND Wait Until Keyword Succeeds 10 min 10 sec Is OS Starting
258
259
260Check If warmReset is Initiated
261 [Documentation] Ping would be still alive, so try SSH to connect
262 ... if fails the ports are down indicating reboot
263 ... is in progress
264
265 # Warm reset adds 3 seconds delay before forcing reboot
266 # To minimize race conditions, we wait for 7 seconds
267 Sleep 7s
268 ${alive}= Run Keyword and Return Status
269 ... Open Connection And Log In
270 Return From Keyword If '${alive}' == '${False}' ${False}
271 [Return] ${True}
272
273
274Initialize DBUS cmd
275 [Documentation] Initialize dbus string with property string to extract
276 [Arguments] ${boot_property}
277
278 # Description of argument(s):
279 # boot_property Property string.
280
281 ${cmd}= Catenate ${dbuscmdBase} ${dbuscmdGet} ${dbuscmdString}
282 ${cmd}= Catenate ${cmd}${boot_property}
283 Set Global Variable ${dbuscmd} ${cmd}
284
285
286Create OS Console Command String
287 [Documentation] Return a command string to start OS console logging.
288
289 # First make sure that the ssh_pw program is available.
290 ${cmd}= Catenate which ssh_pw 2>/dev/null || find
291 ... ${EXECDIR} -name 'ssh_pw'
292
Michael Walshc108e422019-03-28 12:27:18 -0500293 Dprint Issuing ${cmd}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600294 ${rc} ${output}= Run And Return Rc And Output ${cmd}
295 Rdpvars rc output
296
297 Should Be Equal As Integers 0 ${rc} msg=Could not find ssh_pw.
298
299 ${ssh_pw_file_path}= Set Variable ${output}
300
301 ${cmd}= Catenate ${ssh_pw_file_path} ${OPENBMC_PASSWORD} -p 2200
302 ... -o "StrictHostKeyChecking no" ${OPENBMC_USERNAME}@${OPENBMC_HOST}
303
304 [Return] ${cmd}
305
306
307Get SOL Console Pid
308 [Documentation] Get the pid of the active SOL console job.
Michael Walsh07ed7942019-11-08 14:41:30 -0600309 [Arguments] ${expect_running}=${0} ${log_file_path}=${EMPTY}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600310
311 # Description of argument(s):
Michael Walsh07ed7942019-11-08 14:41:30 -0600312 # expect_running If set and if no SOL console job is found, print debug info and fail.
313 # log_file_path Needed to print debug info if expect_running is set and no pid is found.
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600314
315 # Find the pid of the active system console logging session (if any).
316 ${search_string}= Create OS Console Command String
317 # At least in some cases, ps output does not show double quotes so we must
318 # replace them in our search string with the regexes to indicate that they
319 # are optional.
320 ${search_string}= Replace String ${search_string} " ["]?
321 ${ps_cmd}= Catenate ps axwwo user,pid,cmd
322 ${cmd_buf}= Catenate echo $(${ps_cmd} | egrep '${search_string}' |
323 ... egrep -v grep | cut -c10-14)
Michael Walshc108e422019-03-28 12:27:18 -0500324 Dprint Issuing ${cmd_buf}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600325 ${rc} ${os_con_pid}= Run And Return Rc And Output ${cmd_buf}
326 Rdpvars os_con_pid
327 # If rc is not zero it just means that there is no OS Console process
328 # running.
329
330 Return From Keyword If '${os_con_pid}' != '${EMPTY}' ${os_con_pid}
331 Return From Keyword If '${expect_running}' == '${0}' ${os_con_pid}
332
333 Cmd Fnc cat ${log_file_path} ; echo ; ${ps_cmd} quiet=${0}
334 ... print_output=${1} show_err=${1}
Michael Walsh07ed7942019-11-08 14:41:30 -0600335 Valid Value os_con_pid
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600336
337
338Stop SOL Console Logging
339 [Documentation] Stop system console logging and return log output.
340 [Arguments] ${log_file_path}=${EMPTY}
341 ... ${targ_file_path}=${EXECDIR}${/}logs${/}
342 ... ${return_data}=${1}
343
344 # If there are muliple system console processes, they will all be stopped.
345 # If there is no existing log file this keyword will return an error
346 # message to that effect (and write that message to targ_file_path, if
347 # specified).
348 # NOTE: This keyword will not fail if there is no running system console
349 # process.
350
351 # Description of arguments:
352 # log_file_path The file path that was used to call "Start SOL
353 # Console Logging". See that keyword (above) for details.
354 # targ_file_path If specified, the file path to which the source
355 # file path (i.e. "log_file_path") should be copied.
356 # return_data If this is set to ${1}, this keyword will return the SOL
357 # data to the caller as a unicode string.
358
359 ${log_file_path}= Create OS Console File Path ${log_file_path}
360
361 ${os_con_pid}= Get SOL Console Pid
362
363 ${cmd_buf}= Catenate kill -9 ${os_con_pid}
Michael Walshc108e422019-03-28 12:27:18 -0500364 Run Keyword If '${os_con_pid}' != '${EMPTY}' Dprint Issuing ${cmd_buf}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600365 ${rc} ${output}= Run Keyword If '${os_con_pid}' != '${EMPTY}'
366 ... Run And Return Rc And Output ${cmd_buf}
367 Run Keyword If '${os_con_pid}' != '${EMPTY}' Rdpvars rc output
368
369 Run Keyword If '${targ_file_path}' != '${EMPTY}'
370 ... Run Keyword And Ignore Error
371 ... Copy File ${log_file_path} ${targ_file_path}
372
373 ${output}= Set Variable ${EMPTY}
374 ${loc_quiet}= Evaluate ${debug}^1
375 ${rc} ${output}= Run Keyword If '${return_data}' == '${1}'
376 ... Cmd Fnc cat ${log_file_path} 2>/dev/null quiet=${loc_quiet}
377 ... print_output=${0} show_err=${0}
378
379 [Return] ${output}
380
381
382Start SOL Console Logging
383 [Documentation] Start system console log to file.
384 [Arguments] ${log_file_path}=${EMPTY} ${return_data}=${1}
385
386 # This keyword will first call "Stop SOL Console Logging". Only then will
387 # it start SOL console logging. The data returned by "Stop SOL Console
388 # Logging" will in turn be returned by this keyword.
389
390 # Description of arguments:
391 # log_file_path The file path to which system console log data should be
392 # written. Note that this path is taken to be a location
393 # on the machine where this program is running rather than
394 # on the Open BMC system.
395 # return_data If this is set to ${1}, this keyword will return any SOL
396 # data to the caller as a unicode string.
397
398 ${log_file_path}= Create OS Console File Path ${log_file_path}
399
400 ${log_output}= Stop SOL Console Logging ${log_file_path}
401 ... return_data=${return_data}
402
403 # Validate by making sure we can create the file. Problems creating the
404 # file would not be noticed by the subsequent ssh command because we fork
405 # the command.
406 Create File ${log_file_path}
407 ${sub_cmd_buf}= Create OS Console Command String
408 # Routing stderr to stdout so that any startup error text will go to the
409 # output file.
410 # TODO: Doesn't work with tox so reverting temporarily.
411 # nohup detaches the process completely from our pty.
412 #${cmd_buf}= Catenate nohup ${sub_cmd_buf} &> ${log_file_path} &
413 ${cmd_buf}= Catenate ${sub_cmd_buf} > ${log_file_path} 2>&1 &
Michael Walshc108e422019-03-28 12:27:18 -0500414 Dprint Issuing ${cmd_buf}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600415 ${rc} ${output}= Run And Return Rc And Output ${cmd_buf}
416 # Because we are forking this command, we essentially will never get a
417 # non-zero return code or any output.
418 Should Be Equal ${rc} ${0}
419
420 Wait Until Keyword Succeeds 10 seconds 0 seconds
Michael Walsh07ed7942019-11-08 14:41:30 -0600421 ... Get SOL Console Pid ${1} ${log_file_path}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600422
423 [Return] ${log_output}
424
425
426Get Time Stamp
427 [Documentation] Get the current time stamp data
428 ${cur_time}= Get Current Date result_format=%Y%m%d%H%M%S%f
429 [Return] ${cur_time}
430
431
432Start Journal Log
433 [Documentation] Start capturing journal log to a file in /tmp using
434 ... journalctl command. By default journal log is collected
435 ... at /tmp/journal_log else user input location.
436 ... The File is appended with datetime.
437 [Arguments] ${file_path}=/tmp/journal_log ${filter}=${EMPTY}
438
439 # Description of arguments:
440 # file_path The file path of the journal file.
441
442 ${cur_time}= Get Time Stamp
443 Set Global Variable ${LOG_TIME} ${cur_time}
444 Open Connection And Log In
445 Start Command
446 ... journalctl -f ${filter} > ${file_path}-${LOG_TIME}
447 Log Journal Log Started: ${file_path}-${LOG_TIME}
448
449
450Stop Journal Log
451 [Documentation] Stop journalctl process if its running.
452 ... By default return log from /tmp/journal_log else
453 ... user input location.
454 [Arguments] ${file_path}=/tmp/journal_log
455
456 # Description of arguments:
457 # file_path The file path of the journal file.
458
459 Open Connection And Log In
460
461 ${rc}=
462 ... Execute Command
463 ... ps | grep journalctl | grep -v grep
464 ... return_stdout=False return_rc=True
465
466 Return From Keyword If '${rc}' == '${1}'
467 ... No journal log process running
468
469 ${output} ${stderr}=
470 ... Execute Command killall journalctl
471 ... return_stderr=True
472 Should Be Empty ${stderr}
473
474 ${journal_log} ${stderr}=
475 ... Execute Command
476 ... cat ${file_path}-${LOG_TIME}
477 ... return_stderr=True
478 Should Be Empty ${stderr}
479
480 Log ${journal_log}
481
482 Execute Command rm ${file_path}-${LOG_TIME}
483
484 [Return] ${journal_log}
485
486
487Mac Address To Hex String
488 [Documentation] Converts MAC address into hex format.
489 ... Example
490 ... Given the following MAC: 00:01:6C:80:02:78
491 ... This keyword will return: 0x00 0x01 0x6C 0x80 0x02 0x78
492 ... Description of arguments:
493 ... i_macaddress MAC address in the following format
494 ... 00:01:6C:80:02:78
495 [Arguments] ${i_macaddress}
496
497 # Description of arguments:
498 # i_macaddress The MAC address.
499
500 ${mac_hex}= Catenate 0x${i_macaddress.replace(':', ' 0x')}
501 [Return] ${mac_hex}
502
503
504IP Address To Hex String
505 [Documentation] Converts IP address into hex format.
506 ... Example:
507 ... Given the following IP: 10.3.164.100
508 ... This keyword will return: 0xa 0x3 0xa4 0xa0
509 [Arguments] ${i_ipaddress}
510
511 # Description of arguments:
512 # i_macaddress The IP address in the format 10.10.10.10.
513
514 @{ip}= Split String ${i_ipaddress} .
515 ${index}= Set Variable ${0}
516
517 :FOR ${item} IN @{ip}
518 \ ${hex}= Convert To Hex ${item} prefix=0x lowercase=yes
519 \ Set List Value ${ip} ${index} ${hex}
520 \ ${index}= Set Variable ${index + 1}
521 ${ip_hex}= Catenate @{ip}
522
523 [Return] ${ip_hex}
524
525
526BMC CPU Performance Check
527 [Documentation] Minimal 10% of proc should be free in this instance
528
529 ${bmc_cpu_usage_output} ${stderr} ${rc}= BMC Execute Command
530 ... ${bmc_cpu_usage_cmd}
531 ${bmc_cpu_usage_output} ${stderr} ${rc}= BMC Execute Command
532 ... ${bmc_cpu_usage_cmd}
533 ${bmc_cpu_percentage}= Fetch From Left ${bmc_cpu_usage_output} %
Sivas SRR14426bc2019-04-10 10:14:46 -0500534 Rprint Vars bmc_cpu_percentage
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600535 Should be true ${bmc_cpu_percentage} < 90
536
537
538BMC Mem Performance Check
539 [Documentation] Minimal 10% of memory should be free in this instance
540
541 ${bmc_mem_free_output} ${stderr} ${rc}= BMC Execute Command
542 ... ${bmc_mem_free_cmd}
543
544 ${bmc_mem_total_output} ${stderr} ${rc}= BMC Execute Command
545 ... ${bmc_mem_total_cmd}
546 ${bmc_mem_free_output} ${stderr} ${rc}= BMC Execute Command
547 ... ${bmc_mem_free_cmd}
548
549 ${bmc_mem_total_output} ${stderr} ${rc}= BMC Execute Command
550 ... ${bmc_mem_total_cmd}
551
552 ${bmc_mem_percentage}= Evaluate ${bmc_mem_free_output}*100
553 ${bmc_mem_percentage}= Evaluate
554 ... ${bmc_mem_percentage}/${bmc_mem_total_output}
Sivas SRR14426bc2019-04-10 10:14:46 -0500555 Rprint Vars bmc_mem_percentage
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600556 Should be true ${bmc_mem_percentage} > 10
557
558
559BMC File System Usage Check
560 [Documentation] Check the file system space. 4 file system should be
561 ... 100% full which is expected
562 # Filesystem Size Used Available Use% Mounted on
563 # /dev/root 14.4M 14.4M 0 100% /
564 # /dev/ubiblock0_0 14.4M 14.4M 0 100% /media/rofs-c9249b0e
565 # /dev/ubiblock8_0 19.6M 19.6M 0 100% /media/pnor-ro-8764baa3
566 # /dev/ubiblock4_0 14.4M 14.4M 0 100% /media/rofs-407816c
567 # /dev/ubiblock8_4 21.1M 21.1M 0 100% /media/pnor-ro-cecc64c4
568 ${bmc_fs_usage_output} ${stderr} ${rc}= BMC Execute Command
569 ... ${bmc_file_system_usage_cmd}
570 ${bmc_pnor_fs_usage_output} ${stderr} ${rc}= BMC Execute Command
571 ... ${total_pnor_ro_file_system_cmd}
572 ${bmc_bmc_fs_usage_output} ${stderr} ${rc}= BMC Execute Command
573 ... ${total_bmc_ro_file_system_cmd}
574 ${total_bmc_pnor_image}= Evaluate
575 ... ${bmc_pnor_fs_usage_output}+${bmc_bmc_fs_usage_output}
576 # Considering /dev/root also in total 100% used file system
577 ${total_full_fs}= Evaluate ${total_bmc_pnor_image}+1
Sivas SRR14426bc2019-04-10 10:14:46 -0500578 Rprint Vars bmc_fs_usage_output bmc_pnor_fs_usage_output bmc_bmc_fs_usage_output
579 ... total_full_fs
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600580 Should Be True ${bmc_fs_usage_output}==${total_full_fs}
581
582
583Check BMC CPU Performance
584 [Documentation] Minimal 10% of proc should be free in 3 sample
George Keishing9a089442018-12-10 09:38:50 -0600585 :FOR ${var} IN RANGE 1 4
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600586 \ BMC CPU Performance check
587
588
589Check BMC Mem Performance
590 [Documentation] Minimal 10% of memory should be free
591
George Keishing9a089442018-12-10 09:38:50 -0600592 :FOR ${var} IN RANGE 1 4
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600593 \ BMC Mem Performance check
594
595
596Check BMC File System Performance
597 [Documentation] Check for file system usage for 4 times
598
George Keishing9a089442018-12-10 09:38:50 -0600599 :FOR ${var} IN RANGE 1 4
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600600 \ BMC File System Usage check
601
602
603Get URL List
604 [Documentation] Return list of URLs under given URL.
605 [Arguments] ${openbmc_url}
606
607 # Description of argument(s):
608 # openbmc_url URL for list operation (e.g.
609 # /xyz/openbmc_project/inventory).
610
Steven Sombaraaaab222018-12-19 13:16:23 -0600611 ${url_list}= Read Properties ${openbmc_url}list quiet=${1}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600612 Sort List ${url_list}
613
614 [Return] ${url_list}
615
616
617Check Zombie Process
618 [Documentation] Check if any defunct process exist or not on BMC
619 ${count} ${stderr} ${rc}= Execute Command ps -o stat | grep Z | wc -l
620 ... return_stderr=True return_rc=True
621 Should Be True ${count}==0
622 Should Be Empty ${stderr}
623
624
625Prune Journal Log
626 [Documentation] Prune archived journal logs.
627 [Arguments] ${vacuum_size}=1M
628
629 # This keyword can be used to prevent the journal
630 # log from filling up the /run filesystem.
631 # This command will retain only the latest logs
632 # of the user specified size.
633
634 # Description of argument(s):
635 # vacuum_size Size of journal.
636
637 Open Connection And Log In
638 ${output} ${stderr} ${rc}=
639 ... Execute Command
640 ... journalctl --vacuum-size=${vacuum_size}
641 ... return_stderr=True return_rc=True
642
643 Should Be Equal ${rc} ${0} msg=${stderr}
644
645
646Get System Power Policy
647 [Documentation] Returns the BMC power policy.
648
649 # Set the bmc_power_policy_method to either 'Old' or 'New'.
650 Set Power Policy Method
651 ${cmd_buf}= Create List ${bmc_power_policy_method} Get Power Policy
652 # Run the appropriate keyword.
653 ${currentPolicy}= Run Keyword @{cmd_buf}
654
655 [Return] ${currentPolicy}
656
657
658Set BMC Reset Reference Time
659 [Documentation] Set current boot time as a reference and increment
660 ... boot count.
661
662 ${cur_btime}= Get BMC Boot Time
663 Run Keyword If ${BOOT_TIME} == ${0} and ${BOOT_COUNT} == ${0}
664 ... Set Global Variable ${BOOT_TIME} ${cur_btime}
665 ... ELSE IF ${cur_btime} > ${BOOT_TIME}
666 ... Run Keywords Set Global Variable ${BOOT_TIME} ${cur_btime}
667 ... AND
668 ... Set Global Variable ${BOOT_COUNT} ${BOOT_COUNT + 1}
669
670
671Get BMC Boot Time
672 [Documentation] Returns boot time from /proc/stat.
673
674 Open Connection And Log In
675 ${output} ${stderr}=
676 ... Execute Command egrep '^btime ' /proc/stat | cut -f 2 -d ' '
677 ... return_stderr=True
678 Should Be Empty ${stderr}
679 ${btime}= Convert To Integer ${output}
680 [Return] ${btime}
681
682
683Enable Core Dump On BMC
684 [Documentation] Enable core dump collection.
685 ${core_pattern} ${stderr} ${rc}= BMC Execute Command
686 ... echo '/tmp/core_%e.%p' | tee /proc/sys/kernel/core_pattern
687 Should Be Equal As Strings ${core_pattern} /tmp/core_%e.%p
688
689
690Get Number Of BMC Core Dump Files
691 [Documentation] Returns number of core dump files on BMC.
692 Open Connection And Log In
693 ${num_of_core_dump}= Execute Command
694 ... ls /tmp/core* 2>/dev/null | wc -l
695 [Return] ${num_of_core_dump}
696
697
698Set Core Dump File Size Unlimited
699 [Documentation] Set core dump file size to unlimited.
700 BMC Execute Command ulimit -c unlimited
701
702
703Check For Core Dumps
704 [Documentation] Check for any core dumps exist.
705 ${output}= Get Number Of BMC Core Dump Files
706 Run Keyword If ${output} > 0
707 ... Log **Warning** BMC core dump files exist level=WARN
708
709
710Configure Initial Settings
711 [Documentation] Restore old IP and route.
712 ... This keyword requires initial settings viz IP address,
713 ... Network Mask, default gatway and serial console IP and port
714 ... information which should be provided in command line.
715
716 [Arguments] ${host}=${OPENBMC_HOST} ${mask}=${NET_MASK}
717 ... ${gw_ip}=${GW_IP}
718
719 # Description of arguments:
720 # host IP address of the OS Host.
721 # mask Network mask.
722 # gu_ip Gateway IP address or hostname.
723
724 # Open telnet connection and ignore the error, in case telnet session is
725 # already opened by the program calling this keyword.
726 Run Keyword And Ignore Error Open Telnet Connection to BMC Serial Console
727 Telnet.write ifconfig eth0 ${host} netmask ${mask}
728 Telnet.write route add default gw ${gw_ip}
729
730
731Install Debug Tarball On BMC
732 [Documentation] Copy the debug tar file to BMC and install.
733 [Arguments] ${tarball_file_path}=${default_tarball}
734 ... ${targ_tarball_dir_path}=/tmp/tarball/
735
736 # Description of arguments:
737 # tarball_file_path Path of the debug tarball file.
738 # The tar file is downloaded from the build page
739 # https://openpower.xyz/job/openbmc-build/
740 # obmc-phosphor-debug-tarball-witherspoon.tar.xz
741 #
742 # targ_tarball_dir_path The directory path where the tarball is to be
743 # installed.
744
745 OperatingSystem.File Should Exist ${tarball_file_path}
746 ... msg=${tarball_file_path} doesn't exist.
747
748 # Upload the file to BMC.
749 Import Library SCPLibrary WITH NAME scp
750 Open Connection for SCP
751 scp.Put File ${tarball_file_path} /tmp/debug-tarball.tar.xz
752
753 # Create tarball directory and install.
754 BMC Execute Command mkdir -p ${targ_tarball_dir_path}
755 BMC Execute Command
756 ... tar -xf /tmp/debug-tarball.tar.xz -C ${targ_tarball_dir_path}
757
758 # Remove the tarball file from BMC.
759 BMC Execute Command rm -f /tmp/debug-tarball.tar.xz
760
761
762Get BMC Boot Count
763 [Documentation] Returns BMC boot count based on boot time.
764 ${cur_btime}= Get BMC Boot Time
765
766 # Set global variable BOOT_TIME to current boot time if current boot time
767 # is changed. Also increase value of global variable BOOT_COUNT by 1.
768 Run Keyword If ${cur_btime} > ${BOOT_TIME}
769 ... Run Keywords Set Global Variable ${BOOT_TIME} ${cur_btime}
770 ... AND
771 ... Set Global Variable ${BOOT_COUNT} ${BOOT_COUNT + 1}
772
773 [Return] ${BOOT_COUNT}
774
775
776Set BMC Boot Count
777 [Documentation] Set BMC boot count to given value.
778 [Arguments] ${count}
779
780 # Description of arguments:
781 # count boot count value.
782 ${cur_btime}= Get BMC Boot Time
783
784 # Set global variable BOOT_COUNT to given value.
785 Set Global Variable ${BOOT_COUNT} ${count}
786
787 # Set BOOT_TIME variable to current boot time.
788 Set Global Variable ${BOOT_COUNT} ${count}
789
790
791Delete Error Log Entry
792 [Documentation] Delete error log entry.
793 [Arguments] ${entry_path}
794
795 # Description of argument(s):
796 # entry_path Delete an error log entry.
797 # Ex. /xyz/openbmc_project/logging/entry/1
798
799 # Skip delete if entry URI is a callout.
800 # Examples:
801 # /xyz/openbmc_project/logging/entry/1/callout
802 # /xyz/openbmc_project/logging/entry/1/callouts/0
803 ${callout_entry}= Run Keyword And Return Status
804 ... Should Match Regexp ${entry_path} /callout[s]?(/|$)
805 Return From Keyword If ${callout_entry}
806
807 ${data}= Create Dictionary data=@{EMPTY}
808 ${resp}= Openbmc Delete Request ${entry_path} data=${data}
809 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
810
811
812Get BMC Version
813 [Documentation] Returns BMC version from /etc/os-release.
814 ... e.g. "v1.99.6-141-ge662190"
815
816 ${cmd}= Set Variable grep ^VERSION_ID= /etc/os-release | cut -f 2 -d '='
817 ${output} ${stderr} ${rc}= BMC Execute Command ${cmd}
818 [Return] ${output}
819
820
821Get PNOR Version
822 [Documentation] Returns the PNOR version from the BMC.
823
824 ${pnor_attrs}= Get PNOR Attributes
825 [Return] ${pnor_attrs['version']}
826
827
828Get PNOR Attributes
829 [Documentation] Return PNOR software attributes as a dictionary.
830
831 # This keyword parses /var/lib/phosphor-software-manager/pnor/ro/pnor.toc
832 # into key/value pairs.
833
834 ${outbuf} ${stderr} ${rc}= BMC Execute Command
835 ... cat /var/lib/phosphor-software-manager/pnor/ro/pnor.toc
836 ${pnor_attrs}= Key Value Outbuf To Dict ${outbuf} delim==
837
838 [Return] ${pnor_attrs}
839
840
Sivas SRR17dae1a2019-03-24 23:52:02 -0500841GET BMC PNOR Version
842 [Documentation] Return BMC & PNOR version from openbmc shell.
843
844 ${bmc_version}= GET BMC Version
845 ${pnor_version}= GET PNOR Version
846 Log ${bmc_version}
847 Rprint Vars bmc_version
848 Log ${pnor_version}
849 Rprint Vars pnor_version
850
851 [Return] ${bmc_version} ${pnor_version}
852
853
Sushil Singh5ea86d02019-07-11 02:05:16 -0500854Redfish Get BMC Version
855 [Documentation] Get BMC version via Redfish.
856
857 ${output}= Redfish.Get Attribute ${REDFISH_BASE_URI}Managers/bmc FirmwareVersion
858 [Return] ${output}
859
860Redfish Get Host Version
861 [Documentation] Get host version via Redfish.
862
863 ${output}= Redfish.Get Attribute ${REDFISH_BASE_URI}Systems/system BiosVersion
864 [Return] ${output}
865
866
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600867Copy Address Translation Utils To HOST OS
868 [Documentation] Copy address translation utils to host OS.
869
870 OperatingSystem.File Should Exist ${probe_cpu_tool_path}
871 ... msg=${probe_cpu_tool_path} doesn't exist.
872 OperatingSystem.File Should Exist ${probe_cpu_tool_path}
873 ... msg=${probe_cpu_tool_path} doesn't exist.
874
875 scp.Open connection ${OS_HOST} username=${OS_USERNAME}
876 ... password=${OS_PASSWORD}
877 scp.Put File ${probe_cpu_tool_path} ${target_file_path}
878 scp.Put File ${scom_addrs_tool_path} ${target_file_path}
879
880
881Verify BMC RTC And UTC Time Drift
882 [Documentation] Verify that the RTC and UTC time difference is less than
883 ... the given time_drift_max.
884 [Arguments] ${time_diff_max}=${10}
885
886 # Description of argument(s):
887 # time_diff_max The max allowable RTC and UTC time difference in seconds.
888
889 # Example:
890 # time_dict:
891 # [local_time]: Fri 2017-11-03 152756 UTC
892 # [local_time_seconds]: 1509740876
893 # [universal_time]: Fri 2017-11-03 152756 UTC
894 # [universal_time_seconds]: 1509740876
895 # [rtc_time]: Fri 2016-05-20 163403
896 # [rtc_time_seconds]: 1463780043
897 # [time_zone]: n/a (UTC, +0000)
898 # [network_time_on]: yes
899 # [ntp_synchronized]: no
900 # [rtc_in_local_tz]: no
901
902 ${time}= Get BMC Date Time
903 ${time_diff}= Evaluate
904 ... ${time['universal_time_seconds']} - ${time['rtc_time_seconds']}
905 Should Be True ${time_diff} < ${time_diff_max}
906
907
908Validate IP On BMC
909 [Documentation] Validate IP address is present in set of IP addresses.
910 [Arguments] ${ip_address} ${ip_data}
911
912 # Description of argument(s):
913 # ip_address IP address to check (e.g. xx.xx.xx.xx).
914 # ip_data Set of the IP addresses present.
915
916 Should Contain Match ${ip_data} ${ip_address}/*
917 ... msg=${ip_address} not found in the list provided.
918
919
920Remove Journald Logs
921 [Documentation] Remove all journald logs and restart service.
922
923 ${cmd}= Catenate systemctl stop systemd-journald.service &&
924 ... rm -rf /var/log/journal && systemctl start systemd-journald.service
925
926 BMC Execute Command ${cmd}
927
928
929Check For Regex In Journald
930 [Documentation] Parse the journal log and check for regex string.
931 [Arguments] ${regex}=${ERROR_REGEX} ${error_check}=${0} ${boot}=${EMPTY}
932
933 # Description of argument(s):
934 # regex Strings to be filter.
935 # error_check Check for errors.
936 # boot Argument to check current or persistent full boot log
937 # (e.g. "-b").
938
939 ${journal_log} ${stderr} ${rc}= BMC Execute Command
940 ... journalctl --no-pager ${boot} | egrep '${regex}' ignore_err=1
941
942 Run Keyword If ${error_check} == ${0}
943 ... Should Be Empty ${journal_log}
944 ... ELSE
945 ... Should Not Be Empty ${journal_log}
946
947
948Get Service Attribute
949 [Documentation] Get service attribute policy output.
950 [Arguments] ${option} ${servicename}
951
952 # Description of argument(s):
953 # option systemctl supported options
954 # servicename Qualified service name
955 ${cmd}= Set Variable
956 ... systemctl -p ${option} show ${servicename} | cut -d = -f2
957 ${attr} ${stderr} ${rc}= BMC Execute Command ${cmd}
958 [Return] ${attr}
Anusha Dathatrieda09492019-02-26 08:30:13 -0600959
960
961Verify Watchdog Enabled
962 [Documentation] Check that watchdog settings are enabled.
963
964 ${properties}= Read Properties /xyz/openbmc_project/watchdog/host0
965 Should Be Equal As Strings ${properties["Enabled"]} ${True}
966 Should Not Be Equal As Strings ${properties["TimeRemaining"]} 0