blob: fef4675eb433ebe17d7517bee6e4cafbbc212a3a [file] [log] [blame]
Steven Sombarf60cbcf2018-12-07 08:12:18 -06001*** Settings ***
2
3Documentation Utilities for Robot keywords that do not use REST.
4
Sandhya Somashekar839a0c22019-01-31 05:05:43 -06005Resource ../lib/resource.robot
Steven Sombarf60cbcf2018-12-07 08:12:18 -06006Resource ../lib/connection_client.robot
7Resource ../lib/boot_utils.robot
8Library String
9Library DateTime
10Library Process
11Library OperatingSystem
12Library gen_print.py
13Library gen_robot_print.py
14Library gen_cmd.py
15Library gen_robot_keyword.py
16Library bmc_ssh_utils.py
17Library utils.py
18Library var_funcs.py
19Library SCPLibrary WITH NAME scp
20
21*** Variables ***
22
23${pflash_cmd} /usr/sbin/pflash -r /dev/stdout -P VERSION
24
25${dbuscmdBase}
26... dbus-send --system --print-reply --dest=${OPENBMC_BASE_DBUS}.settings.Host
27${dbuscmdGet}
28... ${SETTINGS_URI}host0 org.freedesktop.DBus.Properties.Get
29${dbuscmdString}= string:"xyz.openbmc_project.settings.Host" string:
30
31# Assign default value to QUIET for programs which may not define it.
32${QUIET} ${0}
33
34${bmc_mem_free_cmd}= free | tr -s ' ' | sed '/^Mem/!d' | cut -d" " -f4
35${bmc_mem_total_cmd}= free | tr -s ' ' | sed '/^Mem/!d' | cut -d" " -f2
36${bmc_cpu_usage_cmd}= top -n 1 | grep CPU: | cut -c 7-9
37${HOST_SETTING} ${SETTINGS_URI}host0
38
39# /run/initramfs/ro associate filesystem should be 100% full always
40${bmc_file_system_usage_cmd}= df -h | cut -c 52-54 | grep 100 | wc -l
41${total_pnor_ro_file_system_cmd}= df -h | grep /media/pnor-ro | wc -l
42${total_bmc_ro_file_system_cmd}= df -h | grep /media/rofs | wc -l
43
44${BOOT_TIME} ${0}
45${BOOT_COUNT} ${0}
46${count} ${0}
47${devicetree_base} /sys/firmware/devicetree/base/model
48
49# Initialize default debug value to 0.
50${DEBUG} ${0}
51
52${probe_cpu_tool_path} ${EXECDIR}/tools/ras/probe_cpus.sh
53${scom_addrs_tool_path} ${EXECDIR}/tools/ras/scom_addr_p9.sh
54${target_file_path} /root/
55
56${default_tarball} ${EXECDIR}/obmc-phosphor-debug-tarball-witherspoon.tar.xz
57
58# These variables are used to straddle between new and old methods of setting
59# values.
60${bmc_power_policy_method} ${EMPTY}
61@{valid_power_policy_vars} RESTORE_LAST_STATE ALWAYS_POWER_ON
62... ALWAYS_POWER_OFF
63
64
65*** Keywords ***
66
67Check BMC Performance
68 [Documentation] Check BMC basic CPU Mem File system performance.
69
70 Check BMC CPU Performance
71 Check BMC Mem Performance
72 Check BMC File System Performance
73
74
75Verify PNOR Update
76 [Documentation] Verify that the PNOR is not corrupted.
77 # Example:
78 # FFS: Flash header not found. Code: 100
79 # Error 100 opening ffs !
80
81 ${stdout} ${stderr} ${rc}=
82 ... BMC Execute Command /usr/sbin/pflash -h | egrep -q skip
83 ... ignore_err=${1}
84 ${pflash_cmd}= Set Variable If ${rc} == ${0} ${pflash_cmd} --skip=4096
85 ... ${pflash_cmd}
86 ${pnor_info}= BMC Execute Command ${pflash_cmd}
87 Should Not Contain Any ${pnor_info} Flash header not found Error
88
89
90Get BMC System Model
91 [Documentation] Get the BMC model from the device tree and return it.
92
93 ${bmc_model} ${stderr} ${rc}= BMC Execute Command
94 ... cat ${devicetree_base} | cut -d " " -f 1 return_stderr=True
95 ... test_mode=0
96 Should Be Empty ${stderr}
97 Should Not Be Empty ${bmc_model} msg=BMC model is empty.
98 [Return] ${bmc_model}
99
100
101Verify BMC System Model
102 [Documentation] Verify the BMC model with ${OPENBMC_MODEL}.
103 [Arguments] ${bmc_model}
104
105 # Description of argument(s):
106 # bmc_model System model (e.g. "witherspoon").
107
108 ${tmp_bmc_model}= Fetch From Right ${OPENBMC_MODEL} /
109 ${tmp_bmc_model}= Fetch From Left ${tmp_bmc_model} .
110 ${ret}= Run Keyword And Return Status Should Contain ${bmc_model}
111 ... ${tmp_bmc_model} ignore_case=True
112 [Return] ${ret}
113
114
115Wait For Host To Ping
116 [Documentation] Wait for the given host to ping.
117 [Arguments] ${host} ${timeout}=${OPENBMC_REBOOT_TIMEOUT}min
118 ... ${interval}=5 sec
119
120 # Description of argument(s):
121 # host The host name or IP of the host to ping.
122 # timeout The amount of time after which ping attempts cease.
123 # This should be expressed in Robot Framework's time format
124 # (e.g. "10 seconds").
125 # interval The amount of time in between attempts to ping.
126 # This should be expressed in Robot Framework's time format
127 # (e.g. "5 seconds").
128
129 Wait Until Keyword Succeeds ${timeout} ${interval} Ping Host ${host}
130
131
132Ping Host
133 [Documentation] Ping the given host.
134 [Arguments] ${host}
135
136 # Description of argument(s):
137 # host The host name or IP of the host to ping.
138
139 Should Not Be Empty ${host} msg=No host provided
140 ${RC} ${output}= Run and return RC and Output ping -c 4 ${host}
141 Log RC: ${RC}\nOutput:\n${output}
142 Should be equal ${RC} ${0}
143
144
145Check OS
146 [Documentation] Attempts to ping the host OS and then checks that the host
147 ... OS is up by running an SSH command.
148
149 [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME}
150 ... ${os_password}=${OS_PASSWORD} ${quiet}=${QUIET}
151 ... ${print_string}=${EMPTY}
152 [Teardown] SSHLibrary.Close Connection
153
154 # Description of argument(s):
155 # os_host The DNS name/IP of the OS host associated with our BMC.
156 # os_username The username to be used to sign on to the OS host.
157 # os_password The password to be used to sign on to the OS host.
158 # quiet Indicates whether this keyword should write to console.
159 # print_string A string to be printed before checking the OS.
160
161 rprint ${print_string}
162
163 # Attempt to ping the OS. Store the return code to check later.
164 ${ping_rc}= Run Keyword and Return Status Ping Host ${os_host}
165
166 SSHLibrary.Open connection ${os_host}
167
168 ${status} ${msg}= Run Keyword And Ignore Error Login ${os_username}
169 ... ${os_password}
170 ${err_msg1}= Sprint Error ${msg}
171 ${err_msg}= Catenate SEPARATOR= \n ${err_msg1}
172 Run Keyword If '${status}' == 'FAIL' Fail msg=${err_msg}
173 ${output} ${stderr} ${rc}= Execute Command uptime return_stderr=True
174 ... return_rc=True
175
176 ${temp_msg}= Catenate Could not execute a command on the operating
177 ... system.\n
178 ${err_msg1}= Sprint Error ${temp_msg}
179 ${err_msg}= Catenate SEPARATOR= \n ${err_msg1}
180
181 # If the return code returned by "Execute Command" is non-zero, this
182 # keyword will fail.
183 Should Be Equal ${rc} ${0} msg=${err_msg}
184 # We will likewise fail if there is any stderr data.
185 Should Be Empty ${stderr}
186
187 ${temp_msg}= Set Variable Could not ping the operating system.\n
188 ${err_msg1}= Sprint Error ${temp_msg}
189 ${err_msg}= Catenate SEPARATOR= \n ${err_msg1}
190 # We will likewise fail if the OS did not ping, as we could SSH but not
191 # ping
192 Should Be Equal As Strings ${ping_rc} ${TRUE} msg=${err_msg}
193
194
195Wait for OS
196 [Documentation] Waits for the host OS to come up via calls to "Check OS".
197 [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME}
198 ... ${os_password}=${OS_PASSWORD} ${timeout}=${OS_WAIT_TIMEOUT}
199 ... ${quiet}=${0}
200 [Teardown] rprintn
201
202 # Description of argument(s):
203 # os_host The DNS name or IP of the OS host associated with our
204 # BMC.
205 # os_username The username to be used to sign on to the OS host.
206 # os_password The password to be used to sign on to the OS host.
207 # timeout The timeout in seconds indicating how long you're
208 # willing to wait for the OS to respond.
209 # quiet Indicates whether this keyword should write to console.
210
211 # The interval to be used between calls to "Check OS".
212 ${interval}= Set Variable 5
213
214 ${message}= Catenate Checking every ${interval} seconds for up to
215 ... ${timeout} seconds for the operating system to communicate.
216 rqprint_timen ${message}
217
218 Wait Until Keyword Succeeds ${timeout} sec ${interval} Check OS
219 ... ${os_host} ${os_username} ${os_password}
220 ... print_string=\#
221
222 rqprintn
223
224 rqprint_timen The operating system is now communicating.
225
226
227Copy PNOR to BMC
228 [Documentation] Copy the PNOR image to the BMC.
229 Import Library SCPLibrary WITH NAME scp
230 Open Connection for SCP
231 Log Copying ${PNOR_IMAGE_PATH} to /tmp
232 scp.Put File ${PNOR_IMAGE_PATH} /tmp
233
234
235Is OS Starting
236 [Documentation] Check if boot progress is OS starting.
237 ${boot_progress}= Get Boot Progress
238 Should Be Equal ${boot_progress} OSStart
239
240
241Is OS Off
242 [Documentation] Check if boot progress is "Off".
243 ${boot_progress}= Get Boot Progress
244 Should Be Equal ${boot_progress} Off
245
246
247Get Boot Progress To OS Starting State
248 [Documentation] Get the system to a boot progress state of 'FW Progress,
249 ... Starting OS'.
250
251 ${boot_progress}= Get Boot Progress
252 Run Keyword If '${boot_progress}' == 'OSStart'
253 ... Log Host is already in OS starting state
254 ... ELSE
255 ... Run Keywords Initiate Host PowerOff AND Initiate Host Boot
256 ... AND Wait Until Keyword Succeeds 10 min 10 sec Is OS Starting
257
258
259Check If warmReset is Initiated
260 [Documentation] Ping would be still alive, so try SSH to connect
261 ... if fails the ports are down indicating reboot
262 ... is in progress
263
264 # Warm reset adds 3 seconds delay before forcing reboot
265 # To minimize race conditions, we wait for 7 seconds
266 Sleep 7s
267 ${alive}= Run Keyword and Return Status
268 ... Open Connection And Log In
269 Return From Keyword If '${alive}' == '${False}' ${False}
270 [Return] ${True}
271
272
273Initialize DBUS cmd
274 [Documentation] Initialize dbus string with property string to extract
275 [Arguments] ${boot_property}
276
277 # Description of argument(s):
278 # boot_property Property string.
279
280 ${cmd}= Catenate ${dbuscmdBase} ${dbuscmdGet} ${dbuscmdString}
281 ${cmd}= Catenate ${cmd}${boot_property}
282 Set Global Variable ${dbuscmd} ${cmd}
283
284
285Create OS Console Command String
286 [Documentation] Return a command string to start OS console logging.
287
288 # First make sure that the ssh_pw program is available.
289 ${cmd}= Catenate which ssh_pw 2>/dev/null || find
290 ... ${EXECDIR} -name 'ssh_pw'
291
292 Rdpissuing ${cmd}
293 ${rc} ${output}= Run And Return Rc And Output ${cmd}
294 Rdpvars rc output
295
296 Should Be Equal As Integers 0 ${rc} msg=Could not find ssh_pw.
297
298 ${ssh_pw_file_path}= Set Variable ${output}
299
300 ${cmd}= Catenate ${ssh_pw_file_path} ${OPENBMC_PASSWORD} -p 2200
301 ... -o "StrictHostKeyChecking no" ${OPENBMC_USERNAME}@${OPENBMC_HOST}
302
303 [Return] ${cmd}
304
305
306Get SOL Console Pid
307 [Documentation] Get the pid of the active SOL console job.
308 [Arguments] ${expect_running}=${0}
309
310 # Description of argument(s):
311 # expect_running If set and if no SOL console job is found, print debug
312 # info and fail.
313
314 # Find the pid of the active system console logging session (if any).
315 ${search_string}= Create OS Console Command String
316 # At least in some cases, ps output does not show double quotes so we must
317 # replace them in our search string with the regexes to indicate that they
318 # are optional.
319 ${search_string}= Replace String ${search_string} " ["]?
320 ${ps_cmd}= Catenate ps axwwo user,pid,cmd
321 ${cmd_buf}= Catenate echo $(${ps_cmd} | egrep '${search_string}' |
322 ... egrep -v grep | cut -c10-14)
323 Rdpissuing ${cmd_buf}
324 ${rc} ${os_con_pid}= Run And Return Rc And Output ${cmd_buf}
325 Rdpvars os_con_pid
326 # If rc is not zero it just means that there is no OS Console process
327 # running.
328
329 Return From Keyword If '${os_con_pid}' != '${EMPTY}' ${os_con_pid}
330 Return From Keyword If '${expect_running}' == '${0}' ${os_con_pid}
331
332 Cmd Fnc cat ${log_file_path} ; echo ; ${ps_cmd} quiet=${0}
333 ... print_output=${1} show_err=${1}
334
335 Should Not Be Empty ${os_con_pid}
336
337
338Stop SOL Console Logging
339 [Documentation] Stop system console logging and return log output.
340 [Arguments] ${log_file_path}=${EMPTY}
341 ... ${targ_file_path}=${EXECDIR}${/}logs${/}
342 ... ${return_data}=${1}
343
344 # If there are muliple system console processes, they will all be stopped.
345 # If there is no existing log file this keyword will return an error
346 # message to that effect (and write that message to targ_file_path, if
347 # specified).
348 # NOTE: This keyword will not fail if there is no running system console
349 # process.
350
351 # Description of arguments:
352 # log_file_path The file path that was used to call "Start SOL
353 # Console Logging". See that keyword (above) for details.
354 # targ_file_path If specified, the file path to which the source
355 # file path (i.e. "log_file_path") should be copied.
356 # return_data If this is set to ${1}, this keyword will return the SOL
357 # data to the caller as a unicode string.
358
359 ${log_file_path}= Create OS Console File Path ${log_file_path}
360
361 ${os_con_pid}= Get SOL Console Pid
362
363 ${cmd_buf}= Catenate kill -9 ${os_con_pid}
364 Run Keyword If '${os_con_pid}' != '${EMPTY}' Rdpissuing ${cmd_buf}
365 ${rc} ${output}= Run Keyword If '${os_con_pid}' != '${EMPTY}'
366 ... Run And Return Rc And Output ${cmd_buf}
367 Run Keyword If '${os_con_pid}' != '${EMPTY}' Rdpvars rc output
368
369 Run Keyword If '${targ_file_path}' != '${EMPTY}'
370 ... Run Keyword And Ignore Error
371 ... Copy File ${log_file_path} ${targ_file_path}
372
373 ${output}= Set Variable ${EMPTY}
374 ${loc_quiet}= Evaluate ${debug}^1
375 ${rc} ${output}= Run Keyword If '${return_data}' == '${1}'
376 ... Cmd Fnc cat ${log_file_path} 2>/dev/null quiet=${loc_quiet}
377 ... print_output=${0} show_err=${0}
378
379 [Return] ${output}
380
381
382Start SOL Console Logging
383 [Documentation] Start system console log to file.
384 [Arguments] ${log_file_path}=${EMPTY} ${return_data}=${1}
385
386 # This keyword will first call "Stop SOL Console Logging". Only then will
387 # it start SOL console logging. The data returned by "Stop SOL Console
388 # Logging" will in turn be returned by this keyword.
389
390 # Description of arguments:
391 # log_file_path The file path to which system console log data should be
392 # written. Note that this path is taken to be a location
393 # on the machine where this program is running rather than
394 # on the Open BMC system.
395 # return_data If this is set to ${1}, this keyword will return any SOL
396 # data to the caller as a unicode string.
397
398 ${log_file_path}= Create OS Console File Path ${log_file_path}
399
400 ${log_output}= Stop SOL Console Logging ${log_file_path}
401 ... return_data=${return_data}
402
403 # Validate by making sure we can create the file. Problems creating the
404 # file would not be noticed by the subsequent ssh command because we fork
405 # the command.
406 Create File ${log_file_path}
407 ${sub_cmd_buf}= Create OS Console Command String
408 # Routing stderr to stdout so that any startup error text will go to the
409 # output file.
410 # TODO: Doesn't work with tox so reverting temporarily.
411 # nohup detaches the process completely from our pty.
412 #${cmd_buf}= Catenate nohup ${sub_cmd_buf} &> ${log_file_path} &
413 ${cmd_buf}= Catenate ${sub_cmd_buf} > ${log_file_path} 2>&1 &
414 Rdpissuing ${cmd_buf}
415 ${rc} ${output}= Run And Return Rc And Output ${cmd_buf}
416 # Because we are forking this command, we essentially will never get a
417 # non-zero return code or any output.
418 Should Be Equal ${rc} ${0}
419
420 Wait Until Keyword Succeeds 10 seconds 0 seconds
421 ... Get SOL Console Pid ${1}
422
423 [Return] ${log_output}
424
425
426Get Time Stamp
427 [Documentation] Get the current time stamp data
428 ${cur_time}= Get Current Date result_format=%Y%m%d%H%M%S%f
429 [Return] ${cur_time}
430
431
432Start Journal Log
433 [Documentation] Start capturing journal log to a file in /tmp using
434 ... journalctl command. By default journal log is collected
435 ... at /tmp/journal_log else user input location.
436 ... The File is appended with datetime.
437 [Arguments] ${file_path}=/tmp/journal_log ${filter}=${EMPTY}
438
439 # Description of arguments:
440 # file_path The file path of the journal file.
441
442 ${cur_time}= Get Time Stamp
443 Set Global Variable ${LOG_TIME} ${cur_time}
444 Open Connection And Log In
445 Start Command
446 ... journalctl -f ${filter} > ${file_path}-${LOG_TIME}
447 Log Journal Log Started: ${file_path}-${LOG_TIME}
448
449
450Stop Journal Log
451 [Documentation] Stop journalctl process if its running.
452 ... By default return log from /tmp/journal_log else
453 ... user input location.
454 [Arguments] ${file_path}=/tmp/journal_log
455
456 # Description of arguments:
457 # file_path The file path of the journal file.
458
459 Open Connection And Log In
460
461 ${rc}=
462 ... Execute Command
463 ... ps | grep journalctl | grep -v grep
464 ... return_stdout=False return_rc=True
465
466 Return From Keyword If '${rc}' == '${1}'
467 ... No journal log process running
468
469 ${output} ${stderr}=
470 ... Execute Command killall journalctl
471 ... return_stderr=True
472 Should Be Empty ${stderr}
473
474 ${journal_log} ${stderr}=
475 ... Execute Command
476 ... cat ${file_path}-${LOG_TIME}
477 ... return_stderr=True
478 Should Be Empty ${stderr}
479
480 Log ${journal_log}
481
482 Execute Command rm ${file_path}-${LOG_TIME}
483
484 [Return] ${journal_log}
485
486
487Mac Address To Hex String
488 [Documentation] Converts MAC address into hex format.
489 ... Example
490 ... Given the following MAC: 00:01:6C:80:02:78
491 ... This keyword will return: 0x00 0x01 0x6C 0x80 0x02 0x78
492 ... Description of arguments:
493 ... i_macaddress MAC address in the following format
494 ... 00:01:6C:80:02:78
495 [Arguments] ${i_macaddress}
496
497 # Description of arguments:
498 # i_macaddress The MAC address.
499
500 ${mac_hex}= Catenate 0x${i_macaddress.replace(':', ' 0x')}
501 [Return] ${mac_hex}
502
503
504IP Address To Hex String
505 [Documentation] Converts IP address into hex format.
506 ... Example:
507 ... Given the following IP: 10.3.164.100
508 ... This keyword will return: 0xa 0x3 0xa4 0xa0
509 [Arguments] ${i_ipaddress}
510
511 # Description of arguments:
512 # i_macaddress The IP address in the format 10.10.10.10.
513
514 @{ip}= Split String ${i_ipaddress} .
515 ${index}= Set Variable ${0}
516
517 :FOR ${item} IN @{ip}
518 \ ${hex}= Convert To Hex ${item} prefix=0x lowercase=yes
519 \ Set List Value ${ip} ${index} ${hex}
520 \ ${index}= Set Variable ${index + 1}
521 ${ip_hex}= Catenate @{ip}
522
523 [Return] ${ip_hex}
524
525
526BMC CPU Performance Check
527 [Documentation] Minimal 10% of proc should be free in this instance
528
529 ${bmc_cpu_usage_output} ${stderr} ${rc}= BMC Execute Command
530 ... ${bmc_cpu_usage_cmd}
531 ${bmc_cpu_usage_output} ${stderr} ${rc}= BMC Execute Command
532 ... ${bmc_cpu_usage_cmd}
533 ${bmc_cpu_percentage}= Fetch From Left ${bmc_cpu_usage_output} %
534 Should be true ${bmc_cpu_percentage} < 90
535
536
537BMC Mem Performance Check
538 [Documentation] Minimal 10% of memory should be free in this instance
539
540 ${bmc_mem_free_output} ${stderr} ${rc}= BMC Execute Command
541 ... ${bmc_mem_free_cmd}
542
543 ${bmc_mem_total_output} ${stderr} ${rc}= BMC Execute Command
544 ... ${bmc_mem_total_cmd}
545 ${bmc_mem_free_output} ${stderr} ${rc}= BMC Execute Command
546 ... ${bmc_mem_free_cmd}
547
548 ${bmc_mem_total_output} ${stderr} ${rc}= BMC Execute Command
549 ... ${bmc_mem_total_cmd}
550
551 ${bmc_mem_percentage}= Evaluate ${bmc_mem_free_output}*100
552 ${bmc_mem_percentage}= Evaluate
553 ... ${bmc_mem_percentage}/${bmc_mem_total_output}
554 Should be true ${bmc_mem_percentage} > 10
555
556
557BMC File System Usage Check
558 [Documentation] Check the file system space. 4 file system should be
559 ... 100% full which is expected
560 # Filesystem Size Used Available Use% Mounted on
561 # /dev/root 14.4M 14.4M 0 100% /
562 # /dev/ubiblock0_0 14.4M 14.4M 0 100% /media/rofs-c9249b0e
563 # /dev/ubiblock8_0 19.6M 19.6M 0 100% /media/pnor-ro-8764baa3
564 # /dev/ubiblock4_0 14.4M 14.4M 0 100% /media/rofs-407816c
565 # /dev/ubiblock8_4 21.1M 21.1M 0 100% /media/pnor-ro-cecc64c4
566 ${bmc_fs_usage_output} ${stderr} ${rc}= BMC Execute Command
567 ... ${bmc_file_system_usage_cmd}
568 ${bmc_pnor_fs_usage_output} ${stderr} ${rc}= BMC Execute Command
569 ... ${total_pnor_ro_file_system_cmd}
570 ${bmc_bmc_fs_usage_output} ${stderr} ${rc}= BMC Execute Command
571 ... ${total_bmc_ro_file_system_cmd}
572 ${total_bmc_pnor_image}= Evaluate
573 ... ${bmc_pnor_fs_usage_output}+${bmc_bmc_fs_usage_output}
574 # Considering /dev/root also in total 100% used file system
575 ${total_full_fs}= Evaluate ${total_bmc_pnor_image}+1
576 Should Be True ${bmc_fs_usage_output}==${total_full_fs}
577
578
579Check BMC CPU Performance
580 [Documentation] Minimal 10% of proc should be free in 3 sample
George Keishing9a089442018-12-10 09:38:50 -0600581 :FOR ${var} IN RANGE 1 4
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600582 \ BMC CPU Performance check
583
584
585Check BMC Mem Performance
586 [Documentation] Minimal 10% of memory should be free
587
George Keishing9a089442018-12-10 09:38:50 -0600588 :FOR ${var} IN RANGE 1 4
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600589 \ BMC Mem Performance check
590
591
592Check BMC File System Performance
593 [Documentation] Check for file system usage for 4 times
594
George Keishing9a089442018-12-10 09:38:50 -0600595 :FOR ${var} IN RANGE 1 4
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600596 \ BMC File System Usage check
597
598
599Get URL List
600 [Documentation] Return list of URLs under given URL.
601 [Arguments] ${openbmc_url}
602
603 # Description of argument(s):
604 # openbmc_url URL for list operation (e.g.
605 # /xyz/openbmc_project/inventory).
606
Steven Sombaraaaab222018-12-19 13:16:23 -0600607 ${url_list}= Read Properties ${openbmc_url}list quiet=${1}
Steven Sombarf60cbcf2018-12-07 08:12:18 -0600608 Sort List ${url_list}
609
610 [Return] ${url_list}
611
612
613Check Zombie Process
614 [Documentation] Check if any defunct process exist or not on BMC
615 ${count} ${stderr} ${rc}= Execute Command ps -o stat | grep Z | wc -l
616 ... return_stderr=True return_rc=True
617 Should Be True ${count}==0
618 Should Be Empty ${stderr}
619
620
621Prune Journal Log
622 [Documentation] Prune archived journal logs.
623 [Arguments] ${vacuum_size}=1M
624
625 # This keyword can be used to prevent the journal
626 # log from filling up the /run filesystem.
627 # This command will retain only the latest logs
628 # of the user specified size.
629
630 # Description of argument(s):
631 # vacuum_size Size of journal.
632
633 Open Connection And Log In
634 ${output} ${stderr} ${rc}=
635 ... Execute Command
636 ... journalctl --vacuum-size=${vacuum_size}
637 ... return_stderr=True return_rc=True
638
639 Should Be Equal ${rc} ${0} msg=${stderr}
640
641
642Get System Power Policy
643 [Documentation] Returns the BMC power policy.
644
645 # Set the bmc_power_policy_method to either 'Old' or 'New'.
646 Set Power Policy Method
647 ${cmd_buf}= Create List ${bmc_power_policy_method} Get Power Policy
648 # Run the appropriate keyword.
649 ${currentPolicy}= Run Keyword @{cmd_buf}
650
651 [Return] ${currentPolicy}
652
653
654Set BMC Reset Reference Time
655 [Documentation] Set current boot time as a reference and increment
656 ... boot count.
657
658 ${cur_btime}= Get BMC Boot Time
659 Run Keyword If ${BOOT_TIME} == ${0} and ${BOOT_COUNT} == ${0}
660 ... Set Global Variable ${BOOT_TIME} ${cur_btime}
661 ... ELSE IF ${cur_btime} > ${BOOT_TIME}
662 ... Run Keywords Set Global Variable ${BOOT_TIME} ${cur_btime}
663 ... AND
664 ... Set Global Variable ${BOOT_COUNT} ${BOOT_COUNT + 1}
665
666
667Get BMC Boot Time
668 [Documentation] Returns boot time from /proc/stat.
669
670 Open Connection And Log In
671 ${output} ${stderr}=
672 ... Execute Command egrep '^btime ' /proc/stat | cut -f 2 -d ' '
673 ... return_stderr=True
674 Should Be Empty ${stderr}
675 ${btime}= Convert To Integer ${output}
676 [Return] ${btime}
677
678
679Enable Core Dump On BMC
680 [Documentation] Enable core dump collection.
681 ${core_pattern} ${stderr} ${rc}= BMC Execute Command
682 ... echo '/tmp/core_%e.%p' | tee /proc/sys/kernel/core_pattern
683 Should Be Equal As Strings ${core_pattern} /tmp/core_%e.%p
684
685
686Get Number Of BMC Core Dump Files
687 [Documentation] Returns number of core dump files on BMC.
688 Open Connection And Log In
689 ${num_of_core_dump}= Execute Command
690 ... ls /tmp/core* 2>/dev/null | wc -l
691 [Return] ${num_of_core_dump}
692
693
694Set Core Dump File Size Unlimited
695 [Documentation] Set core dump file size to unlimited.
696 BMC Execute Command ulimit -c unlimited
697
698
699Check For Core Dumps
700 [Documentation] Check for any core dumps exist.
701 ${output}= Get Number Of BMC Core Dump Files
702 Run Keyword If ${output} > 0
703 ... Log **Warning** BMC core dump files exist level=WARN
704
705
706Configure Initial Settings
707 [Documentation] Restore old IP and route.
708 ... This keyword requires initial settings viz IP address,
709 ... Network Mask, default gatway and serial console IP and port
710 ... information which should be provided in command line.
711
712 [Arguments] ${host}=${OPENBMC_HOST} ${mask}=${NET_MASK}
713 ... ${gw_ip}=${GW_IP}
714
715 # Description of arguments:
716 # host IP address of the OS Host.
717 # mask Network mask.
718 # gu_ip Gateway IP address or hostname.
719
720 # Open telnet connection and ignore the error, in case telnet session is
721 # already opened by the program calling this keyword.
722 Run Keyword And Ignore Error Open Telnet Connection to BMC Serial Console
723 Telnet.write ifconfig eth0 ${host} netmask ${mask}
724 Telnet.write route add default gw ${gw_ip}
725
726
727Install Debug Tarball On BMC
728 [Documentation] Copy the debug tar file to BMC and install.
729 [Arguments] ${tarball_file_path}=${default_tarball}
730 ... ${targ_tarball_dir_path}=/tmp/tarball/
731
732 # Description of arguments:
733 # tarball_file_path Path of the debug tarball file.
734 # The tar file is downloaded from the build page
735 # https://openpower.xyz/job/openbmc-build/
736 # obmc-phosphor-debug-tarball-witherspoon.tar.xz
737 #
738 # targ_tarball_dir_path The directory path where the tarball is to be
739 # installed.
740
741 OperatingSystem.File Should Exist ${tarball_file_path}
742 ... msg=${tarball_file_path} doesn't exist.
743
744 # Upload the file to BMC.
745 Import Library SCPLibrary WITH NAME scp
746 Open Connection for SCP
747 scp.Put File ${tarball_file_path} /tmp/debug-tarball.tar.xz
748
749 # Create tarball directory and install.
750 BMC Execute Command mkdir -p ${targ_tarball_dir_path}
751 BMC Execute Command
752 ... tar -xf /tmp/debug-tarball.tar.xz -C ${targ_tarball_dir_path}
753
754 # Remove the tarball file from BMC.
755 BMC Execute Command rm -f /tmp/debug-tarball.tar.xz
756
757
758Get BMC Boot Count
759 [Documentation] Returns BMC boot count based on boot time.
760 ${cur_btime}= Get BMC Boot Time
761
762 # Set global variable BOOT_TIME to current boot time if current boot time
763 # is changed. Also increase value of global variable BOOT_COUNT by 1.
764 Run Keyword If ${cur_btime} > ${BOOT_TIME}
765 ... Run Keywords Set Global Variable ${BOOT_TIME} ${cur_btime}
766 ... AND
767 ... Set Global Variable ${BOOT_COUNT} ${BOOT_COUNT + 1}
768
769 [Return] ${BOOT_COUNT}
770
771
772Set BMC Boot Count
773 [Documentation] Set BMC boot count to given value.
774 [Arguments] ${count}
775
776 # Description of arguments:
777 # count boot count value.
778 ${cur_btime}= Get BMC Boot Time
779
780 # Set global variable BOOT_COUNT to given value.
781 Set Global Variable ${BOOT_COUNT} ${count}
782
783 # Set BOOT_TIME variable to current boot time.
784 Set Global Variable ${BOOT_COUNT} ${count}
785
786
787Delete Error Log Entry
788 [Documentation] Delete error log entry.
789 [Arguments] ${entry_path}
790
791 # Description of argument(s):
792 # entry_path Delete an error log entry.
793 # Ex. /xyz/openbmc_project/logging/entry/1
794
795 # Skip delete if entry URI is a callout.
796 # Examples:
797 # /xyz/openbmc_project/logging/entry/1/callout
798 # /xyz/openbmc_project/logging/entry/1/callouts/0
799 ${callout_entry}= Run Keyword And Return Status
800 ... Should Match Regexp ${entry_path} /callout[s]?(/|$)
801 Return From Keyword If ${callout_entry}
802
803 ${data}= Create Dictionary data=@{EMPTY}
804 ${resp}= Openbmc Delete Request ${entry_path} data=${data}
805 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
806
807
808Get BMC Version
809 [Documentation] Returns BMC version from /etc/os-release.
810 ... e.g. "v1.99.6-141-ge662190"
811
812 ${cmd}= Set Variable grep ^VERSION_ID= /etc/os-release | cut -f 2 -d '='
813 ${output} ${stderr} ${rc}= BMC Execute Command ${cmd}
814 [Return] ${output}
815
816
817Get PNOR Version
818 [Documentation] Returns the PNOR version from the BMC.
819
820 ${pnor_attrs}= Get PNOR Attributes
821 [Return] ${pnor_attrs['version']}
822
823
824Get PNOR Attributes
825 [Documentation] Return PNOR software attributes as a dictionary.
826
827 # This keyword parses /var/lib/phosphor-software-manager/pnor/ro/pnor.toc
828 # into key/value pairs.
829
830 ${outbuf} ${stderr} ${rc}= BMC Execute Command
831 ... cat /var/lib/phosphor-software-manager/pnor/ro/pnor.toc
832 ${pnor_attrs}= Key Value Outbuf To Dict ${outbuf} delim==
833
834 [Return] ${pnor_attrs}
835
836
837Copy Address Translation Utils To HOST OS
838 [Documentation] Copy address translation utils to host OS.
839
840 OperatingSystem.File Should Exist ${probe_cpu_tool_path}
841 ... msg=${probe_cpu_tool_path} doesn't exist.
842 OperatingSystem.File Should Exist ${probe_cpu_tool_path}
843 ... msg=${probe_cpu_tool_path} doesn't exist.
844
845 scp.Open connection ${OS_HOST} username=${OS_USERNAME}
846 ... password=${OS_PASSWORD}
847 scp.Put File ${probe_cpu_tool_path} ${target_file_path}
848 scp.Put File ${scom_addrs_tool_path} ${target_file_path}
849
850
851Verify BMC RTC And UTC Time Drift
852 [Documentation] Verify that the RTC and UTC time difference is less than
853 ... the given time_drift_max.
854 [Arguments] ${time_diff_max}=${10}
855
856 # Description of argument(s):
857 # time_diff_max The max allowable RTC and UTC time difference in seconds.
858
859 # Example:
860 # time_dict:
861 # [local_time]: Fri 2017-11-03 152756 UTC
862 # [local_time_seconds]: 1509740876
863 # [universal_time]: Fri 2017-11-03 152756 UTC
864 # [universal_time_seconds]: 1509740876
865 # [rtc_time]: Fri 2016-05-20 163403
866 # [rtc_time_seconds]: 1463780043
867 # [time_zone]: n/a (UTC, +0000)
868 # [network_time_on]: yes
869 # [ntp_synchronized]: no
870 # [rtc_in_local_tz]: no
871
872 ${time}= Get BMC Date Time
873 ${time_diff}= Evaluate
874 ... ${time['universal_time_seconds']} - ${time['rtc_time_seconds']}
875 Should Be True ${time_diff} < ${time_diff_max}
876
877
878Validate IP On BMC
879 [Documentation] Validate IP address is present in set of IP addresses.
880 [Arguments] ${ip_address} ${ip_data}
881
882 # Description of argument(s):
883 # ip_address IP address to check (e.g. xx.xx.xx.xx).
884 # ip_data Set of the IP addresses present.
885
886 Should Contain Match ${ip_data} ${ip_address}/*
887 ... msg=${ip_address} not found in the list provided.
888
889
890Remove Journald Logs
891 [Documentation] Remove all journald logs and restart service.
892
893 ${cmd}= Catenate systemctl stop systemd-journald.service &&
894 ... rm -rf /var/log/journal && systemctl start systemd-journald.service
895
896 BMC Execute Command ${cmd}
897
898
899Check For Regex In Journald
900 [Documentation] Parse the journal log and check for regex string.
901 [Arguments] ${regex}=${ERROR_REGEX} ${error_check}=${0} ${boot}=${EMPTY}
902
903 # Description of argument(s):
904 # regex Strings to be filter.
905 # error_check Check for errors.
906 # boot Argument to check current or persistent full boot log
907 # (e.g. "-b").
908
909 ${journal_log} ${stderr} ${rc}= BMC Execute Command
910 ... journalctl --no-pager ${boot} | egrep '${regex}' ignore_err=1
911
912 Run Keyword If ${error_check} == ${0}
913 ... Should Be Empty ${journal_log}
914 ... ELSE
915 ... Should Not Be Empty ${journal_log}
916
917
918Get Service Attribute
919 [Documentation] Get service attribute policy output.
920 [Arguments] ${option} ${servicename}
921
922 # Description of argument(s):
923 # option systemctl supported options
924 # servicename Qualified service name
925 ${cmd}= Set Variable
926 ... systemctl -p ${option} show ${servicename} | cut -d = -f2
927 ${attr} ${stderr} ${rc}= BMC Execute Command ${cmd}
928 [Return] ${attr}