blob: e550491cbaf6a70500e96b8518ca91c993887d98 [file] [log] [blame]
Chris Austenb29d2e82016-06-07 12:25:35 -05001*** Settings ***
Steven Sombar0acb3412018-01-24 09:38:42 -06002
3Documentation Utilities for Robot keywords.
4
Chris Austenb29d2e82016-06-07 12:25:35 -05005Resource ../lib/resource.txt
6Resource ../lib/rest_client.robot
George Keishing5e870cd2016-08-24 10:05:47 -05007Resource ../lib/connection_client.robot
Michael Walsh391ba9b2017-08-24 11:36:24 -05008Library String
George Keishing30c12ff2016-09-02 10:25:29 -05009Library DateTime
10Library Process
Chris Austenb29d2e82016-06-07 12:25:35 -050011Library OperatingSystem
Michael Walsha6723f22016-11-22 11:12:01 -060012Library gen_print.py
13Library gen_robot_print.py
Michael Walsh5f3f4142017-05-22 17:09:47 -050014Library gen_cmd.py
Sunil Macd110a2017-05-23 04:14:32 -050015Library gen_robot_keyword.py
Michael Walshe53e47a2017-06-30 17:03:24 -050016Library bmc_ssh_utils.py
Michael Walshfdc5ced2017-08-17 13:15:15 -050017Library utils.py
Charles Paul Hofer35816152017-10-02 11:55:11 -050018Library var_funcs.py
Sridevi Ramesh0d88ab32017-09-21 11:07:28 -050019Library SCPLibrary WITH NAME scp
Chris Austenb29d2e82016-06-07 12:25:35 -050020
21*** Variables ***
Steven Sombar0acb3412018-01-24 09:38:42 -060022
23${pflash_cmd} /usr/sbin/pflash -r /dev/stdout -P VERSION
24${SYSTEM_SHUTDOWN_TIME} ${5}
25
Michael Walsha6723f22016-11-22 11:12:01 -060026${dbuscmdBase}
George Keishing85ca05e2016-11-30 09:47:18 -060027... dbus-send --system --print-reply --dest=${OPENBMC_BASE_DBUS}.settings.Host
Michael Walsha6723f22016-11-22 11:12:01 -060028${dbuscmdGet}
George Keishing9485af92017-01-04 10:21:43 -060029... ${SETTINGS_URI}host0 org.freedesktop.DBus.Properties.Get
Steven Sombar0acb3412018-01-24 09:38:42 -060030${dbuscmdString}= string:"xyz.openbmc_project.settings.Host" string:
Michael Walsha6723f22016-11-22 11:12:01 -060031
32# Assign default value to QUIET for programs which may not define it.
33${QUIET} ${0}
Steven Sombar0acb3412018-01-24 09:38:42 -060034
Sivas SRRea85d1f2016-11-13 22:44:28 -060035${bmc_mem_free_cmd}= free | tr -s ' ' | sed '/^Mem/!d' | cut -d" " -f4
36${bmc_mem_total_cmd}= free | tr -s ' ' | sed '/^Mem/!d' | cut -d" " -f2
37${bmc_cpu_usage_cmd}= top -n 1 | grep CPU: | cut -c 7-9
Sridevi Ramesh1699d372016-12-06 00:20:22 -060038${HOST_SETTING} ${SETTINGS_URI}host0
Steven Sombar0acb3412018-01-24 09:38:42 -060039
Sivas SRR352bc162017-02-13 22:12:29 -060040# /run/initramfs/ro associate filesystem should be 100% full always
Sivas SRR5ff4d692017-11-23 02:53:52 -060041${bmc_file_system_usage_cmd}= df -h | cut -c 52-54 | grep 100 | wc -l
42${total_pnor_ro_file_system_cmd}= df -h | grep /media/pnor-ro | wc -l
43${total_bmc_ro_file_system_cmd}= df -h | grep /media/rofs | wc -l
Chris Austenb29d2e82016-06-07 12:25:35 -050044
George Keishingcce9df22017-01-24 06:19:33 -060045${BOOT_TIME} ${0}
46${BOOT_COUNT} ${0}
Sweta Potthuri025e0122017-02-21 00:42:48 -060047${count} ${0}
Sivas SRR8254db62017-02-07 09:39:46 -060048${devicetree_base} /sys/firmware/devicetree/base/model
George Keishingcce9df22017-01-24 06:19:33 -060049
George Keishingb7c9b542017-05-23 08:19:58 -050050# Initialize default debug value to 0.
51${DEBUG} ${0}
52
Steven Sombar0acb3412018-01-24 09:38:42 -060053${probe_cpu_tool_path} ${EXECDIR}/tools/ras/probe_cpus.sh
54${scom_addrs_tool_path} ${EXECDIR}/tools/ras/scom_addr_p9.sh
55${target_file_path} /root/
56
57${default_tarball} ${EXECDIR}/obmc-phosphor-debug-tarball-witherspoon.tar.xz
58
Michael Walshfdc5ced2017-08-17 13:15:15 -050059# These variables are used to straddle between new and old methods of setting
60# values.
Steven Sombar0acb3412018-01-24 09:38:42 -060061${boot_prog_method} ${EMPTY}
Michael Walshfdc5ced2017-08-17 13:15:15 -050062${power_policy_setup} ${0}
63${bmc_power_policy_method} ${EMPTY}
64@{valid_power_policy_vars} RESTORE_LAST_STATE ALWAYS_POWER_ON
65... ALWAYS_POWER_OFF
66
Sridevi Ramesh0d88ab32017-09-21 11:07:28 -050067
Chris Austenb29d2e82016-06-07 12:25:35 -050068*** Keywords ***
Sridevi Ramesh1699d372016-12-06 00:20:22 -060069
Sivas SRRc0957412017-10-06 13:02:08 -050070Check BMC Performance
71 [Documentation] Check BMC basic CPU Mem File system performance.
72
73 Check BMC CPU Performance
74 Check BMC Mem Performance
75 Check BMC File System Performance
76
Steven Sombar0acb3412018-01-24 09:38:42 -060077
Sivas SRRd97aa9c2017-05-22 11:19:25 -050078Verify PNOR Update
79 [Documentation] Verify that the PNOR is not corrupted.
80 # Example:
81 # FFS: Flash header not found. Code: 100
82 # Error 100 opening ffs !
83
84 Open Connection And Log In
85 ${pnor_info}= Execute Command On BMC ${pflash_cmd}
86 Should Not Contain Any ${pnor_info} Flash header not found Error
87
Steven Sombar0acb3412018-01-24 09:38:42 -060088
Sivas SRR8254db62017-02-07 09:39:46 -060089Get BMC System Model
Steven Sombar0acb3412018-01-24 09:38:42 -060090 [Documentation] Get the BMC model from the device tree and return it.
Sivas SRR8254db62017-02-07 09:39:46 -060091
Michael Walshe53e47a2017-06-30 17:03:24 -050092 ${bmc_model} ${stderr} ${rc}= BMC Execute Command
George Keishing90ab1772017-02-24 05:40:55 -060093 ... cat ${devicetree_base} | cut -d " " -f 1 return_stderr=True
Michael Walsh71182ab2017-10-20 14:44:25 -050094 ... test_mode=0
Sivas SRR8254db62017-02-07 09:39:46 -060095 Should Be Empty ${stderr}
Steven Sombar0acb3412018-01-24 09:38:42 -060096 Should Not Be Empty ${bmc_model} msg=BMC model is empty.
Sivas SRR8254db62017-02-07 09:39:46 -060097 [Return] ${bmc_model}
98
Steven Sombar0acb3412018-01-24 09:38:42 -060099
Sivas SRR8254db62017-02-07 09:39:46 -0600100Verify BMC System Model
101 [Documentation] Verify the BMC model with ${OPENBMC_MODEL}.
102 [Arguments] ${bmc_model}
103
Steven Sombar0acb3412018-01-24 09:38:42 -0600104 # Description of argument(s):
105 # bmc_model System model (e.g. "witherspoon").
106
Sivas SRR8254db62017-02-07 09:39:46 -0600107 ${tmp_bmc_model}= Fetch From Right ${OPENBMC_MODEL} /
108 ${tmp_bmc_model}= Fetch From Left ${tmp_bmc_model} .
109 ${ret}= Run Keyword And Return Status Should Contain ${bmc_model}
110 ... ${tmp_bmc_model} ignore_case=True
111 [Return] ${ret}
112
Steven Sombar0acb3412018-01-24 09:38:42 -0600113
Chris Austenb29d2e82016-06-07 12:25:35 -0500114Wait For Host To Ping
root442f0ef2016-08-04 20:23:05 +0000115 [Arguments] ${host} ${timeout}=${OPENBMC_REBOOT_TIMEOUT}min
116 ... ${interval}=5 sec
117
Steven Sombar0acb3412018-01-24 09:38:42 -0600118 # Description of argument(s):
119 # host The host name or IP of the host to ping.
120 # timeout The amount of time after which ping attempts cease.
121 # This should be expressed in Robot Framework's time format
122 # (e.g. "10 seconds").
root442f0ef2016-08-04 20:23:05 +0000123 # interval The amount of time in between attempts to ping.
Steven Sombar0acb3412018-01-24 09:38:42 -0600124 # This should be expressed in Robot Framework's time format
125 # (e.g. "5 seconds").
root442f0ef2016-08-04 20:23:05 +0000126
127 Wait Until Keyword Succeeds ${timeout} ${interval} Ping Host ${host}
Chris Austenb29d2e82016-06-07 12:25:35 -0500128
Steven Sombar0acb3412018-01-24 09:38:42 -0600129
Chris Austenb29d2e82016-06-07 12:25:35 -0500130Ping Host
131 [Arguments] ${host}
Steven Sombar0acb3412018-01-24 09:38:42 -0600132
133 # Description of argument(s):
134 # host The host name or IP of the host to ping.
135
George Keishing8a84f952016-08-25 04:54:53 -0500136 Should Not Be Empty ${host} msg=No host provided
Michael Walsha6723f22016-11-22 11:12:01 -0600137 ${RC} ${output}= Run and return RC and Output ping -c 4 ${host}
Chris Austenb29d2e82016-06-07 12:25:35 -0500138 Log RC: ${RC}\nOutput:\n${output}
139 Should be equal ${RC} ${0}
140
Steven Sombar0acb3412018-01-24 09:38:42 -0600141
Chris Austenb29d2e82016-06-07 12:25:35 -0500142Get Boot Progress
Michael Walshc972d4d2017-08-16 15:38:03 -0500143 [Documentation] Get the boot progress and return it.
Michael Walsha6723f22016-11-22 11:12:01 -0600144 [Arguments] ${quiet}=${QUIET}
145
Michael Walshc972d4d2017-08-16 15:38:03 -0500146 # Description of argument(s):
147 # quiet Indicates whether this keyword should run without any output to
148 # the console.
149
150 Set Boot Progress Method
151 ${state}= Run Keyword If '${boot_prog_method}' == 'New'
152 ... New Get Boot Progress quiet=${quiet}
153 ... ELSE
154 ... Old Get Boot Progress quiet=${quiet}
155
156 [Return] ${state}
157
Steven Sombar0acb3412018-01-24 09:38:42 -0600158
Michael Walshc972d4d2017-08-16 15:38:03 -0500159Set Boot Progress Method
160 [Documentation] Set the boot_prog_method to either 'Old' or 'New'.
161
162 # The boot progress data has moved from an 'org' location to an 'xyz'
163 # location. This keyword will determine whether the new method of getting
164 # the boot progress is valid and will set the global boot_prog_method
165 # variable accordingly. If boot_prog_method is already set (either by a
166 # prior call to this function or via a -v parm), this keyword will simply
167 # return.
168
Michael Walsh98d8c9e2017-10-04 12:24:29 -0500169 # Note: There are interim builds that contain boot_progress in both the
170 # old and the new location values. It is nearly impossible for this
171 # keyword to determine whether the old boot_progress or the new one is
172 # active. When using such builds where the old boot_progress is active,
173 # the only recourse users will have is that they may specify
174 # -v boot_prog_method:Old to force old behavior on such builds.
Michael Walshc972d4d2017-08-16 15:38:03 -0500175
176 Run Keyword If '${boot_prog_method}' != '${EMPTY}' Return From Keyword
177
178 ${new_status} ${new_value}= Run Keyword And Ignore Error
179 ... New Get Boot Progress
180 # If the new style read fails, the method must necessarily be "Old".
181 Run Keyword If '${new_status}' == 'PASS'
182 ... Run Keywords
183 ... Set Global Variable ${boot_prog_method} New AND
184 ... Rqpvars boot_prog_method AND
185 ... Return From Keyword
186
187 # Default method is "Old".
188 Set Global Variable ${boot_prog_method} Old
189 Rqpvars boot_prog_method
190
Steven Sombar0acb3412018-01-24 09:38:42 -0600191
Michael Walshc972d4d2017-08-16 15:38:03 -0500192Old Get Boot Progress
193 [Documentation] Get the boot progress the old way (via org location).
194 [Arguments] ${quiet}=${QUIET}
195
196 # Description of argument(s):
197 # quiet Indicates whether this keyword should run without any output to
198 # the console.
199
George Keishing85ca05e2016-11-30 09:47:18 -0600200 ${state}= Read Attribute ${OPENBMC_BASE_URI}sensors/host/BootProgress
Michael Walsha6723f22016-11-22 11:12:01 -0600201 ... value quiet=${quiet}
Michael Walshc972d4d2017-08-16 15:38:03 -0500202
Gunnar Millsc9ea9362016-12-13 16:21:13 -0600203 [Return] ${state}
Chris Austenb29d2e82016-06-07 12:25:35 -0500204
Steven Sombar0acb3412018-01-24 09:38:42 -0600205
Michael Walshc972d4d2017-08-16 15:38:03 -0500206New Get Boot Progress
207 [Documentation] Get the boot progress the new way (via xyz location).
208 [Arguments] ${quiet}=${QUIET}
209
210 # Description of argument(s):
211 # quiet Indicates whether this keyword should run without any output to
212 # the console.
213
214 ${state}= Read Attribute ${HOST_STATE_URI} BootProgress quiet=${quiet}
215
216 [Return] ${state.rsplit('.', 1)[1]}
217
Steven Sombar0acb3412018-01-24 09:38:42 -0600218
Chris Austenb29d2e82016-06-07 12:25:35 -0500219Is Power On
root442f0ef2016-08-04 20:23:05 +0000220 ${state}= Get Power State
221 Should be equal ${state} ${1}
Chris Austenb29d2e82016-06-07 12:25:35 -0500222
Steven Sombar0acb3412018-01-24 09:38:42 -0600223
Chris Austenb29d2e82016-06-07 12:25:35 -0500224Is Power Off
root442f0ef2016-08-04 20:23:05 +0000225 ${state}= Get Power State
226 Should be equal ${state} ${0}
Chris Austenb29d2e82016-06-07 12:25:35 -0500227
Steven Sombar0acb3412018-01-24 09:38:42 -0600228
root442f0ef2016-08-04 20:23:05 +0000229Initiate Power On
230 [Documentation] Initiates the power on and waits until the Is Power On
231 ... keyword returns that the power state has switched to on.
Michael Walsha6723f22016-11-22 11:12:01 -0600232 [Arguments] ${wait}=${1}
233
Steven Sombar0acb3412018-01-24 09:38:42 -0600234 # Description of argument(s):
235 # wait Indicates whether to wait for a powered on state after issuing
236 # the power on command.
237
Chris Austenb29d2e82016-06-07 12:25:35 -0500238 @{arglist}= Create List
239 ${args}= Create Dictionary data=@{arglist}
George Keishing85ca05e2016-11-30 09:47:18 -0600240 ${resp}= Call Method ${OPENBMC_BASE_URI}control/chassis0/ powerOn
Michael Walsha6723f22016-11-22 11:12:01 -0600241 ... data=${args}
Chris Austenb29d2e82016-06-07 12:25:35 -0500242 should be equal as strings ${resp.status_code} ${HTTP_OK}
Chris Austenb29d2e82016-06-07 12:25:35 -0500243
Michael Walsha6723f22016-11-22 11:12:01 -0600244 # Does caller want to wait for power on status?
245 Run Keyword If '${wait}' == '${0}' Return From Keyword
246 Wait Until Keyword Succeeds 3 min 10 sec Is Power On
Rahul Maheshwarif684ba72016-10-25 07:24:41 -0500247
Steven Sombar0acb3412018-01-24 09:38:42 -0600248
root442f0ef2016-08-04 20:23:05 +0000249Initiate Power Off
250 [Documentation] Initiates the power off and waits until the Is Power Off
251 ... keyword returns that the power state has switched to off.
Steven Sombar0acb3412018-01-24 09:38:42 -0600252
Chris Austenb29d2e82016-06-07 12:25:35 -0500253 @{arglist}= Create List
254 ${args}= Create Dictionary data=@{arglist}
George Keishing85ca05e2016-11-30 09:47:18 -0600255 ${resp}= Call Method ${OPENBMC_BASE_URI}control/chassis0/ powerOff
Michael Walsha6723f22016-11-22 11:12:01 -0600256 ... data=${args}
Chris Austenb29d2e82016-06-07 12:25:35 -0500257 should be equal as strings ${resp.status_code} ${HTTP_OK}
Michael Walsha6723f22016-11-22 11:12:01 -0600258 Wait Until Keyword Succeeds 1 min 10 sec Is Power Off
Chris Austenb29d2e82016-06-07 12:25:35 -0500259
Steven Sombar0acb3412018-01-24 09:38:42 -0600260
Michael Walsh81843772017-03-02 14:44:44 -0600261Initiate OS Host Power Off
262 [Documentation] Initiate an OS reboot.
263 [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME}
Michael Walshdfc75372018-01-11 11:47:25 -0600264 ... ${os_password}=${OS_PASSWORD} ${hard}=${0}
Michael Walsh81843772017-03-02 14:44:44 -0600265
Michael Walshdfc75372018-01-11 11:47:25 -0600266 # Description of argument(s):
Michael Walsh81843772017-03-02 14:44:44 -0600267 # os_host The DNS name or IP of the OS.
268 # os_username The username to be used to sign in to the OS.
269 # os_password The password to be used to sign in to the OS.
Michael Walshdfc75372018-01-11 11:47:25 -0600270 # hard Indicates whether to do a hard vs. soft power off.
271
272 ${time_string}= Run Keyword If ${hard} Set Variable ${SPACE}now
273 ... ELSE Set Variable ${EMPTY}
Michael Walsh81843772017-03-02 14:44:44 -0600274
Michael Walsh4549cd12017-09-07 18:02:36 -0500275 ${cmd_buf}= Run Keyword If '${os_username}' == 'root'
Michael Walshdfc75372018-01-11 11:47:25 -0600276 ... Set Variable shutdown${time_string}
Michael Walsh4549cd12017-09-07 18:02:36 -0500277 ... ELSE
Michael Walshdfc75372018-01-11 11:47:25 -0600278 ... Set Variable echo ${os_password} | sudo -S shutdown${time_string}
Michael Walsh4549cd12017-09-07 18:02:36 -0500279
280 ${output} ${stderr} ${rc}= OS Execute Command
281 ... ${cmd_buf} fork=${1}
Michael Walsh81843772017-03-02 14:44:44 -0600282
Steven Sombar0acb3412018-01-24 09:38:42 -0600283
Michael Walsh81843772017-03-02 14:44:44 -0600284Initiate OS Host Reboot
285 [Documentation] Initiate an OS reboot.
286 [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME}
287 ... ${os_password}=${OS_PASSWORD}
288
Michael Walshf1a59fb2017-09-01 17:07:13 -0500289 # Description of argument(s):
290 # os_host The host name or IP address of the OS.
Michael Walsh81843772017-03-02 14:44:44 -0600291 # os_username The username to be used to sign in to the OS.
292 # os_password The password to be used to sign in to the OS.
293
Michael Walshf1a59fb2017-09-01 17:07:13 -0500294 ${cmd_buf}= Run Keyword If '${os_username}' == 'root'
295 ... Set Variable reboot
296 ... ELSE
297 ... Set Variable echo ${os_password} | sudo -S reboot
298
299 ${output} ${stderr} ${rc}= OS Execute Command
300 ... ${cmd_buf} fork=${1}
Michael Walsh81843772017-03-02 14:44:44 -0600301
Steven Sombar0acb3412018-01-24 09:38:42 -0600302
Michael Walsh81843772017-03-02 14:44:44 -0600303Initiate Auto Reboot
304 [Documentation] Initiate an auto reboot.
Michael Walsh4ffd1a12018-03-14 10:35:44 -0500305 [Arguments] ${milliseconds}=5000
306
307 # Description of argument(s):
308 # milliseconds The number of milliseconds for the watchdog timer.
Michael Walsh81843772017-03-02 14:44:44 -0600309
310 # Set the auto reboot policy.
Sweta Potthuri62a23572017-07-28 04:05:13 -0500311 Set Auto Reboot ${1}
Michael Walsh4ffd1a12018-03-14 10:35:44 -0500312 # Set the watchdog timer.
313 Trigger Host Watchdog Error ${milliseconds}
Michael Walsh81843772017-03-02 14:44:44 -0600314
Steven Sombar0acb3412018-01-24 09:38:42 -0600315
Chris Austenb29d2e82016-06-07 12:25:35 -0500316Trigger Warm Reset
Steven Sombar0acb3412018-01-24 09:38:42 -0600317 [Documentation] Initiate a warm reset.
318
Chris Austenb29d2e82016-06-07 12:25:35 -0500319 log to console "Triggering warm reset"
Michael Walsha6723f22016-11-22 11:12:01 -0600320 ${data}= create dictionary data=@{EMPTY}
George Keishing85ca05e2016-11-30 09:47:18 -0600321 ${resp}= openbmc post request
322 ... ${OPENBMC_BASE_URI}control/bmc0/action/warmReset data=${data}
Chris Austenb29d2e82016-06-07 12:25:35 -0500323 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
George Keishingb3700812016-08-31 03:03:30 -0500324 ${session_active}= Check If warmReset is Initiated
325 Run Keyword If '${session_active}' == '${True}'
326 ... Fail msg=warm reset didn't occur
327
Chris Austenb29d2e82016-06-07 12:25:35 -0500328 Sleep ${SYSTEM_SHUTDOWN_TIME}min
Rahul Maheshwari5f253c42017-01-30 05:19:51 -0600329 Check If BMC Is Up
Michael Walsh49ab0f42016-07-20 11:44:33 -0500330
Steven Sombar0acb3412018-01-24 09:38:42 -0600331
Michael Walsh49ab0f42016-07-20 11:44:33 -0500332Check OS
root442f0ef2016-08-04 20:23:05 +0000333 [Documentation] Attempts to ping the host OS and then checks that the host
334 ... OS is up by running an SSH command.
Michael Walsh49ab0f42016-07-20 11:44:33 -0500335
336 [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME}
Michael Walsha6723f22016-11-22 11:12:01 -0600337 ... ${os_password}=${OS_PASSWORD} ${quiet}=${QUIET}
338 ... ${print_string}=${EMPTY}
Michael Walshac275512017-03-07 11:39:28 -0600339 [Teardown] SSHLibrary.Close Connection
Michael Walsh49ab0f42016-07-20 11:44:33 -0500340
Steven Sombar0acb3412018-01-24 09:38:42 -0600341 # Description of argument(s):
Michael Walsh49ab0f42016-07-20 11:44:33 -0500342 # os_host The DNS name/IP of the OS host associated with our BMC.
343 # os_username The username to be used to sign on to the OS host.
344 # os_password The password to be used to sign on to the OS host.
Michael Walsha6723f22016-11-22 11:12:01 -0600345 # quiet Indicates whether this keyword should write to console.
346 # print_string A string to be printed before checking the OS.
347
348 rprint ${print_string}
Michael Walsh49ab0f42016-07-20 11:44:33 -0500349
root442f0ef2016-08-04 20:23:05 +0000350 # Attempt to ping the OS. Store the return code to check later.
351 ${ping_rc}= Run Keyword and Return Status Ping Host ${os_host}
352
Michael Walshac275512017-03-07 11:39:28 -0600353 SSHLibrary.Open connection ${os_host}
Michael Walsh49ab0f42016-07-20 11:44:33 -0500354
Michael Walsha6723f22016-11-22 11:12:01 -0600355 ${status} ${msg}= Run Keyword And Ignore Error Login ${os_username}
356 ... ${os_password}
357 ${err_msg1}= Sprint Error ${msg}
358 ${err_msg}= Catenate SEPARATOR= \n ${err_msg1}
359 Run Keyword If '${status}' == 'FAIL' Fail msg=${err_msg}
Michael Walsh49ab0f42016-07-20 11:44:33 -0500360 ${output} ${stderr} ${rc}= Execute Command uptime return_stderr=True
361 ... return_rc=True
362
Michael Walsha6723f22016-11-22 11:12:01 -0600363 ${temp_msg}= Catenate Could not execute a command on the operating
364 ... system.\n
365 ${err_msg1}= Sprint Error ${temp_msg}
366 ${err_msg}= Catenate SEPARATOR= \n ${err_msg1}
367
368 # If the return code returned by "Execute Command" is non-zero, this
369 # keyword will fail.
370 Should Be Equal ${rc} ${0} msg=${err_msg}
Michael Walsh49ab0f42016-07-20 11:44:33 -0500371 # We will likewise fail if there is any stderr data.
372 Should Be Empty ${stderr}
373
Michael Walsha6723f22016-11-22 11:12:01 -0600374 ${temp_msg}= Set Variable Could not ping the operating system.\n
375 ${err_msg1}= Sprint Error ${temp_msg}
376 ${err_msg}= Catenate SEPARATOR= \n ${err_msg1}
377 # We will likewise fail if the OS did not ping, as we could SSH but not
378 # ping
379 Should Be Equal As Strings ${ping_rc} ${TRUE} msg=${err_msg}
root442f0ef2016-08-04 20:23:05 +0000380
Steven Sombar0acb3412018-01-24 09:38:42 -0600381
Michael Walsh49ab0f42016-07-20 11:44:33 -0500382Wait for OS
383 [Documentation] Waits for the host OS to come up via calls to "Check OS".
384 [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME}
385 ... ${os_password}=${OS_PASSWORD} ${timeout}=${OS_WAIT_TIMEOUT}
Michael Walsha6723f22016-11-22 11:12:01 -0600386 ... ${quiet}=${0}
387 [Teardown] rprintn
Michael Walsh49ab0f42016-07-20 11:44:33 -0500388
Steven Sombar0acb3412018-01-24 09:38:42 -0600389 # Description of argument(s):
Michael Walsh49ab0f42016-07-20 11:44:33 -0500390 # os_host The DNS name or IP of the OS host associated with our
391 # BMC.
392 # os_username The username to be used to sign on to the OS host.
393 # os_password The password to be used to sign on to the OS host.
394 # timeout The timeout in seconds indicating how long you're
395 # willing to wait for the OS to respond.
Michael Walsha6723f22016-11-22 11:12:01 -0600396 # quiet Indicates whether this keyword should write to console.
Michael Walsh49ab0f42016-07-20 11:44:33 -0500397
398 # The interval to be used between calls to "Check OS".
399 ${interval}= Set Variable 5
400
Michael Walsha6723f22016-11-22 11:12:01 -0600401 ${message}= Catenate Checking every ${interval} seconds for up to
402 ... ${timeout} seconds for the operating system to communicate.
403 rqprint_timen ${message}
404
Michael Walsh49ab0f42016-07-20 11:44:33 -0500405 Wait Until Keyword Succeeds ${timeout} sec ${interval} Check OS
406 ... ${os_host} ${os_username} ${os_password}
Michael Walsha6723f22016-11-22 11:12:01 -0600407 ... print_string=\#
408
409 rqprintn
410
411 rqprint_timen The operating system is now communicating.
root442f0ef2016-08-04 20:23:05 +0000412
root442f0ef2016-08-04 20:23:05 +0000413
414Get Power State
415 [Documentation] Returns the power state as an integer. Either 0 or 1.
Michael Walsha6723f22016-11-22 11:12:01 -0600416 [Arguments] ${quiet}=${QUIET}
417
Steven Sombar0acb3412018-01-24 09:38:42 -0600418 # Description of argument(s):
419 # quiet Indicates whether this keyword should run without any output to
420 # the console.
421
root442f0ef2016-08-04 20:23:05 +0000422 @{arglist}= Create List
423 ${args}= Create Dictionary data=@{arglist}
Michael Walsha6723f22016-11-22 11:12:01 -0600424
George Keishing85ca05e2016-11-30 09:47:18 -0600425 ${resp}= Call Method ${OPENBMC_BASE_URI}control/chassis0/ getPowerState
Michael Walsha6723f22016-11-22 11:12:01 -0600426 ... data=${args} quiet=${quiet}
root442f0ef2016-08-04 20:23:05 +0000427 Should be equal as strings ${resp.status_code} ${HTTP_OK}
428 ${content}= to json ${resp.content}
Steven Sombar0acb3412018-01-24 09:38:42 -0600429
Gunnar Millsc9ea9362016-12-13 16:21:13 -0600430 [Return] ${content["data"]}
Jay Azurine4c52eb2016-08-16 20:51:10 -0500431
George Keishinga8d68e22017-12-18 09:30:30 -0600432
433Clear BMC Gard Record
434 [Documentation] Clear gard records from the system.
435
436 @{arglist}= Create List
437 ${args}= Create Dictionary data=@{arglist}
George Keishing85ca05e2016-11-30 09:47:18 -0600438 ${resp}= Call Method
George Keishinga8d68e22017-12-18 09:30:30 -0600439 ... ${OPENPOWER_CONTROL}gard Reset data=${args}
440 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
441
Jay Azurine4c52eb2016-08-16 20:51:10 -0500442
443Copy PNOR to BMC
444 Import Library SCPLibrary WITH NAME scp
445 Open Connection for SCP
446 Log Copying ${PNOR_IMAGE_PATH} to /tmp
447 scp.Put File ${PNOR_IMAGE_PATH} /tmp
448
Steven Sombar0acb3412018-01-24 09:38:42 -0600449
Jay Azurine4c52eb2016-08-16 20:51:10 -0500450Flash PNOR
451 [Documentation] Calls flash bios update method to flash PNOR image
Gunnar Mills38032802016-12-12 13:43:40 -0600452 [Arguments] ${pnor_image}
Steven Sombar0acb3412018-01-24 09:38:42 -0600453
454 # Description of argument(s):
455 # pnor_image The filename and path of the PNOR image
456 # (e.g. "/home/image/zaius.pnor").
457
Jay Azurine4c52eb2016-08-16 20:51:10 -0500458 @{arglist}= Create List ${pnor_image}
459 ${args}= Create Dictionary data=@{arglist}
George Keishinge24fdbc2017-10-31 13:41:35 -0500460 ${resp}= Call Method /org/openbmc/control/flash/bios/ update
Michael Walsha6723f22016-11-22 11:12:01 -0600461 ... data=${args}
Jay Azurine4c52eb2016-08-16 20:51:10 -0500462 should be equal as strings ${resp.status_code} ${HTTP_OK}
463 Wait Until Keyword Succeeds 2 min 10 sec Is PNOR Flashing
464
Steven Sombar0acb3412018-01-24 09:38:42 -0600465
Jay Azurine4c52eb2016-08-16 20:51:10 -0500466Get Flash BIOS Status
467 [Documentation] Returns the status of the flash BIOS API as a string. For
468 ... example 'Flashing', 'Flash Done', etc
George Keishinge24fdbc2017-10-31 13:41:35 -0500469 ${data}= Read Properties /org/openbmc/control/flash/bios
Gunnar Millsc9ea9362016-12-13 16:21:13 -0600470 [Return] ${data['status']}
Jay Azurine4c52eb2016-08-16 20:51:10 -0500471
Steven Sombar0acb3412018-01-24 09:38:42 -0600472
Jay Azurine4c52eb2016-08-16 20:51:10 -0500473Is PNOR Flashing
474 [Documentation] Get BIOS 'Flashing' status. This indicates that PNOR
475 ... flashing has started.
476 ${status}= Get Flash BIOS Status
Rahul Maheshwari449dfb72017-05-08 02:06:48 -0500477 Should Contain ${status} Flashing
Jay Azurine4c52eb2016-08-16 20:51:10 -0500478
Steven Sombar0acb3412018-01-24 09:38:42 -0600479
Jay Azurine4c52eb2016-08-16 20:51:10 -0500480Is PNOR Flash Done
481 [Documentation] Get BIOS 'Flash Done' status. This indicates that the
482 ... PNOR flashing has completed.
Leah McNutt9ba32272016-11-17 15:48:39 +0000483 ${status}= Get Flash BIOS Status
Jay Azurine4c52eb2016-08-16 20:51:10 -0500484 should be equal as strings ${status} Flash Done
485
George Keishing5e870cd2016-08-24 10:05:47 -0500486
Rahul Maheshwari2c725042017-01-29 22:55:28 -0600487Is OS Starting
488 [Documentation] Check if boot progress is OS starting.
489 ${boot_progress}= Get Boot Progress
Michael Walsh01975fa2017-08-20 20:51:36 -0500490 Should Be Equal ${boot_progress} OSStart
Rahul Maheshwari2c725042017-01-29 22:55:28 -0600491
Steven Sombar0acb3412018-01-24 09:38:42 -0600492
George Keishing0af24412017-03-10 13:33:23 -0600493Is OS Off
494 [Documentation] Check if boot progress is "Off".
495 ${boot_progress}= Get Boot Progress
496 Should Be Equal ${boot_progress} Off
497
Steven Sombar0acb3412018-01-24 09:38:42 -0600498
Rahul Maheshwariffd30de2017-06-27 05:57:45 -0500499Get Boot Progress To OS Starting State
500 [Documentation] Get the system to a boot progress state of 'FW Progress,
501 ... Starting OS'.
502
503 ${boot_progress}= Get Boot Progress
Michael Walsh01975fa2017-08-20 20:51:36 -0500504 Run Keyword If '${boot_progress}' == 'OSStart'
Rahul Maheshwariffd30de2017-06-27 05:57:45 -0500505 ... Log Host is already in OS starting state
506 ... ELSE
507 ... Run Keywords Initiate Host PowerOff AND Initiate Host Boot
508 ... AND Wait Until Keyword Succeeds 10 min 10 sec Is OS Starting
509
Steven Sombar0acb3412018-01-24 09:38:42 -0600510
George Keishing5e870cd2016-08-24 10:05:47 -0500511Verify Ping and REST Authentication
Michael Walsha6723f22016-11-22 11:12:01 -0600512 ${l_ping}= Run Keyword And Return Status
George Keishing5e870cd2016-08-24 10:05:47 -0500513 ... Ping Host ${OPENBMC_HOST}
George Keishingc4d3dc02016-09-19 03:45:55 -0500514 Run Keyword If '${l_ping}' == '${False}'
515 ... Fail msg=Ping Failed
George Keishing5e870cd2016-08-24 10:05:47 -0500516
Michael Walsha6723f22016-11-22 11:12:01 -0600517 ${l_rest}= Run Keyword And Return Status
George Keishing5e870cd2016-08-24 10:05:47 -0500518 ... Initialize OpenBMC
George Keishingc4d3dc02016-09-19 03:45:55 -0500519 Run Keyword If '${l_rest}' == '${False}'
520 ... Fail msg=REST Authentication Failed
George Keishing5e870cd2016-08-24 10:05:47 -0500521
522 # Just to make sure the SSH is working for SCP
523 Open Connection And Log In
524 ${system} ${stderr}= Execute Command hostname return_stderr=True
525 Should Be Empty ${stderr}
526
Steven Sombar0acb3412018-01-24 09:38:42 -0600527
George Keishingc4d3dc02016-09-19 03:45:55 -0500528Check If BMC is Up
529 [Documentation] Wait for Host to be online. Checks every X seconds
530 ... interval for Y minutes and fails if timed out.
531 ... Default MAX timedout is 10 min, interval 10 seconds.
Gunnar Mills38032802016-12-12 13:43:40 -0600532 [Arguments] ${max_timeout}=${OPENBMC_REBOOT_TIMEOUT} min
George Keishingc4d3dc02016-09-19 03:45:55 -0500533 ... ${interval}=10 sec
534
Steven Sombar0acb3412018-01-24 09:38:42 -0600535 # Description of argument(s):
536 # max_timeout Maximum time to wait.
537 # This should be expressed in Robot Framework's time format
538 # (e.g. "10 minutes").
539 # interfal Interval to wait between status checks.
540 # This should be expressed in Robot Framework's time format
541 # (e.g. "5 seconds").
542
George Keishingc4d3dc02016-09-19 03:45:55 -0500543 Wait Until Keyword Succeeds
544 ... ${max_timeout} ${interval} Verify Ping and REST Authentication
545
George Keishingb3700812016-08-31 03:03:30 -0500546
547Check If warmReset is Initiated
548 [Documentation] Ping would be still alive, so try SSH to connect
549 ... if fails the ports are down indicating reboot
550 ... is in progress
George Keishing3c05d352016-12-13 06:54:50 -0600551
552 # Warm reset adds 3 seconds delay before forcing reboot
553 # To minimize race conditions, we wait for 7 seconds
554 Sleep 7s
George Keishingb3700812016-08-31 03:03:30 -0500555 ${alive}= Run Keyword and Return Status
556 ... Open Connection And Log In
557 Return From Keyword If '${alive}' == '${False}' ${False}
Gunnar Millsc9ea9362016-12-13 16:21:13 -0600558 [Return] ${True}
George Keishingb3700812016-08-31 03:03:30 -0500559
Steven Sombar0acb3412018-01-24 09:38:42 -0600560
George Keishing06ae4aa2016-08-30 01:41:28 -0500561Flush REST Sessions
562 [Documentation] Removes all the active session objects
563 Delete All Sessions
George Keishingb3700812016-08-31 03:03:30 -0500564
Steven Sombar0acb3412018-01-24 09:38:42 -0600565
Sivas SRRe1143ae2016-08-26 22:31:02 -0500566Initialize DBUS cmd
567 [Documentation] Initialize dbus string with property string to extract
Gunnar Mills38032802016-12-12 13:43:40 -0600568 [Arguments] ${boot_property}
Steven Sombar0acb3412018-01-24 09:38:42 -0600569
570 # Description of argument(s):
571 # boot_property Property string.
572
Michael Walsha6723f22016-11-22 11:12:01 -0600573 ${cmd}= Catenate ${dbuscmdBase} ${dbuscmdGet} ${dbuscmdString}
574 ${cmd}= Catenate ${cmd}${boot_property}
Sivas SRRe1143ae2016-08-26 22:31:02 -0500575 Set Global Variable ${dbuscmd} ${cmd}
576
Steven Sombar0acb3412018-01-24 09:38:42 -0600577
David Shawba2d2c22017-01-23 16:56:38 -0600578Create OS Console File Path
579 [Documentation] Create OS console file path name and return it.
580 [Arguments] ${log_file_path}=${EMPTY}
George Keishing30c12ff2016-09-02 10:25:29 -0500581
Gunnar Mills28e403b2017-10-25 16:16:38 -0500582 # Description of arguments:
David Shawba2d2c22017-01-23 16:56:38 -0600583 # file_path The caller's candidate value. If this value is ${EMPTY}, this
584 # keyword will compose a file path name. Otherwise, this
585 # keyword will use the caller's file_path value. In either
586 # case, the value will be returned.
Gunnar Mills9c760342016-12-08 10:12:06 -0600587
George Keishing3deb7d12017-09-14 07:50:00 -0500588 ${default_file_path}= Catenate /tmp/${OPENBMC_HOST}_os_console.txt
David Shawba2d2c22017-01-23 16:56:38 -0600589 ${log_file_path}= Set Variable If '${log_file_path}' == '${EMPTY}'
590 ... ${default_file_path} ${log_file_path}
Michael Walsh01bd3682017-01-10 11:21:05 -0600591
David Shawba2d2c22017-01-23 16:56:38 -0600592 [Return] ${log_file_path}
Gunnar Mills9c760342016-12-08 10:12:06 -0600593
Steven Sombar0acb3412018-01-24 09:38:42 -0600594
David Shawba2d2c22017-01-23 16:56:38 -0600595Create OS Console Command String
596 [Documentation] Return a command string to start OS console logging.
Gunnar Mills9c760342016-12-08 10:12:06 -0600597
David Shawba2d2c22017-01-23 16:56:38 -0600598 # First make sure that the ssh_pw program is available.
Steven Sombar0acb3412018-01-24 09:38:42 -0600599 ${cmd}= Catenate which ssh_pw 2>/dev/null || find
600 ... ${EXECDIR} -name 'ssh_pw'
601
602 Rdpissuing ${cmd}
603 ${rc} ${output}= Run And Return Rc And Output ${cmd}
David Shawba2d2c22017-01-23 16:56:38 -0600604 Rdpvars rc output
Gunnar Mills9c760342016-12-08 10:12:06 -0600605
Michael Walsh98d8c9e2017-10-04 12:24:29 -0500606 Should Be Equal As Integers 0 ${rc} msg=Could not find ssh_pw.
607
608 ${ssh_pw_file_path}= Set Variable ${output}
George Keishingb45d9522017-06-29 04:20:59 -0500609
Steven Sombar0acb3412018-01-24 09:38:42 -0600610 ${cmd}= Catenate ${ssh_pw_file_path} ${OPENBMC_PASSWORD} -p 2200
George Keishing1f9ec7f2017-06-06 08:59:19 -0500611 ... -o "StrictHostKeyChecking no" ${OPENBMC_USERNAME}@${OPENBMC_HOST}
George Keishing30c12ff2016-09-02 10:25:29 -0500612
Steven Sombar0acb3412018-01-24 09:38:42 -0600613 [Return] ${cmd}
614
George Keishing30c12ff2016-09-02 10:25:29 -0500615
Michael Walshcadf1342017-08-28 13:49:25 -0500616Get SOL Console Pid
Michael Walsh70419172017-12-06 14:20:52 -0600617 [Documentation] Get the pid of the active SOL console job.
618 [Arguments] ${expect_running}=${0}
619
620 # Description of argument(s):
621 # expect_running If set and if no SOL console job is found, print debug
622 # info and fail.
Michael Walshcadf1342017-08-28 13:49:25 -0500623
624 # Find the pid of the active system console logging session (if any).
625 ${search_string}= Create OS Console Command String
626 # At least in some cases, ps output does not show double quotes so we must
627 # replace them in our search string with the regexes to indicate that they
628 # are optional.
629 ${search_string}= Replace String ${search_string} " ["]?
Michael Walsh70419172017-12-06 14:20:52 -0600630 ${ps_cmd}= Catenate ps axwwo user,pid,cmd
631 ${cmd_buf}= Catenate echo $(${ps_cmd} | egrep '${search_string}' |
632 ... egrep -v grep | cut -c10-14)
Michael Walshcadf1342017-08-28 13:49:25 -0500633 Rdpissuing ${cmd_buf}
634 ${rc} ${os_con_pid}= Run And Return Rc And Output ${cmd_buf}
635 Rdpvars os_con_pid
636 # If rc is not zero it just means that there is no OS Console process
637 # running.
638
Michael Walsh70419172017-12-06 14:20:52 -0600639 Return From Keyword If '${os_con_pid}' != '${EMPTY}' ${os_con_pid}
640 Return From Keyword If '${expect_running}' == '${0}' ${os_con_pid}
641
642 Cmd Fnc cat ${log_file_path} ; echo ; ${ps_cmd} quiet=${0}
643 ... print_output=${1} show_err=${1}
644
645 Should Not Be Empty ${os_con_pid}
Michael Walshcadf1342017-08-28 13:49:25 -0500646
647
George Keishing30c12ff2016-09-02 10:25:29 -0500648Stop SOL Console Logging
David Shawba2d2c22017-01-23 16:56:38 -0600649 [Documentation] Stop system console logging and return log output.
George Keishing72992e42017-06-16 06:42:36 -0500650 [Arguments] ${log_file_path}=${EMPTY}
651 ... ${targ_file_path}=${EXECDIR}${/}logs${/}
Michael Walsh5f3f4142017-05-22 17:09:47 -0500652 ... ${return_data}=${1}
David Shawba2d2c22017-01-23 16:56:38 -0600653
654 # If there are muliple system console processes, they will all be stopped.
655 # If there is no existing log file this keyword will return an error
656 # message to that effect (and write that message to targ_file_path, if
657 # specified).
658 # NOTE: This keyword will not fail if there is no running system console
659 # process.
Michael Walsh01bd3682017-01-10 11:21:05 -0600660
661 # Description of arguments:
David Shawba2d2c22017-01-23 16:56:38 -0600662 # log_file_path The file path that was used to call "Start SOL
663 # Console Logging". See that keyword (above) for details.
664 # targ_file_path If specified, the file path to which the source
665 # file path (i.e. "log_file_path") should be copied.
Michael Walsh5f3f4142017-05-22 17:09:47 -0500666 # return_data If this is set to ${1}, this keyword will return the SOL
667 # data to the caller as a unicode string.
George Keishing30c12ff2016-09-02 10:25:29 -0500668
David Shawba2d2c22017-01-23 16:56:38 -0600669 ${log_file_path}= Create OS Console File Path ${log_file_path}
Michael Walshcadf1342017-08-28 13:49:25 -0500670
671 ${os_con_pid}= Get SOL Console Pid
George Keishing30c12ff2016-09-02 10:25:29 -0500672
David Shawba2d2c22017-01-23 16:56:38 -0600673 ${cmd_buf}= Catenate kill -9 ${os_con_pid}
674 Run Keyword If '${os_con_pid}' != '${EMPTY}' Rdpissuing ${cmd_buf}
675 ${rc} ${output}= Run Keyword If '${os_con_pid}' != '${EMPTY}'
676 ... Run And Return Rc And Output ${cmd_buf}
677 Run Keyword If '${os_con_pid}' != '${EMPTY}' Rdpvars rc output
George Keishing30c12ff2016-09-02 10:25:29 -0500678
Michael Walsh01bd3682017-01-10 11:21:05 -0600679 Run Keyword If '${targ_file_path}' != '${EMPTY}'
David Shawba2d2c22017-01-23 16:56:38 -0600680 ... Run Keyword And Ignore Error
681 ... Copy File ${log_file_path} ${targ_file_path}
George Keishing30c12ff2016-09-02 10:25:29 -0500682
Michael Walsh5f3f4142017-05-22 17:09:47 -0500683 ${output}= Set Variable ${EMPTY}
684 ${loc_quiet}= Evaluate ${debug}^1
685 ${rc} ${output}= Run Keyword If '${return_data}' == '${1}'
Michael Walsh98d8c9e2017-10-04 12:24:29 -0500686 ... Cmd Fnc cat ${log_file_path} 2>/dev/null quiet=${loc_quiet}
687 ... print_output=${0} show_err=${0}
Michael Walsh5f3f4142017-05-22 17:09:47 -0500688
David Shawba2d2c22017-01-23 16:56:38 -0600689 [Return] ${output}
690
Steven Sombar0acb3412018-01-24 09:38:42 -0600691
David Shawba2d2c22017-01-23 16:56:38 -0600692Start SOL Console Logging
693 [Documentation] Start system console log to file.
Michael Walsh5f3f4142017-05-22 17:09:47 -0500694 [Arguments] ${log_file_path}=${EMPTY} ${return_data}=${1}
David Shawba2d2c22017-01-23 16:56:38 -0600695
696 # This keyword will first call "Stop SOL Console Logging". Only then will
697 # it start SOL console logging. The data returned by "Stop SOL Console
698 # Logging" will in turn be returned by this keyword.
699
700 # Description of arguments:
Michael Walsh5f3f4142017-05-22 17:09:47 -0500701 # log_file_path The file path to which system console log data should be
702 # written. Note that this path is taken to be a location
703 # on the machine where this program is running rather than
704 # on the Open BMC system.
705 # return_data If this is set to ${1}, this keyword will return any SOL
706 # data to the caller as a unicode string.
David Shawba2d2c22017-01-23 16:56:38 -0600707
708 ${log_file_path}= Create OS Console File Path ${log_file_path}
709
710 ${log_output}= Stop SOL Console Logging ${log_file_path}
Michael Walsh5f3f4142017-05-22 17:09:47 -0500711 ... return_data=${return_data}
David Shawba2d2c22017-01-23 16:56:38 -0600712
713 # Validate by making sure we can create the file. Problems creating the
714 # file would not be noticed by the subsequent ssh command because we fork
715 # the command.
716 Create File ${log_file_path}
717 ${sub_cmd_buf}= Create OS Console Command String
718 # Routing stderr to stdout so that any startup error text will go to the
719 # output file.
George Keishing8c3c1df2017-09-13 13:50:20 -0500720 # TODO: Doesn't work with tox so reverting temporarily.
Michael Walsh460042e2017-09-05 16:55:49 -0500721 # nohup detaches the process completely from our pty.
George Keishing8c3c1df2017-09-13 13:50:20 -0500722 #${cmd_buf}= Catenate nohup ${sub_cmd_buf} &> ${log_file_path} &
723 ${cmd_buf}= Catenate ${sub_cmd_buf} > ${log_file_path} 2>&1 &
David Shawba2d2c22017-01-23 16:56:38 -0600724 Rdpissuing ${cmd_buf}
725 ${rc} ${output}= Run And Return Rc And Output ${cmd_buf}
726 # Because we are forking this command, we essentially will never get a
727 # non-zero return code or any output.
728 Should Be Equal ${rc} ${0}
729
Michael Walsh70419172017-12-06 14:20:52 -0600730 Wait Until Keyword Succeeds 10 seconds 0 seconds
731 ... Get SOL Console Pid ${1}
Michael Walshcadf1342017-08-28 13:49:25 -0500732
David Shawba2d2c22017-01-23 16:56:38 -0600733 [Return] ${log_output}
George Keishing30c12ff2016-09-02 10:25:29 -0500734
Michael Walsh70419172017-12-06 14:20:52 -0600735
George Keishing30c12ff2016-09-02 10:25:29 -0500736Get Time Stamp
737 [Documentation] Get the current time stamp data
738 ${cur_time}= Get Current Date result_format=%Y%m%d%H%M%S%f
Sridevi Rameshb96a0c32016-10-19 05:09:52 -0500739 [Return] ${cur_time}
George Keishing30c12ff2016-09-02 10:25:29 -0500740
George Keishing1b150202016-09-29 08:51:58 -0500741
Rahul Maheshwarif684ba72016-10-25 07:24:41 -0500742Start Journal Log
743 [Documentation] Start capturing journal log to a file in /tmp using
744 ... journalctl command. By default journal log is collected
745 ... at /tmp/journal_log else user input location.
746 ... The File is appended with datetime.
747 [Arguments] ${file_path}=/tmp/journal_log
748
Steven Sombar0acb3412018-01-24 09:38:42 -0600749 # Description of arguments:
750 # file_path The file path of the journal file.
Rahul Maheshwarif684ba72016-10-25 07:24:41 -0500751
752 ${cur_time}= Get Time Stamp
753 Set Global Variable ${LOG_TIME} ${cur_time}
754 Start Command
755 ... journalctl -f > ${file_path}-${LOG_TIME}
756 Log Journal Log Started: ${file_path}-${LOG_TIME}
757
David Shawba2d2c22017-01-23 16:56:38 -0600758
Rahul Maheshwarif684ba72016-10-25 07:24:41 -0500759Stop Journal Log
760 [Documentation] Stop journalctl process if its running.
761 ... By default return log from /tmp/journal_log else
762 ... user input location.
763 [Arguments] ${file_path}=/tmp/journal_log
764
Steven Sombar0acb3412018-01-24 09:38:42 -0600765 # Description of arguments:
766 # file_path The file path of the journal file.
767
Rahul Maheshwarif684ba72016-10-25 07:24:41 -0500768 Open Connection And Log In
769
770 ${rc}=
771 ... Execute Command
772 ... ps ax | grep journalctl | grep -v grep
773 ... return_stdout=False return_rc=True
774
775 Return From Keyword If '${rc}' == '${1}'
776 ... No journal log process running
777
778 ${output} ${stderr}=
779 ... Execute Command killall journalctl
780 ... return_stderr=True
781 Should Be Empty ${stderr}
782
783 ${journal_log} ${stderr}=
784 ... Execute Command
785 ... cat ${file_path}-${LOG_TIME}
786 ... return_stderr=True
787 Should Be Empty ${stderr}
788
789 Log ${journal_log}
790
791 Execute Command rm ${file_path}-${LOG_TIME}
792
793 [Return] ${journal_log}
Rahul Maheshwari757d80c2016-10-17 01:09:55 -0500794
Steven Sombar0acb3412018-01-24 09:38:42 -0600795
Rahul Maheshwari757d80c2016-10-17 01:09:55 -0500796Mac Address To Hex String
797 [Documentation] Converts MAC address into hex format.
798 ... Example
799 ... Given the following MAC: 00:01:6C:80:02:78
800 ... This keyword will return: 0x00 0x01 0x6C 0x80 0x02 0x78
801 ... Description of arguments:
Michael Walsh01bd3682017-01-10 11:21:05 -0600802 ... i_macaddress MAC address in the following format
803 ... 00:01:6C:80:02:78
Rahul Maheshwari757d80c2016-10-17 01:09:55 -0500804 [Arguments] ${i_macaddress}
805
Steven Sombar0acb3412018-01-24 09:38:42 -0600806 # Description of arguments:
807 # i_macaddress The MAC address.
808
Rahul Maheshwari757d80c2016-10-17 01:09:55 -0500809 ${mac_hex}= Catenate 0x${i_macaddress.replace(':', ' 0x')}
Gunnar Millsc9ea9362016-12-13 16:21:13 -0600810 [Return] ${mac_hex}
Rahul Maheshwari757d80c2016-10-17 01:09:55 -0500811
Steven Sombar0acb3412018-01-24 09:38:42 -0600812
Rahul Maheshwari757d80c2016-10-17 01:09:55 -0500813IP Address To Hex String
814 [Documentation] Converts IP address into hex format.
815 ... Example:
816 ... Given the following IP: 10.3.164.100
817 ... This keyword will return: 0xa 0x3 0xa4 0xa0
Rahul Maheshwari757d80c2016-10-17 01:09:55 -0500818 [Arguments] ${i_ipaddress}
819
Steven Sombar0acb3412018-01-24 09:38:42 -0600820 # Description of arguments:
821 # i_macaddress The IP address in the format 10.10.10.10.
822
Rahul Maheshwari757d80c2016-10-17 01:09:55 -0500823 @{ip}= Split String ${i_ipaddress} .
824 ${index}= Set Variable ${0}
825
826 :FOR ${item} IN @{ip}
827 \ ${hex}= Convert To Hex ${item} prefix=0x lowercase=yes
828 \ Set List Value ${ip} ${index} ${hex}
829 \ ${index}= Set Variable ${index + 1}
830 ${ip_hex}= Catenate @{ip}
Steven Sombar0acb3412018-01-24 09:38:42 -0600831
Gunnar Millsc9ea9362016-12-13 16:21:13 -0600832 [Return] ${ip_hex}
Sivas SRRea85d1f2016-11-13 22:44:28 -0600833
Steven Sombar0acb3412018-01-24 09:38:42 -0600834
Sivas SRRea85d1f2016-11-13 22:44:28 -0600835BMC CPU Performance Check
836 [Documentation] Minimal 10% of proc should be free in this instance
837
Sivas SRRa13eb202017-11-03 10:01:56 -0500838 ${bmc_cpu_usage_output} ${stderr} ${rc}= BMC Execute Command
839 ... ${bmc_cpu_usage_cmd}
Sivas SRR5ff4d692017-11-23 02:53:52 -0600840 ${bmc_cpu_usage_output} ${stderr} ${rc}= BMC Execute Command
841 ... ${bmc_cpu_usage_cmd}
Sivas SRRea85d1f2016-11-13 22:44:28 -0600842 ${bmc_cpu_percentage}= Fetch From Left ${bmc_cpu_usage_output} %
843 Should be true ${bmc_cpu_percentage} < 90
844
Steven Sombar0acb3412018-01-24 09:38:42 -0600845
Sivas SRRea85d1f2016-11-13 22:44:28 -0600846BMC Mem Performance Check
847 [Documentation] Minimal 10% of memory should be free in this instance
848
Sivas SRRa13eb202017-11-03 10:01:56 -0500849 ${bmc_mem_free_output} ${stderr} ${rc}= BMC Execute Command
850 ... ${bmc_mem_free_cmd}
Sivas SRRea85d1f2016-11-13 22:44:28 -0600851
Sivas SRRa13eb202017-11-03 10:01:56 -0500852 ${bmc_mem_total_output} ${stderr} ${rc}= BMC Execute Command
853 ... ${bmc_mem_total_cmd}
Sivas SRR5ff4d692017-11-23 02:53:52 -0600854 ${bmc_mem_free_output} ${stderr} ${rc}= BMC Execute Command
855 ... ${bmc_mem_free_cmd}
Sivas SRRea85d1f2016-11-13 22:44:28 -0600856
Sivas SRR5ff4d692017-11-23 02:53:52 -0600857 ${bmc_mem_total_output} ${stderr} ${rc}= BMC Execute Command
858 ... ${bmc_mem_total_cmd}
859
860 ${bmc_mem_percentage}= Evaluate ${bmc_mem_free_output}*100
Sivas SRRea85d1f2016-11-13 22:44:28 -0600861 ${bmc_mem_percentage}= Evaluate
862 ... ${bmc_mem_percentage}/${bmc_mem_total_output}
863 Should be true ${bmc_mem_percentage} > 10
864
Steven Sombar0acb3412018-01-24 09:38:42 -0600865
Sivas SRRaca55712016-12-21 04:32:35 -0600866BMC File System Usage Check
Sivas SRRc0957412017-10-06 13:02:08 -0500867 [Documentation] Check the file system space. 4 file system should be
868 ... 100% full which is expected
Steven Sombar0acb3412018-01-24 09:38:42 -0600869 # Filesystem Size Used Available Use% Mounted on
870 # /dev/root 14.4M 14.4M 0 100% /
871 # /dev/ubiblock0_0 14.4M 14.4M 0 100% /media/rofs-c9249b0e
872 # /dev/ubiblock8_0 19.6M 19.6M 0 100% /media/pnor-ro-8764baa3
873 # /dev/ubiblock4_0 14.4M 14.4M 0 100% /media/rofs-407816c
874 # /dev/ubiblock8_4 21.1M 21.1M 0 100% /media/pnor-ro-cecc64c4
Sivas SRR5ff4d692017-11-23 02:53:52 -0600875 ${bmc_fs_usage_output} ${stderr} ${rc}= BMC Execute Command
876 ... ${bmc_file_system_usage_cmd}
877 ${bmc_pnor_fs_usage_output} ${stderr} ${rc}= BMC Execute Command
878 ... ${total_pnor_ro_file_system_cmd}
879 ${bmc_bmc_fs_usage_output} ${stderr} ${rc}= BMC Execute Command
880 ... ${total_bmc_ro_file_system_cmd}
881 ${total_bmc_pnor_image}= Evaluate
882 ... ${bmc_pnor_fs_usage_output}+${bmc_bmc_fs_usage_output}
883 # Considering /dev/root also in total 100% used file system
884 ${total_full_fs}= Evaluate ${total_bmc_pnor_image}+1
885 Should Be True ${bmc_fs_usage_output}==${total_full_fs}
Sivas SRRaca55712016-12-21 04:32:35 -0600886
Steven Sombar0acb3412018-01-24 09:38:42 -0600887
Sivas SRRea85d1f2016-11-13 22:44:28 -0600888Check BMC CPU Performance
889 [Documentation] Minimal 10% of proc should be free in 3 sample
890 :FOR ${var} IN Range 1 4
891 \ BMC CPU Performance check
892
Steven Sombar0acb3412018-01-24 09:38:42 -0600893
Sivas SRRea85d1f2016-11-13 22:44:28 -0600894Check BMC Mem Performance
895 [Documentation] Minimal 10% of memory should be free
896
897 :FOR ${var} IN Range 1 4
898 \ BMC Mem Performance check
899
Steven Sombar0acb3412018-01-24 09:38:42 -0600900
Sivas SRRaca55712016-12-21 04:32:35 -0600901Check BMC File System Performance
902 [Documentation] Check for file system usage for 4 times
903
904 :FOR ${var} IN Range 1 4
905 \ BMC File System Usage check
906
Steven Sombar0acb3412018-01-24 09:38:42 -0600907
Rahul Maheshwari63c105b2017-05-12 05:13:52 -0500908Get URL List
909 [Documentation] Return list of URLs under given URL.
910 [Arguments] ${openbmc_url}
Steven Sombar0acb3412018-01-24 09:38:42 -0600911
Rahul Maheshwari63c105b2017-05-12 05:13:52 -0500912 # Description of argument(s):
Michael Walsh5f3f4142017-05-22 17:09:47 -0500913 # openbmc_url URL for list operation (e.g.
914 # /xyz/openbmc_project/inventory).
Rahul Maheshwari63c105b2017-05-12 05:13:52 -0500915
George Keishing335f5362017-06-30 15:11:20 -0500916 ${url_list}= Read Properties ${openbmc_url}/list quiet=${1}
George Keishing184a89a2017-06-27 07:12:16 -0500917 Sort List ${url_list}
Steven Sombar0acb3412018-01-24 09:38:42 -0600918
Rahul Maheshwari63c105b2017-05-12 05:13:52 -0500919 [Return] ${url_list}
920
Steven Sombar0acb3412018-01-24 09:38:42 -0600921
Rahul Maheshwari9a8d3b12016-12-05 04:06:16 -0600922Get Endpoint Paths
923 [Documentation] Returns all url paths ending with given endpoint
924 ... Example:
925 ... Given the following endpoint: cpu
Michael Walsh01bd3682017-01-10 11:21:05 -0600926 ... This keyword will return: list of all urls ending with
927 ... cpu -
Rahul Maheshwari9a8d3b12016-12-05 04:06:16 -0600928 ... /org/openbmc/inventory/system/chassis/motherboard/cpu0,
929 ... /org/openbmc/inventory/system/chassis/motherboard/cpu1
Rahul Maheshwari9a8d3b12016-12-05 04:06:16 -0600930 [Arguments] ${path} ${endpoint}
931
Steven Sombar0acb3412018-01-24 09:38:42 -0600932 # Description of arguments:
933 # path URL path for enumeration.
934 # endpoint Endpoint string (url path ending).
935
George Keishingc7b161b2017-12-01 09:58:12 -0600936 ${resp}= Read Properties ${path}/enumerate timeout=30
937 Log Dictionary ${resp}
Rahul Maheshwari9a8d3b12016-12-05 04:06:16 -0600938
George Keishingc7b161b2017-12-01 09:58:12 -0600939 ${list}= Get Dictionary Keys ${resp}
940 # For a given string, look for prefix and suffix for matching expression.
941 # Start of string followed by zero or more of any character followed by
942 # any digit or lower case character.
943 ${resp}= Get Matches ${list} regexp=^.*[0-9a-z_].${endpoint}[0-9a-z]*$
Steven Sombar0acb3412018-01-24 09:38:42 -0600944
George Keishingc7b161b2017-12-01 09:58:12 -0600945 [Return] ${resp}
Sridevi Rameshb96a0c32016-10-19 05:09:52 -0500946
Steven Sombar0acb3412018-01-24 09:38:42 -0600947
Sridevi Rameshb96a0c32016-10-19 05:09:52 -0500948Check Zombie Process
949 [Documentation] Check if any defunct process exist or not on BMC
Michael Walsh01bd3682017-01-10 11:21:05 -0600950 ${count} ${stderr} ${rc}= Execute Command ps -o stat | grep Z | wc -l
Sridevi Rameshb96a0c32016-10-19 05:09:52 -0500951 ... return_stderr=True return_rc=True
952 Should Be True ${count}==0
953 Should Be Empty ${stderr}
George Keishing5327d012016-12-08 08:43:29 -0600954
Steven Sombar0acb3412018-01-24 09:38:42 -0600955
George Keishing5327d012016-12-08 08:43:29 -0600956Prune Journal Log
957 [Documentation] Prune archived journal logs.
958 [Arguments] ${vacuum_size}=1M
959
960 # This keyword can be used to prevent the journal
961 # log from filling up the /run filesystem.
962 # This command will retain only the latest logs
963 # of the user specified size.
964
Steven Sombar0acb3412018-01-24 09:38:42 -0600965 # Description of argument(s):
966 # vacuum_size Size of journal.
967
George Keishing5327d012016-12-08 08:43:29 -0600968 Open Connection And Log In
969 ${output} ${stderr} ${rc}=
970 ... Execute Command
971 ... journalctl --vacuum-size=${vacuum_size}
972 ... return_stderr=True return_rc=True
973
974 Should Be Equal ${rc} ${0} msg=${stderr}
Sridevi Ramesh1699d372016-12-06 00:20:22 -0600975
Steven Sombar0acb3412018-01-24 09:38:42 -0600976
Sridevi Ramesh1699d372016-12-06 00:20:22 -0600977Set BMC Power Policy
978 [Documentation] Set the given BMC power policy.
Michael Walshfdc5ced2017-08-17 13:15:15 -0500979 [Arguments] ${policy}
980
981 # Note that this function will translate the old style "RESTORE_LAST_STATE"
982 # policy to the new style "xyz.openbmc_project.Control.Power.RestorePolicy.
983 # Policy.Restore" for you.
Sridevi Ramesh1699d372016-12-06 00:20:22 -0600984
George Keishing1c3a9662017-08-11 10:16:36 -0500985 # Description of argument(s):
Michael Walshfdc5ced2017-08-17 13:15:15 -0500986 # policy Power restore policy (e.g "RESTORE_LAST_STATE",
987 # ${RESTORE_LAST_STATE}).
988
989 # Set the bmc_power_policy_method to either 'Old' or 'New'.
990 Set Power Policy Method
991 # This translation helps bridge between old and new method for calling.
992 ${policy}= Translate Power Policy Value ${policy}
993 # Run the appropriate keyword.
994 Run Key ${bmc_power_policy_method} Set Power Policy \ ${policy}
995 ${currentPolicy}= Get System Power Policy
996 Should Be Equal ${currentPolicy} ${policy}
997
Steven Sombar0acb3412018-01-24 09:38:42 -0600998
Michael Walshfdc5ced2017-08-17 13:15:15 -0500999New Set Power Policy
1000 [Documentation] Set the given BMC power policy (new method).
1001 [Arguments] ${policy}
1002
1003 # Description of argument(s):
George Keishingefc3ff22017-12-12 11:49:25 -06001004 # policy Power restore policy (e.g. ${ALWAYS_POWER_OFF}).
George Keishing1c3a9662017-08-11 10:16:36 -05001005
1006 ${valueDict}= Create Dictionary data=${policy}
1007 Write Attribute
Michael Walshfdc5ced2017-08-17 13:15:15 -05001008 ... ${POWER_RESTORE_URI} PowerRestorePolicy data=${valueDict}
1009
Steven Sombar0acb3412018-01-24 09:38:42 -06001010
Michael Walshfdc5ced2017-08-17 13:15:15 -05001011Old Set Power Policy
1012 [Documentation] Set the given BMC power policy (old method).
1013 [Arguments] ${policy}
1014
1015 # Description of argument(s):
George Keishingefc3ff22017-12-12 11:49:25 -06001016 # policy Power restore policy (e.g. "ALWAYS_POWER_OFF").
Michael Walshfdc5ced2017-08-17 13:15:15 -05001017
1018 ${valueDict}= create dictionary data=${policy}
1019 Write Attribute ${HOST_SETTING} power_policy data=${valueDict}
George Keishingcce9df22017-01-24 06:19:33 -06001020
Steven Sombar0acb3412018-01-24 09:38:42 -06001021
George Keishing4d6f3d62017-02-06 09:50:41 -06001022Get System Power Policy
Steven Sombar0acb3412018-01-24 09:38:42 -06001023 [Documentation] Returns the BMC power policy.
Michael Walshfdc5ced2017-08-17 13:15:15 -05001024
1025 # Set the bmc_power_policy_method to either 'Old' or 'New'.
1026 Set Power Policy Method
1027 ${cmd_buf}= Create List ${bmc_power_policy_method} Get Power Policy
1028 # Run the appropriate keyword.
1029 ${currentPolicy}= Run Keyword @{cmd_buf}
Steven Sombar0acb3412018-01-24 09:38:42 -06001030
Michael Walshfdc5ced2017-08-17 13:15:15 -05001031 [Return] ${currentPolicy}
1032
Steven Sombar0acb3412018-01-24 09:38:42 -06001033
Michael Walshfdc5ced2017-08-17 13:15:15 -05001034New Get Power Policy
Steven Sombar0acb3412018-01-24 09:38:42 -06001035 [Documentation] Returns the BMC power policy (new method).
George Keishing1c3a9662017-08-11 10:16:36 -05001036 ${currentPolicy}= Read Attribute ${POWER_RESTORE_URI} PowerRestorePolicy
Steven Sombar0acb3412018-01-24 09:38:42 -06001037
George Keishing4d6f3d62017-02-06 09:50:41 -06001038 [Return] ${currentPolicy}
George Keishingcce9df22017-01-24 06:19:33 -06001039
Steven Sombar0acb3412018-01-24 09:38:42 -06001040
Michael Walshfdc5ced2017-08-17 13:15:15 -05001041Old Get Power Policy
Steven Sombar0acb3412018-01-24 09:38:42 -06001042 [Documentation] Returns the BMC power policy (old method).
Michael Walshfdc5ced2017-08-17 13:15:15 -05001043 ${currentPolicy}= Read Attribute ${HOST_SETTING} power_policy
Steven Sombar0acb3412018-01-24 09:38:42 -06001044
Michael Walshfdc5ced2017-08-17 13:15:15 -05001045 [Return] ${currentPolicy}
1046
Steven Sombar0acb3412018-01-24 09:38:42 -06001047
Rahul Maheshwari1cbf0042017-02-13 05:17:37 -06001048Get Auto Reboot
1049 [Documentation] Returns auto reboot setting.
Sweta Potthuri62a23572017-07-28 04:05:13 -05001050 ${setting}= Read Attribute ${CONTROL_HOST_URI}/auto_reboot AutoReboot
Steven Sombar0acb3412018-01-24 09:38:42 -06001051
Rahul Maheshwari1cbf0042017-02-13 05:17:37 -06001052 [Return] ${setting}
1053
Steven Sombar0acb3412018-01-24 09:38:42 -06001054
Rahul Maheshwari1cbf0042017-02-13 05:17:37 -06001055Set Auto Reboot
1056 [Documentation] Set the given auto reboot setting.
1057 [Arguments] ${setting}
Steven Sombar0acb3412018-01-24 09:38:42 -06001058
1059 # Description of argument(s):
1060 # setting The reboot setting, 1 for enabling and 0 for disabling.
Rahul Maheshwari1cbf0042017-02-13 05:17:37 -06001061
1062 ${valueDict}= Set Variable ${setting}
1063 ${data}= Create Dictionary data=${valueDict}
Sweta Potthuri62a23572017-07-28 04:05:13 -05001064 Write Attribute ${CONTROL_HOST_URI}/auto_reboot AutoReboot data=${data}
Rahul Maheshwari1cbf0042017-02-13 05:17:37 -06001065 ${current_setting}= Get Auto Reboot
Sweta Potthuri62a23572017-07-28 04:05:13 -05001066 Should Be Equal As Integers ${current_setting} ${setting}
Rahul Maheshwari1cbf0042017-02-13 05:17:37 -06001067
1068
George Keishingcce9df22017-01-24 06:19:33 -06001069Set BMC Reset Reference Time
1070 [Documentation] Set current boot time as a reference and increment
George Keishing80dd5af2017-02-08 07:01:25 -06001071 ... boot count.
George Keishingcce9df22017-01-24 06:19:33 -06001072
1073 ${cur_btime}= Get BMC Boot Time
George Keishing80dd5af2017-02-08 07:01:25 -06001074 Run Keyword If ${BOOT_TIME} == ${0} and ${BOOT_COUNT} == ${0}
George Keishingcce9df22017-01-24 06:19:33 -06001075 ... Set Global Variable ${BOOT_TIME} ${cur_btime}
George Keishing80dd5af2017-02-08 07:01:25 -06001076 ... ELSE IF ${cur_btime} > ${BOOT_TIME}
1077 ... Run Keywords Set Global Variable ${BOOT_TIME} ${cur_btime}
George Keishingcce9df22017-01-24 06:19:33 -06001078 ... AND
1079 ... Set Global Variable ${BOOT_COUNT} ${BOOT_COUNT + 1}
1080
Steven Sombar0acb3412018-01-24 09:38:42 -06001081
George Keishingcce9df22017-01-24 06:19:33 -06001082Get BMC Boot Time
Steven Sombar0acb3412018-01-24 09:38:42 -06001083 [Documentation] Returns boot time from /proc/stat.
George Keishingcce9df22017-01-24 06:19:33 -06001084
1085 Open Connection And Log In
1086 ${output} ${stderr}=
1087 ... Execute Command egrep '^btime ' /proc/stat | cut -f 2 -d ' '
1088 ... return_stderr=True
1089 Should Be Empty ${stderr}
1090 ${btime}= Convert To Integer ${output}
1091 [Return] ${btime}
George Keishingede30002017-02-02 09:33:07 -06001092
Steven Sombar0acb3412018-01-24 09:38:42 -06001093
George Keishingede30002017-02-02 09:33:07 -06001094Execute Command On BMC
1095 [Documentation] Execute given command on BMC and return output.
1096 [Arguments] ${command}
Steven Sombar0acb3412018-01-24 09:38:42 -06001097
1098 # Description of argument(s):
1099 # command The command to execute on the BMC.
1100
George Keishingede30002017-02-02 09:33:07 -06001101 ${stdout} ${stderr}= Execute Command ${command} return_stderr=True
1102 Should Be Empty ${stderr}
1103 [Return] ${stdout}
David Shawba2d2c22017-01-23 16:56:38 -06001104
Sridevi Rameshea36b412017-03-09 04:08:02 -06001105
George Keishing7a520222017-02-27 09:44:30 -06001106Enable Core Dump On BMC
1107 [Documentation] Enable core dump collection.
1108 Open Connection And Log In
1109 ${core_pattern}= Execute Command On BMC
1110 ... echo '/tmp/core_%e.%p' | tee /proc/sys/kernel/core_pattern
1111 Should Be Equal As Strings ${core_pattern} /tmp/core_%e.%p
Rahul Maheshwarie95622c2017-02-24 10:04:29 -06001112
Steven Sombar0acb3412018-01-24 09:38:42 -06001113
Sivas SRR01262bd2017-03-19 10:00:23 -05001114Get Number Of BMC Core Dump Files
Steven Sombar0acb3412018-01-24 09:38:42 -06001115 [Documentation] Returns number of core dump files on BMC.
Sivas SRR01262bd2017-03-19 10:00:23 -05001116 Open Connection And Log In
1117 ${num_of_core_dump}= Execute Command
1118 ... ls /tmp/core* 2>/dev/null | wc -l
1119 [Return] ${num_of_core_dump}
1120
Steven Sombar0acb3412018-01-24 09:38:42 -06001121
Sivas SRR01262bd2017-03-19 10:00:23 -05001122Set Core Dump File Size Unlimited
1123 [Documentation] Set core dump file size to unlimited.
1124 Open Connection And Log In
1125 Execute Command On BMC
1126 ... ulimit -c unlimited
1127
Steven Sombar0acb3412018-01-24 09:38:42 -06001128
Sivas SRR01262bd2017-03-19 10:00:23 -05001129Check For Core Dumps
1130 [Documentation] Check for any core dumps exist.
1131 ${output}= Get Number Of BMC Core Dump Files
1132 Run Keyword If ${output} > 0
1133 ... Log **Warning** BMC core dump files exist level=WARN
1134
Steven Sombar0acb3412018-01-24 09:38:42 -06001135
Rahul Maheshwarie95622c2017-02-24 10:04:29 -06001136Trigger Host Watchdog Error
George Keishing37cfa6b2017-06-15 10:25:55 -05001137 [Documentation] Inject host watchdog timeout error via REST.
Rahul Maheshwarie95622c2017-02-24 10:04:29 -06001138 [Arguments] ${milliseconds}=1000 ${sleep_time}=5s
Steven Sombar0acb3412018-01-24 09:38:42 -06001139
Michael Walshb5839d02017-04-12 16:11:20 -05001140 # Description of argument(s):
1141 # milliseconds The time watchdog timer value in milliseconds (e.g. 1000 =
1142 # 1 second).
Rahul Maheshwarie95622c2017-02-24 10:04:29 -06001143 # sleep_time Time delay for host watchdog error to get injected.
1144 # Default is 5 seconds.
1145
Michael Walsh4ffd1a12018-03-14 10:35:44 -05001146 ${data}= Create Dictionary
1147 ... data=xyz.openbmc_project.State.Watchdog.Action.PowerCycle
1148 ${status} ${result}= Run Keyword And Ignore Error
1149 ... Read Attribute ${HOST_WATCHDOG_URI} ExpireAction
1150 Run Keyword If '${status}' == 'PASS'
1151 ... Write Attribute ${HOST_WATCHDOG_URI} ExpireAction data=${data}
George Keishing37cfa6b2017-06-15 10:25:55 -05001152
1153 ${data}= Create Dictionary data=${milliseconds}
Michael Walsh4ffd1a12018-03-14 10:35:44 -05001154 Write Attribute ${HOST_WATCHDOG_URI} Interval data=${data}
1155
1156 ${data}= Create Dictionary data=${True}
1157 Write Attribute ${HOST_WATCHDOG_URI} Enabled data=${data}
George Keishing37cfa6b2017-06-15 10:25:55 -05001158
Rahul Maheshwarie95622c2017-02-24 10:04:29 -06001159 Sleep ${sleep_time}
Prashanth Kattiae7c2282017-03-15 07:43:46 -05001160
Steven Sombar0acb3412018-01-24 09:38:42 -06001161
Prashanth Kattiae7c2282017-03-15 07:43:46 -05001162Login To OS Host
Steven Sombar0acb3412018-01-24 09:38:42 -06001163 [Documentation] Login to OS Host and return the Login response code.
Prashanth Kattiae7c2282017-03-15 07:43:46 -05001164 [Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME}
1165 ... ${os_password}=${OS_PASSWORD}
Steven Sombar0acb3412018-01-24 09:38:42 -06001166
Gunnar Mills28e403b2017-10-25 16:16:38 -05001167 # Description of arguments:
Prashanth Kattiae7c2282017-03-15 07:43:46 -05001168 # ${os_host} IP address of the OS Host.
1169 # ${os_username} OS Host Login user name.
1170 # ${os_password} OS Host Login passwrd.
1171
1172 ${os_state}= Run Keyword And Return Status Ping Host ${os_host}
1173 Run Keyword If '${os_state}' == 'False'
1174 ... Run Keywords Initiate Host Reboot AND
1175 ... Is Host Running AND
1176 ... Wait for OS ${os_host} ${os_username} ${os_password}
1177
George Keishing3f223b82017-06-28 03:50:35 -05001178 SSHLibrary.Open Connection ${os_host}
Sridevi Rameshea36b412017-03-09 04:08:02 -06001179 ${resp}= Login ${os_username} ${os_password}
1180 [Return] ${resp}
Prashanth Katti884ee062017-03-16 05:05:03 -05001181
Steven Sombar0acb3412018-01-24 09:38:42 -06001182
Prashanth Katti884ee062017-03-16 05:05:03 -05001183Configure Initial Settings
1184 [Documentation] Restore old IP and route.
1185 ... This keyword requires initial settings viz IP address,
1186 ... Network Mask, default gatway and serial console IP and port
1187 ... information which should be provided in command line.
1188
Michael Walshb5839d02017-04-12 16:11:20 -05001189 [Arguments] ${host}=${OPENBMC_HOST} ${mask}=${NET_MASK}
1190 ... ${gw_ip}=${GW_IP}
Prashanth Katti884ee062017-03-16 05:05:03 -05001191
Steven Sombar0acb3412018-01-24 09:38:42 -06001192 # Description of arguments:
1193 # host IP address of the OS Host.
1194 # mask Network mask.
1195 # gu_ip Gateway IP address or hostname.
1196
Michael Walshb5839d02017-04-12 16:11:20 -05001197 # Open telnet connection and ignore the error, in case telnet session is
1198 # already opened by the program calling this keyword.
Prashanth Katti884ee062017-03-16 05:05:03 -05001199 Run Keyword And Ignore Error Open Telnet Connection to BMC Serial Console
1200 Telnet.write ifconfig eth0 ${host} netmask ${mask}
1201 Telnet.write route add default gw ${gw_ip}
George Keishingd05b1082017-03-24 05:27:32 -05001202
Steven Sombar0acb3412018-01-24 09:38:42 -06001203
George Keishingd05b1082017-03-24 05:27:32 -05001204Install Debug Tarball On BMC
George Keishing55441822017-05-18 14:17:59 -05001205 [Documentation] Copy the debug tar file to BMC and install.
Steven Sombar0acb3412018-01-24 09:38:42 -06001206 [Arguments] ${tarball_file_path}=${default_tarball}
1207 ... ${targ_tarball_dir_path}=/tmp/tarball/
George Keishingd05b1082017-03-24 05:27:32 -05001208
George Keishing55441822017-05-18 14:17:59 -05001209 # Description of arguments:
1210 # tarball_file_path Path of the debug tarball file.
1211 # The tar file is downloaded from the build page
1212 # https://openpower.xyz/job/openbmc-build/
1213 # obmc-phosphor-debug-tarball-witherspoon.tar.xz
1214 #
1215 # targ_tarball_dir_path The directory path where the tarball is to be
1216 # installed.
1217
1218 OperatingSystem.File Should Exist ${tarball_file_path}
1219 ... msg=${tarball_file_path} doesn't exist.
1220
1221 # Upload the file to BMC.
George Keishingd05b1082017-03-24 05:27:32 -05001222 Import Library SCPLibrary WITH NAME scp
1223 Open Connection for SCP
George Keishing55441822017-05-18 14:17:59 -05001224 scp.Put File ${tarball_file_path} /tmp/debug-tarball.tar.xz
1225
1226 # Create tarball directory and install.
George Keishing95c371a2017-09-14 15:42:33 -05001227 BMC Execute Command mkdir -p ${targ_tarball_dir_path}
1228 BMC Execute Command
George Keishing55441822017-05-18 14:17:59 -05001229 ... tar -xf /tmp/debug-tarball.tar.xz -C ${targ_tarball_dir_path}
1230
George Keishing95c371a2017-09-14 15:42:33 -05001231 # Remove the tarball file from BMC.
1232 BMC Execute Command rm -f /tmp/debug-tarball.tar.xz
George Keishingd05b1082017-03-24 05:27:32 -05001233
George Keishing55441822017-05-18 14:17:59 -05001234
Sweta Potthuri025e0122017-02-21 00:42:48 -06001235Get BMC Boot Count
Steven Sombar0acb3412018-01-24 09:38:42 -06001236 [Documentation] Returns BMC boot count based on boot time.
Sweta Potthuri025e0122017-02-21 00:42:48 -06001237 ${cur_btime}= Get BMC Boot Time
1238
1239 # Set global variable BOOT_TIME to current boot time if current boot time
1240 # is changed. Also increase value of global variable BOOT_COUNT by 1.
1241 Run Keyword If ${cur_btime} > ${BOOT_TIME}
1242 ... Run Keywords Set Global Variable ${BOOT_TIME} ${cur_btime}
1243 ... AND
1244 ... Set Global Variable ${BOOT_COUNT} ${BOOT_COUNT + 1}
Steven Sombar0acb3412018-01-24 09:38:42 -06001245
Sweta Potthuri025e0122017-02-21 00:42:48 -06001246 [Return] ${BOOT_COUNT}
1247
Steven Sombar0acb3412018-01-24 09:38:42 -06001248
Sweta Potthuri025e0122017-02-21 00:42:48 -06001249Set BMC Boot Count
1250 [Documentation] Set BMC boot count to given value.
1251 [Arguments] ${count}
1252
1253 # Description of arguments:
1254 # count boot count value.
1255 ${cur_btime}= Get BMC Boot Time
1256
1257 # Set global variable BOOT_COUNT to given value.
1258 Set Global Variable ${BOOT_COUNT} ${count}
1259
1260 # Set BOOT_TIME variable to current boot time.
1261 Set Global Variable ${BOOT_COUNT} ${count}
Michael Walsh689fbbe2017-04-04 17:58:03 -05001262
Steven Sombar0acb3412018-01-24 09:38:42 -06001263
Sweta Potthurie5a9fc72017-05-03 07:02:46 -05001264Get System LED State
1265 [Documentation] Return the state of given system LED.
1266 [Arguments] ${led_name}
1267
1268 # Description of argument(s):
1269 # led_name System LED name (e.g. heartbeat, identify, beep).
1270
1271 ${state}= Read Attribute ${LED_PHYSICAL_URI}${led_name} State
1272 [Return] ${state.rsplit('.', 1)[1]}
1273
George Keishing3987a1e2017-09-20 09:13:02 -05001274
Steven Sombar5f3af442018-02-14 08:30:15 -06001275Set System LED State
1276 [Documentation] Set given system LED via REST.
1277 [Arguments] ${led_name} ${led_state}
1278 # Description of argument(s):
1279 # led_name System LED name (e.g. heartbeat, identify, beep).
1280 # led_state LED state to be set (e.g. On, Off).
1281
1282 ${args}= Create Dictionary
1283 ... data=xyz.openbmc_project.Led.Physical.Action.${led_state}
1284 Write Attribute ${LED_PHYSICAL_URI}${led_name} State data=${args}
1285
1286 Verify LED State ${led_name} ${led_state}
1287
1288
1289Verify LED State
1290 [Documentation] Checks if LED is in given state.
1291 [Arguments] ${led_name} ${led_state}
1292 # Description of argument(s):
1293 # led_name System LED name (e.g. heartbeat, identify, beep).
1294 # led_state LED state to be verified (e.g. On, Off).
1295
1296 ${state}= Get System LED State ${led_name}
1297 Should Be Equal ${state} ${led_state}
1298
1299
Steven Sombarb89d5802018-02-15 14:48:49 -06001300Get LED State XYZ
1301 [Documentation] Returns state of given LED.
1302 [Arguments] ${led_name}
1303
1304 # Description of argument(s):
1305 # led_name Name of LED.
1306
1307 ${state}= Read Attribute ${LED_GROUPS_URI}${led_name} Asserted
1308 # Returns the state of the LED, either On or Off.
1309 [Return] ${state}
1310
1311
George Keishingd2d90af2017-06-27 08:17:34 -05001312Delete Error Logs
Michael Walsh689fbbe2017-04-04 17:58:03 -05001313 [Documentation] Delete error logs.
1314
George Keishingd2d90af2017-06-27 08:17:34 -05001315 # Check if error logs entries exist, if not return.
George Keishing335f5362017-06-30 15:11:20 -05001316 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${/}list quiet=${1}
George Keishingd2d90af2017-06-27 08:17:34 -05001317 Return From Keyword If ${resp.status_code} == ${HTTP_NOT_FOUND}
1318
1319 # Get the list of error logs entries and delete them all.
George Keishing335f5362017-06-30 15:11:20 -05001320 ${elog_entries}= Get URL List ${BMC_LOGGING_ENTRY}
George Keishingd2d90af2017-06-27 08:17:34 -05001321 :FOR ${entry} IN @{elog_entries}
1322 \ Delete Error Log Entry ${entry}
George Keishingdb34c2f2017-04-18 13:20:25 -05001323
George Keishing3987a1e2017-09-20 09:13:02 -05001324
George Keishingd2d90af2017-06-27 08:17:34 -05001325Delete Error Log Entry
George Keishingdb34c2f2017-04-18 13:20:25 -05001326 [Documentation] Delete error log entry.
1327 [Arguments] ${entry_path}
1328
1329 # Description of argument(s):
1330 # entry_path Delete an error log entry.
1331 # Ex. /xyz/openbmc_project/logging/entry/1
1332
George Keishing6a8ed0f2017-07-11 22:54:42 -05001333 # Skip delete if entry URI is a callout.
1334 # Example: /xyz/openbmc_project/logging/entry/1/callout
1335 Return From Keyword If '${entry_path.rsplit('/', 1)[1]}' == 'callout'
1336
George Keishingdb34c2f2017-04-18 13:20:25 -05001337 ${data}= Create Dictionary data=@{EMPTY}
1338 ${resp}= Openbmc Delete Request ${entry_path} data=${data}
1339 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
Rahul Maheshwari43be6932017-05-01 02:37:06 -05001340
1341
George Keishingcc71c3c2017-09-20 09:38:03 -05001342Delete All Error Logs
George Keishing3987a1e2017-09-20 09:13:02 -05001343 [Documentation] Delete all error log entries using "DeleteAll" interface.
1344
1345 ${data}= Create Dictionary data=@{EMPTY}
1346 ${resp}= Openbmc Post Request ${BMC_LOGGING_URI}action/DeleteAll
1347 ... data=${data}
1348 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
1349
1350
Rahul Maheshwari13256d92017-06-15 10:05:40 -05001351Get BMC Version
1352 [Documentation] Returns BMC version from /etc/os-release.
1353 ... e.g. "v1.99.6-141-ge662190"
1354
1355 Open Connection And Log In
1356 ${cmd}= Set Variable grep ^VERSION_ID= /etc/os-release | cut -f 2 -d '='
1357 ${output}= Execute Command On BMC ${cmd}
1358 [Return] ${output}
1359
George Keishingad5817a2017-06-28 10:19:03 -05001360
Charles Paul Hofere43fb2f2017-09-26 15:36:18 -05001361Get PNOR Version
Steven Sombar0acb3412018-01-24 09:38:42 -06001362 [Documentation] Returns the PNOR version from the BMC.
Charles Paul Hofere43fb2f2017-09-26 15:36:18 -05001363
1364 ${pnor_attrs}= Get PNOR Attributes
1365 [Return] ${pnor_attrs['version']}
1366
1367
1368Get PNOR Attributes
1369 [Documentation] Return PNOR software attributes as a dictionary.
1370
Michael Walsh98d8c9e2017-10-04 12:24:29 -05001371 # This keyword parses /var/lib/phosphor-software-manager/pnor/ro/pnor.toc
1372 # into key/value pairs.
Charles Paul Hofere43fb2f2017-09-26 15:36:18 -05001373
1374 ${outbuf} ${stderr} ${rc}= BMC Execute Command
1375 ... cat /var/lib/phosphor-software-manager/pnor/ro/pnor.toc
1376 ${pnor_attrs}= Key Value Outbuf To Dict ${outbuf} delim==
1377
1378 [Return] ${pnor_attrs}
1379
1380
George Keishingad5817a2017-06-28 10:19:03 -05001381Get Elog URL List
1382 [Documentation] Return error log entry list of URLs.
1383
1384 ${url_list}= Read Properties /xyz/openbmc_project/logging/entry/
1385 Sort List ${url_list}
1386 [Return] ${url_list}
1387
1388
Steven Sombaraaf72c42017-09-05 12:02:00 -05001389Read Turbo Setting Via REST
1390 [Documentation] Return turbo setting via REST.
1391
1392 ${resp}= OpenBMC Get Request ${SENSORS_URI}host/TurboAllowed
1393 ${jsondata}= To JSON ${resp.content}
1394 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
1395 [Return] ${jsondata["data"]["value"]}
1396
1397
Steven Sombaraaf72c42017-09-05 12:02:00 -05001398Set Turbo Setting Via REST
1399 [Documentation] Set turbo setting via REST.
1400 [Arguments] ${setting}
Steven Sombar0acb3412018-01-24 09:38:42 -06001401
Steven Sombaraaf72c42017-09-05 12:02:00 -05001402 # Description of argument(s):
Steven Sombar0acb3412018-01-24 09:38:42 -06001403 # setting Value which needs to be set.(i.e. False or True).
Steven Sombaraaf72c42017-09-05 12:02:00 -05001404
1405 ${valueDict}= Create Dictionary data=${setting}
1406 Write Attribute ${SENSORS_URI}host/TurboAllowed value data=${valueDict}
George Keishingef74a8c2017-10-20 10:23:52 -05001407
Steven Sombar0acb3412018-01-24 09:38:42 -06001408
George Keishingef74a8c2017-10-20 10:23:52 -05001409Set Control Boot Mode
1410 [Documentation] Set given boot mode on the boot object path attribute.
1411 [Arguments] ${boot_path} ${boot_mode}
1412
1413 # Description of argument(s):
1414 # boot_path Boot object path.
1415 # Example:
1416 # /xyz/openbmc_project/control/host0/boot
1417 # /xyz/openbmc_project/control/host0/boot/one_time
1418 # boot_mode Boot mode which need to be set.
1419 # Example:
1420 # "xyz.openbmc_project.Control.Boot.Mode.Modes.Regular"
1421
1422 ${valueDict}= Create Dictionary data=${boot_mode}
1423 Write Attribute ${boot_path} BootMode data=${valueDict}
Sridevi Ramesh0d88ab32017-09-21 11:07:28 -05001424
Steven Sombar0acb3412018-01-24 09:38:42 -06001425
Sridevi Ramesh0d88ab32017-09-21 11:07:28 -05001426Copy Address Translation Utils To HOST OS
1427 [Documentation] Copy address translation utils to host OS.
1428
1429 OperatingSystem.File Should Exist ${probe_cpu_tool_path}
1430 ... msg=${probe_cpu_tool_path} doesn't exist.
1431 OperatingSystem.File Should Exist ${probe_cpu_tool_path}
1432 ... msg=${probe_cpu_tool_path} doesn't exist.
1433
1434 scp.Open connection ${OS_HOST} username=${OS_USERNAME}
1435 ... password=${OS_PASSWORD}
1436 scp.Put File ${probe_cpu_tool_path} ${target_file_path}
1437 scp.Put File ${scom_addrs_tool_path} ${target_file_path}
George Keishing034902d2017-11-10 13:15:38 -06001438
1439
1440Verify BMC RTC And UTC Time Drift
1441 [Documentation] Verify that the RTC and UTC time difference is less than
1442 ... the given time_drift_max.
1443 [Arguments] ${time_diff_max}=${10}
1444
1445 # Description of argument(s):
1446 # time_diff_max The max allowable RTC and UTC time difference in seconds.
1447
1448 # Example:
1449 # time_dict:
1450 # [local_time]: Fri 2017-11-03 152756 UTC
1451 # [local_time_seconds]: 1509740876
1452 # [universal_time]: Fri 2017-11-03 152756 UTC
1453 # [universal_time_seconds]: 1509740876
1454 # [rtc_time]: Fri 2016-05-20 163403
1455 # [rtc_time_seconds]: 1463780043
1456 # [time_zone]: n/a (UTC, +0000)
1457 # [network_time_on]: yes
1458 # [ntp_synchronized]: no
1459 # [rtc_in_local_tz]: no
1460
Steven Sombar0acb3412018-01-24 09:38:42 -06001461 ${time}= Get BMC Date Time
George Keishing034902d2017-11-10 13:15:38 -06001462 ${time_diff}= Evaluate
Steven Sombar0acb3412018-01-24 09:38:42 -06001463 ... ${time['universal_time_seconds']} - ${time['rtc_time_seconds']}
George Keishing034902d2017-11-10 13:15:38 -06001464 Should Be True ${time_diff} < ${time_diff_max}
Sweta Potthuri7d3af3c2018-01-29 03:07:07 -06001465
Sweta Potthurif39022d2018-02-06 03:40:07 -06001466
Sweta Potthuri7d3af3c2018-01-29 03:07:07 -06001467Watchdog Object Should Exist
1468 [Documentation] Check that watchdog object exists.
1469
1470 ${resp}= OpenBMC Get Request ${WATCHDOG_URI}host0
1471 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
1472 ... msg=Expected watchdog object does not exist.
Sweta Potthurif39022d2018-02-06 03:40:07 -06001473
1474
1475Validate IP On BMC
1476 [Documentation] Validate IP address is present in set of IP addresses.
1477 [Arguments] ${ip_address} ${ip_data}
1478
1479 # Description of argument(s):
1480 # ip_address IP address to check (e.g. xx.xx.xx.xx).
1481 # ip_data Set of the IP addresses present.
1482
1483 Should Contain Match ${ip_data} ${ip_address}/*
1484 ... msg=${ip_address} not found in the list provided.
George Keishingb9f407b2018-02-20 00:07:46 -06001485
1486
1487Remove Journald Logs
1488 [Documentation] Remove all journald logs and restart service.
1489
1490 ${cmd}= Catenate systemctl stop systemd-journald.service &&
1491 ... rm -rf /var/log/journal && systemctl start systemd-journald.service
1492
1493 BMC Execute Command ${cmd}
1494
George Keishing78ce8dc2018-03-30 11:49:06 -05001495
1496Verify Identify LED State
1497 [Documentation] Verify the identify LED state
1498 ... matches caller's expectations.
1499 [Arguments] ${expected_state}
1500
1501 # Description of argument(s):
1502 # expected_state The LED state expected by the caller ("Blink" or "Off").
1503
1504 ${resp}= Read Attribute ${LED_PHYSICAL_URI}/front_id State
1505 Should Be Equal ${resp}
1506 ... xyz.openbmc_project.Led.Physical.Action.${expected_state}
1507 ... msg=Unexpected LED state.
1508
1509 ${resp}= Read Attribute ${LED_PHYSICAL_URI}/rear_id State
1510 Should Be Equal ${resp}
1511 ... xyz.openbmc_project.Led.Physical.Action.${expected_state}
1512 ... msg=Unexpected LED state.
1513
Sweta Potthuri39255032018-03-28 10:12:14 -05001514
1515Verify The Attribute
1516 [Arguments] ${uri} ${attribute_name} ${attribute_value}
1517
1518 # Description of argument(s):
1519 # uri URI path
1520 # (e.g. "/xyz/openbmc_project/control/host0/TPMEnable").
1521 # attribute_name Name of attribute to be verified (e.g. "TPMEnable").
1522 # attribute_value The expected value of attribute (e.g. "1", "0", etc.)
1523
1524 ${output}= Read Attribute ${uri} ${attribute_name}
1525 Should Be Equal ${attribute_value} ${output}
1526 ... msg=Attribute "${attribute_name} does not have the expected value.