blob: 0fd79ea0be1620afb9950a69ced607357362fab9 [file] [log] [blame]
George Keishing6a514ef2018-08-27 05:19:19 -05001*** Settings ***
2Documentation Remote logging test for rsyslog.
3
4# Program arguments:
5# REMOTE_LOG_SERVER_HOST The host name or IP address of the remote
George Keishing67133672018-08-31 10:23:15 -05006# logging server.
George Keishing6a514ef2018-08-27 05:19:19 -05007# REMOTE_LOG_SERVER_PORT The port number for the remote logging server.
George Keishing67133672018-08-31 10:23:15 -05008# REMOTE_USERNAME The username for the remote logging server.
9# REMOTE_PASSWORD The password for the remote logging server.
George Keishing6a514ef2018-08-27 05:19:19 -050010
George Keishing76a5daf2018-09-14 06:00:01 -050011Library String
George Keishing6a514ef2018-08-27 05:19:19 -050012Resource ../lib/openbmc_ffdc.robot
George Keishing76a5daf2018-09-14 06:00:01 -050013Resource ../lib/boot_utils.robot
George Keishingbbfc3942018-09-19 10:15:48 -050014Resource ../lib/remote_logging_utils.robot
George Keishing9a0d4802018-08-31 13:29:40 -050015Library ../lib/gen_misc.py
George Keishing6a514ef2018-08-27 05:19:19 -050016
17Suite Setup Suite Setup Execution
George Keishing216e5842018-08-31 14:58:15 -050018Test Setup Test Setup Execution
George Keishing6a514ef2018-08-27 05:19:19 -050019Test Teardown FFDC On Test Case Fail
20
George Keishing2cec1862018-09-03 12:59:25 -050021*** Variables ***
22
George Keishingc1ca5e92018-09-03 11:19:23 -050023# Strings to check from journald.
George Keishing13451ed2018-09-06 12:46:16 -050024${BMC_STOP_MSG} Stopping Phosphor IPMI BT DBus Bridge
25${BMC_START_MSG} Starting Flush Journal to Persistent Storage
26${BMC_BOOT_MSG} Startup finished in
27${BMC_SYSLOG_REGEX} dropbear|vrm-control.sh
George Keishingeacb1882018-09-20 00:11:00 -050028${RSYSLOG_REGEX} start|exiting on signal 15|there are no active actions configured
George Keishing13451ed2018-09-06 12:46:16 -050029${RSYSLOG_RETRY_REGEX} suspended, next retry
George Keishingc1ca5e92018-09-03 11:19:23 -050030
George Keishing6a514ef2018-08-27 05:19:19 -050031*** Test Cases ***
32
George Keishingeb9fc592018-09-18 07:58:17 -050033Test BMC Hostname Service And Verify
34 [Documentation] Write to hostname interface and verify via REST and
35 ... 'hostname' command.
36 [Tags] Test_BMC_Hostname_Service_And_Verify
37
38 ${openbmc_host_name} ${openbmc_ip} ${openbmc_short_name}=
39 ... Get Host Name IP host=${OPENBMC_HOST} short_name=1
40
41 ${host_name_dict}= Create Dictionary data=${openbmc_short_name}
42 Write Attribute ${NETWORK_MANAGER}config HostName data=${host_name_dict}
43 ... verify=${TRUE} expected_value=${openbmc_short_name}
44
45 ${hostname} ${stderr} ${rc}= BMC Execute Command hostname
46
47 Should Be Equal As Strings ${hostname} ${openbmc_short_name}
48 ... msg=The hostname interface ${openbmc_short_name} and command value ${hostname} do not match.
49
50 # Override the suite hostname variable if this test is executed.
51 Set Suite Variable ${bmc_hostname} ${openbmc_short_name}
52
53
George Keishing13a64e62018-09-10 13:07:36 -050054Verify REST Logging On BMC Journal When Disabled
55 [Documentation] Enable REST logging and verify from journald.
George Keishingb77a3572018-09-13 23:26:28 -050056 [Tags] Verify_REST_Logging_On_BMC_Journal_When_Disabled
George Keishing13a64e62018-09-10 13:07:36 -050057
58 ${log_dict}= Create Dictionary data=${False}
59 Write Attribute ${BMC_LOGGING_URI}${/}rest_api_logs Enabled data=${log_dict}
60 ... verify=${True} expected_value=${False}
61
George Keishingeacb1882018-09-20 00:11:00 -050062 # If it was enabled prior, this REST footprint will show up.
63 # Takes around 5 seconds for the REST to restart service when policy is changed.
64 Sleep 10s
65
George Keishing4df837f2018-09-19 14:06:01 -050066 ${login_footprint}= Catenate user:root POST http://127.0.0.1:8081/login json:None 200 OK
67 # Example: Just get the message part of the syslog
68 # user:root POST http://127.0.0.1:8081/login json:None 200 OK
69 ${cmd}= Catenate SEPARATOR= --no-pager | egrep '${login_footprint}'
70 ... | awk -F': ' '{print $2}'
71
72 Start Journal Log filter=${cmd}
George Keishing13a64e62018-09-10 13:07:36 -050073 Initialize OpenBMC
George Keishing4df837f2018-09-19 14:06:01 -050074 Sleep 5s
75 ${bmc_journald}= Stop Journal Log
George Keishing13a64e62018-09-10 13:07:36 -050076
George Keishing4df837f2018-09-19 14:06:01 -050077 Should Be Empty ${bmc_journald}
George Keishing13a64e62018-09-10 13:07:36 -050078 ... msg=${bmc_journald} contains unexpected REST entries.
79
80
George Keishingb77a3572018-09-13 23:26:28 -050081Verify REST Logging On BMC Journal When Enabled
George Keishing13a64e62018-09-10 13:07:36 -050082 [Documentation] Enable REST logging and verify from journald.
George Keishingb77a3572018-09-13 23:26:28 -050083 [Tags] Verify_REST_Logging_On_BMC_Journal_When_Enabled
George Keishing13a64e62018-09-10 13:07:36 -050084
85 ${log_dict}= Create Dictionary data=${True}
86 Write Attribute ${BMC_LOGGING_URI}${/}rest_api_logs Enabled data=${log_dict}
87 ... verify=${True} expected_value=${True}
88
89 # Sep 10 14:34:35 witherspoon phosphor-gevent[1288]: 127.0.0.1 user:root POST http://127.0.0.1:8081/login json:None 200 OK
90 Initialize OpenBMC
91
92 ${bmc_journald} ${stderr} ${rc}= BMC Execute Command
93 ... journalctl --no-pager
94
95 Should Contain ${bmc_journald} user:root POST http://127.0.0.1:8081/login json:None 200 OK
96 ... msg=${bmc_journald} doesn't contains REST entries.
97
98
George Keishing38733f32018-09-20 05:56:50 -050099Test Remote API Valid Config Combination
100 [Documentation] Verify valid combination of address and port.
101 [Tags] Test_Remote_API_Valid_Config_Combination
102 [Template] Verify Configure Remote Logging Server
103 # Forego normal test setup:
104 [Setup] No Operation
105
106 # Address Port Expected result
107 ${EMPTY} ${REMOTE_LOG_SERVER_PORT} ${True}
108 ${REMOTE_LOG_SERVER_HOST} ${REMOTE_LOG_SERVER_PORT} ${True}
109 remotelog.xzy.com ${REMOTE_LOG_SERVER_PORT} ${True}
110 ${REMOTE_LOG_SERVER_HOST} ${0} ${True}
111
112
113Test Remote API Invalid Config Combination
114 [Documentation] Verify invalid combination of address and port.
115 [Tags] Test_Remote_API_Invalid_Config_Combination
116 [Template] Verify Configure Remote Logging Server
117 # Forego normal test setup:
118 [Setup] No Operation
119
120 # Address Port Expected result
121 ${0} ${REMOTE_LOG_SERVER_PORT} ${False}
122 "0" ${REMOTE_LOG_SERVER_PORT} ${False}
123 ${REMOTE_LOG_SERVER_HOST} ${EMPTY} ${False}
124 ${REMOTE_LOG_SERVER_HOST} "0" ${False}
125
126
George Keishing6a514ef2018-08-27 05:19:19 -0500127Test Remote Logging REST Interface And Verify Config
128 [Documentation] Test remote logging interface and configuration.
129 [Tags] Test_Remote_Logging_REST_Interface_And_Verify_Config
130
George Keishing6a514ef2018-08-27 05:19:19 -0500131 Verify Rsyslog Config On BMC
132
George Keishingbbfc3942018-09-19 10:15:48 -0500133 Configure Remote Log Server With Parameters remote_host=${EMPTY} remote_port=0
George Keishing6a514ef2018-08-27 05:19:19 -0500134 Verify Rsyslog Config On BMC remote_host=remote-host remote_port=port
135
136
George Keishing13451ed2018-09-06 12:46:16 -0500137Test Remote Logging Invalid Port Config And Verify BMC Journald
138 [Documentation] Test remote logging interface and configuration.
139 [Tags] Test_Remote_Logging_Invalid_Port_Config_And_Verify_BMC_Journald
140
141 # Invalid port derived by (REMOTE_LOG_SERVER_PORT + 1) port config setting.
142 ${INVALID_PORT}= Evaluate ${REMOTE_LOG_SERVER_PORT} + ${1}
George Keishingbbfc3942018-09-19 10:15:48 -0500143 Configure Remote Log Server With Parameters
George Keishing13451ed2018-09-06 12:46:16 -0500144 ... remote_host=${REMOTE_LOG_SERVER_HOST} remote_port=${INVALID_PORT}
145
146 Sleep 3s
George Keishing96904c42018-09-20 09:25:30 -0500147 # rsyslogd[1870]: action 'action 0' suspended, next retry is Fri Sep 14 05:47:39 2018 [v8.29.0 try http://www.rsyslog.com/e/2007 ]
George Keishing13451ed2018-09-06 12:46:16 -0500148 ${bmc_journald} ${stderr} ${rc}= BMC Execute Command
149 ... journalctl -b --no-pager | egrep 'rsyslog.*${RSYSLOG_RETRY_REGEX}'
150
151 Should Contain ${bmc_journald} ${RSYSLOG_RETRY_REGEX}
152 ... msg=${bmc_journald} doesn't contain rsyslog retry entries.
153
154
George Keishingc1ca5e92018-09-03 11:19:23 -0500155Verify Rsyslog Does Not Log On BMC
156 [Documentation] Check that rsyslog journald doesn't log on BMC.
157 [Tags] Verify_Rsyslog_Does_Not_Log_On_BMC
158
159 # Expected filter rsyslog entries.
160 # Example:
George Keishing96904c42018-09-20 09:25:30 -0500161 # syslogd[3356]: [origin software="rsyslogd" swVersion="8.29.0" x-pid="3356" x-info="http://www.rsyslog.com"] exiting on signal 15.
162 # rsyslogd[3364]: [origin software="rsyslogd" swVersion="8.29.0" x-pid="3364" x-info="http://www.rsyslog.com"] start
George Keishingc1ca5e92018-09-03 11:19:23 -0500163 ${bmc_journald} ${stderr} ${rc}= BMC Execute Command
George Keishing3ad0c432018-09-21 00:51:08 -0500164 ... journalctl -b --no-pager | egrep 'rsyslog' | egrep -Ev '${RSYSLOG_REGEX}|${RSYSLOG_RETRY_REGEX}'
George Keishingc1ca5e92018-09-03 11:19:23 -0500165 ... ignore_err=${1}
166
167 Should Be Empty ${bmc_journald}
168 ... msg=${bmc_journald} contains unexpected rsyslog entries.
169
170
George Keishing67133672018-08-31 10:23:15 -0500171Verfiy BMC Journald Synced To Remote Logging Server
172 [Documentation] Check that BMC journald is sync to remote rsyslog.
173 [Tags] Verfiy_BMC_Journald_Synced_To_Remote_Logging_Server
174
George Keishing67133672018-08-31 10:23:15 -0500175 # Restart BMC dump service and get the last entry of the journald.
176 # Example:
George Keishing96904c42018-09-20 09:25:30 -0500177 # systemd[1]: Started Phosphor Dump Manager.
George Keishing67133672018-08-31 10:23:15 -0500178 BMC Execute Command
179 ... systemctl restart xyz.openbmc_project.Dump.Manager.service
180
181 ${bmc_journald} ${stderr} ${rc}= BMC Execute Command
George Keishing82ea1d42018-09-11 13:45:36 -0500182 ... journalctl --no-pager | grep 'Started Phosphor Dump Manager'
George Keishing67133672018-08-31 10:23:15 -0500183
George Keishing96904c42018-09-20 09:25:30 -0500184 # systemd[1]: Started Phosphor Dump Manager.
George Keishing82ea1d42018-09-11 13:45:36 -0500185 ${cmd}= Catenate SEPARATOR= egrep '${bmc_hostname}.*Started Phosphor Dump Manager' /var/log/syslog
George Keishing67133672018-08-31 10:23:15 -0500186 ${remote_journald}= Remote Logging Server Execute Command command=${cmd}
187
George Keishing216e5842018-08-31 14:58:15 -0500188 # TODO: rsyslog configuration and time date template to match BMC journald.
189 # Compare the BMC journlad log. Example:
190 # systemd[1]: Started Phosphor Dump Manager.
George Keishing82ea1d42018-09-11 13:45:36 -0500191 Should Contain ${remote_journald} ${bmc_journald.split('${bmc_hostname}')[1][0]}
George Keishing216e5842018-08-31 14:58:15 -0500192 ... msg= ${bmc_journald} doesn't match remote rsyslog:${remote_journald}.
George Keishing67133672018-08-31 10:23:15 -0500193
194
George Keishing2cec1862018-09-03 12:59:25 -0500195Verify Journald Post BMC Reset
196 [Documentation] Check that BMC journald is sync'ed to remote rsyslog after
197 ... BMC reset.
198 [Tags] Verify_Journald_Post_BMC_Reset
199
200 ${hostname} ${stderr} ${rc}= BMC Execute Command hostname
201 OBMC Reboot (off)
202
203 ${cmd}= Catenate grep ${hostname} /var/log/syslog |
204 ... egrep '${BMC_STOP_MSG}|${BMC_START_MSG}|${BMC_BOOT_MSG}'
205 ${remote_journald}= Remote Logging Server Execute Command command=${cmd}
206
207 # 1. Last reboot message to verify.
208 Should Contain ${remote_journald} ${BMC_STOP_MSG}
209 ... msg=The remote journald doesn't contain the IPMI shutdown message: ${BMC_STOP_MSG}.
210
211 # 2. Earliest booting message on journald.
212 Should Contain ${remote_journald} ${BMC_START_MSG}
213 ... msg=The remote journald doesn't contain the start message: ${BMC_START_MSG}.
214
215 # 3. Unique boot to standby message.
216 # Startup finished in 9.961s (kernel) + 1min 59.039s (userspace) = 2min 9.000s
217 ${bmc_journald} ${stderr} ${rc}= BMC Execute Command
218 ... journalctl --no-pager | egrep '${BMC_BOOT_MSG}' | tail -1
219
220 Should Contain ${remote_journald}
221 ... ${bmc_journald.split('${hostname}')[1]}
222 ... msg=The remote journald doesn't contain the boot message: ${BMC_BOOT_MSG}.
223
224
George Keishingcc18f7c2018-09-03 09:40:11 -0500225Verify BMC Journald Contains No Credential Data
George Keishingcb3b9b62018-09-14 12:11:19 -0500226 [Documentation] Check that BMC journald doesn't log any credential data.
George Keishingcc18f7c2018-09-03 09:40:11 -0500227 [Tags] Verify_BMC_Journald_Contains_No_Credential_Data
228
229 Open Connection And Log In
230 ${bmc_journald} ${stderr} ${rc}= BMC Execute Command
231 ... journalctl -o json-pretty | cat
232
233 Should Not Contain Any ${bmc_journald} ${OPENBMC_PASSWORD}
234 ... msg=Journald logs BMC credentials/password ${OPENBMC_PASSWORD}.
235
236
George Keishing9a0d4802018-08-31 13:29:40 -0500237Audit BMC SSH Login And Remote Logging
238 [Documentation] Check that the SSH login to BMC is logged and synced to
239 ... remote logging server.
240 [Tags] Audit_BMC_SSH_Login_And_Remote_Logging
241
George Keishingaef483e2018-09-19 07:57:22 -0500242 ${login_footprint}= Catenate Started SSH Per-Connection Server
243 # Example: Just get the message part of the syslog
244 # Started SSH Per-Connection Server (xx.xx.xx.xx:51292)
245 ${cmd}= Catenate SEPARATOR= --no-pager | egrep '${login_footprint}'
246 ... | awk -F': ' '{print $2}'
George Keishing9a0d4802018-08-31 13:29:40 -0500247
George Keishingaef483e2018-09-19 07:57:22 -0500248 Start Journal Log filter=${cmd}
George Keishing9a0d4802018-08-31 13:29:40 -0500249 Open Connection And Log In
George Keishingaef483e2018-09-19 07:57:22 -0500250 Sleep 5s
251 ${bmc_journald}= Stop Journal Log
252 @{ssh_entry}= Split To Lines ${bmc_journald}
George Keishing9a0d4802018-08-31 13:29:40 -0500253
George Keishingaef483e2018-09-19 07:57:22 -0500254 ${cmd}= Catenate SEPARATOR= egrep -E '*${bmc_hostname}.*${login_footprint}' /var/log/syslog
George Keishing9a0d4802018-08-31 13:29:40 -0500255
256 ${remote_journald}= Remote Logging Server Execute Command command=${cmd}
257
George Keishingaef483e2018-09-19 07:57:22 -0500258 Should Contain ${remote_journald} ${ssh_entry[0]}
George Keishing9a0d4802018-08-31 13:29:40 -0500259 ... msg=${remote_journald} don't contain ${bmc_journald} entry.
260
261
George Keishing76a5daf2018-09-14 06:00:01 -0500262Boot Host And Verify Data Is Synced To Remote Server
263 [Documentation] Boot host and verify the power on sequence logs are synced
264 ... to remote logging server.
265 [Tags] Boot_Host_And_Verify_Data_Is_Synced_To_Remote_Server
266
267 ${cmd}= Catenate SEPARATOR= --no-pager | egrep -Ev '${BMC_SYSLOG_REGEX}'
268 ... | awk -F': ' '{print $2}'
269
270 # Example: Just get the message part of the syslog
271 # Started OpenPOWER OCC Active Disable.
272 Start Journal Log filter=${cmd}
273
274 # Irrespective of the outcome, the journald should be synced.
275 Run Keyword And Ignore Error REST Power On
276 ${bmc_journald}= Stop Journal Log
277
278 ${cmd}= Catenate SEPARATOR= egrep '${bmc_hostname}' /var/log/syslog
279 ${remote_journald}= Remote Logging Server Execute Command command=${cmd}
280
281 @{lines}= Split To Lines ${bmc_journald}
282 :FOR ${line} IN @{lines}
283 \ Log To Console \n ${line}
284 \ Should Contain ${remote_journald} ${line}
285 ... mgs=${line} line doesn't contain in ${remote_journald}.
286
287
George Keishing6a514ef2018-08-27 05:19:19 -0500288*** Keywords ***
289
290Suite Setup Execution
291 [Documentation] Do the suite setup.
292
293 Should Not Be Empty ${REMOTE_LOG_SERVER_HOST}
294 Should Not Be Empty ${REMOTE_LOG_SERVER_PORT}
George Keishing67133672018-08-31 10:23:15 -0500295 Should Not Be Empty ${REMOTE_USERNAME}
296 Should Not Be Empty ${REMOTE_PASSWORD}
George Keishing6a514ef2018-08-27 05:19:19 -0500297 Ping Host ${REMOTE_LOG_SERVER_HOST}
George Keishing67133672018-08-31 10:23:15 -0500298 Remote Logging Server Execute Command true
George Keishing6a514ef2018-08-27 05:19:19 -0500299 Remote Logging Interface Should Exist
300
George Keishing216e5842018-08-31 14:58:15 -0500301 ${hostname} ${stderr} ${rc}= BMC Execute Command /bin/hostname
302 Set Suite Variable ${bmc_hostname} ${hostname}
George Keishingbbfc3942018-09-19 10:15:48 -0500303 Configure Remote Log Server With Parameters
George Keishing216e5842018-08-31 14:58:15 -0500304
305
306Test Setup Execution
307 [Documentation] Do the test setup.
308
George Keishingb77a3572018-09-13 23:26:28 -0500309 # Retain only the past 1 second log:
310 BMC Execute Command journalctl --vacuum-time=1s
George Keishing13a64e62018-09-10 13:07:36 -0500311
312 ${config_status}= Run Keyword And Return Status
George Keishing216e5842018-08-31 14:58:15 -0500313 ... Get Remote Log Server Configured
314
George Keishingbbfc3942018-09-19 10:15:48 -0500315 Run Keyword If ${config_status}==${FALSE}
316 ... Configure Remote Log Server With Parameters
George Keishing216e5842018-08-31 14:58:15 -0500317
318 ${ActiveState}= Get Service Attribute ActiveState rsyslog.service
319 Should Be Equal active ${ActiveState}
320 ... msg=rsyslog logging service not in active state.
321
George Keishing6a514ef2018-08-27 05:19:19 -0500322
323Remote Logging Interface Should Exist
324 [Documentation] Check that the remote logging URI exist.
325
326 ${resp}= OpenBMC Get Request ${REMOTE_LOGGING_URI}
327 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
George Keishing38733f32018-09-20 05:56:50 -0500328
329
330Verify Configure Remote Logging Server
331 [Documentation] Configure the remote logging REST interface on BMC.
332 [Arguments] ${remote_host} ${remote_port} ${expectation}
333
334 # Description of argument(s):
335 # remote_host The host name or IP address of the remote logging server
336 # (e.g. "xx.xx.xx.xx").
337 # remote_port Remote ryslog server port number (e.g. "514").
338 # expectation Expect boolean True/False.
339
340
341 ${status}= Run Keyword And Return Status
George Keishingccd6d092018-09-21 00:39:43 -0500342 ... Configure Remote Log Server With Parameters remote_host=${remote_host} remote_port=${remote_port}
George Keishing38733f32018-09-20 05:56:50 -0500343
344 Should Be Equal ${status} ${expectation}
345 ... msg=Test result ${status} and expectation ${expectation} do not match.