blob: 4e585750276587324747b3a30c8784d5af2d0957 [file] [log] [blame]
Michael Tritz935c18e2017-03-08 11:45:41 -06001*** Settings ***
2Documentation Test Error logging.
3
4Resource ../lib/connection_client.robot
5Resource ../lib/openbmc_ffdc.robot
6Resource ../lib/utils.robot
7Resource ../lib/state_manager.robot
Rahul Maheshwari92972e72017-06-22 00:59:59 -05008Resource ../lib/ipmi_client.robot
George Keishing1d82dd12017-07-24 01:16:23 -05009Resource ../lib/boot_utils.robot
Sivas SRRcf6714f2018-03-26 10:51:29 -050010Resource ../lib/logging_utils.robot
Michael Tritz935c18e2017-03-08 11:45:41 -060011
George Keishing95c371a2017-09-14 15:42:33 -050012Test Setup Test Setup Execution
George Keishingfa2fc162018-01-08 07:41:13 -060013Test Teardown Test Teardown Execution
George Keishing184a89a2017-06-27 07:12:16 -050014Suite Teardown Delete Error Logs And Verify
Michael Tritz935c18e2017-03-08 11:45:41 -060015
George Keishing1d82dd12017-07-24 01:16:23 -050016*** Variables ***
17
18${stack_mode} skip
19
Michael Tritz935c18e2017-03-08 11:45:41 -060020*** Test Cases ***
21
George Keishing1d82dd12017-07-24 01:16:23 -050022Error Log Check After BMC Reboot
23 [Documentation] Check error log after BMC rebooted.
24 [Tags] Error_Log_Check_At_BMC_Ready
25 # 1. Power off.
26 # 2. Delete error logs.
27 # 3. Reboot BMC.
Gunnar Mills917ba1a2018-04-08 16:42:12 -050028 # 4. Check if error log exists.
George Keishing1d82dd12017-07-24 01:16:23 -050029
George Keishing3f4a3eb2017-09-26 10:33:16 -050030 Smart Power Off
George Keishing1d82dd12017-07-24 01:16:23 -050031 Delete Error Logs And Verify
George Keishing3f4a3eb2017-09-26 10:33:16 -050032 OBMC Reboot (off) stack_mode=normal
George Keishing1d82dd12017-07-24 01:16:23 -050033 Error Logs Should Not Exist
34
35
36Error Log Check After Host Poweron
37 [Documentation] Check error log after host has booted.
38 [Tags] Error_Log_Check_At_Host_Booted
39 # 1. Delete error logs
40 # 1. Power on.
Gunnar Mills917ba1a2018-04-08 16:42:12 -050041 # 3. Check if error log exists.
George Keishing1d82dd12017-07-24 01:16:23 -050042
43 Delete Error Logs And Verify
44 REST Power On
45 Error Logs Should Not Exist
46
47
Michael Tritz935c18e2017-03-08 11:45:41 -060048Create Test Error And Verify
49 [Documentation] Create error logs and verify via REST.
50 [Tags] Create_Test_Error_And_Verify
51
52 Create Test Error Log
53 Verify Test Error Log
54
55
56Test Error Persistency On Restart
George Keishing184a89a2017-06-27 07:12:16 -050057 [Documentation] Restart logging service and verify error logs.
Michael Tritz935c18e2017-03-08 11:45:41 -060058 [Tags] Test_Error_Persistency_On_Restart
59
60 Create Test Error Log
61 Verify Test Error Log
Joy Onyerikwu2ef965a2018-05-24 15:23:27 -050062 BMC Execute Command
Michael Tritz935c18e2017-03-08 11:45:41 -060063 ... systemctl restart xyz.openbmc_project.Logging.service
64 Sleep 10s reason=Wait for logging service to restart properly.
George Keishing184a89a2017-06-27 07:12:16 -050065 Verify Test Error Log
Michael Tritz935c18e2017-03-08 11:45:41 -060066
67
George Keishing742b0602017-07-28 14:20:53 -050068Test Error Entry Numbering Reset On Restart
69 [Documentation] Restarts logging service and verify error logs entry start
70 ... from entry "Id" 1.
71 # 1. Create error log.
72 # 2. Verify error log.
73 # 3. Delete error log.
74 # 4. Restart logging service.
75 # 5. Create error log.
76 # 6. Verify new error log entry starts with Id entry 1.
77
78 [Tags] Test_Error_Entry_Numbering_Reset_On_Restart
79 # Example Error logs:
80 # "/xyz/openbmc_project/logging/entry/1": {
81 # "AdditionalData": [
82 # "STRING=FOO"
83 # ],
84 # "Id": 1, <--- Entry value should be 1.
85 # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
86 # "Resolved": 0,
87 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
88 # "Timestamp": 1490818990051,
89 # "associations": []
90 # },
91
92 Create Test Error Log
93 Verify Test Error Log
George Keishingcc71c3c2017-09-20 09:38:03 -050094 Delete All Error Logs
Joy Onyerikwu2ef965a2018-05-24 15:23:27 -050095 BMC Execute Command
George Keishing742b0602017-07-28 14:20:53 -050096 ... systemctl restart xyz.openbmc_project.Logging.service
97 Sleep 10s reason=Wait for logging service to restart properly.
98 Create Test Error Log
99 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
100 ${entry_id}= Read Attribute ${elog_entry[0]} Id
101 Should Be Equal ${entry_id} ${1}
102
103
Michael Tritz935c18e2017-03-08 11:45:41 -0600104Test Error Persistency On Reboot
George Keishing184a89a2017-06-27 07:12:16 -0500105 [Documentation] Reboot BMC and verify error logs.
Michael Tritz935c18e2017-03-08 11:45:41 -0600106 [Tags] Test_Error_Persistency_On_Reboot
107
108 Create Test Error Log
109 Verify Test Error Log
110 Initiate BMC Reboot
111 Wait Until Keyword Succeeds 10 min 10 sec
112 ... Is BMC Ready
George Keishing184a89a2017-06-27 07:12:16 -0500113 Verify Test Error Log
Michael Tritz935c18e2017-03-08 11:45:41 -0600114
115
George Keishing04bc4812017-03-30 03:33:31 -0500116Create Test Error And Verify Resolved Field
117 [Documentation] Create error log and verify "Resolved"
118 ... field is 0.
119 [Tags] Create_Test_Error_And_Verify_Resolved_Field
120
121 # Example Error log:
122 # "/xyz/openbmc_project/logging/entry/1": {
123 # "AdditionalData": [
124 # "STRING=FOO"
125 # ],
126 # "Id": 1,
127 # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
128 # "Resolved": 0,
129 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
130 # "Timestamp": 1490817164983,
131 # "associations": []
132 # },
133
134 # It's work in progress, but it's mnfg need. To mark an error as
135 # resolved, without deleting the error, mfg will set this bool
136 # property.
137 # In this test context we are making sure "Resolved" field is "0"
138 # by default.
139
George Keishingcc71c3c2017-09-20 09:38:03 -0500140 Delete All Error Logs
George Keishing04bc4812017-03-30 03:33:31 -0500141 Create Test Error Log
George Keishing184a89a2017-06-27 07:12:16 -0500142 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
143 ${resolved}= Read Attribute ${elog_entry[0]} Resolved
George Keishing04bc4812017-03-30 03:33:31 -0500144 Should Be True ${resolved} == 0
145
146
147Create Test Errors And Verify Time Stamp
148 [Documentation] Create error logs and verify time stamp.
149 [Tags] Create_Test_Error_And_Verify_Time_Stamp
150
151 # Example Error logs:
152 # "/xyz/openbmc_project/logging/entry/1": {
153 # "AdditionalData": [
154 # "STRING=FOO"
155 # ],
156 # "Id": 1,
157 # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
158 # "Resolved": 0,
159 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
160 # "Timestamp": 1490818990051, <--- Time stamp
161 # "associations": []
162 # },
163 # "/xyz/openbmc_project/logging/entry/2": {
164 # "AdditionalData": [
165 # "STRING=FOO"
166 # ],
167 # "Id": 2,
168 # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
169 # "Resolved": 0,
170 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
171 # "Timestamp": 1490818992116, <---- Time stamp
172 # "associations": []
173 # },
174
George Keishingcc71c3c2017-09-20 09:38:03 -0500175 Delete All Error Logs
George Keishing04bc4812017-03-30 03:33:31 -0500176 Create Test Error Log
177 Create Test Error Log
178 # The error log generated is associated with the epoc time and unique
179 # for every error and in increasing time stamp.
George Keishing184a89a2017-06-27 07:12:16 -0500180 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
181 ${time_stamp1}= Read Attribute ${elog_entry[0]} Timestamp
182 ${time_stamp2}= Read Attribute ${elog_entry[1]} Timestamp
George Keishing6eb10b82017-06-06 10:02:09 -0500183 Should Be True ${time_stamp2} > ${time_stamp1}
George Keishing04bc4812017-03-30 03:33:31 -0500184
Sweta Potthuri9ce8eeb2017-05-08 04:20:11 -0500185Create Test Error Log And Delete
186 [Documentation] Create an error log and delete it.
187 [Tags] Create_Test_Error_Log_And_Delete
188
189 Delete Error Logs And Verify
190 Create Test Error Log
191 Delete Error Logs And Verify
192
193Create Multiple Test Error Logs And Delete All
194 [Documentation] Create multiple error logs and delete all.
195 [Tags] Create_Multiple_Test_Error_Logs_And_Delete_All
196
197 Delete Error Logs And Verify
198 Create Test Error Log
199 Create Test Error Log
200 Create Test Error Log
201 Delete Error Logs And Verify
202
203Create Two Test Error Logs And Delete One
204 [Documentation] Create two error logs and delete the first entry.
205 [Tags] Create_Two_Test_Error_Logs_And_Delete_One
206
207 Delete Error Logs And Verify
208 Create Test Error Log
George Keishing184a89a2017-06-27 07:12:16 -0500209 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
Sweta Potthuri9ce8eeb2017-05-08 04:20:11 -0500210 Create Test Error Log
George Keishing184a89a2017-06-27 07:12:16 -0500211 Delete Error log Entry ${elog_entry[0]}
212 ${resp}= OpenBMC Get Request ${elog_entry[0]}
Sweta Potthuri9ce8eeb2017-05-08 04:20:11 -0500213 Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND}
George Keishing04bc4812017-03-30 03:33:31 -0500214
Rahul Maheshwari92972e72017-06-22 00:59:59 -0500215
216Verify IPMI SEL Version
217 [Documentation] Verify IPMI SEL's version info.
218 [Tags] Verify_IPMI_SEL_Version
219
220 ${version_info}= Get IPMI SEL Setting Version
221 ${setting_status}= Fetch From Left ${version_info} (
222 ${setting_status}= Evaluate $setting_status.replace(' ','')
223
224 Should Be True ${setting_status} >= 1.5
Rahul Maheshwarib44b6ec2017-07-05 00:23:03 -0500225 Should Contain ${version_info} v2 compliant case_insensitive=True
Rahul Maheshwari92972e72017-06-22 00:59:59 -0500226
227
George Keishing845d9dc2017-06-16 14:49:29 -0500228Verify Watchdog Timedout Error
229 [Documentation] Trigger watchdog timed out and verify errorlog generated.
230 [Tags] Verify_Watchdog_Timedout_Error
231
George Keishingd397e6a2017-08-11 11:19:24 -0500232 REST Power Off
233 REST Power On
234
George Keishing845d9dc2017-06-16 14:49:29 -0500235 # Clear errors if there are any.
George Keishingcc71c3c2017-09-20 09:38:03 -0500236 Delete All Error Logs
George Keishing845d9dc2017-06-16 14:49:29 -0500237
George Keishing845d9dc2017-06-16 14:49:29 -0500238 Trigger Host Watchdog Error
239
George Keishing45921ea2017-11-23 04:32:56 -0600240 # Logging took time to generate the timedout error.
241 Wait Until Keyword Succeeds 2 min 30 sec
242 ... Verify Watchdog Errorlog Content
George Keishing845d9dc2017-06-16 14:49:29 -0500243
244
Rahul Maheshwarib44b6ec2017-07-05 00:23:03 -0500245Verify IPMI SEL Delete
246 [Documentation] Verify IPMI SEL delete operation.
247 [Tags] Verify_IPMI_SEL_Delete
248
249 Delete Error Logs And Verify
250 Create Test Error Log
251
252 ${sel_list}= Run IPMI Standard Command sel list
253 # Example of SEL List:
254 # 4 | 04/21/2017 | 10:51:16 | System Event #0x01 | Undetermined system hardware failure | Asserted
255
256 ${sel_entry}= Fetch from Left ${sel_list} |
257 ${sel_entry}= Evaluate $sel_entry.replace(' ','')
258 ${sel_entry}= Convert To Integer 0x${sel_entry}
259
260 ${sel_delete}= Run IPMI Standard Command sel delete ${sel_entry}
261 Should Be Equal As Strings ${sel_delete} Deleted entry ${sel_entry}
262 ... case_insensitive=True
263
264 ${sel_list}= Run IPMI Standard Command sel list
265 Should Be Equal As Strings ${sel_list} SEL has no entries
266 ... case_insensitive=True
267
268
269Verify Empty SEL
270 [Documentation] Verify empty SEL list.
271 [Tags] Verify_Empty_SEL
272
273 Delete Error Logs And Verify
274
275 ${resp}= Run IPMI Standard Command sel list
276 Should Contain ${resp} SEL has no entries case_insensitive=True
277
278
279Delete Non Existing SEL Entry
280 [Documentation] Delete non existing SEL entry.
281 [Tags] Delete_Non_Existing_SEL_Entry
282
283 Delete Error Logs And Verify
284 ${sel_delete}= Run Keyword And Expect Error *
285 ... Run IPMI Standard Command sel delete 100
286 Should Contain ${sel_delete} Unable to delete entry
287 ... case_insensitive=True
288
289
290Delete Invalid SEL Entry
291 [Documentation] Delete invalid SEL entry.
292 [Tags] Delete_Invalid_SEL_Entry
293
294 ${sel_delete}= Run Keyword And Expect Error *
295 ... Run IPMI Standard Command sel delete abc
296 Should Contain ${sel_delete} Given SEL ID 'abc' is invalid
297 ... case_insensitive=True
298
299
Rahul Maheshwarie89b5092017-06-23 00:24:15 -0500300Verify IPMI SEL Entries
301 [Documentation] Verify IPMI SEL's entries info.
302 [Tags] Verify_IPMI_SEL_Entries
303
304 Delete Error Logs And Verify
305
306 # Generate error logs of random count.
307 ${count}= Evaluate random.randint(1, 5) modules=random
308 Repeat Keyword ${count} Create Test Error Log
309
310 ${sel_entries_count}= Get IPMI SEL Setting Entries
311 Should Be Equal As Strings ${sel_entries_count} ${count}
312
313
314Verify IPMI SEL Last Add Time
315 [Documentation] Verify IPMI SEL's last added timestamp.
316 [Tags] Verify_IPMI_SEL_Last_Add_Time
317
318 Create Test Error Log
319 ${sel_time}= Run IPMI Standard Command sel time get
320 ${sel_time}= Convert Date ${sel_time}
321 ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True
322
323 ${sel_last_add_time}= Get IPMI SEL Setting Last Add Time
324 ${sel_last_add_time}= Convert Date ${sel_last_add_time}
325 ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=True
326
327 ${time-diff}=
328 ... Subtract Date From Date ${sel_last_add_time} ${sel_time}
329
330 # Verify if the delay in current time check and last add SEL time
331 # is less or equals to 2 seconds.
332 Should Be True ${time-diff} <= 2
333
Sweta Potthuri96d82492017-08-23 09:03:38 -0500334Verify Error Logs Capping
335 [Documentation] Verify error logs capping.
336 [Tags] Verify_Error_Logs_Capping
337
338 Delete Error Logs And Verify
George Keishingfa2fc162018-01-08 07:41:13 -0600339 ${cmd}= Catenate for i in {1..201}; do /tmp/tarball/bin/logging-test -c
340 ... AutoTestSimple; done
Joy Onyerikwu2ef965a2018-05-24 15:23:27 -0500341 BMC Execute Command ${cmd}
Sweta Potthuri96d82492017-08-23 09:03:38 -0500342 ${count}= Count Error Entries
George Keishingfa2fc162018-01-08 07:41:13 -0600343 Run Keyword If ${count} > 200
344 ... Fail Error logs created exceeded max capacity 200.
Rahul Maheshwarib44b6ec2017-07-05 00:23:03 -0500345
Sweta Potthurid9e746e2018-01-08 09:38:42 -0600346Test Error Log Rotation
347 [Documentation] Verify creation of 201 error log is replaced by entry id 1.
348 [Tags] Test_Error_Log_Rotation
349
350 Delete Error Logs And Verify
351
352 # Restart service.
353 BMC Execute Command
354 ... systemctl restart xyz.openbmc_project.Logging.service
355 Sleep 10s reason=Wait for logging service to restart properly.
356
357 # Create 200 error logs.
358 ${cmd}= Catenate for i in {1..200}; do /tmp/tarball/bin/logging-test -c
359 ... AutoTestSimple;done
360 BMC Execute Command ${cmd}
361
362 # Check the response for 200th error log.
363 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${200}
364 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
365
366 # Check if error log with id 1 exists.
367 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${1}
368 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
369
370 # Create error log and verify the entry ID is 201 and not 1.
371 Create Test Error Log
372 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${201}
373 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
374
375 # Error log 1 is not present.
376 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${1}
377 Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND}
378
Michael Tritz935c18e2017-03-08 11:45:41 -0600379*** Keywords ***
380
George Keishing95c371a2017-09-14 15:42:33 -0500381Test Setup Execution
382 [Documentation] Do test case setup tasks.
383
384 ${status}= Run Keyword And Return Status Logging Test Binary Exist
385 Run Keyword If ${status} == ${False} Install Tarball
386 Delete Error Logs And Verify
387
388
George Keishingfa2fc162018-01-08 07:41:13 -0600389Test Teardown Execution
George Keishing845d9dc2017-06-16 14:49:29 -0500390 [Documentation] Do the post test teardown.
391 # 1. Capture FFDC on test failure.
392 # 2. Delete error logs.
393 # 3. Close all open SSH connections.
394 # 4. Clear all REST sessions.
395
396 FFDC On Test Case Fail
George Keishingcc71c3c2017-09-20 09:38:03 -0500397 Delete All Error Logs
George Keishing845d9dc2017-06-16 14:49:29 -0500398 Close All Connections