blob: 84104a75e6834cb37bfd49c06cf18673bef348d5 [file] [log] [blame]
George Keishingc37fca62017-11-24 05:07:32 -06001*** Settings ***
2Documentation Error logging utility keywords.
3
4Resource rest_client.robot
George Keishinga6c00892021-10-11 01:09:54 -05005Resource bmc_redfish_utils.robot
George Keishingc37fca62017-11-24 05:07:32 -06006Variables ../data/variables.py
Rahul Maheshwarib8580dd2021-05-10 00:57:33 -05007Variables ../data/pel_variables.py
Michael Walshdb1adf42020-03-20 15:05:46 -05008
9*** Variables ***
10
11
12# Define variables for use by callers of 'Get Error Logs'.
George Keishinga6c00892021-10-11 01:09:54 -050013${low_severity_errlog_regex} \\.(Informational|Notice|Debug|OK)$
Michael Walshdb1adf42020-03-20 15:05:46 -050014&{low_severity_errlog_filter} Severity=${low_severity_errlog_regex}
15&{low_severity_errlog_filter_args} filter_dict=${low_severity_errlog_filter} regex=${True} invert=${True}
16# The following is equivalent to &{low_severity_errlog_filter_args} but the name may be more intuitive for
17# users. Example usage:
18# ${err_logs}= Get Error Logs &{filter_low_severity_errlogs}
19&{filter_low_severity_errlogs} &{low_severity_errlog_filter_args}
20
George Keishingc37fca62017-11-24 05:07:32 -060021*** Keywords ***
22
23Get Logging Entry List
24 [Documentation] Get logging entry and return the object list.
25
26 ${entry_list}= Create List
27 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}list quiet=${1}
28 Return From Keyword If ${resp.status_code} == ${HTTP_NOT_FOUND}
29 ${jsondata}= To JSON ${resp.content}
30
Marissa Garza20ccfc72020-06-19 12:51:10 -050031 FOR ${entry} IN @{jsondata["data"]}
32 Continue For Loop If '${entry.rsplit('/', 1)[1]}' == 'callout'
33 Append To List ${entry_list} ${entry}
34 END
George Keishingc37fca62017-11-24 05:07:32 -060035
36 # Logging entries list.
37 # ['/xyz/openbmc_project/logging/entry/14',
38 # '/xyz/openbmc_project/logging/entry/15']
39 [Return] ${entry_list}
40
41
42Logging Entry Should Exist
43 [Documentation] Find the matching message id and return the entry id.
44 [Arguments] ${message_id}
45
46 # Description of argument(s):
47 # message_id Logging message string.
48 # Example: "xyz.openbmc_project.Common.Error.InternalFailure"
49
George Keishing5ac6e172018-04-09 12:44:15 -050050 @{elog_entries}= Get Logging Entry List
George Keishingc37fca62017-11-24 05:07:32 -060051
Marissa Garza20ccfc72020-06-19 12:51:10 -050052 FOR ${entry} IN @{elog_entries}
53 ${resp}= Read Properties ${entry}
54 ${status}= Run Keyword And Return Status
55 ... Should Be Equal As Strings ${message_id} ${resp["Message"]}
56 Return From Keyword If ${status} == ${TRUE} ${entry}
57 END
George Keishingc37fca62017-11-24 05:07:32 -060058
59 Fail No ${message_id} logging entry found.
60
Steven Sombar95672942018-03-23 11:24:58 -050061
62Get Error Logs
Michael Walshdb1adf42020-03-20 15:05:46 -050063 [Documentation] Return the BMC error logs as a dictionary.
64 [Arguments] ${quiet}=1 &{filter_struct_args}
65
66 # Example of call using pre-defined filter args (defined above).
67
68 # ${err_logs}= Get Error Logs &{filter_low_severity_errlogs}
69
70 # In this example, all error logs with "Severity" fields that are neither Informational, Debug nor
71 # Notice will be returned.
Steven Sombar95672942018-03-23 11:24:58 -050072
73 # Description of argument(s):
Michael Walshdb1adf42020-03-20 15:05:46 -050074 # quiet Indicates whether this keyword should run without any output to the
75 # console, 0 = verbose, 1 = quiet.
76 # filter_struct_args filter_struct args (e.g. filter_dict, regex, etc.) to be passed directly
77 # to the Filter Struct keyword. See its prolog for details.
Steven Sombar95672942018-03-23 11:24:58 -050078
79 # The length of the returned dictionary indicates how many logs there are.
Michael Walshdb1adf42020-03-20 15:05:46 -050080
81 # Use 'Print Error Logs' to print. Example:
82
83 # Print Error Logs ${error_logs} Message.
Steven Sombar95672942018-03-23 11:24:58 -050084
85 ${status} ${error_logs}= Run Keyword And Ignore Error Read Properties
Michael Walshdb1adf42020-03-20 15:05:46 -050086 ... /xyz/openbmc_project/logging/entry/enumerate timeout=30 quiet=${quiet}
87 Return From Keyword If '${status}' == 'FAIL' &{EMPTY}
88 ${num_filter_struct_args}= Get Length ${filter_struct_args}
89 Return From Keyword If '${num_filter_struct_args}' == '${0}' ${error_logs}
90 ${filtered_error_logs}= Filter Struct ${error_logs} &{filter_struct_args}
91 [Return] ${filtered_error_logs}
Sivas SRRcf6714f2018-03-26 10:51:29 -050092
93
94Get IPMI SEL Setting
95 [Documentation] Returns status for given IPMI SEL setting.
96 [Arguments] ${setting}
97 # Description of argument(s):
98 # setting SEL setting which needs to be read(e.g. "Last Add Time").
99
100 ${resp}= Run IPMI Standard Command sel info
101
102 ${setting_line}= Get Lines Containing String ${resp} ${setting}
103 ... case-insensitive
104 ${setting_status}= Fetch From Right ${setting_line} :${SPACE}
105
106 [Return] ${setting_status}
107
108
109Verify Watchdog Errorlog Content
110 [Documentation] Verify watchdog errorlog content.
111 # Example:
112 # "/xyz/openbmc_project/logging/entry/1":
113 # {
114 # "AdditionalData": [],
115 # "Id": 1,
116 # "Message": "org.open_power.Host.Boot.Error.WatchdogTimedOut",
117 # "Resolved": 0,
118 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
119 # "Timestamp": 1492715244828,
George Keishing58520d02020-02-24 10:55:32 -0600120 # "Associations": []
Sivas SRRcf6714f2018-03-26 10:51:29 -0500121 # },
122
123 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
124 ${elog}= Read Properties ${elog_entry[0]}
125 Should Be Equal As Strings
126 ... ${elog["Message"]} org.open_power.Host.Boot.Error.WatchdogTimedOut
Sivas SRR21c2ef22018-04-18 11:01:38 -0500127 ... msg=Watchdog timeout error log was not found.
Sivas SRRcf6714f2018-03-26 10:51:29 -0500128 Should Be Equal As Strings
129 ... ${elog["Severity"]} xyz.openbmc_project.Logging.Entry.Level.Error
Sivas SRR21c2ef22018-04-18 11:01:38 -0500130 ... msg=Watchdog timeout severity unexpected value.
Sivas SRRcf6714f2018-03-26 10:51:29 -0500131
132
133Logging Test Binary Exist
134 [Documentation] Verify existence of prerequisite logging-test.
George Keishing5f0c1fa2022-08-05 13:30:16 -0500135 ${stdout} ${stderr} ${rc}=
136 ... BMC Execute Command test -f /tmp/tarball/bin/logging-test print_out=1
Sivas SRR21c2ef22018-04-18 11:01:38 -0500137 Should Be Empty ${stderr} msg=Logging Test stderr is non-empty.
George Keishing5f0c1fa2022-08-05 13:30:16 -0500138
Sivas SRRcf6714f2018-03-26 10:51:29 -0500139
140Clear Existing Error Logs
141 [Documentation] If error log isn't empty, reboot the BMC to clear the log.
142 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${1}
143 Return From Keyword If ${resp.status_code} == ${HTTP_NOT_FOUND}
144 Initiate BMC Reboot
145 Wait Until Keyword Succeeds 10 min 10 sec
146 ... Is BMC Ready
147 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}${1}
148 Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND}
Sivas SRR21c2ef22018-04-18 11:01:38 -0500149 ... msg=Could not clear BMC error logs.
Sivas SRRcf6714f2018-03-26 10:51:29 -0500150
Rahul Maheshwarib8580dd2021-05-10 00:57:33 -0500151
152Create Test PEL Log
153 [Documentation] Generate test PEL log.
Rahul Maheshwari55e23462022-02-16 23:19:35 -0600154 [Arguments] ${pel_type}=Unrecoverable Error
Rahul Maheshwarib8580dd2021-05-10 00:57:33 -0500155
156 # Description of argument(s):
157 # pel_type The PEL type (e.g. Internal Failure, FRU Callout, Procedural Callout).
158
159 # Test PEL log entry example:
160 # {
161 # "0x5000002D": {
162 # "SRC": "BD8D1002",
163 # "Message": "An application had an internal failure",
164 # "PLID": "0x5000002D",
165 # "CreatorID": "BMC",
166 # "Subsystem": "BMC Firmware",
167 # "Commit Time": "02/25/2020 04:47:09",
168 # "Sev": "Unrecoverable Error",
169 # "CompID": "0x1000"
170 # }
171 # }
172
173 Run Keyword If '${pel_type}' == 'Internal Failure'
174 ... BMC Execute Command ${CMD_INTERNAL_FAILURE}
175 ... ELSE IF '${pel_type}' == 'FRU Callout'
176 ... BMC Execute Command ${CMD_FRU_CALLOUT}
177 ... ELSE IF '${pel_type}' == 'Procedure And Symbolic FRU Callout'
178 ... BMC Execute Command ${CMD_PROCEDURAL_SYMBOLIC_FRU_CALLOUT}
Rahul Maheshwari55e23462022-02-16 23:19:35 -0600179 ... ELSE IF '${pel_type}' == 'Unrecoverable Error'
180 ... BMC Execute Command ${CMD_UNRECOVERABLE_ERROR}
Rahul Maheshwarib8580dd2021-05-10 00:57:33 -0500181
182
Sivas SRRcf6714f2018-03-26 10:51:29 -0500183Create Test Error Log
184 [Documentation] Generate test error log.
185 # Test error log entry example:
186 # "/xyz/openbmc_project/logging/entry/1": {
187 # "AdditionalData": [
188 # "STRING=FOO"
189 # ],
190 # "Id": 1,
191 # "Message": "example.xyz.openbmc_project.Example.Elog.AutoTestSimple",
192 # "Severity": "xyz.openbmc_project.Logging.Entry.Level.Error",
193 # "Timestamp": 1487743963328,
George Keishing58520d02020-02-24 10:55:32 -0600194 # "Associations": []
Sivas SRRcf6714f2018-03-26 10:51:29 -0500195 # }
Joy Onyerikwu9b668972018-05-22 19:10:43 -0500196 BMC Execute Command /tmp/tarball/bin/logging-test -c AutoTestSimple
Sivas SRRcf6714f2018-03-26 10:51:29 -0500197
198Count Error Entries
199 [Documentation] Count Error entries.
200 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}
201 Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
Sivas SRR21c2ef22018-04-18 11:01:38 -0500202 ... msg=Failed to get error logs.
Sivas SRRcf6714f2018-03-26 10:51:29 -0500203 ${jsondata}= To JSON ${resp.content}
204 ${count}= Get Length ${jsondata["data"]}
205 [Return] ${count}
206
207Verify Test Error Log
208 [Documentation] Verify test error log entries.
209 ${elog_entry}= Get URL List ${BMC_LOGGING_ENTRY}
210 ${entry_id}= Read Attribute ${elog_entry[0]} Message
211 Should Be Equal ${entry_id}
212 ... example.xyz.openbmc_project.Example.Elog.AutoTestSimple
Sivas SRR21c2ef22018-04-18 11:01:38 -0500213 ... msg=Error log not from AutoTestSimple.
Sivas SRRcf6714f2018-03-26 10:51:29 -0500214 ${entry_id}= Read Attribute ${elog_entry[0]} Severity
215 Should Be Equal ${entry_id}
216 ... xyz.openbmc_project.Logging.Entry.Level.Error
Sivas SRR21c2ef22018-04-18 11:01:38 -0500217 ... msg=Error log severity mismatch.
Sivas SRRcf6714f2018-03-26 10:51:29 -0500218
219Delete Error Logs And Verify
220 [Documentation] Delete all error logs and verify.
221 Delete All Error Logs
Steven Sombara8800da2018-12-18 16:19:05 -0600222 ${resp}= OpenBMC Get Request ${BMC_LOGGING_ENTRY}list quiet=${1}
Sivas SRRcf6714f2018-03-26 10:51:29 -0500223 Should Be Equal As Strings ${resp.status_code} ${HTTP_NOT_FOUND}
Sivas SRR21c2ef22018-04-18 11:01:38 -0500224 ... msg=Error logs not deleted as expected.
Sivas SRRcf6714f2018-03-26 10:51:29 -0500225
226
227Install Tarball
228 [Documentation] Install tarball on BMC.
Sivas SRRf27afbe2018-05-26 06:45:23 -0500229 Should Not Be Empty ${DEBUG_TARBALL_PATH}
230 ... msg=Debug tarball path value is required.
Sivas SRRcf6714f2018-03-26 10:51:29 -0500231 BMC Execute Command rm -rf /tmp/tarball
232 Install Debug Tarball On BMC ${DEBUG_TARBALL_PATH}
George Keishing87e2a852019-05-29 12:30:14 -0500233
234
235Get Event Logs
236 [Documentation] Get all available EventLog entries.
237
238 #{
239 # "@odata.context": "/redfish/v1/$metadata#LogEntryCollection.LogEntryCollection",
240 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries",
241 # "@odata.type": "#LogEntryCollection.LogEntryCollection",
242 # "Description": "Collection of System Event Log Entries",
243 # "Members": [
244 # {
245 # "@odata.context": "/redfish/v1/$metadata#LogEntry.LogEntry",
246 # "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/1",
247 # "@odata.type": "#LogEntry.v1_4_0.LogEntry",
248 # "Created": "2019-05-29T13:19:27+00:00",
249 # "EntryType": "Event",
250 # "Id": "1",
251 # "Message": "org.open_power.Host.Error.Event",
252 # "Name": "System DBus Event Log Entry",
253 # "Severity": "Critical"
254 # }
255 # ],
256 # "Members@odata.count": 1,
257 # "Name": "System Event Log Entries"
258 #}
259
George Keishing32f6df62019-05-29 13:58:12 -0500260 ${members}= Redfish.Get Attribute ${EVENT_LOG_URI}Entries Members
George Keishing87e2a852019-05-29 12:30:14 -0500261 [Return] ${members}
George Keishing32f6df62019-05-29 13:58:12 -0500262
263
George Keishing438fd3b2021-10-08 02:15:18 -0500264Get Redfish Event Logs
265 [Documentation] Pack the list of all available EventLog entries in dictionary.
George Keishinga6c00892021-10-11 01:09:54 -0500266 [Arguments] ${quiet}=1 &{filter_struct_args}
267
268 # Description of argument(s):
269 # quiet Indicates whether this keyword should run without any output to the
270 # console, 0 = verbose, 1 = quiet.
271 # filter_struct_args filter_struct args (e.g. filter_dict, regex, etc.) to be passed
272 # directly to the Filter Struct keyword. See its prolog for details.
George Keishing438fd3b2021-10-08 02:15:18 -0500273
274 ${packed_dict}= Create Dictionary
275 ${error_logs}= Get Event Logs
276
George Keishing1769aa92021-10-19 10:44:47 -0500277 FOR ${idx} IN @{error_logs}
George Keishing438fd3b2021-10-08 02:15:18 -0500278 Set To Dictionary ${packed_dict} ${idx['@odata.id']}=${idx}
279 END
280
George Keishinga6c00892021-10-11 01:09:54 -0500281 ${num_filter_struct_args}= Get Length ${filter_struct_args}
282 Return From Keyword If '${num_filter_struct_args}' == '${0}' &{packed_dict}
283 ${filtered_error_logs}= Filter Struct ${packed_dict} &{filter_struct_args}
284
285 [Return] ${filtered_error_logs}
George Keishing438fd3b2021-10-08 02:15:18 -0500286
287
Joy Onyerikwud806cc02019-10-01 07:46:18 -0500288Get Event Logs Not Ok
289 [Documentation] Get all event logs where the 'Severity' is not 'OK'.
290
291 ${members}= Get Event Logs
292 ${severe_logs}= Evaluate [elog for elog in $members if elog['Severity'] != 'OK']
293 [Return] ${severe_logs}
294
295
Steven Sombar3468df52019-06-29 11:01:47 -0500296Get Number Of Event Logs
297 [Documentation] Return the number of EventLog members.
298
299 ${members}= Get Event Logs
300 ${num_members}= Get Length ${members}
301 [Return] ${num_members}
302
303
George Keishing32f6df62019-05-29 13:58:12 -0500304Redfish Purge Event Log
305 [Documentation] Do Redfish EventLog purge.
306
Sushil Singheda8bff2019-12-05 00:47:13 -0600307 ${target_action}= redfish_utils.Get Target Actions
308 ... /redfish/v1/Systems/system/LogServices/EventLog/ LogService.ClearLog
309 Redfish.Post ${target_action} body={'target': '${target_action}'}
310 ... valid_status_codes=[${HTTP_OK}, ${HTTP_NO_CONTENT}]
311
Tim Lee26393aa2022-04-15 11:00:37 +0800312
313Event Log Should Not Exist
314 [Documentation] Event log entries should not exist.
315
316 ${elogs}= Get Event Logs
317 Should Be Empty ${elogs} msg=System event log entry is not empty.
George Keishingcdfea1d2022-08-05 10:42:39 -0500318
319
320Redfish Clear PostCodes
321 [Documentation] Do Redfish PostCodes purge from system.
322
323 ${target_action}= redfish_utils.Get Target Actions
324 ... /redfish/v1/Systems/system/LogServices/PostCodes/ LogService.ClearLog
325 Redfish.Post ${target_action} body={'target': '${target_action}'}
326 ... valid_status_codes=[${HTTP_OK}, ${HTTP_NO_CONTENT}]
327
328
329Redfish Get PostCodes
330 [Documentation] Perform Redfish GET request and return the PostCodes entries as a list of dictionaries.
331
332 # Formatted example output from Rprint vars members
333 # members:
334 # [0]:
335 # [@odata.id]: /redfish/v1/Systems/system/LogServices/PostCodes/Entries/B1-1
336 # [@odata.type]: #LogEntry.v1_8_0.LogEntry
337 # [AdditionalDataURI]: /redfish/v1/Systems/system/LogServices/PostCodes/Entries/B1-1/attachment
338 # [Created]: 2022-08-06T04:38:10+00:00
339 # [EntryType]: Event
340 # [Id]: B1-1
341 # [Message]: Message": "Boot Count: 4: TS Offset: 0.0033; POST Code: 0x43
342 # [MessageArgs]:
343 # [0]: 4
344 # [1]: 0.0033
345 # [2]: 0x43
346 # [MessageId]: OpenBMC.0.2.BIOSPOSTCodeASCII
347 # [Name]: POST Code Log Entry
348 # [Severity]: OK
349
350 ${members}= Redfish.Get Attribute /redfish/v1/Systems/system/LogServices/PostCodes/Entries Members
351 ... valid_status_codes=[${HTTP_OK}, ${HTTP_NO_CONTENT}]
352
353 [Return] ${members}