blob: cad299db3579a6f48a62f13c14513d893884b93f [file] [log] [blame]
Rahul Maheshwarifaa5d202020-02-24 23:32:57 -06001*** Settings ***
2Documentation This suite tests Platform Event Log (PEL) functionality of OpenBMC.
3
4Library ../../lib/pel_utils.py
Rahul Maheshwaria17a3392020-03-02 04:51:52 -06005Variables ../../data/pel_variables.py
Rahul Maheshwari8f5256f2020-02-26 23:53:55 -06006Resource ../../lib/list_utils.robot
Rahul Maheshwarif4a01aa2020-08-24 23:22:59 -05007Resource ../../lib/logging_utils.robot
manashsarma1810a5b2020-12-09 20:09:20 -06008Resource ../../lib/connection_client.robot
Rahul Maheshwarifaa5d202020-02-24 23:32:57 -06009Resource ../../lib/openbmc_ffdc.robot
10
Rahul Maheshwarif575c902020-02-26 04:38:40 -060011Test Setup Redfish.Login
12Test Teardown Run Keywords Redfish.Logout AND FFDC On Test Case Fail
Rahul Maheshwarifaa5d202020-02-24 23:32:57 -060013
14
15*** Variables ***
16
Rahul Maheshwari39918682020-07-07 06:42:20 -050017@{mandatory_pel_fileds} Private Header User Header Primary SRC Extended User Header Failing MTMS
Rahul Maheshwarifaa5d202020-02-24 23:32:57 -060018
Rahul Maheshwarib8580dd2021-05-10 00:57:33 -050019
Rahul Maheshwarifaa5d202020-02-24 23:32:57 -060020*** Test Cases ***
21
22Create Test PEL Log And Verify
23 [Documentation] Create PEL log using busctl command and verify via peltool.
24 [Tags] Create_Test_PEL_Log_And_Verify
25
Rahul Maheshwarif575c902020-02-26 04:38:40 -060026 Redfish Purge Event Log
Rahul Maheshwarifaa5d202020-02-24 23:32:57 -060027 Create Test PEL Log
Rahul Maheshwarif575c902020-02-26 04:38:40 -060028 ${pel_id}= Get PEL Log Via BMC CLI
29 Should Not Be Empty ${pel_id} msg=System PEL log entry is empty.
30
31
Rahul Maheshwaria17a3392020-03-02 04:51:52 -060032Verify PEL Log Details
33 [Documentation] Verify PEL log details via peltool.
34 [Tags] Verify_PEL_Log_Details
35
36 Redfish Purge Event Log
37
38 ${bmc_time1}= CLI Get BMC DateTime
39 Create Test PEL Log
40 ${bmc_time2}= CLI Get BMC DateTime
41
42 ${pel_records}= Peltool -l
43
44 # Example output from 'Peltool -l':
45 # pel_records:
46 # [0x50000012]:
47 # [CreatorID]: BMC
48 # [CompID]: 0x1000
49 # [PLID]: 0x50000012
50 # [Subsystem]: BMC Firmware
51 # [Message]: An application had an internal failure
52 # [SRC]: BD8D1002
53 # [Commit Time]: 03/02/2020 09:35:15
54 # [Sev]: Unrecoverable Error
55
56 ${ids}= Get Dictionary Keys ${pel_records}
57 ${id}= Get From List ${ids} 0
58
59 @{pel_fields}= Create List CreatorID Subsystem Message Sev
60 FOR ${field} IN @{pel_fields}
61 Valid Value pel_records['${id}']['${field}'] ['${PEL_DETAILS['${field}']}']
62 END
63
64 Valid Value pel_records['${id}']['PLID'] ['${id}']
65
66 # Verify if "CompID" and "SRC" fields of PEL has alphanumeric value.
67 Should Match Regexp ${pel_records['${id}']['CompID']} [a-zA-Z0-9]
68 Should Match Regexp ${pel_records['${id}']['SRC']} [a-zA-Z0-9]
69
70 ${pel_date_time}= Convert Date ${pel_records['${id}']['Commit Time']}
71 ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=yes
72
73 # Convert BMC and PEL time to epoch time before comparing.
74 ${bmc_time1_epoch}= Convert Date ${bmc_time1} epoch
75 ${pel_time_epoch}= Convert Date ${pel_date_time} epoch
76 ${bmc_time2_epoch}= Convert Date ${bmc_time2} epoch
77
78 Should Be True ${bmc_time1_epoch} <= ${pel_time_epoch} <= ${bmc_time2_epoch}
79
80
Rahul Maheshwari39918682020-07-07 06:42:20 -050081Verify Mandatory Sections Of Error Log PEL
82 [Documentation] Verify mandatory sections of error log PEL.
83 [Tags] Verify_Mandatory_Sections_Of_Error_Log_PEL
84
85 Create Test PEL Log
86
87 ${pel_ids}= Get PEL Log Via BMC CLI
88 ${pel_id}= Get From List ${pel_ids} -1
89 ${pel_output}= Peltool -i ${pel_id}
90 ${pel_sections}= Get Dictionary Keys ${pel_output}
91
92 List Should Contain Sub List ${pel_sections} ${mandatory_pel_fileds}
93
94
Rahul Maheshwarif575c902020-02-26 04:38:40 -060095Verify PEL Log Persistence After BMC Reboot
96 [Documentation] Verify PEL log persistence after BMC reboot.
97 [Tags] Verify_PEL_Log_Persistence_After_BMC_Reboot
98
99 Create Test PEL Log
100 ${pel_before_reboot}= Get PEL Log Via BMC CLI
101
102 Redfish OBMC Reboot (off)
103 ${pel_after_reboot}= Get PEL Log Via BMC CLI
104
105 List Should Contain Sub List ${pel_after_reboot} ${pel_before_reboot}
Rahul Maheshwarifaa5d202020-02-24 23:32:57 -0600106
107
Rahul Maheshwari58b99fc2020-03-03 00:05:55 -0600108Verify PEL ID Numbering
109 [Documentation] Verify PEL ID numbering.
110 [Tags] Verify_PEL_ID_Numbering
111
112 Redfish Purge Event Log
113 Create Test PEL Log
114 Create Test PEL Log
115
116 ${pel_ids}= Get PEL Log Via BMC CLI
117
118 # Example of PEL IDs from PEL logs.
119 # [0x50000012]: <--- First PEL ID
120 # [CreatorID]: BMC
121 # [CompID]: 0x1000
122 # [PLID]: 0x50000012
123 # [Subsystem]: BMC Firmware
124 # [Message]: An application had an internal failure
125 # [SRC]: BD8D1002
126 # [Commit Time]: 03/02/2020 09:35:15
127 # [Sev]: Unrecoverable Error
128 #
129 # [0x50000013]: <--- Second PEL ID
130 # [CreatorID]: BMC
131 # [CompID]: 0x1000
132 # [PLID]: 0x50000013
133 # [Subsystem]: BMC Firmware
134 # [Message]: An application had an internal failure
135 # [SRC]: BD8D1002
136 # [Commit Time]: 03/02/2020 09:35:15
137 # [Sev]: Unrecoverable Error
138
139 Should Be True ${pel_ids[1]} == ${pel_ids[0]}+1
140
Rahul Maheshwari371e4f82020-04-06 22:53:52 -0500141Verify Machine Type Model And Serial Number
142 [Documentation] Verify machine type model and serial number from PEL.
143 [Tags] Verify_Machine_Type_Model_And_Serial_Number
144
145 Create Test PEL Log
146
147 ${pel_ids}= Get PEL Log Via BMC CLI
148 ${id}= Get From List ${pel_ids} -1
Rahul Maheshwari56dba362020-05-12 00:59:48 -0500149
Rahul Maheshwari371e4f82020-04-06 22:53:52 -0500150 ${pel_serial_number}= Get PEL Field Value ${id} Failing MTMS Serial Number
Rahul Maheshwari56dba362020-05-12 00:59:48 -0500151 ${pel_serial_number}= Replace String Using Regexp ${pel_serial_number} ^0+ ${EMPTY}
Rahul Maheshwari371e4f82020-04-06 22:53:52 -0500152 ${pel_machine_type_model}= Get PEL Field Value ${id} Failing MTMS Machine Type Model
Rahul Maheshwari56dba362020-05-12 00:59:48 -0500153 ${pel_machine_type_model}= Replace String Using Regexp ${pel_machine_type_model} ^0+ ${EMPTY}
Rahul Maheshwari371e4f82020-04-06 22:53:52 -0500154
155 # Example of "Machine Type Model" and "Serial Number" fields value from "Failing MTMS" section of PEL.
156 # [Failing MTMS]:
157 # [Created by]: 0x2000
158 # [Machine Type Model]: 1234-ABC <---- Machine type
159 # [Section Version]: 1
160 # [Serial Number]: ABCDEFG <---- Serial number
161 # [Sub-section type]: 0
162
163 ${redfish_machine_model}= Redfish.Get Attribute /redfish/v1/Systems/system/ Model
Rahul Maheshwari56dba362020-05-12 00:59:48 -0500164 ${redfish_machine_model}= Replace String Using Regexp ${redfish_machine_model} ^0+ ${EMPTY}
Rahul Maheshwari371e4f82020-04-06 22:53:52 -0500165 ${redfish_serial_number}= Redfish.Get Attribute /redfish/v1/Systems/system/ SerialNumber
Rahul Maheshwari56dba362020-05-12 00:59:48 -0500166 ${redfish_serial_number}= Replace String Using Regexp ${redfish_serial_number} ^0+ ${EMPTY}
Rahul Maheshwari371e4f82020-04-06 22:53:52 -0500167
168 Valid Value pel_machine_type_model ['${redfish_machine_model}']
169 Valid Value pel_serial_number ['${redfish_serial_number}']
170
171 # Check "Machine Type Model" and "Serial Number" fields value from "Extended User Header" section of PEL.
172 ${pel_machine_type_model}= Get PEL Field Value ${id} Extended User Header Reporting Machine Type
Rahul Maheshwari56dba362020-05-12 00:59:48 -0500173 ${pel_machine_type_model}= Replace String Using Regexp ${pel_machine_type_model} ^0+ ${EMPTY}
Rahul Maheshwari371e4f82020-04-06 22:53:52 -0500174 ${pel_serial_number}= Get PEL Field Value ${id} Extended User Header Reporting Serial Number
Rahul Maheshwari56dba362020-05-12 00:59:48 -0500175 ${pel_serial_number}= Replace String Using Regexp ${pel_serial_number} ^0+ ${EMPTY}
Rahul Maheshwari371e4f82020-04-06 22:53:52 -0500176
177 Valid Value pel_machine_type_model ['${redfish_machine_model}']
178 Valid Value pel_serial_number ['${redfish_serial_number}']
179
Rahul Maheshwari58b99fc2020-03-03 00:05:55 -0600180
Rahul Maheshwari1bea6cc2020-03-31 00:53:14 -0500181Verify Host Off State From PEL
182 [Documentation] Verify Host off state from PEL.
183 [Tags] Verify_Host_Off_State_From_PEL
184
185 Redfish Power Off stack_mode=skip
186 Create Test PEL Log
187
188 ${pel_ids}= Get PEL Log Via BMC CLI
189 ${id}= Get From List ${pel_ids} -1
190 ${pel_host_state}= Get PEL Field Value ${id} User Data HostState
191
192 Valid Value pel_host_state ['Off']
193
194
195Verify BMC Version From PEL
196 [Documentation] Verify BMC Version from PEL.
197 [Tags] Verify_BMC_Version_From_PEL
198
199 Create Test PEL Log
200
201 ${pel_ids}= Get PEL Log Via BMC CLI
202 ${id}= Get From List ${pel_ids} -1
Rahul Maheshwarif6600f82021-05-28 01:25:20 -0500203 ${pel_bmc_version}= Get PEL Field Value ${id} User Data FW Version ID
Rahul Maheshwari1bea6cc2020-03-31 00:53:14 -0500204
205 ${bmc_version}= Get BMC Version
206 Valid Value bmc_version ['${bmc_version}']
207
208
Rahul Maheshwari8f5256f2020-02-26 23:53:55 -0600209Verify PEL Log After Host Poweron
210 [Documentation] Verify PEL log generation while booting host.
211 [Tags] Verify_PEL_Log_After_Host_Poweron
212
213 Redfish Power Off stack_mode=skip
214 Redfish Purge Event Log
215 Redfish Power On stack_mode=skip
216
217 ${pel_informational_error}= Get PEL Log IDs User Header Event Severity Informational Event
218 ${pel_bmc_created_error}= Get PEL Log IDs Private Header Creator Subsystem BMC
219
220 # Get BMC created non-infomational error.
221 ${pel_bmc_error}= Subtract Lists ${pel_bmc_created_error} ${pel_informational_error}
222
223 Should Be Empty ${pel_bmc_error} msg=Unexpected error log generated during Host poweron.
224
225
Rahul Maheshwari089615b2020-04-02 01:01:38 -0500226Verify BMC Event Log ID
227 [Documentation] Verify BMC Event Log ID from PEL.
228 [Tags] Verify_BMC_Event_Log_ID
229
230 Redfish Purge Event Log
231 Create Test PEL Log
232
233 ${pel_ids}= Get PEL Log Via BMC CLI
234 ${pel_bmc_event_log_id}= Get PEL Field Value ${pel_ids[0]} Private Header BMC Event Log Id
235
236 # Example "BMC Event Log Id" field value from "Private Header" section of PEL.
237 # [Private Header]:
238 # [Created at]: 08/24/1928 12:04:06
239 # [Created by]: 0x584D
240 # [Sub-section type]: 0
241 # [Entry Id]: 0x50000BB7
242 # [Platform Log Id]: 0x8200061D
243 # [CSSVER]:
244 # [Section Version]: 1
245 # [Creator Subsystem]: PHYP
246 # [BMC Event Log Id]: 341 <---- BMC event log id value
247 # [Committed at]: 03/25/1920 12:06:22
248
249 ${redfish_event_logs}= Redfish.Get Properties /redfish/v1/Systems/system/LogServices/EventLog/Entries
250
251 # Example of redfish_event_logs output:
252 # redfish_event_logs:
253 # [@odata.id]: /redfish/v1/Systems/system/LogServices/EventLog/Entries
254 # [Name]: System Event Log Entries
255 # [Members@odata.count]: 1
256 # [@odata.type]: #LogEntryCollection.LogEntryCollection
257 # [Description]: Collection of System Event Log Entries
258 # [Members]:
259 # [0]:
260 # [@odata.id]: /redfish/v1/Systems/system/LogServices/EventLog/Entries/235
261 # [Name]: System Event Log Entry
262 # [Severity]: Critical
263 # [EntryType]: Event
264 # [Created]: 2020-04-02T07:25:13+00:00
265 # [@odata.type]: #LogEntry.v1_5_1.LogEntry
266 # [Id]: 235 <----- Event log ID
267 # [Message]: xyz.openbmc_project.Common.Error.InternalFailure
268
269 Valid Value pel_bmc_event_log_id ['${redfish_event_logs['Members'][0]['Id']}']
270
271
Rahul Maheshwari97d7ab82020-08-04 00:01:42 -0500272Verify FRU Callout
273 [Documentation] Verify FRU callout entries from PEL log.
274 [Tags] Verify_FRU_Callout
275
276 Create Test PEL Log FRU Callout
277
278 ${pel_ids}= Get PEL Log Via BMC CLI
279 ${id}= Get From List ${pel_ids} -1
280 ${pel_callout_section}= Get PEL Field Value ${id} Primary SRC Callout Section
281
282 # Example of PEL Callout Section from "peltool -i <id>" command.
283 # [Callouts]:
284 # [0]:
285 # [FRU Type]: Normal Hardware FRU
286 # [Priority]: Mandatory, replace all with this type as a unit
287 # [Location Code]: U78DA.ND1.1234567-P0
288 # [Part Number]: F191014
289 # [CCIN]: 2E2D
290 # [Serial Number]: YL2E2D010000
291 # [Callout Count]: 1
292
293 Valid Value pel_callout_section['Callout Count'] ['1']
294 Valid Value pel_callout_section['Callouts'][0]['FRU Type'] ['Normal Hardware FRU']
295 Should Contain ${pel_callout_section['Callouts'][0]['Priority']} Mandatory
296
297 # Verify Location Code field of PEL callout with motherboard's Location Code.
298 ${busctl_output}= BMC Execute Command ${CMD_INVENTORY_PREFIX} com.ibm.ipzvpd.Location LocationCode
299 Should Be Equal ${pel_callout_section['Callouts'][0]['Location Code']}
300 ... ${busctl_output[0].split('"')[1].strip('"')}
301
302 # TODO: Compare CCIN and part number fields of PEL callout with Redfish or busctl output.
303 Should Match Regexp ${pel_callout_section['Callouts'][0]['CCIN']} [a-zA-Z0-9]
304 Should Match Regexp ${pel_callout_section['Callouts'][0]['Part Number']} [a-zA-Z0-9]
305
306 # Verify Serial Number field of PEL callout with motherboard's Serial Number.
307 ${busctl_output}= BMC Execute Command
308 ... ${CMD_INVENTORY_PREFIX} xyz.openbmc_project.Inventory.Decorator.Asset SerialNumber
309 Should Be Equal ${pel_callout_section['Callouts'][0]['Serial Number']}
310 ... ${busctl_output[0].split('"')[1].strip('"')}
311
312
313Verify Procedure And Symbolic FRU Callout
314 [Documentation] Verify procedure and symbolic FRU callout from PEL log.
315 [Tags] Verify_Procedure_And_Symbolic_FRU_Callout
316
317 Create Test PEL Log Procedure And Symbolic FRU Callout
318
319 ${pel_ids}= Get PEL Log Via BMC CLI
320 ${id}= Get From List ${pel_ids} -1
321 ${pel_callout_section}= Get PEL Field Value ${id} Primary SRC Callout Section
322
323 # Example of PEL Callout Section from "peltool -i <id>" command.
324 # [Callouts]:
325 # [0]:
326 # [Priority]: Mandatory, replace all with this type as a unit
327 # [Procedure Number]: BMCSP02
328 # [FRU Type]: Maintenance Procedure Required
329 # [1]:
330 # [Priority]: Medium Priority
331 # [Part Number]: SVCDOCS
332 # [FRU Type]: Symbolic FRU
333 # [Callout Count]: 2
334
335 Valid Value pel_callout_section['Callout Count'] ['2']
336
337 # Verify procedural callout info.
338
339 Valid Value pel_callout_section['Callouts'][0]['FRU Type'] ['Maintenance Procedure Required']
340 Should Contain ${pel_callout_section['Callouts'][0]['Priority']} Mandatory
341 # Verify if "Procedure Number" field of PEL has an alphanumeric value.
Rahul Maheshwari6acf0ee2020-10-28 05:37:01 -0500342 Should Match Regexp ${pel_callout_section['Callouts'][0]['Procedure']} [a-zA-Z0-9]
Rahul Maheshwari97d7ab82020-08-04 00:01:42 -0500343
344 # Verify procedural callout info.
345
346 Valid Value pel_callout_section['Callouts'][1]['FRU Type'] ['Symbolic FRU']
347 Should Contain ${pel_callout_section['Callouts'][1]['Priority']} Medium Priority
348 # Verify if "Part Number" field of Symbolic FRU has an alphanumeric value.
349 Should Match Regexp ${pel_callout_section['Callouts'][1]['Part Number']} [a-zA-Z0-9]
350
351
Rahul Maheshwarif4a01aa2020-08-24 23:22:59 -0500352Verify PEL Log Entry For Event Log
353 [Documentation] Create an event log and verify PEL log entry in BMC for the same.
354 [Tags] Verify_PEL_Log_Entry_For_Event_Log
355
356 Redfish Purge Event Log
Rahul Maheshwari71d589c2022-04-22 01:29:44 -0500357
358 # Create an unrecoverable error log.
359 Create Test PEL Log Unrecoverable Error
Rahul Maheshwarif4a01aa2020-08-24 23:22:59 -0500360
361 ${elog_entry}= Get Event Logs
362 # Example of Redfish event logs:
363 # elog_entry:
364 # [0]:
Rahul Maheshwari71d589c2022-04-22 01:29:44 -0500365 # [Message]: BD802003 event in subsystem: Platform Firmware
George Keishingb78bca22021-06-29 11:11:19 -0500366 # [Created]: 2020-04-20T01:55:22+00:00
367 # [Id]: 1
368 # [@odata.id]: /redfish/v1/Systems/system/LogServices/EventLog/Entries/1
369 # [@odata.type]: #LogEntry.v1_4_0.LogEntry
370 # [EntryType]: Event
371 # [Severity]: Critical
372 # [Name]: System Event Log Entry
Rahul Maheshwarif4a01aa2020-08-24 23:22:59 -0500373
374 ${redfish_log_time}= Convert Date ${elog_entry[0]["Created"]} epoch
375
376 ${pel_records}= Peltool -l
377 # Example output from 'Peltool -l':
378 # pel_records:
379 # [0x50000023]:
Rahul Maheshwari71d589c2022-04-22 01:29:44 -0500380 # [SRC]: BD802003
George Keishingb78bca22021-06-29 11:11:19 -0500381 # [CreatorID]: BMC
Rahul Maheshwari71d589c2022-04-22 01:29:44 -0500382 # [Message]: This is a test error
383 # [CompID]: 0x2000
384 # [PLID]: 0x500053D9
George Keishingb78bca22021-06-29 11:11:19 -0500385 # [Commit Time]: 04/20/2020 01:55:22
Rahul Maheshwari71d589c2022-04-22 01:29:44 -0500386 # [Subsystem]: Platform Firmware
George Keishingb78bca22021-06-29 11:11:19 -0500387 # [Sev]: Unrecoverable Error
Rahul Maheshwarif4a01aa2020-08-24 23:22:59 -0500388
389 ${ids}= Get Dictionary Keys ${pel_records}
390 ${id}= Get From List ${ids} 0
391 ${pel_log_time}= Convert Date ${pel_records['${id}']['Commit Time']} epoch
392 ... date_format=%m/%d/%Y %H:%M:%S
393
394 # Verify that both Redfish event and PEL has log entry for internal error with same time stamp.
Rahul Maheshwari71d589c2022-04-22 01:29:44 -0500395 Should Contain Any ${pel_records['${id}']['Message']} test error ignore_case=True
Rahul Maheshwari40ade892021-12-20 22:38:21 -0600396 Should Contain ${elog_entry[0]['Message']}
Rahul Maheshwari71d589c2022-04-22 01:29:44 -0500397 ... ${pel_records['${id}']['SRC']} ignore_case=True
Rahul Maheshwarif4a01aa2020-08-24 23:22:59 -0500398
399 Should Be Equal ${redfish_log_time} ${pel_log_time}
400
401
Rahul Maheshwari326d0642020-04-07 06:13:38 -0500402Verify Delete All PEL
403 [Documentation] Verify deleting all PEL logs.
404 [Tags] Verify_Delete_All_PEL
405
406 Create Test PEL Log
407 Create Test PEL Log
408 Peltool --delete-all False
409
410 ${pel_ids}= Get PEL Log Via BMC CLI
411 Should Be Empty ${pel_ids}
412
413
manashsarma4e935382020-10-16 12:00:53 -0500414Verify Informational Error Log
415 [Documentation] Create an informational error log and verify.
416 [Tags] Verify_Informational_Error_Log
417
418 Redfish Purge Event Log
419 # Create an informational error log.
420 BMC Execute Command ${CMD_INFORMATIONAL_ERROR}
421 ${pel_records}= Peltool -lfh
422
423 # An example of information error log data:
424 # {
425 # "0x500006A0": {
426 # "SRC": "BD8D1002",
427 # "Message": "An application had an internal failure",
428 # "PLID": "0x500006A0",
429 # "CreatorID": "BMC",
430 # "Subsystem": "BMC Firmware",
431 # "Commit Time": "10/14/2020 11:41:38",
432 # "Sev": "Informational Event",
433 # "CompID": "0x1000"
434 # }
435 # }
436
437 ${ids}= Get Dictionary Keys ${pel_records}
438 ${id}= Get From List ${ids} 0
439 Should Contain ${pel_records['${id}']['Sev']} Informational
440
441
manashsarma15fe5632020-10-18 03:42:58 -0500442Verify Predictable Error Log
443 [Documentation] Create a predictive error and verify.
444 [Tags] Verify_Predictable_Error_Log
445
446 # Create a predictable error log.
447 BMC Execute Command ${CMD_PREDICTIVE_ERROR}
448 ${pel_records}= Peltool -l
449
450 # An example of predictive error log data:
451 # {
452 # "0x5000069E": {
453 # "SRC": "BD8D1002",
454 # "Message": "An application had an internal failure",
455 # "PLID": "0x5000069E",
456 # "CreatorID": "BMC",
457 # "Subsystem": "BMC Firmware",
458 # "Commit Time": "10/14/2020 11:40:07",
459 # "Sev": "Predictive Error",
460 # "CompID": "0x1000"
461 # }
462 # }
463
464 ${pel_ids}= Get PEL Log Via BMC CLI
465 ${id}= Get From List ${pel_ids} -1
466 Should Contain ${pel_records['${id}']['Sev']} Predictive
467
468
469Verify Unrecoverable Error Log
470 [Documentation] Create an unrecoverable error and verify.
471 [Tags] Verify_Unrecoverable_Error_Log
472
473 # Create an internal failure error log.
474 BMC Execute Command ${CMD_UNRECOVERABLE_ERROR}
475 ${pel_records}= Peltool -l
476
477 # An example of unrecoverable error log data:
478 # {
479 # "0x50000CC5": {
480 # "SRC": "BD8D1002",
481 # "Message": "An application had an internal failure",
482 # "PLID": "0x50000CC5",
483 # "CreatorID": "BMC",
484 # "Subsystem": "BMC Firmware",
485 # "Commit Time": "04/01/2020 16:44:55",
486 # "Sev": "Unrecoverable Error",
487 # "CompID": "0x1000"
488 # }
489 # }
490
491 ${pel_ids}= Get PEL Log Via BMC CLI
492 ${id}= Get From List ${pel_ids} -1
493 Should Contain ${pel_records['${id}']['Sev']} Unrecoverable
494
495
manashsarma595282d2020-10-20 13:22:31 -0500496Verify Error Logging Rotation Policy
497 [Documentation] Verify error logging rotation policy.
498 [Tags] Verify_Error_Logging_Rotation_Policy
499 [Template] Error Logging Rotation Policy
manashsarmad073a0a2020-10-20 12:54:59 -0500500
manashsarma557b6322020-11-29 12:08:32 -0600501 # Error logs to be created % of total logging space when error
502 # log exceeds max limit.
503 Informational BMC 3000 15
504 Predictive BMC 3000 30
505 Unrecoverable BMC 3000 30
506 Informational BMC 1500, Predictive BMC 1500 45
507 Informational BMC 1500, Unrecoverable BMC 1500 45
508 Unrecoverable BMC 1500, Predictive BMC 1500 30
Rahul Maheshwarib4cacb52021-01-18 23:57:29 -0600509
510
511Verify Error Logging Rotation Policy With All Types Of Errors
512 [Documentation] Verify error logging rotation policy with all types of errors.
George Keishing4203fad2022-01-31 12:22:33 -0600513 [Tags] Verify_Error_Logging_Rotation_Policy_With_All_Types_Of_Errors
Rahul Maheshwarib4cacb52021-01-18 23:57:29 -0600514 [Template] Error Logging Rotation Policy
515
516 # Error logs to be created % of total logging space when error
517 # log exceeds max limit.
manashsarma557b6322020-11-29 12:08:32 -0600518 Unrecoverable BMC 1000, Informational BMC 1000, Predictive BMC 1000 45
manashsarmac5131662020-11-12 12:25:59 -0600519
manashsarma1810a5b2020-12-09 20:09:20 -0600520
521Verify Error Logging Rotation Policy With HOST Error Logs
522 [Documentation] Verify error logging rotation policy for non bmc error logs.
523 [Tags] Verify_Error_Logging_Rotation_Policy_With_HOST_Error_Logs
524 [Setup] Run Keywords Open Connection for SCP AND scp.Put File ${UNRECOVERABLE_FILE_PATH}
manashsarma4baa6a72020-12-16 07:13:37 -0600525 ... /tmp/FILE_HOST_UNRECOVERABLE AND scp.Put File ${INFORMATIONAL_FILE_PATH}
526 ... /tmp/FILE_HOST_INFORMATIONAL
manashsarma1810a5b2020-12-09 20:09:20 -0600527 [Template] Error Logging Rotation Policy
528
529 # Error logs to be created % of total logging space when error
530 # log exceeds max limit.
manashsarma4baa6a72020-12-16 07:13:37 -0600531 Informational HOST 3000 15
manashsarma1810a5b2020-12-09 20:09:20 -0600532 Unrecoverable HOST 3000 30
manashsarma4baa6a72020-12-16 07:13:37 -0600533 Informational HOST 1500, Informational BMC 1500 30
534 Informational HOST 1500, Unrecoverable BMC 1500 45
manashsarma1810a5b2020-12-09 20:09:20 -0600535 Unrecoverable HOST 1500, Informational BMC 1500 45
536 Unrecoverable HOST 1500, Predictive BMC 1500 60
Rahul Maheshwarib4cacb52021-01-18 23:57:29 -0600537
538
539Verify Error Logging Rotation Policy With Unrecoverable HOST And BMC Error Logs
540 [Documentation] Verify error logging rotation policy with unrecoverable HOST and BMC error logs.
541 [Tags] Verify_Error_Logging_Rotation_Policy_With_Unrecoverable_HOST_And_BMC_Error_Logs
542 [Setup] Run Keywords Open Connection for SCP AND scp.Put File ${UNRECOVERABLE_FILE_PATH}
Rahul Maheshwarie54d16f2021-02-04 05:20:46 -0600543 ... /tmp/FILE_NBMC_UNRECOVERABLE AND Redfish.Login
Rahul Maheshwarib4cacb52021-01-18 23:57:29 -0600544 [Template] Error Logging Rotation Policy
545
546 # Error logs to be created % of total logging space when error
547 # log exceeds max limit.
manashsarma1810a5b2020-12-09 20:09:20 -0600548 Unrecoverable HOST 1500, Unrecoverable BMC 1500 60
549
550
manashsarma6f1f2a42020-12-15 07:29:41 -0600551Verify Old Logs Are Deleted When Count Crosses Max
552 [Documentation] Verify that when the count crosses max, older logs are deleted.
553 [Tags] Verify_Old_Logs_Are_Deleted_When_Count_Crosses_Max
554
555 Redfish Purge Event Log
manashsarma6f1f2a42020-12-15 07:29:41 -0600556 # Create 3000 error logs.
557 FOR ${count} IN RANGE ${3000}
558 BMC Execute Command ${CMD_PREDICTIVE_ERROR}
559 END
560
561 # Retrieve the IDs of the logs.
562 ${pel_ids}= Get PEL Log Via BMC CLI
manashsarmadd132682021-04-08 07:59:13 -0500563 ${1st_id}= Get From List ${pel_ids} 0
564 ${3000th_id}= Get From List ${pel_ids} 2999
manashsarma6f1f2a42020-12-15 07:29:41 -0600565
566 # Now create 3001st log to cross threshold limit and trigger error logs rotation.
567 BMC Execute Command ${CMD_PREDICTIVE_ERROR}
manashsarmadd132682021-04-08 07:59:13 -0500568
manashsarma6f1f2a42020-12-15 07:29:41 -0600569 # Wait few seconds for error logs rotation to complete.
570 Sleep 10s
571
572 # Now verify that first log is no more available but the 3000th is available.
manashsarmadd132682021-04-08 07:59:13 -0500573 ${status} ${output}= Run Keyword And Ignore Error Peltool -i ${1st_id}
574 Should Be True '${status}' == 'FAIL'
manashsarma6f1f2a42020-12-15 07:29:41 -0600575 Should Contain ${output} PEL not found
manashsarmadd132682021-04-08 07:59:13 -0500576
577 ${status} ${output}= Run Keyword And Ignore Error Peltool -i ${3000th_id}
578 Should Be True '${status}' == 'PASS'
manashsarma6f1f2a42020-12-15 07:29:41 -0600579 Should Not Contain ${output} PEL not found
580
581
manashsarma5f201202020-10-30 12:52:04 -0500582Verify Reverse Order Of PEL Logs
583 [Documentation] Verify PEL command to output PEL logs in reverse order.
George Keishing4203fad2022-01-31 12:22:33 -0600584 [Tags] Verify_Reverse_Order_Of_PEL_Logs
manashsarma5f201202020-10-30 12:52:04 -0500585
586 Redfish Purge Event Log
manashsarmac5131662020-11-12 12:25:59 -0600587
588 # Below commands create unrecoverable error log at first and then the predictable error.
manashsarma5f201202020-10-30 12:52:04 -0500589 BMC Execute Command ${CMD_UNRECOVERABLE_ERROR}
590 BMC Execute Command ${CMD_PREDICTIVE_ERROR}
591
manashsarmac5131662020-11-12 12:25:59 -0600592 # Using peltool -lr, recent PELs appear first. Hence the ID of first PEL is greater than the next.
593 ${pel_records}= peltool -lr
manashsarma5f201202020-10-30 12:52:04 -0500594
manashsarmac5131662020-11-12 12:25:59 -0600595 # It is found that, variables like dictionary always keep items in sorted order that makes
596 # this verification not possible, hence json is used to keep the items original order.
597 ${pel_records}= Convert To String ${pel_records}
598 ${json_string}= Replace String ${pel_records} ' "
599 ${json_object}= Evaluate json.loads('''${json_string}''') json
600
601 ${list}= Convert To List ${json_object}
602
603 ${id1}= Get From List ${list} 0
604 ${id1}= Convert To Integer ${id1}
605 ${id2}= Get From List ${list} 1
606 ${id2}= Convert To Integer ${id2}
607
608 Should Be True ${id1} > ${id2}
manashsarma5f201202020-10-30 12:52:04 -0500609
610
manashsarma6d75bd82020-10-30 07:36:05 -0500611Verify Total PEL Count
612 [Documentation] Verify total PEL count returned by peltool command.
613 [Tags] Verify_Total_PEL_Count
614
615 # Initially remove all logs.
616 Redfish Purge Event Log
617
618 # Generate a random number between 1-20.
619 ${random}= Evaluate random.randint(1, 20) modules=random
620
621 # Generate predictive error log multiple times.
622 FOR ${count} IN RANGE 0 ${random}
623 BMC Execute Command ${CMD_PREDICTIVE_ERROR}
624 END
625
626 # Check PEL log count via peltool command and compare it with actual generated log count.
627 ${pel_records}= peltool -n
628
629 Should Be Equal ${pel_records['Number of PELs found']} ${random}
630
631
manashsarma407deda2020-10-30 10:36:26 -0500632Verify Listing Information Error
633 [Documentation] Verify that information error logs can only be listed using -lfh option of peltool.
634 [Tags] Verify_Listing_Information_Error
635
636 # Initially remove all logs.
637 Redfish Purge Event Log
638 BMC Execute Command ${CMD_INFORMATIONAL_ERROR}
639
640 # Generate informational logs and verify that it would not get listed by peltool's list command.
641 ${pel_records}= peltool -l
manashsarma31d807b2020-11-11 12:49:46 -0600642 ${ids}= Get Dictionary Keys ${pel_records}
643 Should Be Empty ${ids}
manashsarma407deda2020-10-30 10:36:26 -0500644
645 # Verify that information logs get listed using peltool's list command with -lfh option.
646 ${pel_records}= peltool -lfh
manashsarma31d807b2020-11-11 12:49:46 -0600647 ${ids}= Get Dictionary Keys ${pel_records}
648 Should Not Be Empty ${ids}
649 ${id}= Get From List ${ids} 0
manashsarma407deda2020-10-30 10:36:26 -0500650 Should Contain ${pel_records['${id}']['Sev']} Informational
651
652
manashsarma54539ff2020-11-23 02:32:36 -0600653Verify PEL Delete
654 [Documentation] Verify that peltool command can delete PEL log based on id.
655 [Tags] Verify_PEL_Delete
656
657 BMC Execute Command ${CMD_PREDICTIVE_ERROR}
658 ${pel_ids}= Get PEL Log Via BMC CLI
659 ${id}= Get From List ${pel_ids} -1
Rahul Maheshwari5eab56c2021-01-06 23:58:46 -0600660 Peltool -d ${id} False
661 Run Keyword and Expect Error *PEL not found* Peltool -i ${id}
manashsarma54539ff2020-11-23 02:32:36 -0600662
663
Rahul Maheshwarifaa5d202020-02-24 23:32:57 -0600664*** Keywords ***
665
manashsarma595282d2020-10-20 13:22:31 -0500666Error Logging Rotation Policy
667 [Documentation] Verify that when maximum log limit is reached, given error logging type
668 ... are deleted when reached their max allocated space.
669 [Arguments] ${error_log_type} ${max_allocated_space_percentage}
670
671 # Description of argument(s):
manashsarma557b6322020-11-29 12:08:32 -0600672 # error_log Error logs to be created (E.g. Informational BMC 3000
673 # stands for BMC created 3000 informational error logs).
manashsarma595282d2020-10-20 13:22:31 -0500674 # max_allocated_space_percentage The maximum percentage of disk usage for given error
675 # log type when maximum count/log size is reached.
676 # The maximum error log count is 3000.
677
George Keishing6beaeab2022-08-29 09:34:42 -0500678 Redfish.Login
679
manashsarma595282d2020-10-20 13:22:31 -0500680 # Initially remove all logs. Purging is done to ensure that, only specific logs are present
681 # in BMC during the test.
682 Redfish Purge Event Log
683
manashsarma557b6322020-11-29 12:08:32 -0600684 @{lists}= Split String ${error_log_type} ,${SPACE}
manashsarma595282d2020-10-20 13:22:31 -0500685
manashsarma557b6322020-11-29 12:08:32 -0600686 ${length}= Get Length ${lists}
manashsarma595282d2020-10-20 13:22:31 -0500687
manashsarma557b6322020-11-29 12:08:32 -0600688 FOR ${list} IN RANGE ${length}
689 @{words}= Split String ${lists}[${list}] ${SPACE}
690 Create Error Log ${words}[0] ${words}[1] ${words}[2]
manashsarma595282d2020-10-20 13:22:31 -0500691 END
692
manashsarma557b6322020-11-29 12:08:32 -0600693 # Create an additional error log to exceed max error logs limit.
694 BMC Execute Command ${CMD_UNRECOVERABLE_ERROR}
695
manashsarma595282d2020-10-20 13:22:31 -0500696 # Delay for BMC to perform delete older error logs when log limit exceeds.
697 Sleep 10s
698
699 # Verify disk usage is around max allocated space. Maximum usage is around 3MB not exactly 3MB
700 # (for informational log) and around 6 MB for unrecoverable / predictive error log. So, usage
701 # percentage is NOT exactly 15% and 30%. So, an error/accuracy factor 0.5 percent is added.
702
703 ${disk_usage_percentage}= Get Disk Usage For Error Logs
704 ${percent_diff}= Evaluate ${disk_usage_percentage} - ${max_allocated_space_percentage}
705 ${percent_diff}= Evaluate abs(${percent_diff})
manashsarma4baa6a72020-12-16 07:13:37 -0600706
707 ${trimmed_as_expected}= Run Keyword If ${disk_usage_percentage} > ${max_allocated_space_percentage}
708 ... Evaluate ${percent_diff} <= 0.5
709 ... ELSE
710 ... Set Variable True
711
712 # Check PEL log count via peltool command and compare it with actual generated log count.
713 ${pel_records}= peltool -n
714 ${no_pel_records}= Set Variable ${pel_records["Number of PELs found"]}
715 # Number of logs can be 80% of the total logs created after trimming.
716 ${expected_max_record}= Evaluate 3000 * 0.8
717
George Keishingb78bca22021-06-29 11:11:19 -0500718 Run Keyword If ${trimmed_as_expected} == False
719 ... Should Be True ${no_pel_records} <= ${expected_max_record}
manashsarma595282d2020-10-20 13:22:31 -0500720
721
manashsarma557b6322020-11-29 12:08:32 -0600722Create Error Log
George Keishingf9248952021-05-28 07:52:37 -0500723 [Documentation] Create an error log.
manashsarma557b6322020-11-29 12:08:32 -0600724 [Arguments] ${error_severity} ${error_creator} ${count}
725
726 # Description of argument(s):
727 # error_severity Severity of the error (E.g. Informational, Unrecoberable or Predictive)
728 # error_creator Name of error log's creator(E.g BMC, Host Boot)
729 # count Number of error logs to be generated.
730
731 FOR ${i} IN RANGE 0 ${count}
732 ${cmd}= Set Variable If
Rahul Maheshwari5eab56c2021-01-06 23:58:46 -0600733 ... '${error_severity}' == 'Informational' and '${error_creator}' == 'BMC' ${CMD_INFORMATIONAL_ERROR}
734 ... '${error_severity}' == 'Predictive' and '${error_creator}' == 'BMC' ${CMD_PREDICTIVE_ERROR}
735 ... '${error_severity}' == 'Unrecoverable' and '${error_creator}' == 'BMC' ${CMD_UNRECOVERABLE_ERROR}
736 ... '${error_severity}' == 'Unrecoverable' and '${error_creator}' == 'HOST' ${CMD_UNRECOVERABLE_HOST_ERROR}
manashsarma557b6322020-11-29 12:08:32 -0600737 BMC Execute Command ${cmd}
738 END
739
740
manashsarmad073a0a2020-10-20 12:54:59 -0500741Get Disk Usage For Error Logs
742 [Documentation] Get disk usage percentage for error logs.
743
manashsarma407deda2020-10-30 10:36:26 -0500744 ${usage_output} ${stderr} ${rc}= BMC Execute Command du /var/lib/phosphor-logging/errors
manashsarmad073a0a2020-10-20 12:54:59 -0500745
746 ${usage_output}= Fetch From Left ${usage_output} \/
747
George Keishingc91aadd2022-02-23 09:30:48 -0600748 # Convert disk usage unit from KB to MB.
manashsarmad073a0a2020-10-20 12:54:59 -0500749 ${usage_output}= Evaluate ${usage_output} / 1024
750
751 # Logging disk capacity limit is set to 20MB. So calculating the log usage percentage.
752 ${usage_percent}= Evaluate ${usage_output} / 20 * 100
753
754 [return] ${usage_percent}
755
756
Rahul Maheshwari8f5256f2020-02-26 23:53:55 -0600757Get PEL Log IDs
758 [Documentation] Returns the list of PEL log IDs which contains given field's value.
759 [Arguments] ${pel_section} ${pel_field} @{pel_field_value}
760
761 # Description of argument(s):
762 # pel_section The section of PEL (e.g. Private Header, User Header).
763 # pel_field The PEL field (e.g. Event Severity, Event Type).
764 # pel_field_value The list of PEL's field value (e.g. Unrecoverable Error).
765
766 ${pel_ids}= Get PEL Log Via BMC CLI
767 @{pel_id_list}= Create List
768
769 FOR ${id} IN @{pel_ids}
770 ${pel_output}= Peltool -i ${id}
771 # Example of PEL output from "peltool -i <id>" command.
772 # [Private Header]:
773 # [Created at]: 08/24/1928 12:04:06
774 # [Created by]: 0x584D
775 # [Sub-section type]: 0
776 # [Entry Id]: 0x50000BB7
777 # [Platform Log Id]: 0x8200061D
778 # [CSSVER]:
779 # [Section Version]: 1
780 # [Creator Subsystem]: PHYP
781 # [BMC Event Log Id]: 341
782 # [Committed at]: 03/25/1920 12:06:22
783 # [User Header]:
784 # [Log Committed by]: 0x4552
785 # [Action Flags]:
786 # [0]: Report Externally
787 # [Subsystem]: I/O Subsystem
788 # [Event Type]: Miscellaneous, Informational Only
789 # [Sub-section type]: 0
790 # [Event Scope]: Entire Platform
791 # [Event Severity]: Informational Event
792 # [Host Transmission]: Not Sent
793 # [Section Version]: 1
794
795 ${pel_section_output}= Get From Dictionary ${pel_output} ${pel_section}
796 ${pel_field_output}= Get From Dictionary ${pel_section_output} ${pel_field}
797 Run Keyword If '${pel_field_output}' in @{pel_field_value} Append To List ${pel_id_list} ${id}
798 END
799 Sort List ${pel_id_list}
800
801 [Return] ${pel_id_list}
802
803
Rahul Maheshwarif575c902020-02-26 04:38:40 -0600804Get PEL Log Via BMC CLI
805 [Documentation] Returns the list of PEL IDs using BMC CLI.
Rahul Maheshwarifaa5d202020-02-24 23:32:57 -0600806
807 ${pel_records}= Peltool -l
Rahul Maheshwarif575c902020-02-26 04:38:40 -0600808 ${ids}= Get Dictionary Keys ${pel_records}
Rahul Maheshwari1cffc4b2020-04-02 02:58:16 -0500809 Sort List ${ids}
Rahul Maheshwarifaa5d202020-02-24 23:32:57 -0600810
Rahul Maheshwarif575c902020-02-26 04:38:40 -0600811 [Return] ${ids}
Rahul Maheshwari60b58a82020-04-02 02:40:19 -0500812
813
814Get PEL Field Value
815 [Documentation] Returns the value of given PEL's field.
816 [Arguments] ${pel_id} ${pel_section} ${pel_field}
817
818 # Description of argument(s):
819 # pel_id The ID of PEL (e.g. 0x5000002D, 0x5000002E).
820 # pel_section The section of PEL (e.g. Private Header, User Header)
821 # pel_field The PEL field (e.g. Event Severity, Event Type).
822
823 ${pel_output}= Peltool -i ${pel_id}
824
825 # Example of PEL output from "peltool -i <id>" command.
826 # [Private Header]:
827 # [Created at]: 08/24/1928 12:04:06
828 # [Created by]: 0x584D
829 # [Sub-section type]: 0
830 # [Entry Id]: 0x50000BB7
831 # [Platform Log Id]: 0x8200061D
832 # [CSSVER]:
833 # [Section Version]: 1
834 # [Creator Subsystem]: PHYP
835 # [BMC Event Log Id]: 341
836 # [Committed at]: 03/25/1920 12:06:22
837 # [User Header]:
838 # [Log Committed by]: 0x4552
839 # [Action Flags]:
840 # [0]: Report Externally
841 # [Subsystem]: I/O Subsystem
842 # [Event Type]: Miscellaneous, Informational Only
843 # [Sub-section type]: 0
844 # [Event Scope]: Entire Platform
845 # [Event Severity]: Informational Event
846 # [Host Transmission]: Not Sent
847 # [Section Version]: 1
848
849 ${pel_section_output}= Get From Dictionary ${pel_output} ${pel_section}
850 ${pel_field_output}= Get From Dictionary ${pel_section_output} ${pel_field}
851
852 [Return] ${pel_field_output}