Rahul Maheshwari | faa5d20 | 2020-02-24 23:32:57 -0600 | [diff] [blame] | 1 | *** Settings *** |
| 2 | Documentation This suite tests Platform Event Log (PEL) functionality of OpenBMC. |
| 3 | |
| 4 | Library ../../lib/pel_utils.py |
Rahul Maheshwari | a17a339 | 2020-03-02 04:51:52 -0600 | [diff] [blame] | 5 | Variables ../../data/pel_variables.py |
Rahul Maheshwari | 8f5256f | 2020-02-26 23:53:55 -0600 | [diff] [blame] | 6 | Resource ../../lib/list_utils.robot |
Rahul Maheshwari | f4a01aa | 2020-08-24 23:22:59 -0500 | [diff] [blame] | 7 | Resource ../../lib/logging_utils.robot |
manashsarma | 1810a5b | 2020-12-09 20:09:20 -0600 | [diff] [blame] | 8 | Resource ../../lib/connection_client.robot |
Rahul Maheshwari | faa5d20 | 2020-02-24 23:32:57 -0600 | [diff] [blame] | 9 | Resource ../../lib/openbmc_ffdc.robot |
| 10 | |
Rahul Maheshwari | f575c90 | 2020-02-26 04:38:40 -0600 | [diff] [blame] | 11 | Test Setup Redfish.Login |
| 12 | Test Teardown Run Keywords Redfish.Logout AND FFDC On Test Case Fail |
Rahul Maheshwari | faa5d20 | 2020-02-24 23:32:57 -0600 | [diff] [blame] | 13 | |
| 14 | |
| 15 | *** Variables *** |
| 16 | |
dnirmala | 789c802 | 2022-11-18 05:49:08 -0600 | [diff] [blame] | 17 | @{mandatory_pel_fields} Private Header User Header Primary SRC |
| 18 | ... Extended User Header Failing MTMS |
| 19 | @{mandatory_Predictive_pel_fields} Private Header User Header Primary SRC |
| 20 | ... Extended User Header Failing MTMS User Data |
dnirmala | 819be78 | 2022-11-07 04:34:40 -0600 | [diff] [blame] | 21 | @{Mandatory_Informational_Pel_Fields} Private Header User Header Primary SRC |
dnirmala | 789c802 | 2022-11-18 05:49:08 -0600 | [diff] [blame] | 22 | ... Extended User Header User Data |
Rahul Maheshwari | b8580dd | 2021-05-10 00:57:33 -0500 | [diff] [blame] | 23 | |
Rahul Maheshwari | faa5d20 | 2020-02-24 23:32:57 -0600 | [diff] [blame] | 24 | *** Test Cases *** |
| 25 | |
| 26 | Create Test PEL Log And Verify |
| 27 | [Documentation] Create PEL log using busctl command and verify via peltool. |
| 28 | [Tags] Create_Test_PEL_Log_And_Verify |
| 29 | |
Rahul Maheshwari | f575c90 | 2020-02-26 04:38:40 -0600 | [diff] [blame] | 30 | Redfish Purge Event Log |
Rahul Maheshwari | faa5d20 | 2020-02-24 23:32:57 -0600 | [diff] [blame] | 31 | Create Test PEL Log |
Rahul Maheshwari | f575c90 | 2020-02-26 04:38:40 -0600 | [diff] [blame] | 32 | ${pel_id}= Get PEL Log Via BMC CLI |
| 33 | Should Not Be Empty ${pel_id} msg=System PEL log entry is empty. |
| 34 | |
| 35 | |
Rahul Maheshwari | a17a339 | 2020-03-02 04:51:52 -0600 | [diff] [blame] | 36 | Verify PEL Log Details |
| 37 | [Documentation] Verify PEL log details via peltool. |
| 38 | [Tags] Verify_PEL_Log_Details |
| 39 | |
| 40 | Redfish Purge Event Log |
| 41 | |
| 42 | ${bmc_time1}= CLI Get BMC DateTime |
| 43 | Create Test PEL Log |
| 44 | ${bmc_time2}= CLI Get BMC DateTime |
| 45 | |
| 46 | ${pel_records}= Peltool -l |
| 47 | |
| 48 | # Example output from 'Peltool -l': |
| 49 | # pel_records: |
| 50 | # [0x50000012]: |
| 51 | # [CreatorID]: BMC |
| 52 | # [CompID]: 0x1000 |
| 53 | # [PLID]: 0x50000012 |
| 54 | # [Subsystem]: BMC Firmware |
| 55 | # [Message]: An application had an internal failure |
| 56 | # [SRC]: BD8D1002 |
| 57 | # [Commit Time]: 03/02/2020 09:35:15 |
| 58 | # [Sev]: Unrecoverable Error |
| 59 | |
| 60 | ${ids}= Get Dictionary Keys ${pel_records} |
| 61 | ${id}= Get From List ${ids} 0 |
| 62 | |
| 63 | @{pel_fields}= Create List CreatorID Subsystem Message Sev |
| 64 | FOR ${field} IN @{pel_fields} |
| 65 | Valid Value pel_records['${id}']['${field}'] ['${PEL_DETAILS['${field}']}'] |
| 66 | END |
| 67 | |
| 68 | Valid Value pel_records['${id}']['PLID'] ['${id}'] |
| 69 | |
| 70 | # Verify if "CompID" and "SRC" fields of PEL has alphanumeric value. |
| 71 | Should Match Regexp ${pel_records['${id}']['CompID']} [a-zA-Z0-9] |
| 72 | Should Match Regexp ${pel_records['${id}']['SRC']} [a-zA-Z0-9] |
| 73 | |
| 74 | ${pel_date_time}= Convert Date ${pel_records['${id}']['Commit Time']} |
| 75 | ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=yes |
| 76 | |
| 77 | # Convert BMC and PEL time to epoch time before comparing. |
| 78 | ${bmc_time1_epoch}= Convert Date ${bmc_time1} epoch |
| 79 | ${pel_time_epoch}= Convert Date ${pel_date_time} epoch |
| 80 | ${bmc_time2_epoch}= Convert Date ${bmc_time2} epoch |
| 81 | |
| 82 | Should Be True ${bmc_time1_epoch} <= ${pel_time_epoch} <= ${bmc_time2_epoch} |
| 83 | |
| 84 | |
Rahul Maheshwari | 3991868 | 2020-07-07 06:42:20 -0500 | [diff] [blame] | 85 | Verify Mandatory Sections Of Error Log PEL |
| 86 | [Documentation] Verify mandatory sections of error log PEL. |
| 87 | [Tags] Verify_Mandatory_Sections_Of_Error_Log_PEL |
| 88 | |
| 89 | Create Test PEL Log |
| 90 | |
| 91 | ${pel_ids}= Get PEL Log Via BMC CLI |
| 92 | ${pel_id}= Get From List ${pel_ids} -1 |
| 93 | ${pel_output}= Peltool -i ${pel_id} |
| 94 | ${pel_sections}= Get Dictionary Keys ${pel_output} |
| 95 | |
dnirmala | 789c802 | 2022-11-18 05:49:08 -0600 | [diff] [blame] | 96 | List Should Contain Sub List ${pel_sections} ${mandatory_pel_fields} |
Rahul Maheshwari | 3991868 | 2020-07-07 06:42:20 -0500 | [diff] [blame] | 97 | |
| 98 | |
Rahul Maheshwari | f575c90 | 2020-02-26 04:38:40 -0600 | [diff] [blame] | 99 | Verify PEL Log Persistence After BMC Reboot |
| 100 | [Documentation] Verify PEL log persistence after BMC reboot. |
| 101 | [Tags] Verify_PEL_Log_Persistence_After_BMC_Reboot |
| 102 | |
| 103 | Create Test PEL Log |
| 104 | ${pel_before_reboot}= Get PEL Log Via BMC CLI |
| 105 | |
| 106 | Redfish OBMC Reboot (off) |
| 107 | ${pel_after_reboot}= Get PEL Log Via BMC CLI |
| 108 | |
| 109 | List Should Contain Sub List ${pel_after_reboot} ${pel_before_reboot} |
Rahul Maheshwari | faa5d20 | 2020-02-24 23:32:57 -0600 | [diff] [blame] | 110 | |
| 111 | |
Rahul Maheshwari | 58b99fc | 2020-03-03 00:05:55 -0600 | [diff] [blame] | 112 | Verify PEL ID Numbering |
| 113 | [Documentation] Verify PEL ID numbering. |
| 114 | [Tags] Verify_PEL_ID_Numbering |
| 115 | |
| 116 | Redfish Purge Event Log |
| 117 | Create Test PEL Log |
| 118 | Create Test PEL Log |
| 119 | |
| 120 | ${pel_ids}= Get PEL Log Via BMC CLI |
| 121 | |
| 122 | # Example of PEL IDs from PEL logs. |
| 123 | # [0x50000012]: <--- First PEL ID |
| 124 | # [CreatorID]: BMC |
| 125 | # [CompID]: 0x1000 |
| 126 | # [PLID]: 0x50000012 |
| 127 | # [Subsystem]: BMC Firmware |
| 128 | # [Message]: An application had an internal failure |
| 129 | # [SRC]: BD8D1002 |
| 130 | # [Commit Time]: 03/02/2020 09:35:15 |
| 131 | # [Sev]: Unrecoverable Error |
| 132 | # |
| 133 | # [0x50000013]: <--- Second PEL ID |
| 134 | # [CreatorID]: BMC |
| 135 | # [CompID]: 0x1000 |
| 136 | # [PLID]: 0x50000013 |
| 137 | # [Subsystem]: BMC Firmware |
| 138 | # [Message]: An application had an internal failure |
| 139 | # [SRC]: BD8D1002 |
| 140 | # [Commit Time]: 03/02/2020 09:35:15 |
| 141 | # [Sev]: Unrecoverable Error |
| 142 | |
| 143 | Should Be True ${pel_ids[1]} == ${pel_ids[0]}+1 |
| 144 | |
Rahul Maheshwari | 371e4f8 | 2020-04-06 22:53:52 -0500 | [diff] [blame] | 145 | Verify Machine Type Model And Serial Number |
| 146 | [Documentation] Verify machine type model and serial number from PEL. |
| 147 | [Tags] Verify_Machine_Type_Model_And_Serial_Number |
| 148 | |
| 149 | Create Test PEL Log |
| 150 | |
| 151 | ${pel_ids}= Get PEL Log Via BMC CLI |
| 152 | ${id}= Get From List ${pel_ids} -1 |
Rahul Maheshwari | 56dba36 | 2020-05-12 00:59:48 -0500 | [diff] [blame] | 153 | |
Rahul Maheshwari | 371e4f8 | 2020-04-06 22:53:52 -0500 | [diff] [blame] | 154 | ${pel_serial_number}= Get PEL Field Value ${id} Failing MTMS Serial Number |
Rahul Maheshwari | 56dba36 | 2020-05-12 00:59:48 -0500 | [diff] [blame] | 155 | ${pel_serial_number}= Replace String Using Regexp ${pel_serial_number} ^0+ ${EMPTY} |
Rahul Maheshwari | 371e4f8 | 2020-04-06 22:53:52 -0500 | [diff] [blame] | 156 | ${pel_machine_type_model}= Get PEL Field Value ${id} Failing MTMS Machine Type Model |
Rahul Maheshwari | 56dba36 | 2020-05-12 00:59:48 -0500 | [diff] [blame] | 157 | ${pel_machine_type_model}= Replace String Using Regexp ${pel_machine_type_model} ^0+ ${EMPTY} |
Rahul Maheshwari | 371e4f8 | 2020-04-06 22:53:52 -0500 | [diff] [blame] | 158 | |
| 159 | # Example of "Machine Type Model" and "Serial Number" fields value from "Failing MTMS" section of PEL. |
| 160 | # [Failing MTMS]: |
| 161 | # [Created by]: 0x2000 |
| 162 | # [Machine Type Model]: 1234-ABC <---- Machine type |
| 163 | # [Section Version]: 1 |
| 164 | # [Serial Number]: ABCDEFG <---- Serial number |
| 165 | # [Sub-section type]: 0 |
| 166 | |
| 167 | ${redfish_machine_model}= Redfish.Get Attribute /redfish/v1/Systems/system/ Model |
Rahul Maheshwari | 56dba36 | 2020-05-12 00:59:48 -0500 | [diff] [blame] | 168 | ${redfish_machine_model}= Replace String Using Regexp ${redfish_machine_model} ^0+ ${EMPTY} |
Rahul Maheshwari | 371e4f8 | 2020-04-06 22:53:52 -0500 | [diff] [blame] | 169 | ${redfish_serial_number}= Redfish.Get Attribute /redfish/v1/Systems/system/ SerialNumber |
Rahul Maheshwari | 56dba36 | 2020-05-12 00:59:48 -0500 | [diff] [blame] | 170 | ${redfish_serial_number}= Replace String Using Regexp ${redfish_serial_number} ^0+ ${EMPTY} |
Rahul Maheshwari | 371e4f8 | 2020-04-06 22:53:52 -0500 | [diff] [blame] | 171 | |
| 172 | Valid Value pel_machine_type_model ['${redfish_machine_model}'] |
| 173 | Valid Value pel_serial_number ['${redfish_serial_number}'] |
| 174 | |
| 175 | # Check "Machine Type Model" and "Serial Number" fields value from "Extended User Header" section of PEL. |
| 176 | ${pel_machine_type_model}= Get PEL Field Value ${id} Extended User Header Reporting Machine Type |
Rahul Maheshwari | 56dba36 | 2020-05-12 00:59:48 -0500 | [diff] [blame] | 177 | ${pel_machine_type_model}= Replace String Using Regexp ${pel_machine_type_model} ^0+ ${EMPTY} |
Rahul Maheshwari | 371e4f8 | 2020-04-06 22:53:52 -0500 | [diff] [blame] | 178 | ${pel_serial_number}= Get PEL Field Value ${id} Extended User Header Reporting Serial Number |
Rahul Maheshwari | 56dba36 | 2020-05-12 00:59:48 -0500 | [diff] [blame] | 179 | ${pel_serial_number}= Replace String Using Regexp ${pel_serial_number} ^0+ ${EMPTY} |
Rahul Maheshwari | 371e4f8 | 2020-04-06 22:53:52 -0500 | [diff] [blame] | 180 | |
| 181 | Valid Value pel_machine_type_model ['${redfish_machine_model}'] |
| 182 | Valid Value pel_serial_number ['${redfish_serial_number}'] |
| 183 | |
Rahul Maheshwari | 58b99fc | 2020-03-03 00:05:55 -0600 | [diff] [blame] | 184 | |
Rahul Maheshwari | 1bea6cc | 2020-03-31 00:53:14 -0500 | [diff] [blame] | 185 | Verify Host Off State From PEL |
| 186 | [Documentation] Verify Host off state from PEL. |
| 187 | [Tags] Verify_Host_Off_State_From_PEL |
| 188 | |
| 189 | Redfish Power Off stack_mode=skip |
| 190 | Create Test PEL Log |
| 191 | |
| 192 | ${pel_ids}= Get PEL Log Via BMC CLI |
| 193 | ${id}= Get From List ${pel_ids} -1 |
| 194 | ${pel_host_state}= Get PEL Field Value ${id} User Data HostState |
| 195 | |
| 196 | Valid Value pel_host_state ['Off'] |
| 197 | |
| 198 | |
| 199 | Verify BMC Version From PEL |
| 200 | [Documentation] Verify BMC Version from PEL. |
| 201 | [Tags] Verify_BMC_Version_From_PEL |
| 202 | |
| 203 | Create Test PEL Log |
| 204 | |
| 205 | ${pel_ids}= Get PEL Log Via BMC CLI |
| 206 | ${id}= Get From List ${pel_ids} -1 |
Rahul Maheshwari | f6600f8 | 2021-05-28 01:25:20 -0500 | [diff] [blame] | 207 | ${pel_bmc_version}= Get PEL Field Value ${id} User Data FW Version ID |
Rahul Maheshwari | 1bea6cc | 2020-03-31 00:53:14 -0500 | [diff] [blame] | 208 | |
| 209 | ${bmc_version}= Get BMC Version |
| 210 | Valid Value bmc_version ['${bmc_version}'] |
| 211 | |
| 212 | |
Rahul Maheshwari | 8f5256f | 2020-02-26 23:53:55 -0600 | [diff] [blame] | 213 | Verify PEL Log After Host Poweron |
| 214 | [Documentation] Verify PEL log generation while booting host. |
| 215 | [Tags] Verify_PEL_Log_After_Host_Poweron |
| 216 | |
| 217 | Redfish Power Off stack_mode=skip |
| 218 | Redfish Purge Event Log |
| 219 | Redfish Power On stack_mode=skip |
| 220 | |
| 221 | ${pel_informational_error}= Get PEL Log IDs User Header Event Severity Informational Event |
| 222 | ${pel_bmc_created_error}= Get PEL Log IDs Private Header Creator Subsystem BMC |
| 223 | |
| 224 | # Get BMC created non-infomational error. |
| 225 | ${pel_bmc_error}= Subtract Lists ${pel_bmc_created_error} ${pel_informational_error} |
| 226 | |
| 227 | Should Be Empty ${pel_bmc_error} msg=Unexpected error log generated during Host poweron. |
| 228 | |
| 229 | |
Rahul Maheshwari | 089615b | 2020-04-02 01:01:38 -0500 | [diff] [blame] | 230 | Verify BMC Event Log ID |
| 231 | [Documentation] Verify BMC Event Log ID from PEL. |
| 232 | [Tags] Verify_BMC_Event_Log_ID |
| 233 | |
| 234 | Redfish Purge Event Log |
| 235 | Create Test PEL Log |
| 236 | |
| 237 | ${pel_ids}= Get PEL Log Via BMC CLI |
| 238 | ${pel_bmc_event_log_id}= Get PEL Field Value ${pel_ids[0]} Private Header BMC Event Log Id |
| 239 | |
| 240 | # Example "BMC Event Log Id" field value from "Private Header" section of PEL. |
| 241 | # [Private Header]: |
| 242 | # [Created at]: 08/24/1928 12:04:06 |
| 243 | # [Created by]: 0x584D |
| 244 | # [Sub-section type]: 0 |
| 245 | # [Entry Id]: 0x50000BB7 |
| 246 | # [Platform Log Id]: 0x8200061D |
| 247 | # [CSSVER]: |
| 248 | # [Section Version]: 1 |
| 249 | # [Creator Subsystem]: PHYP |
| 250 | # [BMC Event Log Id]: 341 <---- BMC event log id value |
| 251 | # [Committed at]: 03/25/1920 12:06:22 |
| 252 | |
| 253 | ${redfish_event_logs}= Redfish.Get Properties /redfish/v1/Systems/system/LogServices/EventLog/Entries |
| 254 | |
| 255 | # Example of redfish_event_logs output: |
| 256 | # redfish_event_logs: |
| 257 | # [@odata.id]: /redfish/v1/Systems/system/LogServices/EventLog/Entries |
| 258 | # [Name]: System Event Log Entries |
| 259 | # [Members@odata.count]: 1 |
| 260 | # [@odata.type]: #LogEntryCollection.LogEntryCollection |
| 261 | # [Description]: Collection of System Event Log Entries |
| 262 | # [Members]: |
| 263 | # [0]: |
| 264 | # [@odata.id]: /redfish/v1/Systems/system/LogServices/EventLog/Entries/235 |
| 265 | # [Name]: System Event Log Entry |
| 266 | # [Severity]: Critical |
| 267 | # [EntryType]: Event |
| 268 | # [Created]: 2020-04-02T07:25:13+00:00 |
| 269 | # [@odata.type]: #LogEntry.v1_5_1.LogEntry |
| 270 | # [Id]: 235 <----- Event log ID |
| 271 | # [Message]: xyz.openbmc_project.Common.Error.InternalFailure |
| 272 | |
| 273 | Valid Value pel_bmc_event_log_id ['${redfish_event_logs['Members'][0]['Id']}'] |
| 274 | |
| 275 | |
Rahul Maheshwari | 97d7ab8 | 2020-08-04 00:01:42 -0500 | [diff] [blame] | 276 | Verify FRU Callout |
| 277 | [Documentation] Verify FRU callout entries from PEL log. |
| 278 | [Tags] Verify_FRU_Callout |
dnirmala | d395d46 | 2022-11-11 02:45:49 -0600 | [diff] [blame] | 279 | |
| 280 | # Power off the system to avoid unnecessary error logs during test. |
| 281 | Redfish Power Off stack_mode=skip |
Rahul Maheshwari | 97d7ab8 | 2020-08-04 00:01:42 -0500 | [diff] [blame] | 282 | Create Test PEL Log FRU Callout |
| 283 | |
| 284 | ${pel_ids}= Get PEL Log Via BMC CLI |
| 285 | ${id}= Get From List ${pel_ids} -1 |
| 286 | ${pel_callout_section}= Get PEL Field Value ${id} Primary SRC Callout Section |
| 287 | |
| 288 | # Example of PEL Callout Section from "peltool -i <id>" command. |
| 289 | # [Callouts]: |
| 290 | # [0]: |
| 291 | # [FRU Type]: Normal Hardware FRU |
| 292 | # [Priority]: Mandatory, replace all with this type as a unit |
| 293 | # [Location Code]: U78DA.ND1.1234567-P0 |
| 294 | # [Part Number]: F191014 |
| 295 | # [CCIN]: 2E2D |
| 296 | # [Serial Number]: YL2E2D010000 |
| 297 | # [Callout Count]: 1 |
| 298 | |
| 299 | Valid Value pel_callout_section['Callout Count'] ['1'] |
| 300 | Valid Value pel_callout_section['Callouts'][0]['FRU Type'] ['Normal Hardware FRU'] |
| 301 | Should Contain ${pel_callout_section['Callouts'][0]['Priority']} Mandatory |
| 302 | |
| 303 | # Verify Location Code field of PEL callout with motherboard's Location Code. |
| 304 | ${busctl_output}= BMC Execute Command ${CMD_INVENTORY_PREFIX} com.ibm.ipzvpd.Location LocationCode |
| 305 | Should Be Equal ${pel_callout_section['Callouts'][0]['Location Code']} |
| 306 | ... ${busctl_output[0].split('"')[1].strip('"')} |
| 307 | |
| 308 | # TODO: Compare CCIN and part number fields of PEL callout with Redfish or busctl output. |
| 309 | Should Match Regexp ${pel_callout_section['Callouts'][0]['CCIN']} [a-zA-Z0-9] |
| 310 | Should Match Regexp ${pel_callout_section['Callouts'][0]['Part Number']} [a-zA-Z0-9] |
| 311 | |
| 312 | # Verify Serial Number field of PEL callout with motherboard's Serial Number. |
| 313 | ${busctl_output}= BMC Execute Command |
| 314 | ... ${CMD_INVENTORY_PREFIX} xyz.openbmc_project.Inventory.Decorator.Asset SerialNumber |
| 315 | Should Be Equal ${pel_callout_section['Callouts'][0]['Serial Number']} |
| 316 | ... ${busctl_output[0].split('"')[1].strip('"')} |
| 317 | |
| 318 | |
| 319 | Verify Procedure And Symbolic FRU Callout |
| 320 | [Documentation] Verify procedure and symbolic FRU callout from PEL log. |
| 321 | [Tags] Verify_Procedure_And_Symbolic_FRU_Callout |
| 322 | |
| 323 | Create Test PEL Log Procedure And Symbolic FRU Callout |
| 324 | |
| 325 | ${pel_ids}= Get PEL Log Via BMC CLI |
| 326 | ${id}= Get From List ${pel_ids} -1 |
| 327 | ${pel_callout_section}= Get PEL Field Value ${id} Primary SRC Callout Section |
| 328 | |
| 329 | # Example of PEL Callout Section from "peltool -i <id>" command. |
| 330 | # [Callouts]: |
| 331 | # [0]: |
| 332 | # [Priority]: Mandatory, replace all with this type as a unit |
| 333 | # [Procedure Number]: BMCSP02 |
| 334 | # [FRU Type]: Maintenance Procedure Required |
| 335 | # [1]: |
| 336 | # [Priority]: Medium Priority |
| 337 | # [Part Number]: SVCDOCS |
| 338 | # [FRU Type]: Symbolic FRU |
| 339 | # [Callout Count]: 2 |
| 340 | |
| 341 | Valid Value pel_callout_section['Callout Count'] ['2'] |
| 342 | |
| 343 | # Verify procedural callout info. |
| 344 | |
| 345 | Valid Value pel_callout_section['Callouts'][0]['FRU Type'] ['Maintenance Procedure Required'] |
| 346 | Should Contain ${pel_callout_section['Callouts'][0]['Priority']} Mandatory |
| 347 | # Verify if "Procedure Number" field of PEL has an alphanumeric value. |
Rahul Maheshwari | 6acf0ee | 2020-10-28 05:37:01 -0500 | [diff] [blame] | 348 | Should Match Regexp ${pel_callout_section['Callouts'][0]['Procedure']} [a-zA-Z0-9] |
Rahul Maheshwari | 97d7ab8 | 2020-08-04 00:01:42 -0500 | [diff] [blame] | 349 | |
| 350 | # Verify procedural callout info. |
| 351 | |
| 352 | Valid Value pel_callout_section['Callouts'][1]['FRU Type'] ['Symbolic FRU'] |
| 353 | Should Contain ${pel_callout_section['Callouts'][1]['Priority']} Medium Priority |
| 354 | # Verify if "Part Number" field of Symbolic FRU has an alphanumeric value. |
| 355 | Should Match Regexp ${pel_callout_section['Callouts'][1]['Part Number']} [a-zA-Z0-9] |
| 356 | |
| 357 | |
Rahul Maheshwari | f4a01aa | 2020-08-24 23:22:59 -0500 | [diff] [blame] | 358 | Verify PEL Log Entry For Event Log |
| 359 | [Documentation] Create an event log and verify PEL log entry in BMC for the same. |
| 360 | [Tags] Verify_PEL_Log_Entry_For_Event_Log |
| 361 | |
| 362 | Redfish Purge Event Log |
Rahul Maheshwari | 71d589c | 2022-04-22 01:29:44 -0500 | [diff] [blame] | 363 | |
| 364 | # Create an unrecoverable error log. |
| 365 | Create Test PEL Log Unrecoverable Error |
Rahul Maheshwari | f4a01aa | 2020-08-24 23:22:59 -0500 | [diff] [blame] | 366 | |
| 367 | ${elog_entry}= Get Event Logs |
| 368 | # Example of Redfish event logs: |
| 369 | # elog_entry: |
| 370 | # [0]: |
Rahul Maheshwari | 71d589c | 2022-04-22 01:29:44 -0500 | [diff] [blame] | 371 | # [Message]: BD802003 event in subsystem: Platform Firmware |
George Keishing | b78bca2 | 2021-06-29 11:11:19 -0500 | [diff] [blame] | 372 | # [Created]: 2020-04-20T01:55:22+00:00 |
| 373 | # [Id]: 1 |
| 374 | # [@odata.id]: /redfish/v1/Systems/system/LogServices/EventLog/Entries/1 |
| 375 | # [@odata.type]: #LogEntry.v1_4_0.LogEntry |
| 376 | # [EntryType]: Event |
| 377 | # [Severity]: Critical |
| 378 | # [Name]: System Event Log Entry |
Rahul Maheshwari | f4a01aa | 2020-08-24 23:22:59 -0500 | [diff] [blame] | 379 | |
| 380 | ${redfish_log_time}= Convert Date ${elog_entry[0]["Created"]} epoch |
| 381 | |
| 382 | ${pel_records}= Peltool -l |
| 383 | # Example output from 'Peltool -l': |
| 384 | # pel_records: |
| 385 | # [0x50000023]: |
Rahul Maheshwari | 71d589c | 2022-04-22 01:29:44 -0500 | [diff] [blame] | 386 | # [SRC]: BD802003 |
George Keishing | b78bca2 | 2021-06-29 11:11:19 -0500 | [diff] [blame] | 387 | # [CreatorID]: BMC |
Rahul Maheshwari | 71d589c | 2022-04-22 01:29:44 -0500 | [diff] [blame] | 388 | # [Message]: This is a test error |
| 389 | # [CompID]: 0x2000 |
| 390 | # [PLID]: 0x500053D9 |
George Keishing | b78bca2 | 2021-06-29 11:11:19 -0500 | [diff] [blame] | 391 | # [Commit Time]: 04/20/2020 01:55:22 |
Rahul Maheshwari | 71d589c | 2022-04-22 01:29:44 -0500 | [diff] [blame] | 392 | # [Subsystem]: Platform Firmware |
George Keishing | b78bca2 | 2021-06-29 11:11:19 -0500 | [diff] [blame] | 393 | # [Sev]: Unrecoverable Error |
Rahul Maheshwari | f4a01aa | 2020-08-24 23:22:59 -0500 | [diff] [blame] | 394 | |
| 395 | ${ids}= Get Dictionary Keys ${pel_records} |
| 396 | ${id}= Get From List ${ids} 0 |
| 397 | ${pel_log_time}= Convert Date ${pel_records['${id}']['Commit Time']} epoch |
| 398 | ... date_format=%m/%d/%Y %H:%M:%S |
| 399 | |
| 400 | # Verify that both Redfish event and PEL has log entry for internal error with same time stamp. |
Rahul Maheshwari | 71d589c | 2022-04-22 01:29:44 -0500 | [diff] [blame] | 401 | Should Contain Any ${pel_records['${id}']['Message']} test error ignore_case=True |
Rahul Maheshwari | 40ade89 | 2021-12-20 22:38:21 -0600 | [diff] [blame] | 402 | Should Contain ${elog_entry[0]['Message']} |
Rahul Maheshwari | 71d589c | 2022-04-22 01:29:44 -0500 | [diff] [blame] | 403 | ... ${pel_records['${id}']['SRC']} ignore_case=True |
Rahul Maheshwari | f4a01aa | 2020-08-24 23:22:59 -0500 | [diff] [blame] | 404 | |
| 405 | Should Be Equal ${redfish_log_time} ${pel_log_time} |
| 406 | |
| 407 | |
Rahul Maheshwari | 326d064 | 2020-04-07 06:13:38 -0500 | [diff] [blame] | 408 | Verify Delete All PEL |
| 409 | [Documentation] Verify deleting all PEL logs. |
| 410 | [Tags] Verify_Delete_All_PEL |
| 411 | |
| 412 | Create Test PEL Log |
| 413 | Create Test PEL Log |
| 414 | Peltool --delete-all False |
| 415 | |
| 416 | ${pel_ids}= Get PEL Log Via BMC CLI |
| 417 | Should Be Empty ${pel_ids} |
| 418 | |
| 419 | |
manashsarma | 4e93538 | 2020-10-16 12:00:53 -0500 | [diff] [blame] | 420 | Verify Informational Error Log |
| 421 | [Documentation] Create an informational error log and verify. |
| 422 | [Tags] Verify_Informational_Error_Log |
| 423 | |
| 424 | Redfish Purge Event Log |
| 425 | # Create an informational error log. |
| 426 | BMC Execute Command ${CMD_INFORMATIONAL_ERROR} |
| 427 | ${pel_records}= Peltool -lfh |
| 428 | |
| 429 | # An example of information error log data: |
| 430 | # { |
| 431 | # "0x500006A0": { |
| 432 | # "SRC": "BD8D1002", |
| 433 | # "Message": "An application had an internal failure", |
| 434 | # "PLID": "0x500006A0", |
| 435 | # "CreatorID": "BMC", |
| 436 | # "Subsystem": "BMC Firmware", |
| 437 | # "Commit Time": "10/14/2020 11:41:38", |
| 438 | # "Sev": "Informational Event", |
| 439 | # "CompID": "0x1000" |
| 440 | # } |
| 441 | # } |
| 442 | |
| 443 | ${ids}= Get Dictionary Keys ${pel_records} |
| 444 | ${id}= Get From List ${ids} 0 |
| 445 | Should Contain ${pel_records['${id}']['Sev']} Informational |
| 446 | |
| 447 | |
manashsarma | 15fe563 | 2020-10-18 03:42:58 -0500 | [diff] [blame] | 448 | Verify Predictable Error Log |
| 449 | [Documentation] Create a predictive error and verify. |
| 450 | [Tags] Verify_Predictable_Error_Log |
| 451 | |
| 452 | # Create a predictable error log. |
| 453 | BMC Execute Command ${CMD_PREDICTIVE_ERROR} |
| 454 | ${pel_records}= Peltool -l |
| 455 | |
| 456 | # An example of predictive error log data: |
| 457 | # { |
| 458 | # "0x5000069E": { |
| 459 | # "SRC": "BD8D1002", |
| 460 | # "Message": "An application had an internal failure", |
| 461 | # "PLID": "0x5000069E", |
| 462 | # "CreatorID": "BMC", |
| 463 | # "Subsystem": "BMC Firmware", |
| 464 | # "Commit Time": "10/14/2020 11:40:07", |
| 465 | # "Sev": "Predictive Error", |
| 466 | # "CompID": "0x1000" |
| 467 | # } |
| 468 | # } |
| 469 | |
| 470 | ${pel_ids}= Get PEL Log Via BMC CLI |
| 471 | ${id}= Get From List ${pel_ids} -1 |
| 472 | Should Contain ${pel_records['${id}']['Sev']} Predictive |
| 473 | |
| 474 | |
| 475 | Verify Unrecoverable Error Log |
| 476 | [Documentation] Create an unrecoverable error and verify. |
| 477 | [Tags] Verify_Unrecoverable_Error_Log |
| 478 | |
| 479 | # Create an internal failure error log. |
| 480 | BMC Execute Command ${CMD_UNRECOVERABLE_ERROR} |
| 481 | ${pel_records}= Peltool -l |
| 482 | |
| 483 | # An example of unrecoverable error log data: |
| 484 | # { |
| 485 | # "0x50000CC5": { |
| 486 | # "SRC": "BD8D1002", |
| 487 | # "Message": "An application had an internal failure", |
| 488 | # "PLID": "0x50000CC5", |
| 489 | # "CreatorID": "BMC", |
| 490 | # "Subsystem": "BMC Firmware", |
| 491 | # "Commit Time": "04/01/2020 16:44:55", |
| 492 | # "Sev": "Unrecoverable Error", |
| 493 | # "CompID": "0x1000" |
| 494 | # } |
| 495 | # } |
| 496 | |
| 497 | ${pel_ids}= Get PEL Log Via BMC CLI |
| 498 | ${id}= Get From List ${pel_ids} -1 |
| 499 | Should Contain ${pel_records['${id}']['Sev']} Unrecoverable |
| 500 | |
| 501 | |
manashsarma | 595282d | 2020-10-20 13:22:31 -0500 | [diff] [blame] | 502 | Verify Error Logging Rotation Policy |
| 503 | [Documentation] Verify error logging rotation policy. |
| 504 | [Tags] Verify_Error_Logging_Rotation_Policy |
| 505 | [Template] Error Logging Rotation Policy |
manashsarma | d073a0a | 2020-10-20 12:54:59 -0500 | [diff] [blame] | 506 | |
manashsarma | 557b632 | 2020-11-29 12:08:32 -0600 | [diff] [blame] | 507 | # Error logs to be created % of total logging space when error |
| 508 | # log exceeds max limit. |
| 509 | Informational BMC 3000 15 |
| 510 | Predictive BMC 3000 30 |
| 511 | Unrecoverable BMC 3000 30 |
| 512 | Informational BMC 1500, Predictive BMC 1500 45 |
| 513 | Informational BMC 1500, Unrecoverable BMC 1500 45 |
| 514 | Unrecoverable BMC 1500, Predictive BMC 1500 30 |
Rahul Maheshwari | b4cacb5 | 2021-01-18 23:57:29 -0600 | [diff] [blame] | 515 | |
| 516 | |
| 517 | Verify Error Logging Rotation Policy With All Types Of Errors |
| 518 | [Documentation] Verify error logging rotation policy with all types of errors. |
George Keishing | 4203fad | 2022-01-31 12:22:33 -0600 | [diff] [blame] | 519 | [Tags] Verify_Error_Logging_Rotation_Policy_With_All_Types_Of_Errors |
Rahul Maheshwari | b4cacb5 | 2021-01-18 23:57:29 -0600 | [diff] [blame] | 520 | [Template] Error Logging Rotation Policy |
| 521 | |
| 522 | # Error logs to be created % of total logging space when error |
| 523 | # log exceeds max limit. |
manashsarma | 557b632 | 2020-11-29 12:08:32 -0600 | [diff] [blame] | 524 | Unrecoverable BMC 1000, Informational BMC 1000, Predictive BMC 1000 45 |
manashsarma | c513166 | 2020-11-12 12:25:59 -0600 | [diff] [blame] | 525 | |
manashsarma | 1810a5b | 2020-12-09 20:09:20 -0600 | [diff] [blame] | 526 | |
| 527 | Verify Error Logging Rotation Policy With HOST Error Logs |
| 528 | [Documentation] Verify error logging rotation policy for non bmc error logs. |
| 529 | [Tags] Verify_Error_Logging_Rotation_Policy_With_HOST_Error_Logs |
| 530 | [Setup] Run Keywords Open Connection for SCP AND scp.Put File ${UNRECOVERABLE_FILE_PATH} |
manashsarma | 4baa6a7 | 2020-12-16 07:13:37 -0600 | [diff] [blame] | 531 | ... /tmp/FILE_HOST_UNRECOVERABLE AND scp.Put File ${INFORMATIONAL_FILE_PATH} |
| 532 | ... /tmp/FILE_HOST_INFORMATIONAL |
manashsarma | 1810a5b | 2020-12-09 20:09:20 -0600 | [diff] [blame] | 533 | [Template] Error Logging Rotation Policy |
| 534 | |
| 535 | # Error logs to be created % of total logging space when error |
| 536 | # log exceeds max limit. |
manashsarma | 4baa6a7 | 2020-12-16 07:13:37 -0600 | [diff] [blame] | 537 | Informational HOST 3000 15 |
manashsarma | 1810a5b | 2020-12-09 20:09:20 -0600 | [diff] [blame] | 538 | Unrecoverable HOST 3000 30 |
manashsarma | 4baa6a7 | 2020-12-16 07:13:37 -0600 | [diff] [blame] | 539 | Informational HOST 1500, Informational BMC 1500 30 |
| 540 | Informational HOST 1500, Unrecoverable BMC 1500 45 |
manashsarma | 1810a5b | 2020-12-09 20:09:20 -0600 | [diff] [blame] | 541 | Unrecoverable HOST 1500, Informational BMC 1500 45 |
| 542 | Unrecoverable HOST 1500, Predictive BMC 1500 60 |
Rahul Maheshwari | b4cacb5 | 2021-01-18 23:57:29 -0600 | [diff] [blame] | 543 | |
| 544 | |
| 545 | Verify Error Logging Rotation Policy With Unrecoverable HOST And BMC Error Logs |
| 546 | [Documentation] Verify error logging rotation policy with unrecoverable HOST and BMC error logs. |
| 547 | [Tags] Verify_Error_Logging_Rotation_Policy_With_Unrecoverable_HOST_And_BMC_Error_Logs |
| 548 | [Setup] Run Keywords Open Connection for SCP AND scp.Put File ${UNRECOVERABLE_FILE_PATH} |
Rahul Maheshwari | e54d16f | 2021-02-04 05:20:46 -0600 | [diff] [blame] | 549 | ... /tmp/FILE_NBMC_UNRECOVERABLE AND Redfish.Login |
Rahul Maheshwari | b4cacb5 | 2021-01-18 23:57:29 -0600 | [diff] [blame] | 550 | [Template] Error Logging Rotation Policy |
| 551 | |
| 552 | # Error logs to be created % of total logging space when error |
| 553 | # log exceeds max limit. |
manashsarma | 1810a5b | 2020-12-09 20:09:20 -0600 | [diff] [blame] | 554 | Unrecoverable HOST 1500, Unrecoverable BMC 1500 60 |
| 555 | |
| 556 | |
manashsarma | 6f1f2a4 | 2020-12-15 07:29:41 -0600 | [diff] [blame] | 557 | Verify Old Logs Are Deleted When Count Crosses Max |
| 558 | [Documentation] Verify that when the count crosses max, older logs are deleted. |
| 559 | [Tags] Verify_Old_Logs_Are_Deleted_When_Count_Crosses_Max |
| 560 | |
| 561 | Redfish Purge Event Log |
manashsarma | 6f1f2a4 | 2020-12-15 07:29:41 -0600 | [diff] [blame] | 562 | # Create 3000 error logs. |
| 563 | FOR ${count} IN RANGE ${3000} |
| 564 | BMC Execute Command ${CMD_PREDICTIVE_ERROR} |
| 565 | END |
| 566 | |
| 567 | # Retrieve the IDs of the logs. |
| 568 | ${pel_ids}= Get PEL Log Via BMC CLI |
manashsarma | dd13268 | 2021-04-08 07:59:13 -0500 | [diff] [blame] | 569 | ${1st_id}= Get From List ${pel_ids} 0 |
| 570 | ${3000th_id}= Get From List ${pel_ids} 2999 |
manashsarma | 6f1f2a4 | 2020-12-15 07:29:41 -0600 | [diff] [blame] | 571 | |
| 572 | # Now create 3001st log to cross threshold limit and trigger error logs rotation. |
| 573 | BMC Execute Command ${CMD_PREDICTIVE_ERROR} |
manashsarma | dd13268 | 2021-04-08 07:59:13 -0500 | [diff] [blame] | 574 | |
manashsarma | 6f1f2a4 | 2020-12-15 07:29:41 -0600 | [diff] [blame] | 575 | # Wait few seconds for error logs rotation to complete. |
| 576 | Sleep 10s |
| 577 | |
| 578 | # Now verify that first log is no more available but the 3000th is available. |
manashsarma | dd13268 | 2021-04-08 07:59:13 -0500 | [diff] [blame] | 579 | ${status} ${output}= Run Keyword And Ignore Error Peltool -i ${1st_id} |
| 580 | Should Be True '${status}' == 'FAIL' |
manashsarma | 6f1f2a4 | 2020-12-15 07:29:41 -0600 | [diff] [blame] | 581 | Should Contain ${output} PEL not found |
manashsarma | dd13268 | 2021-04-08 07:59:13 -0500 | [diff] [blame] | 582 | |
| 583 | ${status} ${output}= Run Keyword And Ignore Error Peltool -i ${3000th_id} |
| 584 | Should Be True '${status}' == 'PASS' |
manashsarma | 6f1f2a4 | 2020-12-15 07:29:41 -0600 | [diff] [blame] | 585 | Should Not Contain ${output} PEL not found |
| 586 | |
| 587 | |
manashsarma | 5f20120 | 2020-10-30 12:52:04 -0500 | [diff] [blame] | 588 | Verify Reverse Order Of PEL Logs |
| 589 | [Documentation] Verify PEL command to output PEL logs in reverse order. |
George Keishing | 4203fad | 2022-01-31 12:22:33 -0600 | [diff] [blame] | 590 | [Tags] Verify_Reverse_Order_Of_PEL_Logs |
manashsarma | 5f20120 | 2020-10-30 12:52:04 -0500 | [diff] [blame] | 591 | |
| 592 | Redfish Purge Event Log |
manashsarma | c513166 | 2020-11-12 12:25:59 -0600 | [diff] [blame] | 593 | |
| 594 | # Below commands create unrecoverable error log at first and then the predictable error. |
manashsarma | 5f20120 | 2020-10-30 12:52:04 -0500 | [diff] [blame] | 595 | BMC Execute Command ${CMD_UNRECOVERABLE_ERROR} |
| 596 | BMC Execute Command ${CMD_PREDICTIVE_ERROR} |
| 597 | |
manashsarma | c513166 | 2020-11-12 12:25:59 -0600 | [diff] [blame] | 598 | # Using peltool -lr, recent PELs appear first. Hence the ID of first PEL is greater than the next. |
| 599 | ${pel_records}= peltool -lr |
manashsarma | 5f20120 | 2020-10-30 12:52:04 -0500 | [diff] [blame] | 600 | |
manashsarma | c513166 | 2020-11-12 12:25:59 -0600 | [diff] [blame] | 601 | # It is found that, variables like dictionary always keep items in sorted order that makes |
| 602 | # this verification not possible, hence json is used to keep the items original order. |
| 603 | ${pel_records}= Convert To String ${pel_records} |
| 604 | ${json_string}= Replace String ${pel_records} ' " |
| 605 | ${json_object}= Evaluate json.loads('''${json_string}''') json |
| 606 | |
| 607 | ${list}= Convert To List ${json_object} |
| 608 | |
| 609 | ${id1}= Get From List ${list} 0 |
| 610 | ${id1}= Convert To Integer ${id1} |
| 611 | ${id2}= Get From List ${list} 1 |
| 612 | ${id2}= Convert To Integer ${id2} |
| 613 | |
| 614 | Should Be True ${id1} > ${id2} |
manashsarma | 5f20120 | 2020-10-30 12:52:04 -0500 | [diff] [blame] | 615 | |
| 616 | |
manashsarma | 6d75bd8 | 2020-10-30 07:36:05 -0500 | [diff] [blame] | 617 | Verify Total PEL Count |
| 618 | [Documentation] Verify total PEL count returned by peltool command. |
| 619 | [Tags] Verify_Total_PEL_Count |
| 620 | |
| 621 | # Initially remove all logs. |
| 622 | Redfish Purge Event Log |
| 623 | |
| 624 | # Generate a random number between 1-20. |
| 625 | ${random}= Evaluate random.randint(1, 20) modules=random |
| 626 | |
| 627 | # Generate predictive error log multiple times. |
| 628 | FOR ${count} IN RANGE 0 ${random} |
| 629 | BMC Execute Command ${CMD_PREDICTIVE_ERROR} |
| 630 | END |
| 631 | |
| 632 | # Check PEL log count via peltool command and compare it with actual generated log count. |
| 633 | ${pel_records}= peltool -n |
| 634 | |
| 635 | Should Be Equal ${pel_records['Number of PELs found']} ${random} |
| 636 | |
| 637 | |
manashsarma | 407deda | 2020-10-30 10:36:26 -0500 | [diff] [blame] | 638 | Verify Listing Information Error |
| 639 | [Documentation] Verify that information error logs can only be listed using -lfh option of peltool. |
| 640 | [Tags] Verify_Listing_Information_Error |
| 641 | |
| 642 | # Initially remove all logs. |
| 643 | Redfish Purge Event Log |
| 644 | BMC Execute Command ${CMD_INFORMATIONAL_ERROR} |
| 645 | |
| 646 | # Generate informational logs and verify that it would not get listed by peltool's list command. |
| 647 | ${pel_records}= peltool -l |
manashsarma | 31d807b | 2020-11-11 12:49:46 -0600 | [diff] [blame] | 648 | ${ids}= Get Dictionary Keys ${pel_records} |
| 649 | Should Be Empty ${ids} |
manashsarma | 407deda | 2020-10-30 10:36:26 -0500 | [diff] [blame] | 650 | |
| 651 | # Verify that information logs get listed using peltool's list command with -lfh option. |
| 652 | ${pel_records}= peltool -lfh |
manashsarma | 31d807b | 2020-11-11 12:49:46 -0600 | [diff] [blame] | 653 | ${ids}= Get Dictionary Keys ${pel_records} |
| 654 | Should Not Be Empty ${ids} |
| 655 | ${id}= Get From List ${ids} 0 |
manashsarma | 407deda | 2020-10-30 10:36:26 -0500 | [diff] [blame] | 656 | Should Contain ${pel_records['${id}']['Sev']} Informational |
| 657 | |
| 658 | |
manashsarma | 54539ff | 2020-11-23 02:32:36 -0600 | [diff] [blame] | 659 | Verify PEL Delete |
| 660 | [Documentation] Verify that peltool command can delete PEL log based on id. |
| 661 | [Tags] Verify_PEL_Delete |
| 662 | |
| 663 | BMC Execute Command ${CMD_PREDICTIVE_ERROR} |
| 664 | ${pel_ids}= Get PEL Log Via BMC CLI |
| 665 | ${id}= Get From List ${pel_ids} -1 |
Rahul Maheshwari | 5eab56c | 2021-01-06 23:58:46 -0600 | [diff] [blame] | 666 | Peltool -d ${id} False |
| 667 | Run Keyword and Expect Error *PEL not found* Peltool -i ${id} |
manashsarma | 54539ff | 2020-11-23 02:32:36 -0600 | [diff] [blame] | 668 | |
| 669 | |
dnirmala | fe690af | 2022-10-18 05:27:31 -0500 | [diff] [blame] | 670 | Verify Mandatory Fields For Predictive Error |
| 671 | [Documentation] Verify mandatory fields of predictive errors from pel information. |
| 672 | [Tags] Verify_Mandatory_Fields_For_Predictive_Error |
| 673 | |
| 674 | # Inject predictive error. |
| 675 | BMC Execute Command ${CMD_PREDICTIVE_ERROR} |
| 676 | |
| 677 | ${pel_ids}= Get PEL Log Via BMC CLI |
| 678 | ${pel_id}= Get From List ${pel_ids} -1 |
| 679 | ${pel_output}= Peltool -i ${pel_id} |
| 680 | # Get all fields in predictive error log. |
| 681 | ${pel_sections}= Get Dictionary Keys ${pel_output} |
| 682 | |
dnirmala | 789c802 | 2022-11-18 05:49:08 -0600 | [diff] [blame] | 683 | List Should Contain Sub List ${pel_sections} ${mandatory_Predictive_pel_fields} |
dnirmala | fe690af | 2022-10-18 05:27:31 -0500 | [diff] [blame] | 684 | |
| 685 | |
dnirmala | 819be78 | 2022-11-07 04:34:40 -0600 | [diff] [blame] | 686 | Verify Mandatory Fields For Informational Error |
| 687 | [Documentation] Verify mandatory fields of informational error from pel information. |
| 688 | [Tags] Verify_Mandatory_Fields_For_Informational_Error |
| 689 | |
| 690 | # Inject informational error. |
| 691 | BMC Execute Command ${CMD_INFORMATIONAL_ERROR} |
| 692 | ${pel_records}= Peltool -lfh |
| 693 | |
| 694 | ${ids}= Get Dictionary Keys ${pel_records} |
| 695 | ${pel_id}= Get From List ${ids} -1 |
| 696 | ${pel_output}= Peltool -i ${pel_id} |
| 697 | |
| 698 | # Get all fields in the informational error log. |
| 699 | ${pel_sections}= Get Dictionary Keys ${pel_output} |
| 700 | FOR ${section} IN @{Mandatory_Informational_Pel_Fields} |
| 701 | ${contains}= Evaluate "${section}" in "${pel_sections}" |
| 702 | Should Be True ${contains} |
| 703 | END |
| 704 | |
| 705 | |
dnirmala | 789c802 | 2022-11-18 05:49:08 -0600 | [diff] [blame] | 706 | Verify PEL Log Not Offloaded To Host |
| 707 | [Documentation] Verify host transmission state of not offloaded PEL logs. |
| 708 | [Tags] Verify_PEL_Log_Not_Offloaded_To_Host |
| 709 | [Template] Verify PEL Transmission To Host |
| 710 | |
| 711 | # error_type host_state expected_transmission_state |
| 712 | informational_error Off Not Sent |
| 713 | unrecoverable_error Off Not Sent |
| 714 | predictive_error Off Not Sent |
| 715 | |
| 716 | |
| 717 | Verify PEL Log Offloaded To Host |
| 718 | [Documentation] Verify host transmission state of offloaded PEL logs. |
| 719 | [Tags] Verify_PEL_Log_Offloaded_To_Host |
| 720 | [Template] Verify PEL Transmission To Host |
| 721 | |
| 722 | # error_type host_state expected_transmission_state |
| 723 | unrecoverable_error On Acked |
| 724 | predictive_error On Acked |
| 725 | informational_error On Acked |
| 726 | |
| 727 | |
Rahul Maheshwari | faa5d20 | 2020-02-24 23:32:57 -0600 | [diff] [blame] | 728 | *** Keywords *** |
| 729 | |
manashsarma | 595282d | 2020-10-20 13:22:31 -0500 | [diff] [blame] | 730 | Error Logging Rotation Policy |
| 731 | [Documentation] Verify that when maximum log limit is reached, given error logging type |
| 732 | ... are deleted when reached their max allocated space. |
| 733 | [Arguments] ${error_log_type} ${max_allocated_space_percentage} |
| 734 | |
| 735 | # Description of argument(s): |
manashsarma | 557b632 | 2020-11-29 12:08:32 -0600 | [diff] [blame] | 736 | # error_log Error logs to be created (E.g. Informational BMC 3000 |
| 737 | # stands for BMC created 3000 informational error logs). |
manashsarma | 595282d | 2020-10-20 13:22:31 -0500 | [diff] [blame] | 738 | # max_allocated_space_percentage The maximum percentage of disk usage for given error |
| 739 | # log type when maximum count/log size is reached. |
| 740 | # The maximum error log count is 3000. |
| 741 | |
George Keishing | 6beaeab | 2022-08-29 09:34:42 -0500 | [diff] [blame] | 742 | Redfish.Login |
| 743 | |
manashsarma | 595282d | 2020-10-20 13:22:31 -0500 | [diff] [blame] | 744 | # Initially remove all logs. Purging is done to ensure that, only specific logs are present |
| 745 | # in BMC during the test. |
| 746 | Redfish Purge Event Log |
| 747 | |
manashsarma | 557b632 | 2020-11-29 12:08:32 -0600 | [diff] [blame] | 748 | @{lists}= Split String ${error_log_type} ,${SPACE} |
manashsarma | 595282d | 2020-10-20 13:22:31 -0500 | [diff] [blame] | 749 | |
manashsarma | 557b632 | 2020-11-29 12:08:32 -0600 | [diff] [blame] | 750 | ${length}= Get Length ${lists} |
manashsarma | 595282d | 2020-10-20 13:22:31 -0500 | [diff] [blame] | 751 | |
manashsarma | 557b632 | 2020-11-29 12:08:32 -0600 | [diff] [blame] | 752 | FOR ${list} IN RANGE ${length} |
| 753 | @{words}= Split String ${lists}[${list}] ${SPACE} |
| 754 | Create Error Log ${words}[0] ${words}[1] ${words}[2] |
manashsarma | 595282d | 2020-10-20 13:22:31 -0500 | [diff] [blame] | 755 | END |
| 756 | |
manashsarma | 557b632 | 2020-11-29 12:08:32 -0600 | [diff] [blame] | 757 | # Create an additional error log to exceed max error logs limit. |
| 758 | BMC Execute Command ${CMD_UNRECOVERABLE_ERROR} |
| 759 | |
manashsarma | 595282d | 2020-10-20 13:22:31 -0500 | [diff] [blame] | 760 | # Delay for BMC to perform delete older error logs when log limit exceeds. |
| 761 | Sleep 10s |
| 762 | |
| 763 | # Verify disk usage is around max allocated space. Maximum usage is around 3MB not exactly 3MB |
| 764 | # (for informational log) and around 6 MB for unrecoverable / predictive error log. So, usage |
| 765 | # percentage is NOT exactly 15% and 30%. So, an error/accuracy factor 0.5 percent is added. |
| 766 | |
| 767 | ${disk_usage_percentage}= Get Disk Usage For Error Logs |
| 768 | ${percent_diff}= Evaluate ${disk_usage_percentage} - ${max_allocated_space_percentage} |
| 769 | ${percent_diff}= Evaluate abs(${percent_diff}) |
manashsarma | 4baa6a7 | 2020-12-16 07:13:37 -0600 | [diff] [blame] | 770 | |
| 771 | ${trimmed_as_expected}= Run Keyword If ${disk_usage_percentage} > ${max_allocated_space_percentage} |
| 772 | ... Evaluate ${percent_diff} <= 0.5 |
| 773 | ... ELSE |
| 774 | ... Set Variable True |
| 775 | |
| 776 | # Check PEL log count via peltool command and compare it with actual generated log count. |
| 777 | ${pel_records}= peltool -n |
| 778 | ${no_pel_records}= Set Variable ${pel_records["Number of PELs found"]} |
| 779 | # Number of logs can be 80% of the total logs created after trimming. |
| 780 | ${expected_max_record}= Evaluate 3000 * 0.8 |
| 781 | |
George Keishing | b78bca2 | 2021-06-29 11:11:19 -0500 | [diff] [blame] | 782 | Run Keyword If ${trimmed_as_expected} == False |
| 783 | ... Should Be True ${no_pel_records} <= ${expected_max_record} |
manashsarma | 595282d | 2020-10-20 13:22:31 -0500 | [diff] [blame] | 784 | |
| 785 | |
manashsarma | 557b632 | 2020-11-29 12:08:32 -0600 | [diff] [blame] | 786 | Create Error Log |
George Keishing | f924895 | 2021-05-28 07:52:37 -0500 | [diff] [blame] | 787 | [Documentation] Create an error log. |
manashsarma | 557b632 | 2020-11-29 12:08:32 -0600 | [diff] [blame] | 788 | [Arguments] ${error_severity} ${error_creator} ${count} |
| 789 | |
| 790 | # Description of argument(s): |
| 791 | # error_severity Severity of the error (E.g. Informational, Unrecoberable or Predictive) |
| 792 | # error_creator Name of error log's creator(E.g BMC, Host Boot) |
| 793 | # count Number of error logs to be generated. |
| 794 | |
| 795 | FOR ${i} IN RANGE 0 ${count} |
| 796 | ${cmd}= Set Variable If |
Rahul Maheshwari | 5eab56c | 2021-01-06 23:58:46 -0600 | [diff] [blame] | 797 | ... '${error_severity}' == 'Informational' and '${error_creator}' == 'BMC' ${CMD_INFORMATIONAL_ERROR} |
Rahul Maheshwari | 952bad1 | 2022-10-19 01:56:01 -0500 | [diff] [blame] | 798 | ... '${error_severity}' == 'Informational' and '${error_creator}' == 'HOST' ${CMD_INFORMATIONAL_HOST_ERROR} |
Rahul Maheshwari | 5eab56c | 2021-01-06 23:58:46 -0600 | [diff] [blame] | 799 | ... '${error_severity}' == 'Predictive' and '${error_creator}' == 'BMC' ${CMD_PREDICTIVE_ERROR} |
| 800 | ... '${error_severity}' == 'Unrecoverable' and '${error_creator}' == 'BMC' ${CMD_UNRECOVERABLE_ERROR} |
| 801 | ... '${error_severity}' == 'Unrecoverable' and '${error_creator}' == 'HOST' ${CMD_UNRECOVERABLE_HOST_ERROR} |
manashsarma | 557b632 | 2020-11-29 12:08:32 -0600 | [diff] [blame] | 802 | BMC Execute Command ${cmd} |
| 803 | END |
| 804 | |
| 805 | |
manashsarma | d073a0a | 2020-10-20 12:54:59 -0500 | [diff] [blame] | 806 | Get Disk Usage For Error Logs |
| 807 | [Documentation] Get disk usage percentage for error logs. |
| 808 | |
manashsarma | 407deda | 2020-10-30 10:36:26 -0500 | [diff] [blame] | 809 | ${usage_output} ${stderr} ${rc}= BMC Execute Command du /var/lib/phosphor-logging/errors |
manashsarma | d073a0a | 2020-10-20 12:54:59 -0500 | [diff] [blame] | 810 | |
| 811 | ${usage_output}= Fetch From Left ${usage_output} \/ |
| 812 | |
George Keishing | c91aadd | 2022-02-23 09:30:48 -0600 | [diff] [blame] | 813 | # Convert disk usage unit from KB to MB. |
manashsarma | d073a0a | 2020-10-20 12:54:59 -0500 | [diff] [blame] | 814 | ${usage_output}= Evaluate ${usage_output} / 1024 |
| 815 | |
| 816 | # Logging disk capacity limit is set to 20MB. So calculating the log usage percentage. |
| 817 | ${usage_percent}= Evaluate ${usage_output} / 20 * 100 |
| 818 | |
| 819 | [return] ${usage_percent} |
| 820 | |
| 821 | |
Rahul Maheshwari | 8f5256f | 2020-02-26 23:53:55 -0600 | [diff] [blame] | 822 | Get PEL Log IDs |
| 823 | [Documentation] Returns the list of PEL log IDs which contains given field's value. |
| 824 | [Arguments] ${pel_section} ${pel_field} @{pel_field_value} |
| 825 | |
| 826 | # Description of argument(s): |
| 827 | # pel_section The section of PEL (e.g. Private Header, User Header). |
| 828 | # pel_field The PEL field (e.g. Event Severity, Event Type). |
| 829 | # pel_field_value The list of PEL's field value (e.g. Unrecoverable Error). |
| 830 | |
| 831 | ${pel_ids}= Get PEL Log Via BMC CLI |
| 832 | @{pel_id_list}= Create List |
| 833 | |
| 834 | FOR ${id} IN @{pel_ids} |
| 835 | ${pel_output}= Peltool -i ${id} |
| 836 | # Example of PEL output from "peltool -i <id>" command. |
| 837 | # [Private Header]: |
| 838 | # [Created at]: 08/24/1928 12:04:06 |
| 839 | # [Created by]: 0x584D |
| 840 | # [Sub-section type]: 0 |
| 841 | # [Entry Id]: 0x50000BB7 |
| 842 | # [Platform Log Id]: 0x8200061D |
| 843 | # [CSSVER]: |
| 844 | # [Section Version]: 1 |
| 845 | # [Creator Subsystem]: PHYP |
| 846 | # [BMC Event Log Id]: 341 |
| 847 | # [Committed at]: 03/25/1920 12:06:22 |
| 848 | # [User Header]: |
| 849 | # [Log Committed by]: 0x4552 |
| 850 | # [Action Flags]: |
| 851 | # [0]: Report Externally |
| 852 | # [Subsystem]: I/O Subsystem |
| 853 | # [Event Type]: Miscellaneous, Informational Only |
| 854 | # [Sub-section type]: 0 |
| 855 | # [Event Scope]: Entire Platform |
| 856 | # [Event Severity]: Informational Event |
| 857 | # [Host Transmission]: Not Sent |
| 858 | # [Section Version]: 1 |
| 859 | |
| 860 | ${pel_section_output}= Get From Dictionary ${pel_output} ${pel_section} |
| 861 | ${pel_field_output}= Get From Dictionary ${pel_section_output} ${pel_field} |
| 862 | Run Keyword If '${pel_field_output}' in @{pel_field_value} Append To List ${pel_id_list} ${id} |
| 863 | END |
| 864 | Sort List ${pel_id_list} |
| 865 | |
| 866 | [Return] ${pel_id_list} |
| 867 | |
| 868 | |
Rahul Maheshwari | f575c90 | 2020-02-26 04:38:40 -0600 | [diff] [blame] | 869 | Get PEL Log Via BMC CLI |
| 870 | [Documentation] Returns the list of PEL IDs using BMC CLI. |
Rahul Maheshwari | faa5d20 | 2020-02-24 23:32:57 -0600 | [diff] [blame] | 871 | |
| 872 | ${pel_records}= Peltool -l |
Rahul Maheshwari | f575c90 | 2020-02-26 04:38:40 -0600 | [diff] [blame] | 873 | ${ids}= Get Dictionary Keys ${pel_records} |
Rahul Maheshwari | 1cffc4b | 2020-04-02 02:58:16 -0500 | [diff] [blame] | 874 | Sort List ${ids} |
Rahul Maheshwari | faa5d20 | 2020-02-24 23:32:57 -0600 | [diff] [blame] | 875 | |
Rahul Maheshwari | f575c90 | 2020-02-26 04:38:40 -0600 | [diff] [blame] | 876 | [Return] ${ids} |
Rahul Maheshwari | 60b58a8 | 2020-04-02 02:40:19 -0500 | [diff] [blame] | 877 | |
| 878 | |
| 879 | Get PEL Field Value |
| 880 | [Documentation] Returns the value of given PEL's field. |
| 881 | [Arguments] ${pel_id} ${pel_section} ${pel_field} |
| 882 | |
| 883 | # Description of argument(s): |
| 884 | # pel_id The ID of PEL (e.g. 0x5000002D, 0x5000002E). |
| 885 | # pel_section The section of PEL (e.g. Private Header, User Header) |
| 886 | # pel_field The PEL field (e.g. Event Severity, Event Type). |
| 887 | |
| 888 | ${pel_output}= Peltool -i ${pel_id} |
| 889 | |
| 890 | # Example of PEL output from "peltool -i <id>" command. |
| 891 | # [Private Header]: |
| 892 | # [Created at]: 08/24/1928 12:04:06 |
| 893 | # [Created by]: 0x584D |
| 894 | # [Sub-section type]: 0 |
| 895 | # [Entry Id]: 0x50000BB7 |
| 896 | # [Platform Log Id]: 0x8200061D |
| 897 | # [CSSVER]: |
| 898 | # [Section Version]: 1 |
| 899 | # [Creator Subsystem]: PHYP |
| 900 | # [BMC Event Log Id]: 341 |
| 901 | # [Committed at]: 03/25/1920 12:06:22 |
| 902 | # [User Header]: |
| 903 | # [Log Committed by]: 0x4552 |
| 904 | # [Action Flags]: |
| 905 | # [0]: Report Externally |
| 906 | # [Subsystem]: I/O Subsystem |
| 907 | # [Event Type]: Miscellaneous, Informational Only |
| 908 | # [Sub-section type]: 0 |
| 909 | # [Event Scope]: Entire Platform |
| 910 | # [Event Severity]: Informational Event |
| 911 | # [Host Transmission]: Not Sent |
| 912 | # [Section Version]: 1 |
| 913 | |
| 914 | ${pel_section_output}= Get From Dictionary ${pel_output} ${pel_section} |
| 915 | ${pel_field_output}= Get From Dictionary ${pel_section_output} ${pel_field} |
| 916 | |
| 917 | [Return] ${pel_field_output} |
dnirmala | 789c802 | 2022-11-18 05:49:08 -0600 | [diff] [blame] | 918 | |
| 919 | |
| 920 | Verify PEL Transmission To Host |
| 921 | [Documentation] Inject PEL log of given type and verify its host transmission state. |
| 922 | [Arguments] ${error_type} ${host_state} ${expected_transmission_state} |
| 923 | |
| 924 | # Description of argument(s): |
| 925 | # error_type Type of error log to be injected. |
| 926 | # host_state Host state which is required before error log injection. |
| 927 | # expected_transmission_state Expected host transmission state of PEL log. |
| 928 | |
| 929 | # Get system in required state. |
| 930 | Run Keyword If '${host_state}' == 'Off' |
| 931 | ... Redfish Power Off stack_mode=skip |
| 932 | ... ELSE IF '${host_state}' == 'On' |
| 933 | ... Redfish Power On stack_mode=skip |
| 934 | |
| 935 | # Inject required error log. |
| 936 | Run Keyword If "${error_type}" == "informational_error" |
| 937 | ... BMC Execute Command ${CMD_INFORMATIONAL_ERROR} |
| 938 | ... ELSE IF "${error_type}" == "unrecoverable_error" |
| 939 | ... BMC Execute Command ${CMD_UNRECOVERABLE_ERROR} |
| 940 | ... ELSE IF "${error_type}" == "predictive_error" |
| 941 | ... BMC Execute Command ${CMD_PREDICTIVE_ERROR} |
| 942 | |
| 943 | # Check host transmission state for the cases where PEL is |
| 944 | # expected to be offloaded to HOST. |
| 945 | Run Keyword If "${expected_transmission_state}" == "Acked" |
| 946 | ... Wait Until Keyword Succeeds 2 min 10 sec |
| 947 | ... Check If PEL Transmission State Is Expected Acked |
| 948 | |
| 949 | # Adding delay before checking host transmission for the cases where PEL is |
| 950 | # not expected to be offloaded to HOST. |
| 951 | Run Keyword If "${expected_transmission_state}" == "Not sent" |
| 952 | ... Run Keywords Sleep 120s AND |
| 953 | ... Check If PEL Transmission State Is Expected Not sent |
| 954 | |
| 955 | |
| 956 | Check If PEL Transmission State Is Expected |
| 957 | [Documentation] Check if PEL's host transmission state is matching expected state. |
| 958 | [Arguments] ${expected_transmission_state} |
| 959 | |
| 960 | # Description of argument(s): |
| 961 | # expected_transmission_state Expected transmission state of PEL log. |
| 962 | |
| 963 | ${pel_records}= Get Pel Data From Bmc True True |
| 964 | ${ids}= Get Dictionary Keys ${pel_records} |
| 965 | ${pel_id}= Get From List ${ids} -1 |
| 966 | |
| 967 | # Check the transmission details in the error log. |
| 968 | ${host_transmission}= Get PEL Field Value ${pel_id} User Header Host Transmission |
| 969 | Should Be Equal ${host_transmission} ${expected_transmission_state} |