| *** Settings *** |
| Documentation This suite tests Platform Event Log (PEL) functionality of OpenBMC. |
| |
| Library ../../lib/pel_utils.py |
| Variables ../../data/pel_variables.py |
| Resource ../../lib/list_utils.robot |
| Resource ../../lib/logging_utils.robot |
| Resource ../../lib/connection_client.robot |
| Resource ../../lib/openbmc_ffdc.robot |
| |
| Test Setup Redfish.Login |
| Test Teardown Run Keywords Redfish.Logout AND FFDC On Test Case Fail |
| |
| Force Tags Bmc_Pel |
| |
| *** Variables *** |
| |
| @{mandatory_pel_fields} Private Header User Header Primary SRC |
| ... Extended User Header Failing MTMS |
| @{mandatory_Predictive_pel_fields} Private Header User Header Primary SRC |
| ... Extended User Header Failing MTMS User Data |
| @{Mandatory_Informational_Pel_Fields} Private Header User Header Primary SRC |
| ... Extended User Header User Data |
| |
| *** Test Cases *** |
| |
| Create Test PEL Log And Verify |
| [Documentation] Create PEL log using busctl command and verify via peltool. |
| [Tags] Create_Test_PEL_Log_And_Verify |
| |
| Redfish Purge Event Log |
| Create Test PEL Log |
| ${pel_id}= Get PEL Log Via BMC CLI |
| Should Not Be Empty ${pel_id} msg=System PEL log entry is empty. |
| |
| |
| Verify PEL Log Details |
| [Documentation] Verify PEL log details via peltool. |
| [Tags] Verify_PEL_Log_Details |
| |
| Redfish Purge Event Log |
| |
| ${bmc_time1}= CLI Get BMC DateTime |
| Create Test PEL Log |
| ${bmc_time2}= CLI Get BMC DateTime |
| |
| ${pel_records}= Peltool -l |
| |
| # Example output from 'Peltool -l': |
| # pel_records: |
| # [0x50000012]: |
| # [CreatorID]: BMC |
| # [CompID]: 0x1000 |
| # [PLID]: 0x50000012 |
| # [Subsystem]: BMC Firmware |
| # [Message]: An application had an internal failure |
| # [SRC]: BD8D1002 |
| # [Commit Time]: 03/02/2020 09:35:15 |
| # [Sev]: Unrecoverable Error |
| |
| ${ids}= Get Dictionary Keys ${pel_records} |
| ${id}= Get From List ${ids} 0 |
| |
| @{pel_fields}= Create List CreatorID Subsystem Message Sev |
| FOR ${field} IN @{pel_fields} |
| Valid Value pel_records['${id}']['${field}'] ['${PEL_DETAILS['${field}']}'] |
| END |
| |
| Valid Value pel_records['${id}']['PLID'] ['${id}'] |
| |
| # Verify if "CompID" and "SRC" fields of PEL has alphanumeric value. |
| Should Match Regexp ${pel_records['${id}']['CompID']} [a-zA-Z0-9] |
| Should Match Regexp ${pel_records['${id}']['SRC']} [a-zA-Z0-9] |
| |
| ${pel_date_time}= Convert Date ${pel_records['${id}']['Commit Time']} |
| ... date_format=%m/%d/%Y %H:%M:%S exclude_millis=yes |
| |
| # Convert BMC and PEL time to epoch time before comparing. |
| ${bmc_time1_epoch}= Convert Date ${bmc_time1} epoch |
| ${pel_time_epoch}= Convert Date ${pel_date_time} epoch |
| ${bmc_time2_epoch}= Convert Date ${bmc_time2} epoch |
| |
| Should Be True ${bmc_time1_epoch} <= ${pel_time_epoch} <= ${bmc_time2_epoch} |
| |
| |
| Verify Mandatory Sections Of Error Log PEL |
| [Documentation] Verify mandatory sections of error log PEL. |
| [Tags] Verify_Mandatory_Sections_Of_Error_Log_PEL |
| |
| Create Test PEL Log |
| |
| ${pel_ids}= Get PEL Log Via BMC CLI |
| ${pel_id}= Get From List ${pel_ids} -1 |
| ${pel_output}= Peltool -i ${pel_id} |
| ${pel_sections}= Get Dictionary Keys ${pel_output} |
| |
| List Should Contain Sub List ${pel_sections} ${mandatory_pel_fields} |
| |
| |
| Verify PEL Log Persistence After BMC Reboot |
| [Documentation] Verify PEL log persistence after BMC reboot. |
| [Tags] Verify_PEL_Log_Persistence_After_BMC_Reboot |
| |
| Create Test PEL Log |
| ${pel_before_reboot}= Get PEL Log Via BMC CLI |
| |
| Redfish OBMC Reboot (off) |
| ${pel_after_reboot}= Get PEL Log Via BMC CLI |
| |
| List Should Contain Sub List ${pel_after_reboot} ${pel_before_reboot} |
| |
| |
| Verify PEL ID Numbering |
| [Documentation] Verify PEL ID numbering. |
| [Tags] Verify_PEL_ID_Numbering |
| |
| Redfish Purge Event Log |
| Create Test PEL Log |
| Create Test PEL Log |
| |
| ${pel_ids}= Get PEL Log Via BMC CLI |
| |
| # Example of PEL IDs from PEL logs. |
| # [0x50000012]: <--- First PEL ID |
| # [CreatorID]: BMC |
| # [CompID]: 0x1000 |
| # [PLID]: 0x50000012 |
| # [Subsystem]: BMC Firmware |
| # [Message]: An application had an internal failure |
| # [SRC]: BD8D1002 |
| # [Commit Time]: 03/02/2020 09:35:15 |
| # [Sev]: Unrecoverable Error |
| # |
| # [0x50000013]: <--- Second PEL ID |
| # [CreatorID]: BMC |
| # [CompID]: 0x1000 |
| # [PLID]: 0x50000013 |
| # [Subsystem]: BMC Firmware |
| # [Message]: An application had an internal failure |
| # [SRC]: BD8D1002 |
| # [Commit Time]: 03/02/2020 09:35:15 |
| # [Sev]: Unrecoverable Error |
| |
| Should Be True ${pel_ids[1]} == ${pel_ids[0]}+1 |
| |
| Verify Machine Type Model And Serial Number |
| [Documentation] Verify machine type model and serial number from PEL. |
| [Tags] Verify_Machine_Type_Model_And_Serial_Number |
| |
| Create Test PEL Log |
| |
| ${pel_ids}= Get PEL Log Via BMC CLI |
| ${id}= Get From List ${pel_ids} -1 |
| |
| ${pel_serial_number}= Get PEL Field Value ${id} Failing MTMS Serial Number |
| ${pel_serial_number}= Replace String Using Regexp ${pel_serial_number} ^0+ ${EMPTY} |
| ${pel_machine_type_model}= Get PEL Field Value ${id} Failing MTMS Machine Type Model |
| ${pel_machine_type_model}= Replace String Using Regexp ${pel_machine_type_model} ^0+ ${EMPTY} |
| |
| # Example of "Machine Type Model" and "Serial Number" fields value from "Failing MTMS" section of PEL. |
| # [Failing MTMS]: |
| # [Created by]: 0x2000 |
| # [Machine Type Model]: 1234-ABC <---- Machine type |
| # [Section Version]: 1 |
| # [Serial Number]: ABCDEFG <---- Serial number |
| # [Sub-section type]: 0 |
| |
| ${redfish_machine_model}= Redfish.Get Attribute /redfish/v1/Systems/${SYSTEM_ID}/ Model |
| ${redfish_machine_model}= Replace String Using Regexp ${redfish_machine_model} ^0+ ${EMPTY} |
| ${redfish_serial_number}= Redfish.Get Attribute /redfish/v1/Systems/${SYSTEM_ID}/ SerialNumber |
| ${redfish_serial_number}= Replace String Using Regexp ${redfish_serial_number} ^0+ ${EMPTY} |
| |
| Valid Value pel_machine_type_model ['${redfish_machine_model}'] |
| Valid Value pel_serial_number ['${redfish_serial_number}'] |
| |
| # Check "Machine Type Model" and "Serial Number" fields value from "Extended User Header" section of PEL. |
| ${pel_machine_type_model}= Get PEL Field Value ${id} Extended User Header Reporting Machine Type |
| ${pel_machine_type_model}= Replace String Using Regexp ${pel_machine_type_model} ^0+ ${EMPTY} |
| ${pel_serial_number}= Get PEL Field Value ${id} Extended User Header Reporting Serial Number |
| ${pel_serial_number}= Replace String Using Regexp ${pel_serial_number} ^0+ ${EMPTY} |
| |
| Valid Value pel_machine_type_model ['${redfish_machine_model}'] |
| Valid Value pel_serial_number ['${redfish_serial_number}'] |
| |
| |
| Verify Host Off State From PEL |
| [Documentation] Verify Host off state from PEL. |
| [Tags] Verify_Host_Off_State_From_PEL |
| |
| Redfish Power Off stack_mode=skip |
| Create Test PEL Log |
| |
| ${pel_ids}= Get PEL Log Via BMC CLI |
| ${id}= Get From List ${pel_ids} -1 |
| ${pel_host_state}= Get PEL Field Value ${id} User Data HostState |
| |
| Valid Value pel_host_state ['Off'] |
| |
| |
| Verify Host On State From PEL |
| [Documentation] Verify Host on state from PEL. |
| [Tags] Verify_Host_On_State_From_PEL |
| |
| Redfish Power On stack_mode=skip |
| Redfish Purge Event Log |
| |
| Create Test PEL Log |
| |
| ${pel_ids}= Get PEL Log Via BMC CLI |
| ${id}= Get From List ${pel_ids} -1 |
| ${pel_host_state}= Get PEL Field Value ${id} User Data HostState |
| |
| Valid Value pel_host_state ['Running'] |
| |
| |
| Verify BMC Version From PEL |
| [Documentation] Verify BMC Version from PEL. |
| [Tags] Verify_BMC_Version_From_PEL |
| |
| Create Test PEL Log |
| |
| ${pel_ids}= Get PEL Log Via BMC CLI |
| ${id}= Get From List ${pel_ids} -1 |
| ${pel_bmc_version}= Get PEL Field Value ${id} User Data FW Version ID |
| |
| ${bmc_version}= Get BMC Version |
| Valid Value bmc_version ['${bmc_version}'] |
| |
| |
| Verify PEL Log After Host Poweron |
| [Documentation] Verify PEL log generation while booting host. |
| [Tags] Verify_PEL_Log_After_Host_Poweron |
| |
| Redfish Power Off stack_mode=skip |
| Redfish Purge Event Log |
| Redfish Power On stack_mode=skip |
| |
| ${pel_informational_error}= Get PEL Log IDs User Header Event Severity Informational Event |
| ${pel_bmc_created_error}= Get PEL Log IDs Private Header Creator Subsystem BMC |
| |
| # Get BMC created non-infomational error. |
| ${pel_bmc_error}= Subtract Lists ${pel_bmc_created_error} ${pel_informational_error} |
| |
| Should Be Empty ${pel_bmc_error} msg=Unexpected error log generated during Host poweron. |
| |
| |
| Verify BMC Event Log ID |
| [Documentation] Verify BMC Event Log ID from PEL. |
| [Tags] Verify_BMC_Event_Log_ID |
| |
| Redfish Purge Event Log |
| Create Test PEL Log |
| |
| ${pel_ids}= Get PEL Log Via BMC CLI |
| ${pel_bmc_event_log_id}= Get PEL Field Value ${pel_ids[0]} Private Header BMC Event Log Id |
| |
| # Example "BMC Event Log Id" field value from "Private Header" section of PEL. |
| # [Private Header]: |
| # [Created at]: 08/24/1928 12:04:06 |
| # [Created by]: 0x584D |
| # [Sub-section type]: 0 |
| # [Entry Id]: 0x50000BB7 |
| # [Platform Log Id]: 0x8200061D |
| # [CSSVER]: |
| # [Section Version]: 1 |
| # [Creator Subsystem]: PHYP |
| # [BMC Event Log Id]: 341 <---- BMC event log id value |
| # [Committed at]: 03/25/1920 12:06:22 |
| |
| ${redfish_event_logs}= Redfish.Get Properties /redfish/v1/Systems/${SYSTEM_ID}/LogServices/EventLog/Entries |
| |
| # Example of redfish_event_logs output: |
| # redfish_event_logs: |
| # [@odata.id]: /redfish/v1/Systems/system/LogServices/EventLog/Entries |
| # [Name]: System Event Log Entries |
| # [Members@odata.count]: 1 |
| # [@odata.type]: #LogEntryCollection.LogEntryCollection |
| # [Description]: Collection of System Event Log Entries |
| # [Members]: |
| # [0]: |
| # [@odata.id]: /redfish/v1/Systems/system/LogServices/EventLog/Entries/235 |
| # [Name]: System Event Log Entry |
| # [Severity]: Critical |
| # [EntryType]: Event |
| # [Created]: 2020-04-02T07:25:13+00:00 |
| # [@odata.type]: #LogEntry.v1_5_1.LogEntry |
| # [Id]: 235 <----- Event log ID |
| # [Message]: xyz.openbmc_project.Common.Error.InternalFailure |
| |
| Valid Value pel_bmc_event_log_id ['${redfish_event_logs['Members'][0]['Id']}'] |
| |
| |
| Verify FRU Callout |
| [Documentation] Verify FRU callout entries from PEL log. |
| [Tags] Verify_FRU_Callout |
| |
| # Power off the system to avoid unnecessary error logs during test. |
| Redfish Power Off stack_mode=skip |
| Create Test PEL Log FRU Callout |
| |
| ${pel_ids}= Get PEL Log Via BMC CLI |
| ${id}= Get From List ${pel_ids} -1 |
| ${pel_callout_section}= Get PEL Field Value ${id} Primary SRC Callout Section |
| |
| # Example of PEL Callout Section from "peltool -i <id>" command. |
| # [Callouts]: |
| # [0]: |
| # [FRU Type]: Normal Hardware FRU |
| # [Priority]: Mandatory, replace all with this type as a unit |
| # [Location Code]: U78DA.ND1.1234567-P0 |
| # [Part Number]: F191014 |
| # [CCIN]: 2E2D |
| # [Serial Number]: YL2E2D010000 |
| # [Callout Count]: 1 |
| |
| Valid Value pel_callout_section['Callout Count'] ['1'] |
| Valid Value pel_callout_section['Callouts'][0]['FRU Type'] ['Normal Hardware FRU'] |
| Should Contain ${pel_callout_section['Callouts'][0]['Priority']} Mandatory |
| |
| # Verify Location Code field of PEL callout with motherboard's Location Code. |
| ${busctl_output}= BMC Execute Command ${CMD_INVENTORY_PREFIX} com.ibm.ipzvpd.Location LocationCode |
| Should Be Equal ${pel_callout_section['Callouts'][0]['Location Code']} |
| ... ${busctl_output[0].split('"')[1].strip('"')} |
| |
| # TODO: Compare CCIN and part number fields of PEL callout with Redfish or busctl output. |
| Should Match Regexp ${pel_callout_section['Callouts'][0]['CCIN']} [a-zA-Z0-9] |
| Should Match Regexp ${pel_callout_section['Callouts'][0]['Part Number']} [a-zA-Z0-9] |
| |
| # Verify Serial Number field of PEL callout with motherboard's Serial Number. |
| ${busctl_output}= BMC Execute Command |
| ... ${CMD_INVENTORY_PREFIX} xyz.openbmc_project.Inventory.Decorator.Asset SerialNumber |
| Should Be Equal ${pel_callout_section['Callouts'][0]['Serial Number']} |
| ... ${busctl_output[0].split('"')[1].strip('"')} |
| |
| |
| Verify Procedure And Symbolic FRU Callout |
| [Documentation] Verify procedure and symbolic FRU callout from PEL log. |
| [Tags] Verify_Procedure_And_Symbolic_FRU_Callout |
| |
| Create Test PEL Log Procedure And Symbolic FRU Callout |
| |
| ${pel_ids}= Get PEL Log Via BMC CLI |
| ${id}= Get From List ${pel_ids} -1 |
| ${pel_callout_section}= Get PEL Field Value ${id} Primary SRC Callout Section |
| |
| # Example of PEL Callout Section from "peltool -i <id>" command. |
| # [Callouts]: |
| # [0]: |
| # [Priority]: Mandatory, replace all with this type as a unit |
| # [Procedure Number]: BMCSP02 |
| # [FRU Type]: Maintenance Procedure Required |
| # [1]: |
| # [Priority]: Medium Priority |
| # [Part Number]: SVCDOCS |
| # [FRU Type]: Symbolic FRU |
| # [Callout Count]: 2 |
| |
| Valid Value pel_callout_section['Callout Count'] ['2'] |
| |
| # Verify procedural callout info. |
| |
| Valid Value pel_callout_section['Callouts'][0]['FRU Type'] ['Maintenance Procedure Required'] |
| Should Contain ${pel_callout_section['Callouts'][0]['Priority']} Mandatory |
| # Verify if "Procedure Number" field of PEL has an alphanumeric value. |
| Should Match Regexp ${pel_callout_section['Callouts'][0]['Procedure']} [a-zA-Z0-9] |
| |
| # Verify procedural callout info. |
| |
| Valid Value pel_callout_section['Callouts'][1]['FRU Type'] ['Symbolic FRU'] |
| Should Contain ${pel_callout_section['Callouts'][1]['Priority']} Medium Priority |
| # Verify if "Part Number" field of Symbolic FRU has an alphanumeric value. |
| Should Match Regexp ${pel_callout_section['Callouts'][1]['Part Number']} [a-zA-Z0-9] |
| |
| |
| Verify PEL Log Entry For Event Log |
| [Documentation] Create an event log and verify PEL log entry in BMC for the same. |
| [Tags] Verify_PEL_Log_Entry_For_Event_Log |
| |
| Redfish Purge Event Log |
| |
| # Create an unrecoverable error log. |
| Create Test PEL Log Unrecoverable Error |
| |
| ${elog_entry}= Get Event Logs |
| # Example of Redfish event logs: |
| # elog_entry: |
| # [0]: |
| # [Message]: BD802003 event in subsystem: Platform Firmware |
| # [Created]: 2023-05-08T10:19:30.412+00:00 |
| # [Id]: 1 |
| # [@odata.id]: /redfish/v1/Systems/system/LogServices/EventLog/Entries/1 |
| # [@odata.type]: #LogEntry.v1_4_0.LogEntry |
| # [EntryType]: Event |
| # [Severity]: Critical |
| # [Name]: System Event Log Entry |
| |
| ${redfish_log_time}= Convert Date ${elog_entry[0]["Created"].split('.')[0]} epoch |
| |
| ${pel_records}= Peltool -l |
| # Example output from 'Peltool -l': |
| # pel_records: |
| # [0x50000023]: |
| # [SRC]: BD802003 |
| # [CreatorID]: BMC |
| # [Message]: This is a test error |
| # [CompID]: 0x2000 |
| # [PLID]: 0x500053D9 |
| # [Commit Time]: 04/20/2020 01:55:22 |
| # [Subsystem]: Platform Firmware |
| # [Sev]: Unrecoverable Error |
| |
| ${ids}= Get Dictionary Keys ${pel_records} |
| ${id}= Get From List ${ids} 0 |
| ${pel_log_time}= Convert Date ${pel_records['${id}']['Commit Time']} epoch |
| ... date_format=%m/%d/%Y %H:%M:%S |
| |
| # Verify that both Redfish event and PEL has log entry for internal error with same time stamp. |
| Should Contain Any ${pel_records['${id}']['Message']} test error ignore_case=True |
| Should Contain ${elog_entry[0]['Message']} |
| ... ${pel_records['${id}']['SRC']} ignore_case=True |
| |
| Should Be Equal ${redfish_log_time} ${pel_log_time} |
| |
| |
| Verify Delete All PEL |
| [Documentation] Verify deleting all PEL logs. |
| [Tags] Verify_Delete_All_PEL |
| |
| Create Test PEL Log |
| Create Test PEL Log |
| Peltool --delete-all False |
| |
| ${pel_ids}= Get PEL Log Via BMC CLI |
| Should Be Empty ${pel_ids} |
| |
| |
| Verify Informational Error Log |
| [Documentation] Create an informational error log and verify. |
| [Tags] Verify_Informational_Error_Log |
| |
| Redfish Purge Event Log |
| # Create an informational error log. |
| BMC Execute Command ${CMD_INFORMATIONAL_ERROR} |
| ${pel_records}= Peltool -lfh |
| |
| # An example of information error log data: |
| # { |
| # "0x500006A0": { |
| # "SRC": "BD8D1002", |
| # "Message": "An application had an internal failure", |
| # "PLID": "0x500006A0", |
| # "CreatorID": "BMC", |
| # "Subsystem": "BMC Firmware", |
| # "Commit Time": "10/14/2020 11:41:38", |
| # "Sev": "Informational Event", |
| # "CompID": "0x1000" |
| # } |
| # } |
| |
| ${ids}= Get Dictionary Keys ${pel_records} |
| ${id}= Get From List ${ids} 0 |
| Should Contain ${pel_records['${id}']['Sev']} Informational |
| |
| |
| Verify Predictable Error Log |
| [Documentation] Create a predictive error and verify. |
| [Tags] Verify_Predictable_Error_Log |
| |
| # Create a predictable error log. |
| BMC Execute Command ${CMD_PREDICTIVE_ERROR} |
| ${pel_records}= Peltool -l |
| |
| # An example of predictive error log data: |
| # { |
| # "0x5000069E": { |
| # "SRC": "BD8D1002", |
| # "Message": "An application had an internal failure", |
| # "PLID": "0x5000069E", |
| # "CreatorID": "BMC", |
| # "Subsystem": "BMC Firmware", |
| # "Commit Time": "10/14/2020 11:40:07", |
| # "Sev": "Predictive Error", |
| # "CompID": "0x1000" |
| # } |
| # } |
| |
| ${pel_ids}= Get PEL Log Via BMC CLI |
| ${id}= Get From List ${pel_ids} -1 |
| Should Contain ${pel_records['${id}']['Sev']} Predictive |
| |
| |
| Verify Unrecoverable Error Log |
| [Documentation] Create an unrecoverable error and verify. |
| [Tags] Verify_Unrecoverable_Error_Log |
| |
| # Create an internal failure error log. |
| BMC Execute Command ${CMD_UNRECOVERABLE_ERROR} |
| ${pel_records}= Peltool -l |
| |
| # An example of unrecoverable error log data: |
| # { |
| # "0x50000CC5": { |
| # "SRC": "BD8D1002", |
| # "Message": "An application had an internal failure", |
| # "PLID": "0x50000CC5", |
| # "CreatorID": "BMC", |
| # "Subsystem": "BMC Firmware", |
| # "Commit Time": "04/01/2020 16:44:55", |
| # "Sev": "Unrecoverable Error", |
| # "CompID": "0x1000" |
| # } |
| # } |
| |
| ${pel_ids}= Get PEL Log Via BMC CLI |
| ${id}= Get From List ${pel_ids} -1 |
| Should Contain ${pel_records['${id}']['Sev']} Unrecoverable |
| |
| |
| Verify Error Logging Rotation Policy |
| [Documentation] Verify error logging rotation policy. |
| [Tags] Verify_Error_Logging_Rotation_Policy |
| [Template] Error Logging Rotation Policy |
| |
| # Error logs to be created % of total logging space when error |
| # log exceeds max limit. |
| Informational BMC 3000 15 |
| Predictive BMC 3000 30 |
| Unrecoverable BMC 3000 30 |
| Informational BMC 1500, Predictive BMC 1500 45 |
| Informational BMC 1500, Unrecoverable BMC 1500 45 |
| Unrecoverable BMC 1500, Predictive BMC 1500 30 |
| |
| |
| Verify Error Logging Rotation Policy With All Types Of Errors |
| [Documentation] Verify error logging rotation policy with all types of errors. |
| [Tags] Verify_Error_Logging_Rotation_Policy_With_All_Types_Of_Errors |
| [Template] Error Logging Rotation Policy |
| |
| # Error logs to be created % of total logging space when error |
| # log exceeds max limit. |
| Unrecoverable BMC 1000, Informational BMC 1000, Predictive BMC 1000 45 |
| |
| |
| Verify Error Logging Rotation Policy With HOST Error Logs |
| [Documentation] Verify error logging rotation policy for non bmc error logs. |
| [Tags] Verify_Error_Logging_Rotation_Policy_With_HOST_Error_Logs |
| [Setup] Run Keywords Open Connection for SCP AND scp.Put File ${UNRECOVERABLE_FILE_PATH} |
| ... /tmp/FILE_HOST_UNRECOVERABLE AND scp.Put File ${INFORMATIONAL_FILE_PATH} |
| ... /tmp/FILE_HOST_INFORMATIONAL |
| [Template] Error Logging Rotation Policy |
| |
| # Error logs to be created % of total logging space when error |
| # log exceeds max limit. |
| Informational HOST 3000 15 |
| Unrecoverable HOST 3000 30 |
| Informational HOST 1500, Informational BMC 1500 30 |
| Informational HOST 1500, Unrecoverable BMC 1500 45 |
| Unrecoverable HOST 1500, Informational BMC 1500 45 |
| Unrecoverable HOST 1500, Predictive BMC 1500 60 |
| |
| |
| Verify Error Logging Rotation Policy With Unrecoverable HOST And BMC Error Logs |
| [Documentation] Verify error logging rotation policy with unrecoverable HOST and BMC error logs. |
| [Tags] Verify_Error_Logging_Rotation_Policy_With_Unrecoverable_HOST_And_BMC_Error_Logs |
| [Setup] Run Keywords Open Connection for SCP AND scp.Put File ${UNRECOVERABLE_FILE_PATH} |
| ... /tmp/FILE_NBMC_UNRECOVERABLE AND Redfish.Login |
| [Template] Error Logging Rotation Policy |
| |
| # Error logs to be created % of total logging space when error |
| # log exceeds max limit. |
| Unrecoverable HOST 1500, Unrecoverable BMC 1500 60 |
| |
| |
| Verify Old Logs Are Deleted When Count Crosses Max |
| [Documentation] Verify that when the count crosses max, older logs are deleted. |
| [Tags] Verify_Old_Logs_Are_Deleted_When_Count_Crosses_Max |
| |
| Redfish Purge Event Log |
| # Create 3000 error logs. |
| FOR ${count} IN RANGE ${3000} |
| BMC Execute Command ${CMD_PREDICTIVE_ERROR} |
| END |
| |
| # Retrieve the IDs of the logs. |
| ${pel_ids}= Get PEL Log Via BMC CLI |
| ${1st_id}= Get From List ${pel_ids} 0 |
| ${3000th_id}= Get From List ${pel_ids} 2999 |
| |
| # Now create 3001st log to cross threshold limit and trigger error logs rotation. |
| BMC Execute Command ${CMD_PREDICTIVE_ERROR} |
| |
| # Wait few seconds for error logs rotation to complete. |
| Sleep 10s |
| |
| # Now verify that first log is no more available but the 3000th is available. |
| ${status} ${output}= Run Keyword And Ignore Error Peltool -i ${1st_id} |
| Should Be True '${status}' == 'FAIL' |
| Should Contain ${output} PEL not found |
| |
| ${status} ${output}= Run Keyword And Ignore Error Peltool -i ${3000th_id} |
| Should Be True '${status}' == 'PASS' |
| Should Not Contain ${output} PEL not found |
| |
| |
| Verify Reverse Order Of PEL Logs |
| [Documentation] Verify PEL command to output PEL logs in reverse order. |
| [Tags] Verify_Reverse_Order_Of_PEL_Logs |
| |
| Redfish Purge Event Log |
| |
| # Below commands create unrecoverable error log at first and then the predictable error. |
| BMC Execute Command ${CMD_UNRECOVERABLE_ERROR} |
| BMC Execute Command ${CMD_PREDICTIVE_ERROR} |
| |
| # Using peltool -lr, recent PELs appear first. Hence the ID of first PEL is greater than the next. |
| ${pel_records}= peltool -lr |
| |
| # It is found that, variables like dictionary always keep items in sorted order that makes |
| # this verification not possible, hence json is used to keep the items original order. |
| ${pel_records}= Convert To String ${pel_records} |
| ${json_string}= Replace String ${pel_records} ' " |
| ${json_object}= Evaluate json.loads('''${json_string}''') json |
| |
| ${list}= Convert To List ${json_object} |
| |
| ${id1}= Get From List ${list} 0 |
| ${id1}= Convert To Integer ${id1} |
| ${id2}= Get From List ${list} 1 |
| ${id2}= Convert To Integer ${id2} |
| |
| Should Be True ${id1} > ${id2} |
| |
| |
| Verify Total PEL Count |
| [Documentation] Verify total PEL count returned by peltool command. |
| [Tags] Verify_Total_PEL_Count |
| |
| # Initially remove all logs. |
| Redfish Purge Event Log |
| |
| # Generate a random number between 1-20. |
| ${random}= Evaluate random.randint(1, 20) modules=random |
| |
| # Generate predictive error log multiple times. |
| FOR ${count} IN RANGE 0 ${random} |
| BMC Execute Command ${CMD_PREDICTIVE_ERROR} |
| END |
| |
| # Check PEL log count via peltool command and compare it with actual generated log count. |
| ${pel_records}= peltool -n |
| |
| Should Be Equal ${pel_records['Number of PELs found']} ${random} |
| |
| |
| Verify Listing Information Error |
| [Documentation] Verify that information error logs can only be listed using -lfh option of peltool. |
| [Tags] Verify_Listing_Information_Error |
| |
| # Initially remove all logs. |
| Redfish Purge Event Log |
| BMC Execute Command ${CMD_INFORMATIONAL_ERROR} |
| |
| # Generate informational logs and verify that it would not get listed by peltool's list command. |
| ${pel_records}= peltool -l |
| ${ids}= Get Dictionary Keys ${pel_records} |
| Should Be Empty ${ids} |
| |
| # Verify that information logs get listed using peltool's list command with -lfh option. |
| ${pel_records}= peltool -lfh |
| ${ids}= Get Dictionary Keys ${pel_records} |
| Should Not Be Empty ${ids} |
| ${id}= Get From List ${ids} 0 |
| Should Contain ${pel_records['${id}']['Sev']} Informational |
| |
| |
| Verify PEL Delete |
| [Documentation] Verify that peltool command can delete PEL log based on id. |
| [Tags] Verify_PEL_Delete |
| |
| BMC Execute Command ${CMD_PREDICTIVE_ERROR} |
| ${pel_ids}= Get PEL Log Via BMC CLI |
| ${id}= Get From List ${pel_ids} -1 |
| Peltool -d ${id} False |
| Run Keyword and Expect Error *PEL not found* Peltool -i ${id} |
| |
| |
| Verify Mandatory Fields For Predictive Error |
| [Documentation] Verify mandatory fields of predictive errors from pel information. |
| [Tags] Verify_Mandatory_Fields_For_Predictive_Error |
| |
| # Inject predictive error. |
| BMC Execute Command ${CMD_PREDICTIVE_ERROR} |
| |
| ${pel_ids}= Get PEL Log Via BMC CLI |
| ${pel_id}= Get From List ${pel_ids} -1 |
| ${pel_output}= Peltool -i ${pel_id} |
| # Get all fields in predictive error log. |
| ${pel_sections}= Get Dictionary Keys ${pel_output} |
| |
| List Should Contain Sub List ${pel_sections} ${mandatory_Predictive_pel_fields} |
| |
| |
| Verify Mandatory Fields For Informational Error |
| [Documentation] Verify mandatory fields of informational error from pel information. |
| [Tags] Verify_Mandatory_Fields_For_Informational_Error |
| |
| # Inject informational error. |
| BMC Execute Command ${CMD_INFORMATIONAL_ERROR} |
| ${pel_records}= Peltool -lfh |
| |
| ${ids}= Get Dictionary Keys ${pel_records} |
| ${pel_id}= Get From List ${ids} -1 |
| ${pel_output}= Peltool -i ${pel_id} |
| |
| # Get all fields in the informational error log. |
| ${pel_sections}= Get Dictionary Keys ${pel_output} |
| FOR ${section} IN @{Mandatory_Informational_Pel_Fields} |
| ${contains}= Evaluate "${section}" in "${pel_sections}" |
| Should Be True ${contains} |
| END |
| |
| |
| Verify PEL Log Offloaded To Host |
| [Documentation] Verify host transmission state of offloaded PEL logs. |
| [Tags] Verify_PEL_Log_Offloaded_To_Host |
| [Template] Verify PEL Transmission To Host |
| |
| # error_type host_state expected_transmission_state |
| predictive_error On Acked |
| unrecoverable_error On Acked |
| informational_error On Acked |
| |
| |
| Verify PEL Log Not Offloaded To Host |
| [Documentation] Verify host transmission state of not offloaded PEL logs. |
| [Tags] Verify_PEL_Log_Not_Offloaded_To_Host |
| [Template] Verify PEL Transmission To Host |
| |
| # error_type host_state expected_transmission_state |
| predictive_error Off Not Sent |
| unrecoverable_error Off Not Sent |
| informational_error Off Not Sent |
| |
| |
| *** Keywords *** |
| |
| Error Logging Rotation Policy |
| [Documentation] Verify that when maximum log limit is reached, given error logging type |
| ... are deleted when reached their max allocated space. |
| [Arguments] ${error_log_type} ${max_allocated_space_percentage} |
| |
| # Description of argument(s): |
| # error_log Error logs to be created (E.g. Informational BMC 3000 |
| # stands for BMC created 3000 informational error logs). |
| # max_allocated_space_percentage The maximum percentage of disk usage for given error |
| # log type when maximum count/log size is reached. |
| # The maximum error log count is 3000. |
| |
| Redfish.Login |
| |
| # Initially remove all logs. Purging is done to ensure that, only specific logs are present |
| # in BMC during the test. |
| Redfish Purge Event Log |
| |
| @{lists}= Split String ${error_log_type} ,${SPACE} |
| |
| ${length}= Get Length ${lists} |
| |
| FOR ${list} IN RANGE ${length} |
| @{words}= Split String ${lists}[${list}] ${SPACE} |
| Create Error Log ${words}[0] ${words}[1] ${words}[2] |
| END |
| |
| # Create an additional error log to exceed max error logs limit. |
| BMC Execute Command ${CMD_UNRECOVERABLE_ERROR} |
| |
| # Delay for BMC to perform delete older error logs when log limit exceeds. |
| Sleep 10s |
| |
| # Verify disk usage is around max allocated space. Maximum usage is around 3MB not exactly 3MB |
| # (for informational log) and around 6 MB for unrecoverable / predictive error log. So, usage |
| # percentage is NOT exactly 15% and 30%. So, an error/accuracy factor 0.5 percent is added. |
| |
| ${disk_usage_percentage}= Get Disk Usage For Error Logs |
| ${percent_diff}= Evaluate ${disk_usage_percentage} - ${max_allocated_space_percentage} |
| ${percent_diff}= Evaluate abs(${percent_diff}) |
| |
| ${trimmed_as_expected}= Run Keyword If ${disk_usage_percentage} > ${max_allocated_space_percentage} |
| ... Evaluate ${percent_diff} <= 0.5 |
| ... ELSE |
| ... Set Variable True |
| |
| # Check PEL log count via peltool command and compare it with actual generated log count. |
| ${pel_records}= peltool -n |
| ${no_pel_records}= Set Variable ${pel_records["Number of PELs found"]} |
| # Number of logs can be 80% of the total logs created after trimming. |
| ${expected_max_record}= Evaluate 3000 * 0.8 |
| |
| Run Keyword If ${trimmed_as_expected} == False |
| ... Should Be True ${no_pel_records} <= ${expected_max_record} |
| |
| |
| Create Error Log |
| [Documentation] Create an error log. |
| [Arguments] ${error_severity} ${error_creator} ${count} |
| |
| # Description of argument(s): |
| # error_severity Severity of the error (E.g. Informational, Unrecoberable or Predictive) |
| # error_creator Name of error log's creator(E.g BMC, Host Boot) |
| # count Number of error logs to be generated. |
| |
| FOR ${i} IN RANGE 0 ${count} |
| ${cmd}= Set Variable If |
| ... '${error_severity}' == 'Informational' and '${error_creator}' == 'BMC' ${CMD_INFORMATIONAL_ERROR} |
| ... '${error_severity}' == 'Informational' and '${error_creator}' == 'HOST' ${CMD_INFORMATIONAL_HOST_ERROR} |
| ... '${error_severity}' == 'Predictive' and '${error_creator}' == 'BMC' ${CMD_PREDICTIVE_ERROR} |
| ... '${error_severity}' == 'Unrecoverable' and '${error_creator}' == 'BMC' ${CMD_UNRECOVERABLE_ERROR} |
| ... '${error_severity}' == 'Unrecoverable' and '${error_creator}' == 'HOST' ${CMD_UNRECOVERABLE_HOST_ERROR} |
| BMC Execute Command ${cmd} |
| END |
| |
| |
| Get Disk Usage For Error Logs |
| [Documentation] Get disk usage percentage for error logs. |
| |
| ${usage_output} ${stderr} ${rc}= BMC Execute Command du /var/lib/phosphor-logging/errors |
| |
| ${usage_output}= Fetch From Left ${usage_output} \/ |
| |
| # Convert disk usage unit from KB to MB. |
| ${usage_output}= Evaluate ${usage_output} / 1024 |
| |
| # Logging disk capacity limit is set to 20MB. So calculating the log usage percentage. |
| ${usage_percent}= Evaluate ${usage_output} / 20 * 100 |
| |
| [return] ${usage_percent} |
| |
| |
| Get PEL Log IDs |
| [Documentation] Returns the list of PEL log IDs which contains given field's value. |
| [Arguments] ${pel_section} ${pel_field} @{pel_field_value} |
| |
| # Description of argument(s): |
| # pel_section The section of PEL (e.g. Private Header, User Header). |
| # pel_field The PEL field (e.g. Event Severity, Event Type). |
| # pel_field_value The list of PEL's field value (e.g. Unrecoverable Error). |
| |
| ${pel_ids}= Get PEL Log Via BMC CLI |
| @{pel_id_list}= Create List |
| |
| FOR ${id} IN @{pel_ids} |
| ${pel_output}= Peltool -i ${id} |
| # Example of PEL output from "peltool -i <id>" command. |
| # [Private Header]: |
| # [Created at]: 08/24/1928 12:04:06 |
| # [Created by]: 0x584D |
| # [Sub-section type]: 0 |
| # [Entry Id]: 0x50000BB7 |
| # [Platform Log Id]: 0x8200061D |
| # [CSSVER]: |
| # [Section Version]: 1 |
| # [Creator Subsystem]: PHYP |
| # [BMC Event Log Id]: 341 |
| # [Committed at]: 03/25/1920 12:06:22 |
| # [User Header]: |
| # [Log Committed by]: 0x4552 |
| # [Action Flags]: |
| # [0]: Report Externally |
| # [Subsystem]: I/O Subsystem |
| # [Event Type]: Miscellaneous, Informational Only |
| # [Sub-section type]: 0 |
| # [Event Scope]: Entire Platform |
| # [Event Severity]: Informational Event |
| # [Host Transmission]: Not Sent |
| # [Section Version]: 1 |
| |
| ${pel_section_output}= Get From Dictionary ${pel_output} ${pel_section} |
| ${pel_field_output}= Get From Dictionary ${pel_section_output} ${pel_field} |
| Run Keyword If '${pel_field_output}' in @{pel_field_value} Append To List ${pel_id_list} ${id} |
| END |
| Sort List ${pel_id_list} |
| |
| [Return] ${pel_id_list} |
| |
| |
| Get PEL Log Via BMC CLI |
| [Documentation] Returns the list of PEL IDs using BMC CLI. |
| |
| ${pel_records}= Peltool -l |
| ${ids}= Get Dictionary Keys ${pel_records} |
| Sort List ${ids} |
| |
| [Return] ${ids} |
| |
| |
| Get PEL Field Value |
| [Documentation] Returns the value of given PEL's field. |
| [Arguments] ${pel_id} ${pel_section} ${pel_field} |
| |
| # Description of argument(s): |
| # pel_id The ID of PEL (e.g. 0x5000002D, 0x5000002E). |
| # pel_section The section of PEL (e.g. Private Header, User Header) |
| # pel_field The PEL field (e.g. Event Severity, Event Type). |
| |
| ${pel_output}= Peltool -i ${pel_id} |
| |
| # Example of PEL output from "peltool -i <id>" command. |
| # [Private Header]: |
| # [Created at]: 08/24/1928 12:04:06 |
| # [Created by]: 0x584D |
| # [Sub-section type]: 0 |
| # [Entry Id]: 0x50000BB7 |
| # [Platform Log Id]: 0x8200061D |
| # [CSSVER]: |
| # [Section Version]: 1 |
| # [Creator Subsystem]: PHYP |
| # [BMC Event Log Id]: 341 |
| # [Committed at]: 03/25/1920 12:06:22 |
| # [User Header]: |
| # [Log Committed by]: 0x4552 |
| # [Action Flags]: |
| # [0]: Report Externally |
| # [Subsystem]: I/O Subsystem |
| # [Event Type]: Miscellaneous, Informational Only |
| # [Sub-section type]: 0 |
| # [Event Scope]: Entire Platform |
| # [Event Severity]: Informational Event |
| # [Host Transmission]: Not Sent |
| # [Section Version]: 1 |
| |
| ${pel_section_output}= Get From Dictionary ${pel_output} ${pel_section} |
| ${pel_field_output}= Get From Dictionary ${pel_section_output} ${pel_field} |
| |
| [Return] ${pel_field_output} |
| |
| |
| Verify PEL Transmission To Host |
| [Documentation] Inject PEL log of given type and verify its host transmission state. |
| [Arguments] ${error_type} ${host_state} ${expected_transmission_state} |
| |
| # Description of argument(s): |
| # error_type Type of error log to be injected. |
| # host_state Host state which is required before error log injection. |
| # expected_transmission_state Expected host transmission state of PEL log. |
| |
| # Get system in required state. |
| Run Keyword If '${host_state}' == 'Off' |
| ... Redfish Power Off stack_mode=skip |
| ... ELSE IF '${host_state}' == 'On' |
| ... Run Keywords Redfish Power On stack_mode=skip AND |
| ... Wait For Host To Ping ${OS_HOST} 5 min 10 sec |
| |
| Redfish Purge Event Log |
| |
| # Inject required error log. |
| Run Keyword If "${error_type}" == "informational_error" |
| ... BMC Execute Command ${CMD_INFORMATIONAL_ERROR} |
| ... ELSE IF "${error_type}" == "unrecoverable_error" |
| ... BMC Execute Command ${CMD_UNRECOVERABLE_ERROR} |
| ... ELSE IF "${error_type}" == "predictive_error" |
| ... BMC Execute Command ${CMD_PREDICTIVE_ERROR} |
| |
| ${pel_records}= Get Pel Data From Bmc True True |
| ${ids}= Get Dictionary Keys ${pel_records} |
| ${pel_id}= Get From List ${ids} -1 |
| |
| # Check host transmission state for the cases where PEL is |
| # expected to be offloaded to HOST. |
| Run Keyword If "${expected_transmission_state}" == "Acked" |
| ... Wait Until Keyword Succeeds 15 min 10 sec |
| ... Check If PEL Transmission State Is Expected ${pel_id} Acked |
| |
| # Adding delay before checking host transmission for the cases where PEL is |
| # not expected to be offloaded to HOST. |
| Run Keyword If "${expected_transmission_state}" == "Not sent" |
| ... Run Keywords Sleep 120s AND |
| ... Check If PEL Transmission State Is Expected ${pel_id} Not sent |
| |
| |
| Check If PEL Transmission State Is Expected |
| [Documentation] Check if PEL's host transmission state is matching expected state. |
| [Arguments] ${pel_id} ${expected_transmission_state} |
| |
| # Description of argument(s): |
| # expected_transmission_state Expected transmission state of PEL log. |
| |
| # Check the transmission details in the error log. |
| ${host_transmission}= Get PEL Field Value ${pel_id} User Header Host Transmission |
| Should Be Equal ${host_transmission} ${expected_transmission_state} |