Rahul Maheshwari | 2568f87 | 2020-09-23 01:03:39 -0500 | [diff] [blame] | 1 | *** Settings *** |
| 2 | |
| 3 | Documentation Test BMC dump functionality of OpenBMC. |
| 4 | |
George Keishing | 88ec280 | 2021-08-04 03:04:22 -0500 | [diff] [blame] | 5 | Resource ../../lib/bmc_redfish_resource.robot |
| 6 | Resource ../../lib/boot_utils.robot |
Rahul Maheshwari | 2568f87 | 2020-09-23 01:03:39 -0500 | [diff] [blame] | 7 | Resource ../../lib/dump_utils.robot |
George Keishing | 88ec280 | 2021-08-04 03:04:22 -0500 | [diff] [blame] | 8 | Resource ../../lib/openbmc_ffdc.robot |
manashsarma | 7e1f2c5 | 2023-02-03 03:18:40 -0600 | [diff] [blame] | 9 | Variables ../../data/pel_variables.py |
Rahul Maheshwari | 2568f87 | 2020-09-23 01:03:39 -0500 | [diff] [blame] | 10 | |
George Keishing | f514ccb | 2021-02-02 23:55:56 -0600 | [diff] [blame] | 11 | Suite Setup Redfish.Login |
Rahul Maheshwari | 2568f87 | 2020-09-23 01:03:39 -0500 | [diff] [blame] | 12 | Test Setup Redfish Delete All BMC Dumps |
| 13 | Test Teardown Test Teardown Execution |
| 14 | |
George Keishing | 6217ea3 | 2023-10-16 09:58:41 +0530 | [diff] [blame] | 15 | Force Tags BMC_Dumps |
| 16 | |
Tim Lee | de2d130 | 2021-06-18 01:24:11 +0800 | [diff] [blame] | 17 | *** Variables *** |
| 18 | |
| 19 | # Total size of the dump in kilo bytes |
| 20 | ${BMC_DUMP_TOTAL_SIZE} ${1024} |
| 21 | |
| 22 | # Minimum space required for one bmc dump in kilo bytes |
| 23 | ${BMC_DUMP_MIN_SPACE_REQD} ${20} |
manashsarma | bd346b3 | 2022-07-20 06:22:58 -0500 | [diff] [blame] | 24 | ${MAX_DUMP_COUNT} ${20} |
manashsarma | 395f880 | 2022-11-17 03:09:02 -0600 | [diff] [blame] | 25 | ${BMC_DUMP_COLLECTOR_PATH} /var/lib/phosphor-debug-collector/dumps |
Rahul Maheshwari | 2568f87 | 2020-09-23 01:03:39 -0500 | [diff] [blame] | 26 | |
| 27 | *** Test Cases *** |
| 28 | |
manashsarma | 4b5b1fa | 2022-11-16 02:36:58 -0600 | [diff] [blame] | 29 | Verify Error Response For Already Deleted Dump Id |
| 30 | [Documentation] Delete non existing BMC dump and expect an error. |
| 31 | [Tags] Verify_Error_Response_For_Already_Deleted_Dump_Id |
| 32 | |
manashsarma | 9dcabad | 2023-02-02 03:55:17 -0600 | [diff] [blame] | 33 | Redfish Power Off stack_mode=skip |
manashsarma | 4b5b1fa | 2022-11-16 02:36:58 -0600 | [diff] [blame] | 34 | ${dump_id}= Create User Initiated BMC Dump Via Redfish |
manashsarma | 6cc5bcf | 2023-08-22 02:22:48 -0500 | [diff] [blame] | 35 | Wait Until Keyword Succeeds 15 sec 5 sec Redfish Delete BMC Dump ${dump_id} |
manashsarma | 4b5b1fa | 2022-11-16 02:36:58 -0600 | [diff] [blame] | 36 | Run Keyword And Expect Error ValueError: * Redfish Delete BMC Dump ${dump_id} |
| 37 | |
| 38 | |
Rahul Maheshwari | 2568f87 | 2020-09-23 01:03:39 -0500 | [diff] [blame] | 39 | Verify User Initiated BMC Dump When Host Powered Off |
| 40 | [Documentation] Create user initiated BMC dump at host off state and |
| 41 | ... verify dump entry for it. |
| 42 | [Tags] Verify_User_Initiated_BMC_Dump_When_Host_Powered_Off |
| 43 | |
| 44 | Redfish Power Off stack_mode=skip |
Rahul Maheshwari | bd38aa4 | 2021-08-04 01:24:39 -0500 | [diff] [blame] | 45 | ${dump_id}= Create User Initiated BMC Dump Via Redfish |
Rahul Maheshwari | bcefdf2 | 2020-10-16 07:51:34 -0500 | [diff] [blame] | 46 | ${dump_entries}= Get BMC Dump Entries |
Rahul Maheshwari | 2568f87 | 2020-09-23 01:03:39 -0500 | [diff] [blame] | 47 | Length Should Be ${dump_entries} 1 |
| 48 | List Should Contain Value ${dump_entries} ${dump_id} |
| 49 | |
Rahul Maheshwari | 2568f87 | 2020-09-23 01:03:39 -0500 | [diff] [blame] | 50 | |
Rahul Maheshwari | e8e9d0c | 2020-10-30 02:39:01 -0500 | [diff] [blame] | 51 | Verify User Initiated BMC Dump Size |
manashsarma | 7831dc2 | 2022-03-08 01:49:36 -0600 | [diff] [blame] | 52 | [Documentation] Verify user initiated BMC dump size is under 20 MB. |
Rahul Maheshwari | e8e9d0c | 2020-10-30 02:39:01 -0500 | [diff] [blame] | 53 | [Tags] Verify_User_Initiated_BMC_Dump_Size |
| 54 | |
manashsarma | e52174b | 2023-04-03 01:26:56 -0500 | [diff] [blame] | 55 | Redfish Power Off stack_mode=skip |
Rahul Maheshwari | bd38aa4 | 2021-08-04 01:24:39 -0500 | [diff] [blame] | 56 | ${dump_id}= Create User Initiated BMC Dump Via Redfish |
ganesanb | 4d43028 | 2023-04-27 14:33:23 +0000 | [diff] [blame] | 57 | ${resp}= Redfish.Get Properties /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Entries/${dump_id} |
Rahul Maheshwari | e8e9d0c | 2020-10-30 02:39:01 -0500 | [diff] [blame] | 58 | |
| 59 | # Example of response from above Redfish GET request. |
| 60 | # "@odata.type": "#LogEntry.v1_7_0.LogEntry", |
| 61 | # "AdditionalDataSizeBytes": 31644, |
ganesanb | 4d43028 | 2023-04-27 14:33:23 +0000 | [diff] [blame] | 62 | # "AdditionalDataURI": "/redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/attachment/9", |
Rahul Maheshwari | e8e9d0c | 2020-10-30 02:39:01 -0500 | [diff] [blame] | 63 | # "Created": "2020-10-23T06:32:53+00:00", |
| 64 | # "DiagnosticDataType": "Manager", |
| 65 | # "EntryType": "Event", |
| 66 | # "Id": "9", |
| 67 | # "Name": "BMC Dump Entry" |
| 68 | |
manashsarma | 7831dc2 | 2022-03-08 01:49:36 -0600 | [diff] [blame] | 69 | # Max size for dump is 20 MB = 20x1024x1024 Byte. |
| 70 | Should Be True 0 < ${resp["AdditionalDataSizeBytes"]} < 20971520 |
Rahul Maheshwari | e8e9d0c | 2020-10-30 02:39:01 -0500 | [diff] [blame] | 71 | |
| 72 | |
manashsarma | 7e1f2c5 | 2023-02-03 03:18:40 -0600 | [diff] [blame] | 73 | Verify Internal Failure Initiated BMC Dump Size |
| 74 | [Documentation] Verify that the internal failure initiated BMC dump size is under 20 MB. |
| 75 | [Tags] Verify_Internal_Failure_Initiated_BMC_Dump_Size |
| 76 | |
manashsarma | a8efe72 | 2024-02-26 03:24:55 -0600 | [diff] [blame^] | 77 | Redfish Power Off stack_mode=skip |
manashsarma | 7e1f2c5 | 2023-02-03 03:18:40 -0600 | [diff] [blame] | 78 | Redfish Delete All BMC Dumps |
| 79 | |
| 80 | # Create an internal failure error log. |
| 81 | BMC Execute Command ${CMD_INTERNAL_FAILURE} |
| 82 | |
| 83 | # Wait for BMC dump to get generated after injecting internal failure. |
| 84 | Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available |
| 85 | |
| 86 | # Verify that only one BMC dump is generated after injecting error. |
| 87 | ${dump_entries}= Get BMC Dump Entries |
| 88 | ${length}= Get length ${dump_entries} |
| 89 | Should Be Equal As Integers ${length} ${1} |
| 90 | |
| 91 | # Max size for dump is 20 MB = 20x1024x1024 Byte. |
| 92 | ${resp}= Redfish.Get Properties |
ganesanb | 4d43028 | 2023-04-27 14:33:23 +0000 | [diff] [blame] | 93 | ... /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Entries/${dump_entries[0]} |
manashsarma | 7e1f2c5 | 2023-02-03 03:18:40 -0600 | [diff] [blame] | 94 | Should Be True 0 < ${resp["AdditionalDataSizeBytes"]} < 20971520 |
| 95 | |
| 96 | |
manashsarma | bbc68bc | 2022-12-05 09:55:17 -0600 | [diff] [blame] | 97 | Verify Multiple BMC Dump Creation |
| 98 | [Documentation] Verify that multiple BMC dumps can be created one after |
| 99 | ... another successfully. |
| 100 | [Tags] Verify_Multiple_BMC_Dump_Creation |
| 101 | |
manashsarma | e52174b | 2023-04-03 01:26:56 -0500 | [diff] [blame] | 102 | Redfish Power Off stack_mode=skip |
manashsarma | bbc68bc | 2022-12-05 09:55:17 -0600 | [diff] [blame] | 103 | ${dump_count}= Evaluate random.randint(5, 10) modules=random |
| 104 | FOR ${INDEX} IN 1 ${dump_count} |
| 105 | Create User Initiated BMC Dump Via Redfish |
| 106 | END |
| 107 | |
| 108 | |
manashsarma | 395f880 | 2022-11-17 03:09:02 -0600 | [diff] [blame] | 109 | Verify BMC Dump Default Location In BMC |
| 110 | [Documentation] Verify that BMC dump is created in its default location of BMC. |
George Keishing | 224e4c7 | 2022-12-14 05:07:58 -0600 | [diff] [blame] | 111 | [Tags] Verify_BMC_Dump_Default_Location_In_BMC |
manashsarma | 395f880 | 2022-11-17 03:09:02 -0600 | [diff] [blame] | 112 | |
manashsarma | e52174b | 2023-04-03 01:26:56 -0500 | [diff] [blame] | 113 | Redfish Power Off stack_mode=skip |
manashsarma | 395f880 | 2022-11-17 03:09:02 -0600 | [diff] [blame] | 114 | Redfish Delete All BMC Dumps |
| 115 | ${dump_id}= Create User Initiated BMC Dump Via Redfish |
| 116 | ${dump_file} ${stderr} ${rc}= BMC Execute Command |
| 117 | ... ls ${BMC_DUMP_COLLECTOR_PATH}/${dump_id} |
| 118 | Should Be True ${rc} == 0 |
manashsarma | c30e02a | 2023-02-20 06:46:15 -0600 | [diff] [blame] | 119 | Should Contain Any ${dump_file} BMCDUMP obmcdump |
manashsarma | 395f880 | 2022-11-17 03:09:02 -0600 | [diff] [blame] | 120 | |
| 121 | |
George Keishing | 3d85098 | 2022-07-19 11:03:03 -0500 | [diff] [blame] | 122 | Verify User Initiated BMC Dump When Host Booted |
| 123 | [Documentation] Create user initiated BMC dump at host booted state and |
| 124 | ... verify dump entry for it. |
| 125 | [Tags] Verify_User_Initiated_BMC_Dump_When_Host_Booted |
| 126 | |
| 127 | Redfish Power On stack_mode=skip |
| 128 | ${dump_id}= Create User Initiated BMC Dump Via Redfish |
| 129 | ${dump_entries}= Get BMC Dump Entries |
| 130 | Length Should Be ${dump_entries} 1 |
| 131 | List Should Contain Value ${dump_entries} ${dump_id} |
| 132 | |
| 133 | |
manashsarma | 4b5d6ac | 2023-02-08 00:04:03 -0600 | [diff] [blame] | 134 | Verify User Initiated BMC Dump At Host Booting |
| 135 | [Documentation] Create and verify user initiated BMC dump during Host is powwering on |
| 136 | ... or when host booting is in progress. |
| 137 | [Tags] Verify_User_Initiated_BMC_Dump_At_Host_Booting |
| 138 | |
manashsarma | 9712ac5 | 2023-03-09 04:54:41 -0600 | [diff] [blame] | 139 | Redfish Power Off stack_mode=skip |
manashsarma | 4b5d6ac | 2023-02-08 00:04:03 -0600 | [diff] [blame] | 140 | Redfish Delete All BMC Dumps |
| 141 | |
| 142 | # Initiate power on. |
| 143 | Redfish Power Operation On |
| 144 | Wait Until Keyword Succeeds 2 min 5 sec Is Boot Progress Changed |
| 145 | |
| 146 | # Create user initiated BMC dump and verify only one dump is available. |
| 147 | Create User Initiated BMC Dump Via Redfish |
| 148 | ${dump_entries}= Get BMC Dump Entries |
| 149 | Length Should Be ${dump_entries} 1 |
| 150 | |
| 151 | |
Rahul Maheshwari | 2568f87 | 2020-09-23 01:03:39 -0500 | [diff] [blame] | 152 | Verify Dump Persistency On Dump Service Restart |
| 153 | [Documentation] Create user dump, restart dump manager service and verify dump |
| 154 | ... persistency. |
| 155 | [Tags] Verify_Dump_Persistency_On_Dump_Service_Restart |
| 156 | |
manashsarma | e52174b | 2023-04-03 01:26:56 -0500 | [diff] [blame] | 157 | Redfish Power Off stack_mode=skip |
Rahul Maheshwari | bd38aa4 | 2021-08-04 01:24:39 -0500 | [diff] [blame] | 158 | Create User Initiated BMC Dump Via Redfish |
ganesanb | 4d43028 | 2023-04-27 14:33:23 +0000 | [diff] [blame] | 159 | ${dump_entries_before}= redfish_utils.get_member_list /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Entries |
Rahul Maheshwari | 2568f87 | 2020-09-23 01:03:39 -0500 | [diff] [blame] | 160 | |
| 161 | # Restart dump service. |
| 162 | BMC Execute Command systemctl restart xyz.openbmc_project.Dump.Manager.service |
| 163 | Sleep 10s reason=Wait for BMC dump service to restart properly |
| 164 | |
ganesanb | 4d43028 | 2023-04-27 14:33:23 +0000 | [diff] [blame] | 165 | ${dump_entries_after}= redfish_utils.get_member_list /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Entries |
Rahul Maheshwari | 2568f87 | 2020-09-23 01:03:39 -0500 | [diff] [blame] | 166 | Lists Should Be Equal ${dump_entries_before} ${dump_entries_after} |
| 167 | |
| 168 | |
| 169 | Verify Dump Persistency On BMC Reset |
| 170 | [Documentation] Create user dump, reset BMC and verify dump persistency. |
| 171 | [Tags] Verify_Dump_Persistency_On_BMC_Reset |
| 172 | |
manashsarma | 8fa9c89 | 2022-10-03 06:20:58 -0500 | [diff] [blame] | 173 | # Power off host so that dump is not offloaded to host OS. |
| 174 | Redfish Power Off stack_mode=skip |
| 175 | |
Rahul Maheshwari | bd38aa4 | 2021-08-04 01:24:39 -0500 | [diff] [blame] | 176 | Create User Initiated BMC Dump Via Redfish |
ganesanb | 4d43028 | 2023-04-27 14:33:23 +0000 | [diff] [blame] | 177 | ${dump_entries_before}= redfish_utils.get_member_list /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Entries |
Rahul Maheshwari | 2568f87 | 2020-09-23 01:03:39 -0500 | [diff] [blame] | 178 | |
| 179 | # Reset BMC. |
George Keishing | 88ec280 | 2021-08-04 03:04:22 -0500 | [diff] [blame] | 180 | OBMC Reboot (off) stack_mode=skip |
Rahul Maheshwari | 2568f87 | 2020-09-23 01:03:39 -0500 | [diff] [blame] | 181 | |
ganesanb | 4d43028 | 2023-04-27 14:33:23 +0000 | [diff] [blame] | 182 | ${dump_entries_after}= redfish_utils.get_member_list /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Entries |
Rahul Maheshwari | 2568f87 | 2020-09-23 01:03:39 -0500 | [diff] [blame] | 183 | Lists Should Be Equal ${dump_entries_before} ${dump_entries_after} |
| 184 | |
| 185 | |
Rahul Maheshwari | 95cbceb | 2020-10-21 23:25:08 -0500 | [diff] [blame] | 186 | Delete User Initiated BMC Dump And Verify |
| 187 | [Documentation] Delete user initiated BMC dump and verify. |
| 188 | [Tags] Delete_User_Initiated_BMC_Dump_And_Verify |
| 189 | |
manashsarma | e52174b | 2023-04-03 01:26:56 -0500 | [diff] [blame] | 190 | Redfish Power Off stack_mode=skip |
Rahul Maheshwari | bd38aa4 | 2021-08-04 01:24:39 -0500 | [diff] [blame] | 191 | ${dump_id}= Create User Initiated BMC Dump Via Redfish |
manashsarma | 6cc5bcf | 2023-08-22 02:22:48 -0500 | [diff] [blame] | 192 | Wait Until Keyword Succeeds 15 sec 5 sec Redfish Delete BMC Dump ${dump_id} |
Rahul Maheshwari | 95cbceb | 2020-10-21 23:25:08 -0500 | [diff] [blame] | 193 | |
| 194 | ${dump_entries}= Get BMC Dump Entries |
| 195 | Should Be Empty ${dump_entries} |
| 196 | |
| 197 | |
| 198 | Delete All User Initiated BMC Dumps And Verify |
| 199 | [Documentation] Delete all user initiated BMC dumps and verify. |
| 200 | [Tags] Delete_All_User_Initiated_BMC_Dumps_And_Verify |
| 201 | |
manashsarma | 70d840e | 2022-10-18 02:34:45 -0500 | [diff] [blame] | 202 | # Power off host so that dump is not offloaded to host OS. |
| 203 | Redfish Power Off stack_mode=skip |
| 204 | |
Rahul Maheshwari | 95cbceb | 2020-10-21 23:25:08 -0500 | [diff] [blame] | 205 | # Create some BMC dump. |
Rahul Maheshwari | bd38aa4 | 2021-08-04 01:24:39 -0500 | [diff] [blame] | 206 | Create User Initiated BMC Dump Via Redfish |
| 207 | Create User Initiated BMC Dump Via Redfish |
Rahul Maheshwari | 95cbceb | 2020-10-21 23:25:08 -0500 | [diff] [blame] | 208 | |
| 209 | Redfish Delete All BMC Dumps |
| 210 | ${dump_entries}= Get BMC Dump Entries |
| 211 | Should Be Empty ${dump_entries} |
| 212 | |
| 213 | |
Rahul Maheshwari | 4f338ab | 2020-10-21 23:28:40 -0500 | [diff] [blame] | 214 | Create Two User Initiated BMC Dumps |
| 215 | [Documentation] Create two user initiated BMC dumps. |
| 216 | [Tags] Create_Two_User_Initiated_BMC_Dumps |
| 217 | |
manashsarma | e52174b | 2023-04-03 01:26:56 -0500 | [diff] [blame] | 218 | Redfish Power Off stack_mode=skip |
Rahul Maheshwari | bd38aa4 | 2021-08-04 01:24:39 -0500 | [diff] [blame] | 219 | ${dump_id1}= Create User Initiated BMC Dump Via Redfish |
| 220 | ${dump_id2}= Create User Initiated BMC Dump Via Redfish |
Rahul Maheshwari | 4f338ab | 2020-10-21 23:28:40 -0500 | [diff] [blame] | 221 | |
| 222 | ${dump_entries}= Get BMC Dump Entries |
| 223 | Length Should Be ${dump_entries} 2 |
| 224 | Should Contain ${dump_entries} ${dump_id1} |
| 225 | Should Contain ${dump_entries} ${dump_id2} |
| 226 | |
| 227 | |
| 228 | Create Two User Initiated BMC Dumps And Delete One |
| 229 | [Documentation] Create two dumps and delete the first. |
| 230 | [Tags] Create_Two_User_Initiated_BMC_Dumps_And_Delete_One |
| 231 | |
manashsarma | e52174b | 2023-04-03 01:26:56 -0500 | [diff] [blame] | 232 | Redfish Power Off stack_mode=skip |
Rahul Maheshwari | bd38aa4 | 2021-08-04 01:24:39 -0500 | [diff] [blame] | 233 | ${dump_id1}= Create User Initiated BMC Dump Via Redfish |
| 234 | ${dump_id2}= Create User Initiated BMC Dump Via Redfish |
Rahul Maheshwari | 4f338ab | 2020-10-21 23:28:40 -0500 | [diff] [blame] | 235 | |
manashsarma | 6cc5bcf | 2023-08-22 02:22:48 -0500 | [diff] [blame] | 236 | Wait Until Keyword Succeeds 15 sec 5 sec Redfish Delete BMC Dump ${dump_id1} |
Rahul Maheshwari | 4f338ab | 2020-10-21 23:28:40 -0500 | [diff] [blame] | 237 | |
| 238 | ${dump_entries}= Get BMC Dump Entries |
| 239 | Length Should Be ${dump_entries} 1 |
| 240 | List Should Contain Value ${dump_entries} ${dump_id2} |
| 241 | |
| 242 | |
| 243 | Create And Delete User Initiated BMC Dump Multiple Times |
| 244 | [Documentation] Create and delete user initiated BMC dump multiple times. |
| 245 | [Tags] Create_And_Delete_User_Initiated_BMC_Dump_Multiple_Times |
| 246 | |
manashsarma | e52174b | 2023-04-03 01:26:56 -0500 | [diff] [blame] | 247 | Redfish Power Off stack_mode=skip |
Rahul Maheshwari | 4f338ab | 2020-10-21 23:28:40 -0500 | [diff] [blame] | 248 | FOR ${INDEX} IN 1 10 |
Rahul Maheshwari | bd38aa4 | 2021-08-04 01:24:39 -0500 | [diff] [blame] | 249 | ${dump_id}= Create User Initiated BMC Dump Via Redfish |
manashsarma | 6cc5bcf | 2023-08-22 02:22:48 -0500 | [diff] [blame] | 250 | Wait Until Keyword Succeeds 15 sec 5 sec Redfish Delete BMC Dump ${dump_id} |
Rahul Maheshwari | 4f338ab | 2020-10-21 23:28:40 -0500 | [diff] [blame] | 251 | END |
| 252 | |
| 253 | |
Rahul Maheshwari | d709c8d | 2020-11-01 23:03:51 -0600 | [diff] [blame] | 254 | Verify Maximum BMC Dump Creation |
| 255 | [Documentation] Create maximum BMC dump and verify error when dump runs out of space. |
| 256 | [Tags] Verify_Maximum_BMC_Dump_Creation |
| 257 | [Teardown] Redfish Delete All BMC Dumps |
| 258 | |
| 259 | # Maximum allowed space for dump is 1024 KB. BMC typically hold 8-14 dumps |
| 260 | # before running out of this dump space. So trying to create dumps in 20 |
| 261 | # iterations to run out of space. |
manashsarma | bd346b3 | 2022-07-20 06:22:58 -0500 | [diff] [blame] | 262 | # User can key in the Maximum allowed space for bmc dump and how many iteration. |
| 263 | FOR ${n} IN RANGE 0 ${MAX_DUMP_COUNT} |
Rahul Maheshwari | bd38aa4 | 2021-08-04 01:24:39 -0500 | [diff] [blame] | 264 | Create User Initiated BMC Dump Via Redfish |
Rahul Maheshwari | d709c8d | 2020-11-01 23:03:51 -0600 | [diff] [blame] | 265 | ${dump_space}= Get Disk Usage For Dumps |
Tim Lee | de2d130 | 2021-06-18 01:24:11 +0800 | [diff] [blame] | 266 | Exit For Loop If ${dump_space} >= (${BMC_DUMP_TOTAL_SIZE} - ${BMC_DUMP_MIN_SPACE_REQD}) |
Rahul Maheshwari | d709c8d | 2020-11-01 23:03:51 -0600 | [diff] [blame] | 267 | END |
| 268 | |
| 269 | # Check error while creating dump when dump size is full. |
| 270 | ${payload}= Create Dictionary DiagnosticDataType=Manager |
ganesanb | 4d43028 | 2023-04-27 14:33:23 +0000 | [diff] [blame] | 271 | Redfish.Post /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Actions/LogService.CollectDiagnosticData |
Rahul Maheshwari | d709c8d | 2020-11-01 23:03:51 -0600 | [diff] [blame] | 272 | ... body=${payload} valid_status_codes=[${HTTP_INTERNAL_SERVER_ERROR}] |
| 273 | |
| 274 | |
manashsarma | 633cac9 | 2022-11-25 03:27:03 -0600 | [diff] [blame] | 275 | Verify BMC Core Dump When Host Powered Off |
| 276 | [Documentation] Verify BMC core dump after application crash at host powered off state. |
| 277 | [Tags] Verify_BMC_Core_Dump_When_Host_Powered_Off |
| 278 | |
| 279 | Redfish Power Off stack_mode=skip |
| 280 | |
| 281 | # Ensure all dumps are cleaned out. |
| 282 | Redfish Delete All BMC Dumps |
| 283 | Trigger Core Dump |
| 284 | |
| 285 | # Verify that BMC dump is available. |
| 286 | Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available |
| 287 | |
| 288 | |
manashsarma | 849189c | 2022-11-28 04:16:51 -0600 | [diff] [blame] | 289 | Verify Core Dump Size |
| 290 | [Documentation] Verify BMC core dump size is under 20 MB. |
| 291 | [Tags] Verify_Core_Dump_Size |
| 292 | |
| 293 | Redfish Power Off stack_mode=skip |
| 294 | |
| 295 | # Ensure all dumps are cleaned out. |
| 296 | Redfish Delete All BMC Dumps |
| 297 | Trigger Core Dump |
| 298 | |
| 299 | # Verify that BMC dump is available. |
| 300 | Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available |
| 301 | ${dump_entries}= Get BMC Dump Entries |
| 302 | ${resp}= Redfish.Get Properties |
ganesanb | 4d43028 | 2023-04-27 14:33:23 +0000 | [diff] [blame] | 303 | ... /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Entries/${dump_entries[0]} |
manashsarma | 849189c | 2022-11-28 04:16:51 -0600 | [diff] [blame] | 304 | |
| 305 | # Max size for dump is 20 MB = 20x1024x1024 Byte. |
| 306 | Should Be True 0 < ${resp["AdditionalDataSizeBytes"]} < 20971520 |
| 307 | |
| 308 | |
manashsarma | fcbfdf6 | 2022-12-07 06:36:28 -0600 | [diff] [blame] | 309 | Verify Error While Initiating BMC Dump During Dumping State |
| 310 | [Documentation] Verify error while initiating BMC dump during dumping state. |
| 311 | [Tags] Verify_Error_While_Initiating_BMC_Dump_During_Dumping_State |
| 312 | |
manashsarma | e52174b | 2023-04-03 01:26:56 -0500 | [diff] [blame] | 313 | Redfish Power Off stack_mode=skip |
Nandish-Matti | af5bc15 | 2023-01-30 00:49:59 -0600 | [diff] [blame] | 314 | ${task_id}= Create User Initiated BMC Dump Via Redfish ${1} |
manashsarma | fcbfdf6 | 2022-12-07 06:36:28 -0600 | [diff] [blame] | 315 | |
| 316 | # Check error while initiating BMC dump while dump in progress. |
| 317 | ${payload}= Create Dictionary DiagnosticDataType=Manager |
| 318 | Redfish.Post |
ganesanb | 4d43028 | 2023-04-27 14:33:23 +0000 | [diff] [blame] | 319 | ... /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Actions/LogService.CollectDiagnosticData |
manashsarma | fcbfdf6 | 2022-12-07 06:36:28 -0600 | [diff] [blame] | 320 | ... body=${payload} valid_status_codes=[${HTTP_SERVICE_UNAVAILABLE}] |
| 321 | |
Nandish-Matti | af5bc15 | 2023-01-30 00:49:59 -0600 | [diff] [blame] | 322 | # Wait for above initiated dump to complete. Otherwise, on going dump would impact next test. |
| 323 | Wait Until Keyword Succeeds 5 min 15 sec Check Task Completion ${task_id} |
| 324 | |
manashsarma | fcbfdf6 | 2022-12-07 06:36:28 -0600 | [diff] [blame] | 325 | |
manashsarma | aa845b7 | 2022-12-12 05:41:32 -0600 | [diff] [blame] | 326 | Verify BMC Dump Create Errors While Another BMC Dump In Progress |
| 327 | [Documentation] Verify BMC dump creation error until older BMC dump completion. |
| 328 | [Tags] Verify_BMC_Dump_Create_Errors_While_Another_BMC_Dump_In_Progress |
| 329 | |
manashsarma | e52174b | 2023-04-03 01:26:56 -0500 | [diff] [blame] | 330 | Redfish Power Off stack_mode=skip |
| 331 | |
manashsarma | aa845b7 | 2022-12-12 05:41:32 -0600 | [diff] [blame] | 332 | # Initiate a BMC dump that returns without completion. |
| 333 | ${task_id}= Create User Initiated BMC Dump Via Redfish ${1} |
manashsarma | cf757da | 2023-10-12 01:12:23 -0500 | [diff] [blame] | 334 | ${task_dict}= Redfish.Get Properties /redfish/v1/TaskService/Tasks/${task_id} |
manashsarma | aa845b7 | 2022-12-12 05:41:32 -0600 | [diff] [blame] | 335 | ${payload}= Create Dictionary DiagnosticDataType=Manager |
manashsarma | cf757da | 2023-10-12 01:12:23 -0500 | [diff] [blame] | 336 | IF '${task_dict['TaskState']}' != 'Completed' |
manashsarma | 785244c | 2023-03-15 05:56:18 -0500 | [diff] [blame] | 337 | ${resp}= Redfish.Post |
ganesanb | 4d43028 | 2023-04-27 14:33:23 +0000 | [diff] [blame] | 338 | ... /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Actions/LogService.CollectDiagnosticData |
manashsarma | cf757da | 2023-10-12 01:12:23 -0500 | [diff] [blame] | 339 | ... body=${payload} valid_status_codes=[${HTTP_SERVICE_UNAVAILABLE}] |
manashsarma | aa845b7 | 2022-12-12 05:41:32 -0600 | [diff] [blame] | 340 | END |
| 341 | |
manashsarma | aa845b7 | 2022-12-12 05:41:32 -0600 | [diff] [blame] | 342 | # Wait for above initiated dump to complete. Otherwise, on going dump would impact next test. |
manashsarma | cf757da | 2023-10-12 01:12:23 -0500 | [diff] [blame] | 343 | Wait Until Keyword Succeeds 5 min 15 sec Check Task Completion ${task_id} |
manashsarma | aa845b7 | 2022-12-12 05:41:32 -0600 | [diff] [blame] | 344 | |
| 345 | |
manashsarma | adabd3b | 2023-02-08 04:51:32 -0600 | [diff] [blame] | 346 | Verify Core Dump After Terminating Dump Manager Service |
| 347 | [Documentation] Verify initiate core dumps and kill Phosphor-dump-manager. |
| 348 | [Tags] Verify_Core_Dump_After_Terminating_Dump_Manager_Service |
| 349 | |
| 350 | Redfish Power Off stack_mode=skip |
| 351 | |
| 352 | # Remove all available dumps in BMC. |
| 353 | Redfish Delete All BMC Dumps |
| 354 | |
| 355 | # Find the pid of the phosphor-dump-manage process and kill it. |
| 356 | ${cmd_buf}= Catenate kill -s SEGV $(pgrep phosphor-dump-manager) |
| 357 | ${cmd_output} ${stderr} ${rc}= BMC Execute Command ${cmd_buf} |
| 358 | Should Be Equal As Integers ${rc} ${0} |
| 359 | |
| 360 | # Verify that BMC dump is available. |
| 361 | Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available |
| 362 | |
George Keishing | 4611b81 | 2023-02-16 09:44:33 -0600 | [diff] [blame] | 363 | # Verifying that there is only one dump. |
manashsarma | adabd3b | 2023-02-08 04:51:32 -0600 | [diff] [blame] | 364 | ${dump_entries}= Get BMC Dump Entries |
| 365 | ${length}= Get length ${dump_entries} |
| 366 | Should Be Equal As Integers ${length} ${1} |
| 367 | |
| 368 | |
manashsarma | 1202ffe | 2023-09-22 09:01:18 -0500 | [diff] [blame] | 369 | Verify User Initiated BMC Dump Type |
| 370 | [Documentation] Download user initiate BMC dump and validates its type. |
| 371 | [Tags] Verify_User_Initiated_BMC_Dump_Type |
| 372 | |
manashsarma | da7fd92 | 2023-10-05 08:26:13 -0500 | [diff] [blame] | 373 | Redfish Power Off stack_mode=skip |
manashsarma | 1202ffe | 2023-09-22 09:01:18 -0500 | [diff] [blame] | 374 | ${dump_id}= Create User Initiated BMC Dump Via Redfish |
| 375 | |
| 376 | # Download BMC dump and verify its size. |
| 377 | ${resp}= Redfish.Get /redfish/v1/Managers/bmc/LogServices/Dump/Entries/${dump_id} |
manashsarma | 1202ffe | 2023-09-22 09:01:18 -0500 | [diff] [blame] | 378 | ${redfish_dump_creation_timestamp}= Set Variable ${resp.dict["Created"]} |
manashsarma | da7fd92 | 2023-10-05 08:26:13 -0500 | [diff] [blame] | 379 | # Download BMC dump and verify its size. |
| 380 | ${tarfile}= Download BMC Dump ${dump_id} |
manashsarma | 1202ffe | 2023-09-22 09:01:18 -0500 | [diff] [blame] | 381 | |
| 382 | # Extract dump and verify type of dump from summary.log content: |
| 383 | # Wed Aug 30 17:23:29 UTC 2023 Name: BMCDUMP.XXXXXXX.0001005.20230830172329 |
| 384 | # Wed Aug 30 17:23:29 UTC 2023 Epochtime: 1693416209 |
| 385 | # Wed Aug 30 17:23:29 UTC 2023 ID: 0001005 |
| 386 | # Wed Aug 30 17:23:29 UTC 2023 Type: user |
| 387 | ${extracted_dump_folder}= Extract BMC Dump BMC_dump.tar.gz ${redfish_dump_creation_timestamp} |
| 388 | ${contents}= OperatingSystem.Get File ${extracted_dump_folder}/summary.log |
| 389 | Should Match Regexp ${contents} Type:[ ]*user |
| 390 | |
| 391 | # Clean extracted dump files. |
| 392 | Remove Files output output.zst |
| 393 | Remove Directory ${extracted_dump_folder} True |
| 394 | |
| 395 | |
manashsarma | 7223e6dd | 2023-09-21 01:10:24 -0500 | [diff] [blame] | 396 | Verify Retrieve Core Initiated BMC Dump |
| 397 | [Documentation] Verify retrieval of core initiated BMC dump. |
| 398 | [Tags] Verify_Retrieve_Core_Initiated_BMC_Dump |
| 399 | |
| 400 | Redfish Power Off stack_mode=skip |
| 401 | |
| 402 | # Ensure all dumps are cleaned out. |
| 403 | Redfish Delete All BMC Dumps |
| 404 | Trigger Core Dump |
| 405 | |
| 406 | # Verify that BMC dump is available. |
| 407 | Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available |
| 408 | |
| 409 | ${dump_entries}= Get BMC Dump Entries |
| 410 | # Download BMC dump and verify its size. |
| 411 | Download BMC Dump ${dump_entries[0]} |
| 412 | |
| 413 | |
manashsarma | fcc14df | 2023-09-20 11:37:50 -0500 | [diff] [blame] | 414 | Verify Retrieve User Initiated BMC Dump |
| 415 | [Documentation] Verify retrieval of user initiated BMC dump. |
| 416 | [Tags] Verify_Retrieve_User_Initiated_BMC_Dump |
| 417 | |
manashsarma | da7fd92 | 2023-10-05 08:26:13 -0500 | [diff] [blame] | 418 | Redfish Power Off stack_mode=skip |
manashsarma | fcc14df | 2023-09-20 11:37:50 -0500 | [diff] [blame] | 419 | ${dump_id}= Create User Initiated BMC Dump Via Redfish |
| 420 | |
| 421 | # Download BMC dump. |
| 422 | Download BMC Dump ${dump_id} |
| 423 | |
| 424 | |
manashsarma | da7fd92 | 2023-10-05 08:26:13 -0500 | [diff] [blame] | 425 | Verify Core Initiated BMC Dump Type |
| 426 | [Documentation] Download core initiate BMC dump and validates its type. |
| 427 | [Tags] Verify_Core_Initiated_BMC_Dump_Type |
| 428 | |
| 429 | Redfish Power Off stack_mode=skip |
| 430 | |
| 431 | # Ensure all dumps are cleaned out. |
| 432 | Redfish Delete All BMC Dumps |
| 433 | Trigger Core Dump |
| 434 | |
| 435 | # Verify that BMC dump is available. |
| 436 | Wait Until Keyword Succeeds 2 min 10 sec Is BMC Dump Available |
| 437 | |
| 438 | ${dump_entries}= Get BMC Dump Entries |
| 439 | |
| 440 | # Find the timestamp of BMC dump. |
| 441 | ${resp}= Redfish.Get /redfish/v1/Managers/bmc/LogServices/Dump/Entries/${dump_entries[0]} |
| 442 | ${redfish_dump_creation_timestamp}= Set Variable ${resp.dict["Created"]} |
| 443 | |
| 444 | # Download BMC dump and verify its size. |
| 445 | ${tarfile}= Download BMC Dump ${dump_entries[0]} |
| 446 | |
| 447 | # Extract dump and verify type of dump from summary.log content: |
| 448 | # Wed Aug 30 17:23:29 UTC 2023 Name: BMCDUMP.XXXXXXX.0001005.20230830172329 |
| 449 | # Wed Aug 30 17:23:29 UTC 2023 Epochtime: 1693416209 |
| 450 | # Wed Aug 30 17:23:29 UTC 2023 ID: 0001005 |
| 451 | # Wed Aug 30 17:23:29 UTC 2023 Type: core |
| 452 | |
| 453 | ${extracted_dump_folder}= Extract BMC Dump ${tarfile} ${redfish_dump_creation_timestamp} |
| 454 | ${contents}= OperatingSystem.Get File ${extracted_dump_folder}/summary.log |
| 455 | Should Match Regexp ${contents} Type:[ ]*core |
| 456 | |
| 457 | # Clean extracted dump files. |
| 458 | Remove Files output output.zst |
| 459 | Remove Directory ${extracted_dump_folder} True |
| 460 | |
| 461 | |
manashsarma | f0f2995 | 2023-11-16 23:48:01 -0600 | [diff] [blame] | 462 | Verify Core Watchdog Initiated BMC Dump |
| 463 | [Documentation] Verify core watchdog timeout initiated BMC dump. |
| 464 | [Tags] Verify_Core_Watchdog_Initiated_BMC_Dump |
| 465 | |
| 466 | Redfish Delete All BMC Dumps |
| 467 | Redfish Power Off stack_mode=skip |
| 468 | |
| 469 | # Trigger watchdog timeout. |
| 470 | Redfish Initiate Auto Reboot 2000 |
| 471 | |
| 472 | # Wait for BMC dump to get generated after injecting watchdog timeout. |
| 473 | Wait Until Keyword Succeeds 4 min 20 sec Is BMC Dump Available |
| 474 | |
| 475 | # Verify that only one BMC dump is available. |
| 476 | ${dump_entry_list}= Get BMC Dump Entries |
| 477 | ${length}= Get length ${dump_entry_list} |
| 478 | Should Be Equal As Integers ${length} ${1} |
| 479 | |
| 480 | |
Rahul Maheshwari | 2568f87 | 2020-09-23 01:03:39 -0500 | [diff] [blame] | 481 | *** Keywords *** |
| 482 | |
manashsarma | da7fd92 | 2023-10-05 08:26:13 -0500 | [diff] [blame] | 483 | Download BMC Dump |
| 484 | [Documentation] Download BMC dump and verify its size. |
| 485 | [Arguments] ${dump_id} |
| 486 | |
| 487 | # Description of argument(s): |
| 488 | # dump_id An integer value that identifies a particular dump (e.g. 1, 3). |
| 489 | |
| 490 | ${resp}= Redfish.Get /redfish/v1/Managers/bmc/LogServices/Dump/Entries/${dump_id} |
| 491 | ${redfish_bmc_dump_size}= Set Variable ${resp.dict["AdditionalDataSizeBytes"]} |
| 492 | |
| 493 | Initialize OpenBMC |
| 494 | ${headers}= Create Dictionary Content-Type=application/octet-stream X-Auth-Token=${XAUTH_TOKEN} |
| 495 | |
manashsarma | 12b5b79 | 2023-11-02 05:25:19 -0500 | [diff] [blame] | 496 | ${ret}= GET On Session openbmc /redfish/v1/Managers/bmc/LogServices/Dump/Entries/${dump_id}/attachment headers=${headers} |
manashsarma | da7fd92 | 2023-10-05 08:26:13 -0500 | [diff] [blame] | 497 | |
| 498 | Should Be Equal As Numbers ${ret.status_code} 200 |
| 499 | |
| 500 | Create Binary File BMC_dump.tar.gz ${ret.content} |
| 501 | ${downloaded_dump_size}= Get File Size BMC_dump.tar.gz |
| 502 | Should Be Equal ${downloaded_dump_size} ${redfish_bmc_dump_size} |
| 503 | [Return] BMC_dump.tar.gz |
| 504 | |
| 505 | |
manashsarma | 1202ffe | 2023-09-22 09:01:18 -0500 | [diff] [blame] | 506 | Extract BMC Dump |
| 507 | [Documentation] Extract BMC dump from the tar file and returns the name of |
| 508 | ... extracted folder like BMCDUMP.XXXXXXX.0000070.20230706063841. |
| 509 | [Arguments] ${filename} ${bmc_dump_timestamp} |
| 510 | |
| 511 | # Description of argument(s): |
| 512 | # filename name of BMC dump tar file. |
| 513 | # bmc_dump_timestamp timestamp of generated BMC dump. |
| 514 | |
manashsarma | fcc14df | 2023-09-20 11:37:50 -0500 | [diff] [blame] | 515 | OperatingSystem.File Should Exist ${filename} |
manashsarma | 1202ffe | 2023-09-22 09:01:18 -0500 | [diff] [blame] | 516 | ${rc}= Run And Return RC dd if=${filename} of=output.zst bs=1 skip=628 |
| 517 | Should Be True 0 == ${rc} |
| 518 | |
| 519 | ${rc}= Run And Return RC zstd -d output.zst |
| 520 | Should Be True 0 == ${rc} |
| 521 | |
| 522 | ${rc}= Run And Return RC tar -xvf output |
| 523 | Should Be True 0 == ${rc} |
| 524 | |
| 525 | # Find the extracted dump folder identified with BMCDUMP as prefix and |
| 526 | # timestamp of dump generation where timestamp format is : 2023-09-27T08:30:17.000000+00:00. |
| 527 | ${var}= Fetch From Left ${bmc_dump_timestamp} . |
| 528 | ${var}= Remove String ${var} - T : |
| 529 | ${bmc_extraction_folders}= OperatingSystem.List Directories In Directory . BMCDUMP*${var} |
| 530 | ${cnt}= Get length ${bmc_extraction_folders} |
| 531 | should be equal as numbers ${cnt} 1 |
| 532 | |
| 533 | [Return] ${bmc_extraction_folders}[0] |
| 534 | |
| 535 | |
Rahul Maheshwari | bcefdf2 | 2020-10-16 07:51:34 -0500 | [diff] [blame] | 536 | Get BMC Dump Entries |
| 537 | [Documentation] Return BMC dump ids list. |
| 538 | |
ganesanb | 4d43028 | 2023-04-27 14:33:23 +0000 | [diff] [blame] | 539 | ${dump_uris}= redfish_utils.get_member_list /redfish/v1/Managers/${MANAGER_ID}/LogServices/Dump/Entries |
Rahul Maheshwari | bcefdf2 | 2020-10-16 07:51:34 -0500 | [diff] [blame] | 540 | ${dump_ids}= Create List |
| 541 | |
| 542 | FOR ${dump_uri} IN @{dump_uris} |
| 543 | ${dump_id}= Fetch From Right ${dump_uri} / |
| 544 | Append To List ${dump_ids} ${dump_id} |
| 545 | END |
| 546 | |
| 547 | [Return] ${dump_ids} |
| 548 | |
| 549 | |
manashsarma | 633cac9 | 2022-11-25 03:27:03 -0600 | [diff] [blame] | 550 | Is BMC Dump Available |
| 551 | [Documentation] Verify if BMC dump is available. |
| 552 | |
| 553 | ${dump_entries}= Get BMC Dump Entries |
| 554 | |
| 555 | # Verifying that BMC dump is available. |
| 556 | ${length}= Get length ${dump_entries} |
| 557 | Should Be True 0 < ${length} |
| 558 | |
| 559 | |
Rahul Maheshwari | d709c8d | 2020-11-01 23:03:51 -0600 | [diff] [blame] | 560 | Get Disk Usage For Dumps |
| 561 | [Documentation] Return disk usage in kilobyte for BMC dumps. |
| 562 | |
George Keishing | 952bb0f | 2022-12-12 03:13:35 -0600 | [diff] [blame] | 563 | ${usage_output} ${stderr} ${rc}= BMC Execute Command du -s ${BMC_DUMP_COLLECTOR_PATH} |
Rahul Maheshwari | d709c8d | 2020-11-01 23:03:51 -0600 | [diff] [blame] | 564 | |
| 565 | # Example of output from above BMC cli command. |
| 566 | # $ du -s /var/lib/phosphor-debug-collector/dumps |
| 567 | # 516 /var/lib/phosphor-debug-collector/dumps |
| 568 | |
| 569 | ${usage_output}= Fetch From Left ${usage_output} / |
| 570 | ${usage_output}= Convert To Integer ${usage_output} |
| 571 | |
| 572 | [return] ${usage_output} |
| 573 | |
| 574 | |
Rahul Maheshwari | 2568f87 | 2020-09-23 01:03:39 -0500 | [diff] [blame] | 575 | Test Teardown Execution |
| 576 | [Documentation] Do test teardown operation. |
| 577 | |
| 578 | FFDC On Test Case Fail |
| 579 | Close All Connections |