George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 1 | *** Settings *** |
| 2 | Documentation This module will take whatever action is necessary |
| 3 | ... to bring the BMC to a stable, standby state. For our |
| 4 | ... purposes, a stable state is defined as: |
| 5 | ... - BMC is communicating |
| 6 | ... (pinging, sshing and REST commands working) |
| 7 | ... - Power state is 0 (off) |
Rahul Maheshwari | 2c72504 | 2017-01-29 22:55:28 -0600 | [diff] [blame] | 8 | ... - BMC state is "Ready" |
| 9 | ... - HOST state is "Off" |
George Keishing | efc3ff2 | 2017-12-12 11:49:25 -0600 | [diff] [blame] | 10 | ... - Boot policy is "ALWAYS_POWER_OFF" |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 11 | ... Power cycle system via PDU if specified |
George Keishing | 0bd59f1 | 2016-12-14 10:39:21 -0600 | [diff] [blame] | 12 | ... Prune archived journal logs |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 13 | |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 14 | Resource ../lib/utils.robot |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 15 | Resource ../lib/pdu/pdu.robot |
Rahul Maheshwari | 9bd3abc | 2017-02-07 00:51:30 -0600 | [diff] [blame] | 16 | Resource ../lib/state_manager.robot |
Michael Walsh | 1b0ceb2 | 2017-05-31 16:03:01 -0500 | [diff] [blame] | 17 | Resource ../lib/bmc_network_utils.robot |
Sunil M | ec6fe31 | 2017-06-19 04:21:25 -0500 | [diff] [blame] | 18 | Resource ../lib/bmc_cleanup.robot |
George Keishing | 27a1f20 | 2017-08-30 22:40:26 -0500 | [diff] [blame] | 19 | Resource ../lib/dump_utils.robot |
George Keishing | caa718b | 2019-03-10 00:08:33 -0600 | [diff] [blame] | 20 | Resource ../lib/bmc_redfish_resource.robot |
| 21 | Resource ../lib/bmc_redfish_utils.robot |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 22 | Library ../lib/gen_misc.py |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 23 | |
George Keishing | 8dd04af | 2021-03-23 10:36:40 -0500 | [diff] [blame] | 24 | # Force the test to timedout to prevent test hanging. |
| 25 | Test Timeout 10 minutes |
| 26 | |
| 27 | |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 28 | *** Variables *** |
| 29 | ${HOST_SETTING} /org/openbmc/settings/host0 |
| 30 | |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 31 | ${ERROR_REGEX} xyz.openbmc_project.Software.BMC.Updater.service: Failed with result 'core-dump' |
| 32 | |
Gunnar Mills | 7c8923f | 2016-12-12 21:19:52 -0600 | [diff] [blame] | 33 | *** Test Cases *** |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 34 | |
George Keishing | 62f9430 | 2017-01-31 07:00:52 -0600 | [diff] [blame] | 35 | Get To Stable State |
Rahul Maheshwari | 2c72504 | 2017-01-29 22:55:28 -0600 | [diff] [blame] | 36 | [Documentation] BMC cleanup drive to stable state |
| 37 | ... 1. PDU powercycle if specified |
| 38 | ... 1. Ping Test |
| 39 | ... 2. SSH Connection session Test |
| 40 | ... 3. REST Connection session Test |
| 41 | ... 4. Reboot BMC if REST Test failed |
| 42 | ... 5. Get BMC in Ready state if its not in this state |
| 43 | ... 6. Get Host in Off state if its not in this state |
| 44 | ... 7. Update restore policy |
Sunil M | 89fb0f5 | 2017-05-26 05:25:33 -0500 | [diff] [blame] | 45 | ... 8. Verify and Update MAC address. |
George Keishing | 62f9430 | 2017-01-31 07:00:52 -0600 | [diff] [blame] | 46 | [Tags] Get_To_Stable_State |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 47 | |
Rahul Maheshwari | 2c72504 | 2017-01-29 22:55:28 -0600 | [diff] [blame] | 48 | Run Keyword And Ignore Error Powercycle System Via PDU |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 49 | |
George Keishing | 13497eb | 2018-11-07 06:35:14 -0600 | [diff] [blame] | 50 | ${ping_status}= Run Keyword And Return Status |
| 51 | ... Wait For Host To Ping ${OPENBMC_HOST} 2 mins |
| 52 | |
| 53 | # Check if the ping works using 1400 MTU. |
George Keishing | e31e223 | 2020-10-05 12:25:25 -0500 | [diff] [blame] | 54 | #Run Keyword if ${ping_status} == ${True} MTU Ping Test |
George Keishing | 13497eb | 2018-11-07 06:35:14 -0600 | [diff] [blame] | 55 | |
| 56 | Run Keyword if ${ping_status} == ${False} |
| 57 | ... Fail ${OPENBMC_HOST} ping test failed. |
| 58 | |
| 59 | Open Connection And Log In host=${OPENBMC_HOST} |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 60 | |
George Keishing | 8dd04af | 2021-03-23 10:36:40 -0500 | [diff] [blame] | 61 | Run Keyword If ${REDFISH_SUPPORTED} |
| 62 | ... Redfish Clean Up |
George Keishing | 334df29 | 2021-02-08 05:08:47 -0600 | [diff] [blame] | 63 | ... ELSE |
George Keishing | 8dd04af | 2021-03-23 10:36:40 -0500 | [diff] [blame] | 64 | ... REST Clean Up |
| 65 | |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 66 | |
George Keishing | 0bd59f1 | 2016-12-14 10:39:21 -0600 | [diff] [blame] | 67 | Prune Journal Log |
Charles Paul Hofer | 7cdb8b4 | 2018-09-10 12:12:04 -0500 | [diff] [blame] | 68 | Check For Current Boot Application Failures |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 69 | |
| 70 | *** Keywords *** |
| 71 | |
George Keishing | 8dd04af | 2021-03-23 10:36:40 -0500 | [diff] [blame] | 72 | |
| 73 | REST Clean Up |
| 74 | [Documentation] Check states, reboot if needed and poweroff. |
| 75 | |
| 76 | Wait Until Keyword Succeeds 1 min 30 sec Initialize OpenBMC |
| 77 | |
| 78 | ${ready_status}= Run Keyword And Return Status Is BMC Ready |
| 79 | Run Keyword If '${ready_status}' == '${False}' |
| 80 | ... Put BMC State Ready |
| 81 | ... ELSE |
| 82 | ... REST Power Off stack_mode=skip |
| 83 | |
| 84 | Run Keyword And Ignore Error Set BMC Power Policy ${ALWAYS_POWER_OFF} |
| 85 | Run Keyword And Ignore Error Delete All Error Logs |
| 86 | Run Keyword And Ignore Error Delete All Dumps |
| 87 | |
| 88 | |
| 89 | Redfish Clean Up |
| 90 | [Documentation] Check states, reboot if needed and poweroff. |
| 91 | |
| 92 | Wait Until Keyword Succeeds 1 min 30 sec Redfish.Login |
| 93 | |
| 94 | Redfish Power Off stack_mode=skip |
| 95 | |
| 96 | Run Keyword And Ignore Error Redfish Set Power Restore Policy AlwaysOff |
| 97 | Run Keyword And Ignore Error Redfish Purge Event Log |
| 98 | Run Keyword And Ignore Error Redfish Delete All BMC Dumps |
George Keishing | 2ef6a7d | 2021-05-19 09:05:32 -0500 | [diff] [blame] | 99 | Run Keyword And Ignore Error Redfish Delete All System Dumps |
George Keishing | 8dd04af | 2021-03-23 10:36:40 -0500 | [diff] [blame] | 100 | Run Keyword And Ignore Error Delete All Redfish Sessions |
| 101 | |
| 102 | |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 103 | BMC Online Test |
| 104 | [Documentation] BMC ping, SSH, REST connection Test |
| 105 | |
Gunnar Mills | 597ffa0 | 2016-12-06 11:26:53 -0600 | [diff] [blame] | 106 | ${l_status}= Run Keyword and Return Status |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 107 | ... Verify Ping and REST Authentication |
| 108 | Run Keyword If '${l_status}' == '${False}' |
| 109 | ... Fail msg=System not in ideal state to continue [ERROR] |
| 110 | |
| 111 | |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 112 | Update Policy Setting |
| 113 | [Documentation] Update the given restore policy |
Gunnar Mills | 3803280 | 2016-12-12 13:43:40 -0600 | [diff] [blame] | 114 | [Arguments] ${policy} |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 115 | |
Gunnar Mills | 597ffa0 | 2016-12-06 11:26:53 -0600 | [diff] [blame] | 116 | ${valueDict}= create dictionary data=${policy} |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 117 | Write Attribute ${HOST_SETTING} power_policy data=${valueDict} |
| 118 | ${currentPolicy}= Read Attribute ${HOST_SETTING} power_policy |
| 119 | Should Be Equal ${currentPolicy} ${policy} |
| 120 | |
George Keishing | d5123f7 | 2016-12-07 09:54:51 -0600 | [diff] [blame] | 121 | |
| 122 | Trigger Warm Reset via Reboot |
| 123 | [Documentation] Execute reboot command on the remote BMC and |
| 124 | ... returns immediately. This keyword "Start Command" |
| 125 | ... returns nothing and does not wait for the command |
| 126 | ... execution to be finished. |
| 127 | Open Connection And Log In |
| 128 | |
| 129 | Start Command /sbin/reboot |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 130 | |
| 131 | |
| 132 | Powercycle System Via PDU |
Joy Onyerikwu | f4a807b | 2018-06-20 08:43:54 -0500 | [diff] [blame] | 133 | [Documentation] AC cycle the system via PDU. |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 134 | |
| 135 | Validate Parameters |
| 136 | PDU Power Cycle |
| 137 | Check If BMC is Up 5 min 10 sec |
| 138 | |
| 139 | |
Charles Paul Hofer | 7cdb8b4 | 2018-09-10 12:12:04 -0500 | [diff] [blame] | 140 | Check For Current Boot Application Failures |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 141 | [Documentation] Parse the journal log and check for failures. |
| 142 | [Arguments] ${error_regex}=${ERROR_REGEX} |
| 143 | |
| 144 | ${error_regex}= Escape Bash Quotes ${error_regex} |
| 145 | ${journal_log} ${stderr} ${rc}= BMC Execute Command |
Charles Paul Hofer | bcae08b | 2018-08-09 11:33:33 -0500 | [diff] [blame] | 146 | ... journalctl -b --no-pager | egrep '${error_regex}' ignore_err=1 |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 147 | |
| 148 | Should Be Empty ${journal_log} |
| 149 | |
| 150 | |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 151 | Validate Parameters |
Joy Onyerikwu | f4a807b | 2018-06-20 08:43:54 -0500 | [diff] [blame] | 152 | [Documentation] Validate PDU parameters. |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 153 | Should Not Be Empty ${PDU_IP} |
| 154 | Should Not Be Empty ${PDU_TYPE} |
| 155 | Should Not Be Empty ${PDU_SLOT_NO} |
| 156 | Should Not Be Empty ${PDU_USERNAME} |
| 157 | Should Not Be Empty ${PDU_PASSWORD} |
George Keishing | 13497eb | 2018-11-07 06:35:14 -0600 | [diff] [blame] | 158 | |
| 159 | |
| 160 | MTU Ping Test |
| 161 | [Documentation] Ping test using MTU. |
| 162 | [Arguments] ${mtu}=${1400} |
| 163 | |
| 164 | # Description of argument(s): |
| 165 | # mtu The maximum transmission unit (MTU) of a network interface. |
| 166 | |
| 167 | ${rc} ${output}= Run And Return RC And Output |
| 168 | ... ping -M do -s ${mtu} -c 10 ${OPENBMC_HOST} |
| 169 | Should Be Equal As Integers ${rc} 0 |
| 170 | Should Not Contain ${output} 100% packet loss |