| George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 1 | *** Settings *** | 
|  | 2 | Documentation     This module will take whatever action is necessary | 
|  | 3 | ...               to bring the BMC to a stable, standby state.  For our | 
|  | 4 | ...               purposes, a stable state is defined as: | 
|  | 5 | ...                  - BMC is communicating | 
|  | 6 | ...                   (pinging, sshing and REST commands working) | 
|  | 7 | ...                  - Power state is 0 (off) | 
| Rahul Maheshwari | 2c72504 | 2017-01-29 22:55:28 -0600 | [diff] [blame] | 8 | ...                  - BMC state is "Ready" | 
|  | 9 | ...                  - HOST state is "Off" | 
| George Keishing | efc3ff2 | 2017-12-12 11:49:25 -0600 | [diff] [blame] | 10 | ...                  - Boot policy is "ALWAYS_POWER_OFF" | 
| George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 11 | ...               Power cycle system via PDU if specified | 
| George Keishing | 0bd59f1 | 2016-12-14 10:39:21 -0600 | [diff] [blame] | 12 | ...               Prune archived journal logs | 
| George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 13 |  | 
| George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 14 | Resource          ../lib/utils.robot | 
| George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 15 | Resource          ../lib/pdu/pdu.robot | 
| Rahul Maheshwari | 9bd3abc | 2017-02-07 00:51:30 -0600 | [diff] [blame] | 16 | Resource          ../lib/state_manager.robot | 
| Michael Walsh | 1b0ceb2 | 2017-05-31 16:03:01 -0500 | [diff] [blame] | 17 | Resource          ../lib/bmc_network_utils.robot | 
| Sunil M | ec6fe31 | 2017-06-19 04:21:25 -0500 | [diff] [blame] | 18 | Resource          ../lib/bmc_cleanup.robot | 
| George Keishing | 27a1f20 | 2017-08-30 22:40:26 -0500 | [diff] [blame] | 19 | Resource          ../lib/dump_utils.robot | 
| George Keishing | caa718b | 2019-03-10 00:08:33 -0600 | [diff] [blame] | 20 | Resource          ../lib/bmc_redfish_resource.robot | 
|  | 21 | Resource          ../lib/bmc_redfish_utils.robot | 
| George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 22 | Library           ../lib/gen_misc.py | 
| George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 23 |  | 
| George Keishing | 8dd04af | 2021-03-23 10:36:40 -0500 | [diff] [blame] | 24 | # Force the test to timedout to prevent test hanging. | 
|  | 25 | Test Timeout      10 minutes | 
|  | 26 |  | 
|  | 27 |  | 
| George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 28 | *** Variables *** | 
|  | 29 | ${HOST_SETTING}      /org/openbmc/settings/host0 | 
|  | 30 |  | 
| George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 31 | ${ERROR_REGEX}  xyz.openbmc_project.Software.BMC.Updater.service: Failed with result 'core-dump' | 
|  | 32 |  | 
| Gunnar Mills | 7c8923f | 2016-12-12 21:19:52 -0600 | [diff] [blame] | 33 | *** Test Cases *** | 
| George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 34 |  | 
| George Keishing | 62f9430 | 2017-01-31 07:00:52 -0600 | [diff] [blame] | 35 | Get To Stable State | 
| Rahul Maheshwari | 2c72504 | 2017-01-29 22:55:28 -0600 | [diff] [blame] | 36 | [Documentation]  BMC cleanup drive to stable state | 
|  | 37 | ...              1. PDU powercycle if specified | 
|  | 38 | ...              1. Ping Test | 
|  | 39 | ...              2. SSH Connection session Test | 
|  | 40 | ...              3. REST Connection session Test | 
|  | 41 | ...              4. Reboot BMC if REST Test failed | 
|  | 42 | ...              5. Get BMC in Ready state if its not in this state | 
|  | 43 | ...              6. Get Host in Off state if its not in this state | 
|  | 44 | ...              7. Update restore policy | 
| Sunil M | 89fb0f5 | 2017-05-26 05:25:33 -0500 | [diff] [blame] | 45 | ...              8. Verify and Update MAC address. | 
| George Keishing | 62f9430 | 2017-01-31 07:00:52 -0600 | [diff] [blame] | 46 | [Tags]  Get_To_Stable_State | 
| George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 47 |  | 
| Rahul Maheshwari | 2c72504 | 2017-01-29 22:55:28 -0600 | [diff] [blame] | 48 | Run Keyword And Ignore Error  Powercycle System Via PDU | 
| George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 49 |  | 
| George Keishing | 13497eb | 2018-11-07 06:35:14 -0600 | [diff] [blame] | 50 | ${ping_status}=  Run Keyword And Return Status | 
|  | 51 | ...  Wait For Host To Ping  ${OPENBMC_HOST}  2 mins | 
|  | 52 |  | 
|  | 53 | # Check if the ping works using 1400 MTU. | 
| George Keishing | e31e223 | 2020-10-05 12:25:25 -0500 | [diff] [blame] | 54 | #Run Keyword if  ${ping_status} == ${True}  MTU Ping Test | 
| George Keishing | 13497eb | 2018-11-07 06:35:14 -0600 | [diff] [blame] | 55 |  | 
|  | 56 | Run Keyword if  ${ping_status} == ${False} | 
|  | 57 | ...  Fail  ${OPENBMC_HOST} ping test failed. | 
|  | 58 |  | 
|  | 59 | Open Connection And Log In  host=${OPENBMC_HOST} | 
| George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 60 |  | 
| George Keishing | 8dd04af | 2021-03-23 10:36:40 -0500 | [diff] [blame] | 61 | Run Keyword If  ${REDFISH_SUPPORTED} | 
|  | 62 | ...    Redfish Clean Up | 
| George Keishing | 334df29 | 2021-02-08 05:08:47 -0600 | [diff] [blame] | 63 | ...  ELSE | 
| George Keishing | 8dd04af | 2021-03-23 10:36:40 -0500 | [diff] [blame] | 64 | ...    REST Clean Up | 
|  | 65 |  | 
| George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 66 |  | 
| George Keishing | 0bd59f1 | 2016-12-14 10:39:21 -0600 | [diff] [blame] | 67 | Prune Journal Log | 
| Charles Paul Hofer | 7cdb8b4 | 2018-09-10 12:12:04 -0500 | [diff] [blame] | 68 | Check For Current Boot Application Failures | 
| George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 69 |  | 
|  | 70 | *** Keywords *** | 
|  | 71 |  | 
| George Keishing | 8dd04af | 2021-03-23 10:36:40 -0500 | [diff] [blame] | 72 |  | 
|  | 73 | REST Clean Up | 
|  | 74 | [Documentation]  Check states, reboot if needed and poweroff. | 
|  | 75 |  | 
|  | 76 | Wait Until Keyword Succeeds  1 min  30 sec  Initialize OpenBMC | 
|  | 77 |  | 
|  | 78 | ${ready_status}=  Run Keyword And Return Status  Is BMC Ready | 
|  | 79 | Run Keyword If  '${ready_status}' == '${False}' | 
|  | 80 | ...    Put BMC State  Ready | 
|  | 81 | ...  ELSE | 
|  | 82 | ...    REST Power Off  stack_mode=skip | 
|  | 83 |  | 
|  | 84 | Run Keyword And Ignore Error  Set BMC Power Policy  ${ALWAYS_POWER_OFF} | 
|  | 85 | Run Keyword And Ignore Error  Delete All Error Logs | 
|  | 86 | Run Keyword And Ignore Error  Delete All Dumps | 
|  | 87 |  | 
|  | 88 |  | 
|  | 89 | Redfish Clean Up | 
|  | 90 | [Documentation]  Check states, reboot if needed and poweroff. | 
|  | 91 |  | 
|  | 92 | Wait Until Keyword Succeeds  1 min  30 sec  Redfish.Login | 
|  | 93 |  | 
|  | 94 | Redfish Power Off  stack_mode=skip | 
|  | 95 |  | 
|  | 96 | Run Keyword And Ignore Error  Redfish Set Power Restore Policy  AlwaysOff | 
|  | 97 | Run Keyword And Ignore Error  Redfish Purge Event Log | 
|  | 98 | Run Keyword And Ignore Error  Redfish Delete All BMC Dumps | 
| George Keishing | 2ef6a7d | 2021-05-19 09:05:32 -0500 | [diff] [blame] | 99 | Run Keyword And Ignore Error  Redfish Delete All System Dumps | 
| George Keishing | 8dd04af | 2021-03-23 10:36:40 -0500 | [diff] [blame] | 100 | Run Keyword And Ignore Error  Delete All Redfish Sessions | 
|  | 101 |  | 
|  | 102 |  | 
| George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 103 | BMC Online Test | 
|  | 104 | [Documentation]   BMC ping, SSH, REST connection Test | 
|  | 105 |  | 
| Gunnar Mills | 597ffa0 | 2016-12-06 11:26:53 -0600 | [diff] [blame] | 106 | ${l_status}=   Run Keyword and Return Status | 
| George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 107 | ...   Verify Ping and REST Authentication | 
|  | 108 | Run Keyword If  '${l_status}' == '${False}' | 
|  | 109 | ...   Fail  msg=System not in ideal state to continue [ERROR] | 
|  | 110 |  | 
|  | 111 |  | 
| George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 112 | Update Policy Setting | 
|  | 113 | [Documentation]   Update the given restore policy | 
| Gunnar Mills | 3803280 | 2016-12-12 13:43:40 -0600 | [diff] [blame] | 114 | [Arguments]   ${policy} | 
| George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 115 |  | 
| Gunnar Mills | 597ffa0 | 2016-12-06 11:26:53 -0600 | [diff] [blame] | 116 | ${valueDict}=     create dictionary  data=${policy} | 
| George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 117 | Write Attribute    ${HOST_SETTING}    power_policy   data=${valueDict} | 
|  | 118 | ${currentPolicy}=  Read Attribute     ${HOST_SETTING}   power_policy | 
|  | 119 | Should Be Equal    ${currentPolicy}   ${policy} | 
|  | 120 |  | 
| George Keishing | d5123f7 | 2016-12-07 09:54:51 -0600 | [diff] [blame] | 121 |  | 
|  | 122 | Trigger Warm Reset via Reboot | 
|  | 123 | [Documentation]    Execute reboot command on the remote BMC and | 
|  | 124 | ...                returns immediately. This keyword "Start Command" | 
|  | 125 | ...                returns nothing and does not wait for the command | 
|  | 126 | ...                execution to be finished. | 
|  | 127 | Open Connection And Log In | 
|  | 128 |  | 
|  | 129 | Start Command   /sbin/reboot | 
| George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 130 |  | 
|  | 131 |  | 
|  | 132 | Powercycle System Via PDU | 
| Joy Onyerikwu | f4a807b | 2018-06-20 08:43:54 -0500 | [diff] [blame] | 133 | [Documentation]   AC cycle the system via PDU. | 
| George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 134 |  | 
|  | 135 | Validate Parameters | 
|  | 136 | PDU Power Cycle | 
|  | 137 | Check If BMC is Up   5 min    10 sec | 
|  | 138 |  | 
|  | 139 |  | 
| Charles Paul Hofer | 7cdb8b4 | 2018-09-10 12:12:04 -0500 | [diff] [blame] | 140 | Check For Current Boot Application Failures | 
| George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 141 | [Documentation]  Parse the journal log and check for failures. | 
|  | 142 | [Arguments]  ${error_regex}=${ERROR_REGEX} | 
|  | 143 |  | 
|  | 144 | ${error_regex}=  Escape Bash Quotes  ${error_regex} | 
|  | 145 | ${journal_log}  ${stderr}  ${rc}=  BMC Execute Command | 
| Charles Paul Hofer | bcae08b | 2018-08-09 11:33:33 -0500 | [diff] [blame] | 146 | ...  journalctl -b --no-pager | egrep '${error_regex}'  ignore_err=1 | 
| George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 147 |  | 
|  | 148 | Should Be Empty  ${journal_log} | 
|  | 149 |  | 
|  | 150 |  | 
| George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 151 | Validate Parameters | 
| Joy Onyerikwu | f4a807b | 2018-06-20 08:43:54 -0500 | [diff] [blame] | 152 | [Documentation]  Validate PDU parameters. | 
| George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 153 | Should Not Be Empty   ${PDU_IP} | 
|  | 154 | Should Not Be Empty   ${PDU_TYPE} | 
|  | 155 | Should Not Be Empty   ${PDU_SLOT_NO} | 
|  | 156 | Should Not Be Empty   ${PDU_USERNAME} | 
|  | 157 | Should Not Be Empty   ${PDU_PASSWORD} | 
| George Keishing | 13497eb | 2018-11-07 06:35:14 -0600 | [diff] [blame] | 158 |  | 
|  | 159 |  | 
|  | 160 | MTU Ping Test | 
|  | 161 | [Documentation]  Ping test using MTU. | 
|  | 162 | [Arguments]  ${mtu}=${1400} | 
|  | 163 |  | 
|  | 164 | # Description of argument(s): | 
|  | 165 | # mtu   The maximum transmission unit (MTU) of a network interface. | 
|  | 166 |  | 
|  | 167 | ${rc}  ${output}=  Run And Return RC And Output | 
|  | 168 | ...  ping -M do -s ${mtu} -c 10 ${OPENBMC_HOST} | 
|  | 169 | Should Be Equal As Integers  ${rc}  0 | 
|  | 170 | Should Not Contain  ${output}  100% packet loss |