George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 1 | *** Settings *** |
| 2 | Documentation This module will take whatever action is necessary |
| 3 | ... to bring the BMC to a stable, standby state. For our |
| 4 | ... purposes, a stable state is defined as: |
| 5 | ... - BMC is communicating |
| 6 | ... (pinging, sshing and REST commands working) |
| 7 | ... - Power state is 0 (off) |
Rahul Maheshwari | 2c72504 | 2017-01-29 22:55:28 -0600 | [diff] [blame] | 8 | ... - BMC state is "Ready" |
| 9 | ... - HOST state is "Off" |
George Keishing | efc3ff2 | 2017-12-12 11:49:25 -0600 | [diff] [blame] | 10 | ... - Boot policy is "ALWAYS_POWER_OFF" |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 11 | ... Power cycle system via PDU if specified |
George Keishing | 0bd59f1 | 2016-12-14 10:39:21 -0600 | [diff] [blame] | 12 | ... Prune archived journal logs |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 13 | |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 14 | Resource ../lib/utils.robot |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 15 | Resource ../lib/pdu/pdu.robot |
Rahul Maheshwari | 9bd3abc | 2017-02-07 00:51:30 -0600 | [diff] [blame] | 16 | Resource ../lib/state_manager.robot |
Michael Walsh | 1b0ceb2 | 2017-05-31 16:03:01 -0500 | [diff] [blame] | 17 | Resource ../lib/bmc_network_utils.robot |
Sunil M | ec6fe31 | 2017-06-19 04:21:25 -0500 | [diff] [blame] | 18 | Resource ../lib/bmc_cleanup.robot |
George Keishing | 27a1f20 | 2017-08-30 22:40:26 -0500 | [diff] [blame] | 19 | Resource ../lib/dump_utils.robot |
George Keishing | caa718b | 2019-03-10 00:08:33 -0600 | [diff] [blame] | 20 | Resource ../lib/bmc_redfish_resource.robot |
| 21 | Resource ../lib/bmc_redfish_utils.robot |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 22 | Library ../lib/gen_misc.py |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 23 | |
| 24 | *** Variables *** |
| 25 | ${HOST_SETTING} /org/openbmc/settings/host0 |
| 26 | |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 27 | ${ERROR_REGEX} xyz.openbmc_project.Software.BMC.Updater.service: Failed with result 'core-dump' |
| 28 | |
Gunnar Mills | 7c8923f | 2016-12-12 21:19:52 -0600 | [diff] [blame] | 29 | *** Test Cases *** |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 30 | |
George Keishing | 62f9430 | 2017-01-31 07:00:52 -0600 | [diff] [blame] | 31 | Get To Stable State |
Rahul Maheshwari | 2c72504 | 2017-01-29 22:55:28 -0600 | [diff] [blame] | 32 | [Documentation] BMC cleanup drive to stable state |
| 33 | ... 1. PDU powercycle if specified |
| 34 | ... 1. Ping Test |
| 35 | ... 2. SSH Connection session Test |
| 36 | ... 3. REST Connection session Test |
| 37 | ... 4. Reboot BMC if REST Test failed |
| 38 | ... 5. Get BMC in Ready state if its not in this state |
| 39 | ... 6. Get Host in Off state if its not in this state |
| 40 | ... 7. Update restore policy |
Sunil M | 89fb0f5 | 2017-05-26 05:25:33 -0500 | [diff] [blame] | 41 | ... 8. Verify and Update MAC address. |
George Keishing | 62f9430 | 2017-01-31 07:00:52 -0600 | [diff] [blame] | 42 | [Tags] Get_To_Stable_State |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 43 | |
Rahul Maheshwari | 2c72504 | 2017-01-29 22:55:28 -0600 | [diff] [blame] | 44 | Run Keyword And Ignore Error Powercycle System Via PDU |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 45 | |
George Keishing | 13497eb | 2018-11-07 06:35:14 -0600 | [diff] [blame] | 46 | ${ping_status}= Run Keyword And Return Status |
| 47 | ... Wait For Host To Ping ${OPENBMC_HOST} 2 mins |
| 48 | |
| 49 | # Check if the ping works using 1400 MTU. |
George Keishing | e31e223 | 2020-10-05 12:25:25 -0500 | [diff] [blame] | 50 | #Run Keyword if ${ping_status} == ${True} MTU Ping Test |
George Keishing | 13497eb | 2018-11-07 06:35:14 -0600 | [diff] [blame] | 51 | |
| 52 | Run Keyword if ${ping_status} == ${False} |
| 53 | ... Fail ${OPENBMC_HOST} ping test failed. |
| 54 | |
| 55 | Open Connection And Log In host=${OPENBMC_HOST} |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 56 | |
George Keishing | de2acae | 2020-06-17 10:00:37 -0500 | [diff] [blame] | 57 | Run Keyword If ${REDFISH_SUPPORTED} Redfish.Login |
| 58 | |
George Keishing | 59fa048 | 2017-03-02 10:01:09 -0600 | [diff] [blame] | 59 | Wait Until Keyword Succeeds |
| 60 | ... 1 min 30 sec Initialize OpenBMC |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 61 | |
Rahul Maheshwari | 9bd3abc | 2017-02-07 00:51:30 -0600 | [diff] [blame] | 62 | ${ready_status}= Run Keyword And Return Status Is BMC Ready |
| 63 | Run Keyword If '${ready_status}' == '${False}' Put BMC State Ready |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 64 | |
Rahul Maheshwari | 9bd3abc | 2017-02-07 00:51:30 -0600 | [diff] [blame] | 65 | ${host_off_status}= Run Keyword And Return Status Is Host Off |
| 66 | Run Keyword If '${host_off_status}' == '${False}' Initiate Host PowerOff |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 67 | |
George Keishing | 0bd59f1 | 2016-12-14 10:39:21 -0600 | [diff] [blame] | 68 | Prune Journal Log |
| 69 | |
George Keishing | efc3ff2 | 2017-12-12 11:49:25 -0600 | [diff] [blame] | 70 | Run Keyword And Ignore Error Set BMC Power Policy ${ALWAYS_POWER_OFF} |
George Keishing | de2acae | 2020-06-17 10:00:37 -0500 | [diff] [blame] | 71 | Run Keyword And Ignore Error Redfish Set Power Restore Policy AlwaysOff |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 72 | |
Sunil M | 5dc827d | 2017-10-17 03:49:31 -0500 | [diff] [blame] | 73 | Run Keyword And Ignore Error Delete All Error Logs |
George Keishing | cda2c45 | 2020-02-13 09:15:58 -0600 | [diff] [blame] | 74 | Run Keyword And Ignore Error Redfish Purge Event Log |
Sunil M | 5dc827d | 2017-10-17 03:49:31 -0500 | [diff] [blame] | 75 | Run Keyword And Ignore Error Delete All Dumps |
George Keishing | 033a764 | 2020-10-14 02:41:43 -0500 | [diff] [blame^] | 76 | Run Keyword And Ignore Error Redfish Delete All BMC Dumps |
George Keishing | caa718b | 2019-03-10 00:08:33 -0600 | [diff] [blame] | 77 | Run Keyword And Ignore Error Delete All Redfish Sessions |
Charles Paul Hofer | 7cdb8b4 | 2018-09-10 12:12:04 -0500 | [diff] [blame] | 78 | Check For Current Boot Application Failures |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 79 | |
| 80 | *** Keywords *** |
| 81 | |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 82 | BMC Online Test |
| 83 | [Documentation] BMC ping, SSH, REST connection Test |
| 84 | |
Gunnar Mills | 597ffa0 | 2016-12-06 11:26:53 -0600 | [diff] [blame] | 85 | ${l_status}= Run Keyword and Return Status |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 86 | ... Verify Ping and REST Authentication |
| 87 | Run Keyword If '${l_status}' == '${False}' |
| 88 | ... Fail msg=System not in ideal state to continue [ERROR] |
| 89 | |
| 90 | |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 91 | Update Policy Setting |
| 92 | [Documentation] Update the given restore policy |
Gunnar Mills | 3803280 | 2016-12-12 13:43:40 -0600 | [diff] [blame] | 93 | [Arguments] ${policy} |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 94 | |
Gunnar Mills | 597ffa0 | 2016-12-06 11:26:53 -0600 | [diff] [blame] | 95 | ${valueDict}= create dictionary data=${policy} |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 96 | Write Attribute ${HOST_SETTING} power_policy data=${valueDict} |
| 97 | ${currentPolicy}= Read Attribute ${HOST_SETTING} power_policy |
| 98 | Should Be Equal ${currentPolicy} ${policy} |
| 99 | |
George Keishing | d5123f7 | 2016-12-07 09:54:51 -0600 | [diff] [blame] | 100 | |
| 101 | Trigger Warm Reset via Reboot |
| 102 | [Documentation] Execute reboot command on the remote BMC and |
| 103 | ... returns immediately. This keyword "Start Command" |
| 104 | ... returns nothing and does not wait for the command |
| 105 | ... execution to be finished. |
| 106 | Open Connection And Log In |
| 107 | |
| 108 | Start Command /sbin/reboot |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 109 | |
| 110 | |
| 111 | Powercycle System Via PDU |
Joy Onyerikwu | f4a807b | 2018-06-20 08:43:54 -0500 | [diff] [blame] | 112 | [Documentation] AC cycle the system via PDU. |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 113 | |
| 114 | Validate Parameters |
| 115 | PDU Power Cycle |
| 116 | Check If BMC is Up 5 min 10 sec |
| 117 | |
| 118 | |
Charles Paul Hofer | 7cdb8b4 | 2018-09-10 12:12:04 -0500 | [diff] [blame] | 119 | Check For Current Boot Application Failures |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 120 | [Documentation] Parse the journal log and check for failures. |
| 121 | [Arguments] ${error_regex}=${ERROR_REGEX} |
| 122 | |
| 123 | ${error_regex}= Escape Bash Quotes ${error_regex} |
| 124 | ${journal_log} ${stderr} ${rc}= BMC Execute Command |
Charles Paul Hofer | bcae08b | 2018-08-09 11:33:33 -0500 | [diff] [blame] | 125 | ... journalctl -b --no-pager | egrep '${error_regex}' ignore_err=1 |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 126 | |
| 127 | Should Be Empty ${journal_log} |
| 128 | |
| 129 | |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 130 | Validate Parameters |
Joy Onyerikwu | f4a807b | 2018-06-20 08:43:54 -0500 | [diff] [blame] | 131 | [Documentation] Validate PDU parameters. |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 132 | Should Not Be Empty ${PDU_IP} |
| 133 | Should Not Be Empty ${PDU_TYPE} |
| 134 | Should Not Be Empty ${PDU_SLOT_NO} |
| 135 | Should Not Be Empty ${PDU_USERNAME} |
| 136 | Should Not Be Empty ${PDU_PASSWORD} |
George Keishing | 13497eb | 2018-11-07 06:35:14 -0600 | [diff] [blame] | 137 | |
| 138 | |
| 139 | MTU Ping Test |
| 140 | [Documentation] Ping test using MTU. |
| 141 | [Arguments] ${mtu}=${1400} |
| 142 | |
| 143 | # Description of argument(s): |
| 144 | # mtu The maximum transmission unit (MTU) of a network interface. |
| 145 | |
| 146 | ${rc} ${output}= Run And Return RC And Output |
| 147 | ... ping -M do -s ${mtu} -c 10 ${OPENBMC_HOST} |
| 148 | Should Be Equal As Integers ${rc} 0 |
| 149 | Should Not Contain ${output} 100% packet loss |