George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 1 | *** Settings *** |
| 2 | Documentation This module will take whatever action is necessary |
| 3 | ... to bring the BMC to a stable, standby state. For our |
| 4 | ... purposes, a stable state is defined as: |
| 5 | ... - BMC is communicating |
| 6 | ... (pinging, sshing and REST commands working) |
| 7 | ... - Power state is 0 (off) |
Rahul Maheshwari | 2c72504 | 2017-01-29 22:55:28 -0600 | [diff] [blame] | 8 | ... - BMC state is "Ready" |
| 9 | ... - HOST state is "Off" |
George Keishing | efc3ff2 | 2017-12-12 11:49:25 -0600 | [diff] [blame] | 10 | ... - Boot policy is "ALWAYS_POWER_OFF" |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 11 | ... Power cycle system via PDU if specified |
George Keishing | 0bd59f1 | 2016-12-14 10:39:21 -0600 | [diff] [blame] | 12 | ... Prune archived journal logs |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 13 | |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 14 | Resource ../lib/utils.robot |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 15 | Resource ../lib/pdu/pdu.robot |
Rahul Maheshwari | 9bd3abc | 2017-02-07 00:51:30 -0600 | [diff] [blame] | 16 | Resource ../lib/state_manager.robot |
Michael Walsh | 1b0ceb2 | 2017-05-31 16:03:01 -0500 | [diff] [blame] | 17 | Resource ../lib/bmc_network_utils.robot |
Sunil M | ec6fe31 | 2017-06-19 04:21:25 -0500 | [diff] [blame] | 18 | Resource ../lib/bmc_cleanup.robot |
George Keishing | 27a1f20 | 2017-08-30 22:40:26 -0500 | [diff] [blame] | 19 | Resource ../lib/dump_utils.robot |
George Keishing | caa718b | 2019-03-10 00:08:33 -0600 | [diff] [blame] | 20 | Resource ../lib/bmc_redfish_resource.robot |
| 21 | Resource ../lib/bmc_redfish_utils.robot |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 22 | Library ../lib/gen_misc.py |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 23 | |
| 24 | *** Variables *** |
| 25 | ${HOST_SETTING} /org/openbmc/settings/host0 |
| 26 | |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 27 | ${ERROR_REGEX} xyz.openbmc_project.Software.BMC.Updater.service: Failed with result 'core-dump' |
| 28 | |
Gunnar Mills | 7c8923f | 2016-12-12 21:19:52 -0600 | [diff] [blame] | 29 | *** Test Cases *** |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 30 | |
George Keishing | 62f9430 | 2017-01-31 07:00:52 -0600 | [diff] [blame] | 31 | Get To Stable State |
Rahul Maheshwari | 2c72504 | 2017-01-29 22:55:28 -0600 | [diff] [blame] | 32 | [Documentation] BMC cleanup drive to stable state |
| 33 | ... 1. PDU powercycle if specified |
| 34 | ... 1. Ping Test |
| 35 | ... 2. SSH Connection session Test |
| 36 | ... 3. REST Connection session Test |
| 37 | ... 4. Reboot BMC if REST Test failed |
| 38 | ... 5. Get BMC in Ready state if its not in this state |
| 39 | ... 6. Get Host in Off state if its not in this state |
| 40 | ... 7. Update restore policy |
Sunil M | 89fb0f5 | 2017-05-26 05:25:33 -0500 | [diff] [blame] | 41 | ... 8. Verify and Update MAC address. |
George Keishing | 62f9430 | 2017-01-31 07:00:52 -0600 | [diff] [blame] | 42 | [Tags] Get_To_Stable_State |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 43 | |
Rahul Maheshwari | 2c72504 | 2017-01-29 22:55:28 -0600 | [diff] [blame] | 44 | Run Keyword And Ignore Error Powercycle System Via PDU |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 45 | |
George Keishing | 13497eb | 2018-11-07 06:35:14 -0600 | [diff] [blame] | 46 | ${ping_status}= Run Keyword And Return Status |
| 47 | ... Wait For Host To Ping ${OPENBMC_HOST} 2 mins |
| 48 | |
| 49 | # Check if the ping works using 1400 MTU. |
| 50 | Run Keyword if ${ping_status} == ${True} MTU Ping Test |
| 51 | |
| 52 | Run Keyword if ${ping_status} == ${False} |
| 53 | ... Fail ${OPENBMC_HOST} ping test failed. |
| 54 | |
| 55 | Open Connection And Log In host=${OPENBMC_HOST} |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 56 | |
George Keishing | 59fa048 | 2017-03-02 10:01:09 -0600 | [diff] [blame] | 57 | Wait Until Keyword Succeeds |
| 58 | ... 1 min 30 sec Initialize OpenBMC |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 59 | |
Rahul Maheshwari | 9bd3abc | 2017-02-07 00:51:30 -0600 | [diff] [blame] | 60 | ${ready_status}= Run Keyword And Return Status Is BMC Ready |
| 61 | Run Keyword If '${ready_status}' == '${False}' Put BMC State Ready |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 62 | |
Rahul Maheshwari | 9bd3abc | 2017-02-07 00:51:30 -0600 | [diff] [blame] | 63 | ${host_off_status}= Run Keyword And Return Status Is Host Off |
| 64 | Run Keyword If '${host_off_status}' == '${False}' Initiate Host PowerOff |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 65 | |
George Keishing | 0bd59f1 | 2016-12-14 10:39:21 -0600 | [diff] [blame] | 66 | Prune Journal Log |
| 67 | |
George Keishing | efc3ff2 | 2017-12-12 11:49:25 -0600 | [diff] [blame] | 68 | Run Keyword And Ignore Error Set BMC Power Policy ${ALWAYS_POWER_OFF} |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 69 | |
George Keishing | 6bbce53 | 2017-09-21 13:03:55 -0500 | [diff] [blame] | 70 | # TODO: Enable MAC AES check latter. |
| 71 | # Reference : openbmc/openbmc-test-automation#998 |
| 72 | #Run Keyword If '${MAC_ADDRESS}' != '${EMPTY}' |
| 73 | #... Check And Reset MAC |
| 74 | |
| 75 | # TODO: Add new UBI File system cleanup. |
| 76 | # Reference : openbmc/openbmc-test-automation#998 |
| 77 | #${rc}= Execute Command find ${CLEANUP_DIR_PATH} |
| 78 | #... return_stdout=False return_rc=True |
| 79 | #Run Keyword If '${CLEANUP_DIR_PATH}' != '${EMPTY}' and ${rc} == 0 |
| 80 | #... Cleanup Dir |
Sunil M | ec6fe31 | 2017-06-19 04:21:25 -0500 | [diff] [blame] | 81 | |
Sunil M | 5dc827d | 2017-10-17 03:49:31 -0500 | [diff] [blame] | 82 | Run Keyword And Ignore Error Delete All Error Logs |
George Keishing | cda2c45 | 2020-02-13 09:15:58 -0600 | [diff] [blame] | 83 | Run Keyword And Ignore Error Redfish Purge Event Log |
Sunil M | 5dc827d | 2017-10-17 03:49:31 -0500 | [diff] [blame] | 84 | Run Keyword And Ignore Error Delete All Dumps |
George Keishing | caa718b | 2019-03-10 00:08:33 -0600 | [diff] [blame] | 85 | Run Keyword And Ignore Error Delete All Redfish Sessions |
Charles Paul Hofer | 7cdb8b4 | 2018-09-10 12:12:04 -0500 | [diff] [blame] | 86 | Check For Current Boot Application Failures |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 87 | |
| 88 | *** Keywords *** |
| 89 | |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 90 | BMC Online Test |
| 91 | [Documentation] BMC ping, SSH, REST connection Test |
| 92 | |
Gunnar Mills | 597ffa0 | 2016-12-06 11:26:53 -0600 | [diff] [blame] | 93 | ${l_status}= Run Keyword and Return Status |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 94 | ... Verify Ping and REST Authentication |
| 95 | Run Keyword If '${l_status}' == '${False}' |
| 96 | ... Fail msg=System not in ideal state to continue [ERROR] |
| 97 | |
| 98 | |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 99 | Update Policy Setting |
| 100 | [Documentation] Update the given restore policy |
Gunnar Mills | 3803280 | 2016-12-12 13:43:40 -0600 | [diff] [blame] | 101 | [Arguments] ${policy} |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 102 | |
Gunnar Mills | 597ffa0 | 2016-12-06 11:26:53 -0600 | [diff] [blame] | 103 | ${valueDict}= create dictionary data=${policy} |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 104 | Write Attribute ${HOST_SETTING} power_policy data=${valueDict} |
| 105 | ${currentPolicy}= Read Attribute ${HOST_SETTING} power_policy |
| 106 | Should Be Equal ${currentPolicy} ${policy} |
| 107 | |
George Keishing | d5123f7 | 2016-12-07 09:54:51 -0600 | [diff] [blame] | 108 | |
| 109 | Trigger Warm Reset via Reboot |
| 110 | [Documentation] Execute reboot command on the remote BMC and |
| 111 | ... returns immediately. This keyword "Start Command" |
| 112 | ... returns nothing and does not wait for the command |
| 113 | ... execution to be finished. |
| 114 | Open Connection And Log In |
| 115 | |
| 116 | Start Command /sbin/reboot |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 117 | |
| 118 | |
| 119 | Powercycle System Via PDU |
Joy Onyerikwu | f4a807b | 2018-06-20 08:43:54 -0500 | [diff] [blame] | 120 | [Documentation] AC cycle the system via PDU. |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 121 | |
| 122 | Validate Parameters |
| 123 | PDU Power Cycle |
| 124 | Check If BMC is Up 5 min 10 sec |
| 125 | |
| 126 | |
Charles Paul Hofer | 7cdb8b4 | 2018-09-10 12:12:04 -0500 | [diff] [blame] | 127 | Check For Current Boot Application Failures |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 128 | [Documentation] Parse the journal log and check for failures. |
| 129 | [Arguments] ${error_regex}=${ERROR_REGEX} |
| 130 | |
| 131 | ${error_regex}= Escape Bash Quotes ${error_regex} |
| 132 | ${journal_log} ${stderr} ${rc}= BMC Execute Command |
Charles Paul Hofer | bcae08b | 2018-08-09 11:33:33 -0500 | [diff] [blame] | 133 | ... journalctl -b --no-pager | egrep '${error_regex}' ignore_err=1 |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 134 | |
| 135 | Should Be Empty ${journal_log} |
| 136 | |
| 137 | |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 138 | Validate Parameters |
Joy Onyerikwu | f4a807b | 2018-06-20 08:43:54 -0500 | [diff] [blame] | 139 | [Documentation] Validate PDU parameters. |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 140 | Should Not Be Empty ${PDU_IP} |
| 141 | Should Not Be Empty ${PDU_TYPE} |
| 142 | Should Not Be Empty ${PDU_SLOT_NO} |
| 143 | Should Not Be Empty ${PDU_USERNAME} |
| 144 | Should Not Be Empty ${PDU_PASSWORD} |
George Keishing | 13497eb | 2018-11-07 06:35:14 -0600 | [diff] [blame] | 145 | |
| 146 | |
| 147 | MTU Ping Test |
| 148 | [Documentation] Ping test using MTU. |
| 149 | [Arguments] ${mtu}=${1400} |
| 150 | |
| 151 | # Description of argument(s): |
| 152 | # mtu The maximum transmission unit (MTU) of a network interface. |
| 153 | |
| 154 | ${rc} ${output}= Run And Return RC And Output |
| 155 | ... ping -M do -s ${mtu} -c 10 ${OPENBMC_HOST} |
| 156 | Should Be Equal As Integers ${rc} 0 |
| 157 | Should Not Contain ${output} 100% packet loss |