George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 1 | *** Settings *** |
| 2 | Documentation This module will take whatever action is necessary |
| 3 | ... to bring the BMC to a stable, standby state. For our |
| 4 | ... purposes, a stable state is defined as: |
| 5 | ... - BMC is communicating |
| 6 | ... (pinging, sshing and REST commands working) |
| 7 | ... - Power state is 0 (off) |
Rahul Maheshwari | 2c72504 | 2017-01-29 22:55:28 -0600 | [diff] [blame] | 8 | ... - BMC state is "Ready" |
| 9 | ... - HOST state is "Off" |
George Keishing | efc3ff2 | 2017-12-12 11:49:25 -0600 | [diff] [blame] | 10 | ... - Boot policy is "ALWAYS_POWER_OFF" |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 11 | ... Power cycle system via PDU if specified |
George Keishing | 0bd59f1 | 2016-12-14 10:39:21 -0600 | [diff] [blame] | 12 | ... Prune archived journal logs |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 13 | |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 14 | Resource ../lib/utils.robot |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 15 | Resource ../lib/pdu/pdu.robot |
Rahul Maheshwari | 9bd3abc | 2017-02-07 00:51:30 -0600 | [diff] [blame] | 16 | Resource ../lib/state_manager.robot |
Michael Walsh | 1b0ceb2 | 2017-05-31 16:03:01 -0500 | [diff] [blame] | 17 | Resource ../lib/bmc_network_utils.robot |
Sunil M | ec6fe31 | 2017-06-19 04:21:25 -0500 | [diff] [blame] | 18 | Resource ../lib/bmc_cleanup.robot |
George Keishing | 27a1f20 | 2017-08-30 22:40:26 -0500 | [diff] [blame] | 19 | Resource ../lib/dump_utils.robot |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 20 | Library ../lib/gen_misc.py |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 21 | |
| 22 | *** Variables *** |
| 23 | ${HOST_SETTING} /org/openbmc/settings/host0 |
| 24 | |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 25 | ${ERROR_REGEX} xyz.openbmc_project.Software.BMC.Updater.service: Failed with result 'core-dump' |
| 26 | |
Gunnar Mills | 7c8923f | 2016-12-12 21:19:52 -0600 | [diff] [blame] | 27 | *** Test Cases *** |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 28 | |
George Keishing | 62f9430 | 2017-01-31 07:00:52 -0600 | [diff] [blame] | 29 | Get To Stable State |
Rahul Maheshwari | 2c72504 | 2017-01-29 22:55:28 -0600 | [diff] [blame] | 30 | [Documentation] BMC cleanup drive to stable state |
| 31 | ... 1. PDU powercycle if specified |
| 32 | ... 1. Ping Test |
| 33 | ... 2. SSH Connection session Test |
| 34 | ... 3. REST Connection session Test |
| 35 | ... 4. Reboot BMC if REST Test failed |
| 36 | ... 5. Get BMC in Ready state if its not in this state |
| 37 | ... 6. Get Host in Off state if its not in this state |
| 38 | ... 7. Update restore policy |
Sunil M | 89fb0f5 | 2017-05-26 05:25:33 -0500 | [diff] [blame] | 39 | ... 8. Verify and Update MAC address. |
George Keishing | 62f9430 | 2017-01-31 07:00:52 -0600 | [diff] [blame] | 40 | [Tags] Get_To_Stable_State |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 41 | |
Rahul Maheshwari | 2c72504 | 2017-01-29 22:55:28 -0600 | [diff] [blame] | 42 | Run Keyword And Ignore Error Powercycle System Via PDU |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 43 | |
George Keishing | 13497eb | 2018-11-07 06:35:14 -0600 | [diff] [blame^] | 44 | ${ping_status}= Run Keyword And Return Status |
| 45 | ... Wait For Host To Ping ${OPENBMC_HOST} 2 mins |
| 46 | |
| 47 | # Check if the ping works using 1400 MTU. |
| 48 | Run Keyword if ${ping_status} == ${True} MTU Ping Test |
| 49 | |
| 50 | Run Keyword if ${ping_status} == ${False} |
| 51 | ... Fail ${OPENBMC_HOST} ping test failed. |
| 52 | |
| 53 | Open Connection And Log In host=${OPENBMC_HOST} |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 54 | |
George Keishing | 59fa048 | 2017-03-02 10:01:09 -0600 | [diff] [blame] | 55 | Wait Until Keyword Succeeds |
| 56 | ... 1 min 30 sec Initialize OpenBMC |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 57 | |
Rahul Maheshwari | 9bd3abc | 2017-02-07 00:51:30 -0600 | [diff] [blame] | 58 | ${ready_status}= Run Keyword And Return Status Is BMC Ready |
| 59 | Run Keyword If '${ready_status}' == '${False}' Put BMC State Ready |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 60 | |
Rahul Maheshwari | 9bd3abc | 2017-02-07 00:51:30 -0600 | [diff] [blame] | 61 | ${host_off_status}= Run Keyword And Return Status Is Host Off |
| 62 | Run Keyword If '${host_off_status}' == '${False}' Initiate Host PowerOff |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 63 | |
George Keishing | 0bd59f1 | 2016-12-14 10:39:21 -0600 | [diff] [blame] | 64 | Prune Journal Log |
| 65 | |
George Keishing | efc3ff2 | 2017-12-12 11:49:25 -0600 | [diff] [blame] | 66 | Run Keyword And Ignore Error Set BMC Power Policy ${ALWAYS_POWER_OFF} |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 67 | |
George Keishing | 6bbce53 | 2017-09-21 13:03:55 -0500 | [diff] [blame] | 68 | # TODO: Enable MAC AES check latter. |
| 69 | # Reference : openbmc/openbmc-test-automation#998 |
| 70 | #Run Keyword If '${MAC_ADDRESS}' != '${EMPTY}' |
| 71 | #... Check And Reset MAC |
| 72 | |
| 73 | # TODO: Add new UBI File system cleanup. |
| 74 | # Reference : openbmc/openbmc-test-automation#998 |
| 75 | #${rc}= Execute Command find ${CLEANUP_DIR_PATH} |
| 76 | #... return_stdout=False return_rc=True |
| 77 | #Run Keyword If '${CLEANUP_DIR_PATH}' != '${EMPTY}' and ${rc} == 0 |
| 78 | #... Cleanup Dir |
Sunil M | ec6fe31 | 2017-06-19 04:21:25 -0500 | [diff] [blame] | 79 | |
Sunil M | 5dc827d | 2017-10-17 03:49:31 -0500 | [diff] [blame] | 80 | Run Keyword And Ignore Error Delete All Error Logs |
| 81 | Run Keyword And Ignore Error Delete All Dumps |
Charles Paul Hofer | 7cdb8b4 | 2018-09-10 12:12:04 -0500 | [diff] [blame] | 82 | Check For Current Boot Application Failures |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 83 | |
| 84 | *** Keywords *** |
| 85 | |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 86 | BMC Online Test |
| 87 | [Documentation] BMC ping, SSH, REST connection Test |
| 88 | |
Gunnar Mills | 597ffa0 | 2016-12-06 11:26:53 -0600 | [diff] [blame] | 89 | ${l_status}= Run Keyword and Return Status |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 90 | ... Verify Ping and REST Authentication |
| 91 | Run Keyword If '${l_status}' == '${False}' |
| 92 | ... Fail msg=System not in ideal state to continue [ERROR] |
| 93 | |
| 94 | |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 95 | Update Policy Setting |
| 96 | [Documentation] Update the given restore policy |
Gunnar Mills | 3803280 | 2016-12-12 13:43:40 -0600 | [diff] [blame] | 97 | [Arguments] ${policy} |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 98 | |
Gunnar Mills | 597ffa0 | 2016-12-06 11:26:53 -0600 | [diff] [blame] | 99 | ${valueDict}= create dictionary data=${policy} |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 100 | Write Attribute ${HOST_SETTING} power_policy data=${valueDict} |
| 101 | ${currentPolicy}= Read Attribute ${HOST_SETTING} power_policy |
| 102 | Should Be Equal ${currentPolicy} ${policy} |
| 103 | |
George Keishing | d5123f7 | 2016-12-07 09:54:51 -0600 | [diff] [blame] | 104 | |
| 105 | Trigger Warm Reset via Reboot |
| 106 | [Documentation] Execute reboot command on the remote BMC and |
| 107 | ... returns immediately. This keyword "Start Command" |
| 108 | ... returns nothing and does not wait for the command |
| 109 | ... execution to be finished. |
| 110 | Open Connection And Log In |
| 111 | |
| 112 | Start Command /sbin/reboot |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 113 | |
| 114 | |
| 115 | Powercycle System Via PDU |
Joy Onyerikwu | f4a807b | 2018-06-20 08:43:54 -0500 | [diff] [blame] | 116 | [Documentation] AC cycle the system via PDU. |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 117 | |
| 118 | Validate Parameters |
| 119 | PDU Power Cycle |
| 120 | Check If BMC is Up 5 min 10 sec |
| 121 | |
| 122 | |
Charles Paul Hofer | 7cdb8b4 | 2018-09-10 12:12:04 -0500 | [diff] [blame] | 123 | Check For Current Boot Application Failures |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 124 | [Documentation] Parse the journal log and check for failures. |
| 125 | [Arguments] ${error_regex}=${ERROR_REGEX} |
| 126 | |
| 127 | ${error_regex}= Escape Bash Quotes ${error_regex} |
| 128 | ${journal_log} ${stderr} ${rc}= BMC Execute Command |
Charles Paul Hofer | bcae08b | 2018-08-09 11:33:33 -0500 | [diff] [blame] | 129 | ... journalctl -b --no-pager | egrep '${error_regex}' ignore_err=1 |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 130 | |
| 131 | Should Be Empty ${journal_log} |
| 132 | |
| 133 | |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 134 | Validate Parameters |
Joy Onyerikwu | f4a807b | 2018-06-20 08:43:54 -0500 | [diff] [blame] | 135 | [Documentation] Validate PDU parameters. |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 136 | Should Not Be Empty ${PDU_IP} |
| 137 | Should Not Be Empty ${PDU_TYPE} |
| 138 | Should Not Be Empty ${PDU_SLOT_NO} |
| 139 | Should Not Be Empty ${PDU_USERNAME} |
| 140 | Should Not Be Empty ${PDU_PASSWORD} |
George Keishing | 13497eb | 2018-11-07 06:35:14 -0600 | [diff] [blame^] | 141 | |
| 142 | |
| 143 | MTU Ping Test |
| 144 | [Documentation] Ping test using MTU. |
| 145 | [Arguments] ${mtu}=${1400} |
| 146 | |
| 147 | # Description of argument(s): |
| 148 | # mtu The maximum transmission unit (MTU) of a network interface. |
| 149 | |
| 150 | ${rc} ${output}= Run And Return RC And Output |
| 151 | ... ping -M do -s ${mtu} -c 10 ${OPENBMC_HOST} |
| 152 | Should Be Equal As Integers ${rc} 0 |
| 153 | Should Not Contain ${output} 100% packet loss |