George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 1 | *** Settings *** |
| 2 | Documentation This module will take whatever action is necessary |
| 3 | ... to bring the BMC to a stable, standby state. For our |
| 4 | ... purposes, a stable state is defined as: |
| 5 | ... - BMC is communicating |
| 6 | ... (pinging, sshing and REST commands working) |
| 7 | ... - Power state is 0 (off) |
Rahul Maheshwari | 2c72504 | 2017-01-29 22:55:28 -0600 | [diff] [blame] | 8 | ... - BMC state is "Ready" |
| 9 | ... - HOST state is "Off" |
George Keishing | efc3ff2 | 2017-12-12 11:49:25 -0600 | [diff] [blame] | 10 | ... - Boot policy is "ALWAYS_POWER_OFF" |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 11 | ... Power cycle system via PDU if specified |
George Keishing | 0bd59f1 | 2016-12-14 10:39:21 -0600 | [diff] [blame] | 12 | ... Prune archived journal logs |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 13 | |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 14 | Resource ../lib/utils.robot |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 15 | Resource ../lib/pdu/pdu.robot |
Rahul Maheshwari | 9bd3abc | 2017-02-07 00:51:30 -0600 | [diff] [blame] | 16 | Resource ../lib/state_manager.robot |
Michael Walsh | 1b0ceb2 | 2017-05-31 16:03:01 -0500 | [diff] [blame] | 17 | Resource ../lib/bmc_network_utils.robot |
Sunil M | ec6fe31 | 2017-06-19 04:21:25 -0500 | [diff] [blame] | 18 | Resource ../lib/bmc_cleanup.robot |
George Keishing | 27a1f20 | 2017-08-30 22:40:26 -0500 | [diff] [blame] | 19 | Resource ../lib/dump_utils.robot |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 20 | Library ../lib/gen_misc.py |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 21 | |
| 22 | *** Variables *** |
| 23 | ${HOST_SETTING} /org/openbmc/settings/host0 |
| 24 | |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 25 | ${ERROR_REGEX} xyz.openbmc_project.Software.BMC.Updater.service: Failed with result 'core-dump' |
| 26 | |
Gunnar Mills | 7c8923f | 2016-12-12 21:19:52 -0600 | [diff] [blame] | 27 | *** Test Cases *** |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 28 | |
George Keishing | 62f9430 | 2017-01-31 07:00:52 -0600 | [diff] [blame] | 29 | Get To Stable State |
Rahul Maheshwari | 2c72504 | 2017-01-29 22:55:28 -0600 | [diff] [blame] | 30 | [Documentation] BMC cleanup drive to stable state |
| 31 | ... 1. PDU powercycle if specified |
| 32 | ... 1. Ping Test |
| 33 | ... 2. SSH Connection session Test |
| 34 | ... 3. REST Connection session Test |
| 35 | ... 4. Reboot BMC if REST Test failed |
| 36 | ... 5. Get BMC in Ready state if its not in this state |
| 37 | ... 6. Get Host in Off state if its not in this state |
| 38 | ... 7. Update restore policy |
Sunil M | 89fb0f5 | 2017-05-26 05:25:33 -0500 | [diff] [blame] | 39 | ... 8. Verify and Update MAC address. |
George Keishing | 62f9430 | 2017-01-31 07:00:52 -0600 | [diff] [blame] | 40 | [Tags] Get_To_Stable_State |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 41 | |
Rahul Maheshwari | 2c72504 | 2017-01-29 22:55:28 -0600 | [diff] [blame] | 42 | Run Keyword And Ignore Error Powercycle System Via PDU |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 43 | |
George Keishing | 59fa048 | 2017-03-02 10:01:09 -0600 | [diff] [blame] | 44 | Wait For Host To Ping ${OPENBMC_HOST} 2 mins |
| 45 | Run Keyword And Ignore Error |
| 46 | ... Open Connection And Log In host=${OPENBMC_HOST} |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 47 | |
George Keishing | 59fa048 | 2017-03-02 10:01:09 -0600 | [diff] [blame] | 48 | Wait Until Keyword Succeeds |
| 49 | ... 1 min 30 sec Initialize OpenBMC |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 50 | |
Rahul Maheshwari | 9bd3abc | 2017-02-07 00:51:30 -0600 | [diff] [blame] | 51 | ${ready_status}= Run Keyword And Return Status Is BMC Ready |
| 52 | Run Keyword If '${ready_status}' == '${False}' Put BMC State Ready |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 53 | |
Rahul Maheshwari | 9bd3abc | 2017-02-07 00:51:30 -0600 | [diff] [blame] | 54 | ${host_off_status}= Run Keyword And Return Status Is Host Off |
| 55 | Run Keyword If '${host_off_status}' == '${False}' Initiate Host PowerOff |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 56 | |
George Keishing | 0bd59f1 | 2016-12-14 10:39:21 -0600 | [diff] [blame] | 57 | Prune Journal Log |
| 58 | |
George Keishing | efc3ff2 | 2017-12-12 11:49:25 -0600 | [diff] [blame] | 59 | Run Keyword And Ignore Error Set BMC Power Policy ${ALWAYS_POWER_OFF} |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 60 | |
George Keishing | 6bbce53 | 2017-09-21 13:03:55 -0500 | [diff] [blame] | 61 | # TODO: Enable MAC AES check latter. |
| 62 | # Reference : openbmc/openbmc-test-automation#998 |
| 63 | #Run Keyword If '${MAC_ADDRESS}' != '${EMPTY}' |
| 64 | #... Check And Reset MAC |
| 65 | |
| 66 | # TODO: Add new UBI File system cleanup. |
| 67 | # Reference : openbmc/openbmc-test-automation#998 |
| 68 | #${rc}= Execute Command find ${CLEANUP_DIR_PATH} |
| 69 | #... return_stdout=False return_rc=True |
| 70 | #Run Keyword If '${CLEANUP_DIR_PATH}' != '${EMPTY}' and ${rc} == 0 |
| 71 | #... Cleanup Dir |
Sunil M | ec6fe31 | 2017-06-19 04:21:25 -0500 | [diff] [blame] | 72 | |
Sunil M | 5dc827d | 2017-10-17 03:49:31 -0500 | [diff] [blame] | 73 | Run Keyword And Ignore Error Delete All Error Logs |
| 74 | Run Keyword And Ignore Error Delete All Dumps |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 75 | Check For Application Failures |
George Keishing | 457bb3a | 2018-02-22 01:16:10 -0600 | [diff] [blame] | 76 | Run Keyword And Ignore Error Remove Journald Logs |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 77 | |
| 78 | *** Keywords *** |
| 79 | |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 80 | BMC Online Test |
| 81 | [Documentation] BMC ping, SSH, REST connection Test |
| 82 | |
Gunnar Mills | 597ffa0 | 2016-12-06 11:26:53 -0600 | [diff] [blame] | 83 | ${l_status}= Run Keyword and Return Status |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 84 | ... Verify Ping and REST Authentication |
| 85 | Run Keyword If '${l_status}' == '${False}' |
| 86 | ... Fail msg=System not in ideal state to continue [ERROR] |
| 87 | |
| 88 | |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 89 | Update Policy Setting |
| 90 | [Documentation] Update the given restore policy |
Gunnar Mills | 3803280 | 2016-12-12 13:43:40 -0600 | [diff] [blame] | 91 | [Arguments] ${policy} |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 92 | |
Gunnar Mills | 597ffa0 | 2016-12-06 11:26:53 -0600 | [diff] [blame] | 93 | ${valueDict}= create dictionary data=${policy} |
George Keishing | 78ea2dc | 2016-11-18 04:33:45 -0600 | [diff] [blame] | 94 | Write Attribute ${HOST_SETTING} power_policy data=${valueDict} |
| 95 | ${currentPolicy}= Read Attribute ${HOST_SETTING} power_policy |
| 96 | Should Be Equal ${currentPolicy} ${policy} |
| 97 | |
George Keishing | d5123f7 | 2016-12-07 09:54:51 -0600 | [diff] [blame] | 98 | |
| 99 | Trigger Warm Reset via Reboot |
| 100 | [Documentation] Execute reboot command on the remote BMC and |
| 101 | ... returns immediately. This keyword "Start Command" |
| 102 | ... returns nothing and does not wait for the command |
| 103 | ... execution to be finished. |
| 104 | Open Connection And Log In |
| 105 | |
| 106 | Start Command /sbin/reboot |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 107 | |
| 108 | |
| 109 | Powercycle System Via PDU |
Joy Onyerikwu | f4a807b | 2018-06-20 08:43:54 -0500 | [diff] [blame] | 110 | [Documentation] AC cycle the system via PDU. |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 111 | |
| 112 | Validate Parameters |
| 113 | PDU Power Cycle |
| 114 | Check If BMC is Up 5 min 10 sec |
| 115 | |
| 116 | |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 117 | Check For Application Failures |
| 118 | [Documentation] Parse the journal log and check for failures. |
| 119 | [Arguments] ${error_regex}=${ERROR_REGEX} |
| 120 | |
| 121 | ${error_regex}= Escape Bash Quotes ${error_regex} |
| 122 | ${journal_log} ${stderr} ${rc}= BMC Execute Command |
Charles Paul Hofer | bcae08b | 2018-08-09 11:33:33 -0500 | [diff] [blame^] | 123 | ... journalctl -b --no-pager | egrep '${error_regex}' ignore_err=1 |
George Keishing | c7748be | 2018-07-12 05:40:37 -0500 | [diff] [blame] | 124 | |
| 125 | Should Be Empty ${journal_log} |
| 126 | |
| 127 | |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 128 | Validate Parameters |
Joy Onyerikwu | f4a807b | 2018-06-20 08:43:54 -0500 | [diff] [blame] | 129 | [Documentation] Validate PDU parameters. |
George Keishing | 34d4b8d | 2016-12-07 08:35:29 -0600 | [diff] [blame] | 130 | Should Not Be Empty ${PDU_IP} |
| 131 | Should Not Be Empty ${PDU_TYPE} |
| 132 | Should Not Be Empty ${PDU_SLOT_NO} |
| 133 | Should Not Be Empty ${PDU_USERNAME} |
| 134 | Should Not Be Empty ${PDU_PASSWORD} |