Base Tagging and master skip_list file
This is the prequel to new method to use testname tagging
1. README is jared with windows characters. Fixed and added
info how to use the skip list master file argument file
$ OPENBMC_HOST=x.x.x.x tox -e barreleye -- --argumentfile skip_list tests
2. The recent known_issue is kept as it is until we clean
the code using the new method of tagging with testname
3. New file skip_list a master file holder for skip tag introduced
4. Few use cases are tagged by test case name using underscore to
separate the words in the testname
Resolves openbmc/openbmc-test-automation#127
Change-Id: Iaa934bf03631460bd67a620ece76fb198add33fa
Signed-off-by: George Keishing <gkeishin@in.ibm.com>
diff --git a/README.md b/README.md
index 7d0f1fe..9ab2870 100755
--- a/README.md
+++ b/README.md
@@ -1,117 +1,117 @@
-#openbmc-automation
-
-Quickstart
-----------
-
-To run openbmc-automation first you need to install the prerequisite python
-packages which will help to invoke tests through tox. Note that tox
-version 2.3.1 or greater is required.
-
-Install the python dependencies for tox
-```shell
- $ easy_install tox
- $ easy_install pip
-```
-
-Initilize the following environment variable which will used while testing
-```shell
- $ export OPENBMC_HOST=<openbmc machine ip address>
- $ export OPENBMC_PASSWORD=<openbmc password>
- $ export OPENBMC_USERNAME=<openbmc username>
- $ export OPENBMC_MODEL=[./data/Barreleye.py, ./data/Palmetto.py, etc]
+#openbmc-automation
+
+Quickstart
+----------
+
+To run openbmc-automation first you need to install the prerequisite python
+packages which will help to invoke tests through tox. Note that tox
+version 2.3.1 or greater is required.
+
+Install the python dependencies for tox
+```shell
+ $ easy_install tox
+ $ easy_install pip
+```
+
+Initilize the following environment variable which will used while testing
+```shell
+ $ export OPENBMC_HOST=<openbmc machine ip address>
+ $ export OPENBMC_PASSWORD=<openbmc password>
+ $ export OPENBMC_USERNAME=<openbmc username>
+ $ export OPENBMC_MODEL=[./data/Barreleye.py, ./data/Palmetto.py, etc]
$ export IPMI_COMMAND=<Dbus/External>
$ export IPMI_PASSWORD=<External IPMI password>
-```
-
-There are two different set of test suite existing based on the usage.
-The test suites are distinctly separated by directory as under
- tests/
- extended/
-
-`tests`: directory contains the general test cases
-
-`extended`: directory contains the use cases for new IP network testing,PDU,
-BIOS and BMC code update.
-
-```shell
-Use Following Variables for networking test cases
-===========================================================
- $export NEW_BMC_IP=<openbmc machine ip address>
- $export NEW_SUBNET_MASK=<openbmc new subnet mask>
- $export NEW_GATEWAY=<openbmc new gateway>
-==========================================================
-
- Use following parameters for PDU:
- $ export PDU_IP=<PDU IP address>
- $ export PDU_USERNAME=<PDU username>
- $ export PDU_PASSWORD=<PDU password>
- $ export PDU_TYPE=<PDU type>
- $ export PDU_SLOT_NO=<SLOT number>
-
- for PDU_TYPE we support only synaccess at the moment
-
-Use following variables for syslog test cases
-==========================================================
- $ export SYSLOG_IP_ADDRESS=<remote syslog system ip>
- $ export SYSLOG_PORT=<remote syslog system port>
-
-Use the following variables for Qemu test run
-==========================================================
- $ export SSH_PORT=<ssh port number>
- $ export HTTPS_PORT=<https port number>
-
-Use the following variables for BIOS update testing
-==========================================================
- $ export PNOR_IMAGE_PATH=<path to>/<machine>.pnor
-
-```
-
-Run tests
-```shell
- $ tox -e tests
-```
-
-How to test individual test
-```shell
- One specific test
- $ tox -e custom -- -t '"DIMM0 no fault"' tests/test_sensors.robot
-
- No preset environment variables, one test case from one test suite
- $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- -t '"DIMM0 no fault"' tests/test_sensors.robot
-
- No preset environment variables, one test suite for a palmetto system
- $ OPENBMC_HOST=x.x.x.x tox -e palmetto -- tests/test_sensors.robot
-
- No preset environment variables, the entire test suite for a barreleye system
- $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- tests
-
- No preset environment variables, the entire test suite excluding test
- cases failing due to known issues
- $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- -e known_issue tests
-```
-
-It can also be run by pasing variables from the cli...
-```shell
- Run one test suite using using pybot
- $ pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v OPENBMC_MODEL:<model path> tests/test_time.robot
-
- Run entire test suite using using pybot
- $ pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v OPENBMC_MODEL:<model path> tests
-
- Run entire test suite using external ipmitool
- $ pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v IPMI_COMMAND:External -v IPMI_PASSWORD:PASSW0RD -v OPENBMC_MODEL:<model path> tests
-```
-
-Run extended tests
-```shell
- Set the preset environment variables, run test suite for a barreleye system
- $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- extended/test_power_restore.robot
-
- Similarly for Network, PDU and update BIOS
-
- For BMC code update, download the system type *.all.tar image from https://openpower.xyz
- and run as follows:
-
- For Barreleye system
- python -m robot -v OPENBMC_HOST:x.x.x.x -v FILE_PATH:downloaded_path/barreleye-xxxx.all.tar extended/code_update/update_bmc.robot
-```
+```
+
+There are two different set of test suite existing based on the usage.
+The test suites are distinctly separated by directory as under
+ tests/
+ extended/
+
+`tests`: directory contains the general test cases
+
+`extended`: directory contains the use cases for new IP network testing,PDU,
+BIOS and BMC code update.
+
+```shell
+Use Following Variables for networking test cases
+===========================================================
+ $export NEW_BMC_IP=<openbmc machine ip address>
+ $export NEW_SUBNET_MASK=<openbmc new subnet mask>
+ $export NEW_GATEWAY=<openbmc new gateway>
+==========================================================
+
+ Use following parameters for PDU:
+ $ export PDU_IP=<PDU IP address>
+ $ export PDU_USERNAME=<PDU username>
+ $ export PDU_PASSWORD=<PDU password>
+ $ export PDU_TYPE=<PDU type>
+ $ export PDU_SLOT_NO=<SLOT number>
+
+ for PDU_TYPE we support only synaccess at the moment
+
+Use following variables for syslog test cases
+==========================================================
+ $ export SYSLOG_IP_ADDRESS=<remote syslog system ip>
+ $ export SYSLOG_PORT=<remote syslog system port>
+
+Use the following variables for Qemu test run
+==========================================================
+ $ export SSH_PORT=<ssh port number>
+ $ export HTTPS_PORT=<https port number>
+
+Use the following variables for BIOS update testing
+==========================================================
+ $ export PNOR_IMAGE_PATH=<path to>/<machine>.pnor
+
+```
+
+Run tests
+```shell
+ $ tox -e tests
+```
+
+How to test individual test
+```shell
+ One specific test
+ $ tox -e custom -- -t '"DIMM0 no fault"' tests/test_sensors.robot
+
+ No preset environment variables, one test case from one test suite
+ $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- -t '"DIMM0 no fault"' tests/test_sensors.robot
+
+ No preset environment variables, one test suite for a palmetto system
+ $ OPENBMC_HOST=x.x.x.x tox -e palmetto -- tests/test_sensors.robot
+
+ No preset environment variables, the entire test suite for a barreleye system
+ $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- tests
+
+ No preset environment variables, the entire test suite excluding test
+ cases failing due to known issues. The skip_list file contains tag list which are to be skipped
+ $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- --argumentfile skip_list tests
+```
+
+It can also be run by pasing variables from the cli...
+```shell
+ Run one test suite using using pybot
+ $ pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v OPENBMC_MODEL:<model path> tests/test_time.robot
+
+ Run entire test suite using using pybot
+ $ pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v OPENBMC_MODEL:<model path> tests
+
+ Run entire test suite using external ipmitool
+ $ pybot -v OPENBMC_HOST:<ip> -v OPENBMC_USERNAME:root -v OPENBMC_PASSWORD:0penBmc -v IPMI_COMMAND:External -v IPMI_PASSWORD:PASSW0RD -v OPENBMC_MODEL:<model path> tests
+```
+
+Run extended tests
+```shell
+ Set the preset environment variables, run test suite for a barreleye system
+ $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- extended/test_power_restore.robot
+
+ Similarly for Network, PDU and update BIOS
+
+ For BMC code update, download the system type *.all.tar image from https://openpower.xyz
+ and run as follows:
+
+ For Barreleye system
+ python -m robot -v OPENBMC_HOST:x.x.x.x -v FILE_PATH:downloaded_path/barreleye-xxxx.all.tar extended/code_update/update_bmc.robot
+```
diff --git a/skip_list b/skip_list
new file mode 100644
index 0000000..e5b3b48
--- /dev/null
+++ b/skip_list
@@ -0,0 +1,9 @@
+-e known_issue
+#https://github.com/openbmc/openbmc-test-automation/issues/110
+-e Test_non_SSL_Connection_to_port_443
+#https://github.com/openbmc/openbmc-test-automation/issues/123
+-e Execute_ipmi_BT_capabilities_command
+#https://github.com/openbmc/openbmc/issues/539
+-e Test_Wrong_Reservation_ID
+#https://github.com/openbmc/openbmc/issues/539
+-e Test_Correct_Reservation_ID
diff --git a/tests/security/test_ssl.robot b/tests/security/test_ssl.robot
index b15df1e..bed46a6 100644
--- a/tests/security/test_ssl.robot
+++ b/tests/security/test_ssl.robot
@@ -32,6 +32,7 @@
[Documentation] This testcase is for test to check OpenBMC machine
... will not accepts the non-secure connection that is with http to
... port 443 and expect 400 in response
+ [Tags] Test_non_SSL_Connection_to_port_443
Create Session openbmc http://${OPENBMC_HOST}:443/
${resp}= Get Request openbmc /list
Should Be Equal As Strings ${resp.status_code} ${HTTP_BAD_REQUEST}
diff --git a/tests/test_esel.robot b/tests/test_esel.robot
index 9aa9756..803a78e 100644
--- a/tests/test_esel.robot
+++ b/tests/test_esel.robot
@@ -12,10 +12,11 @@
*** Test Cases ***
Test Wrong Reservation_ID
- [Documentation] This testcase is to test BMC can handle multi-requestor's
- ... oem partial add command with incorrect reservation id.
- ... It simulates sending partial add command with fake content
- ... and wrong Reservation ID. This command will be rejected.
+ [Documentation] This testcase is to test BMC can handle multi-requestor's
+ ... oem partial add command with incorrect reservation id.
+ ... It simulates sending partial add command with fake content
+ ... and wrong Reservation ID. This command will be rejected.
+ [Tags] Test_Wrong_Reservation_ID
${rev_id_1} = Run IPMI Command Returned 0x0a 0x42
${rev_id_ls} = Get Substring ${rev_id_1} 1 3
@@ -25,10 +26,11 @@
Should Contain ${output} Reservation cancelled
Test Correct Reservation_ID
- [Documentation] This testcase is to test BMC can handle multi-requestor's
- ... oem partial add command with correct reservation id. It
- ... simulates sending partial add command with fake content
- ... and correct Reservation ID. This command will be accepted.
+ [Documentation] This testcase is to test BMC can handle multi-requestor's
+ ... oem partial add command with correct reservation id. It
+ ... simulates sending partial add command with fake content
+ ... and correct Reservation ID. This command will be accepted.
+ [Tags] Test_Correct_Reservation_ID
Run IPMI command 0x0a 0x42
${rev_id_2} = Run IPMI Command Returned 0x0a 0x42
diff --git a/tests/test_sensors.robot b/tests/test_sensors.robot
index 213d22f..32c88fd 100644
--- a/tests/test_sensors.robot
+++ b/tests/test_sensors.robot
@@ -24,6 +24,7 @@
Response Should Be Equal "hello"
Execute ipmi BT capabilities command
+ [Tags] Execute_ipmi_BT_capabilities_command
Run IPMI command 0x06 0x36
response Should Be Equal " 01 40 40 0a 01"