Handling continuously failing test cases while running entire test bucket.
This change includes:
- Added known_issue tag for test cases failing due to existing issues.
And where fix for the issue is expected to take little long time.
- With each tag addition also added github issue number.
Benefits:
- Failing test cases can be excluded while running entire bucket
using --exclude known_issue as suffix
- Continuously failing test cases will have related issue number
in document section.
Change-Id: I18c40b97a25eff947a5922e4abfacde07fe19a6b
Signed-off-by: Rahul Maheshwari <rahulmaheshwari@in.ibm.com>
diff --git a/README.md b/README.md
index 2b1befb..0f2e82a 100755
--- a/README.md
+++ b/README.md
@@ -84,6 +84,10 @@
No preset environment variables, the entire test suite for a barreleye system
$ OPENBMC_HOST=x.x.x.x tox -e barreleye -- tests
+
+ No preset environment variables, the entire test suite excluding test
+ cases failing due to known issues
+ $ OPENBMC_HOST=x.x.x.x tox -e barreleye -- tests --exclude known_issue
```
It can also be run by pasing variables from the cli...
diff --git a/tests/test_bootpolicy.robot b/tests/test_bootpolicy.robot
index bc635a6..cfcf44d 100644
--- a/tests/test_bootpolicy.robot
+++ b/tests/test_bootpolicy.robot
@@ -84,8 +84,8 @@
Onetime boot order after warm reset
[Documentation] This testcase is to verify that boot policy and order does not change
... after warm reset on a system with onetime boot policy.
-
- [Tags] chassisboot
+ ... Existing Issue: https://github.com/openbmc/openbmc/issues/519
+ [Tags] chassisboot known_issue
Initiate Power On
@@ -104,7 +104,8 @@
Permanent boot order after warm reset
[Documentation] This testcase is to verify that boot policy and order does not change
... after warm reset on a system with permanent boot policy.
- [Tags] chassisboot
+ ... Existing Issue: https://github.com/openbmc/openbmc/issues/519
+ [Tags] chassisboot known_issue
Initiate Power On
@@ -123,6 +124,8 @@
Set boot policy to invalid value
[Documentation] This testcase is to verify that proper error message is prompted
... when invalid value to provided to boot policy.
+ ... Existing Issue: https://github.com/openbmc/openbmc/issues/552
+ [Tags] known_issue
Set Boot Policy abc
diff --git a/tests/test_generic_conf.robot b/tests/test_generic_conf.robot
index b27d32a..bdd6c97 100755
--- a/tests/test_generic_conf.robot
+++ b/tests/test_generic_conf.robot
@@ -38,29 +38,33 @@
[Documentation] ***BAD PATH***
... This test case set the power values with string of characters
... Expectation is to return error.
+ ... Existing Issue: https://github.com/openbmc/openbmc/issues/552
+ [Tags] known_issue
${valueToBeSet}= Set Variable abcdefg
${valueDict}= create dictionary data=${valueToBeSet}
Write Attribute /org/openbmc/settings/host0 power_cap data=${valueDict}
- ${value}= Read Attribute /org/openbmc/settings/host0 power_cap
+ ${value}= Read Attribute /org/openbmc/settings/host0 power_cap
should not be true '${value}'=='${valueToBeSet}'
Set the power with greater then MAX_POWER_VALUE
[Documentation] ***BAD PATH***
- ... This test case sets the power value which is greater
+ ... This test case sets the power value which is greater
... then MAX_ALLOWED_VALUE,Expectation is to return error
+ ... Existing Issue: https://github.com/openbmc/openbmc/issues/552
+ [Tags] known_issue
${valueToBeSet}= Set Variable ${1010}
${valueDict}= create dictionary data=${valueToBeSet}
Write Attribute /org/openbmc/settings/host0 power_cap data=${valueDict}
${value}= Read Attribute /org/openbmc/settings/host0 power_cap
- should not be equal ${value} ${valueToBeSet}
+ should not be equal ${value} ${valueToBeSet}
Set the power with MIN_POWER_VALUE
[Documentation] ***BAD PATH***
- ... This test case sets the power value less then
+ ... This test case sets the power value less then
... MIN_ALLOWED_VALUE,Expectation is it should get error.
${valueToBeSet}= Set Variable ${MIN_POWER_VALUE}
@@ -74,7 +78,7 @@
[Documentation] ***GOOD PATH***
... This test case sets the power value with MAX_POWER_VALUE
... and it should be set.
-
+
${valueToBeSet}= Set Variable ${MAX_POWER_VALUE}
${valueDict}= create dictionary data=${valueToBeSet}
Write Attribute /org/openbmc/settings/host0 power_cap data=${valueDict}
@@ -84,8 +88,10 @@
Set the boot flags with string
[Documentation] ***BAD PATH***
- ... This test case sets the boot flag with some invalid string
+ ... This test case sets the boot flag with some invalid string
... Expectation is it should not be set.
+ ... Existing Issue: https://github.com/openbmc/openbmc/issues/552
+ [Tags] known_issue
${valueToBeSet}= Set Variable 3ab56f
${valueDict} = create dictionary data=${valueToBeSet}
diff --git a/tests/test_occ_powercap.robot b/tests/test_occ_powercap.robot
index 02f1a26..31471f9 100644
--- a/tests/test_occ_powercap.robot
+++ b/tests/test_occ_powercap.robot
@@ -22,6 +22,9 @@
[Documentation] This testcase is to test get/set powercap feature.
... In the testcase we are reading min, max value and then
... try set the random in that range.
+ ... Existing Issue: https://github.com/openbmc/openbmc/issues/552
+ [Tags] known_issue
+
${min}= Get Minimum PowerCap
log ${min}
${max}= Get Maximum PowerCap
@@ -38,6 +41,9 @@
Set Less Than Minimum PowerCAP
[Documentation] Test set powercap with less than min powercap value
+ ... Existing Issue: https://github.com/openbmc/openbmc/issues/552
+ [Tags] known_issue
+
${org_power_cap}= Get PowerCap
${min}= Get Minimum PowerCap
${sample_invalid_pcap}= Evaluate ${min}-${100}
@@ -49,6 +55,9 @@
Set More Than Maximum PowerCAP
[Documentation] Test set powercap with more than max powercap value
+ ... Existing Issue: https://github.com/openbmc/openbmc/issues/552
+ [Tags] known_issue
+
${org_power_cap}= Get PowerCap
${min}= Get Maximum PowerCap
${sample_invalid_pcap}= Evaluate ${min}+${100}
diff --git a/tests/test_warmreset.robot b/tests/test_warmreset.robot
index 5f38d05..6009cb0 100644
--- a/tests/test_warmreset.robot
+++ b/tests/test_warmreset.robot
@@ -18,6 +18,10 @@
*** Test Cases ***
Test WarmReset via REST
+ [Documentation] This testcase is to verify warm reset using REST.
+ ... Existing Issue: https://github.com/openbmc/openbmc/issues/519
+ [Tags] known_issue
+
${warm_test_file}= Set Variable /tmp/before_warmreset
Open Connection And Log In
${stdout} ${stderr} ${rc}=