Add CI approved test cases tag infrastructure
Use Robot Test Framework's use of Tags to create a subset of tests that
can be run on real and simulated hardware for a basic set of validations
against each commit.
Our yocto qemu session has real basic support and can not pass very many
of the robot tests. Not because there is a bug but simply because the
simulation does not support many of the openbmc rest interfaces (i.e.
inventory, sensors). I found some REST and Event Log test suites that
are available in QEMU to start testing now with. When we move the QEMU
simulation to more of a palmetto based model more test cases will get
the CI tag. Eventually we will be able to go to a negative model where
we mark the tests that should not be part of a CI run rather then this
small set of includes.
The tag of 'CI' was chosen over a 'QEMUCI' because I fear the use of
tags could get out of hand (BARRELEYECI vs WEDGECI vs BARRELEYE_SIM_CI,
etc).
Invoking....
To test in QEMU first launch the session, find the IP and then run...
OPENBMC_HOST=<ip> tox -e qemu -- --include CI tests
You can also run the CI suite against real hardware...
OPENBMC_HOST=<ip> tox -e barreleye -- --include CI tests
Change-Id: I2c5fbcf2457d958edf7b89c70d7990f5f97e8b75
Signed-off-by: Chris Austen <austenc@us.ibm.com>
diff --git a/tests/test_eventlog.robot b/tests/test_eventlog.robot
index 660d449..8280f78 100644
--- a/tests/test_eventlog.robot
+++ b/tests/test_eventlog.robot
@@ -19,11 +19,13 @@
valid path to logs
[Documentation] Test list all events
+ [Tags] CI
${resp} = openbmc get request /org/openbmc/records/events/
should be equal as strings ${resp.status_code} ${HTTP_OK}
clear any logs
[Documentation] Test delete all events
+ [Tags] CI
${resp} = openbmc post request /org/openbmc/records/events/action/clear data=${NIL}
should be equal as strings ${resp.status_code} ${HTTP_OK}
${resp} = openbmc get request /org/openbmc/records/events/
@@ -32,22 +34,26 @@
write a log
[Documentation] Test create event
+ [Tags] CI
create a test log
Message attribute should match
[Documentation] Check message attribute for created event
+ [Tags] CI
${uri} = create a test log
${content} = Read Attribute ${uri} message
Should Be Equal ${content} A Test event log just happened
Severity attribute should match
[Documentation] Check severity attribute for created event
+ [Tags] CI
${uri} = create a test log
${content}= Read Attribute ${uri} severity
Should Be Equal ${content} Info
data_bytes attribute should match
[Documentation] Check data_bytes attribute for created event
+ [Tags] CI
@{data_list} = Create List ${48} ${0} ${19} ${127} ${136} ${255}
${uri} = create a test log
${content} = Read Attribute ${uri} debug_data
@@ -55,6 +61,7 @@
delete the log
[Documentation] Test the delete event
+ [Tags] CI
${uri} = create a test log
${deluri} = catenate SEPARATOR= ${uri} /action/delete
${resp} = openbmc post request ${deluri} data=${NIL}
@@ -64,6 +71,7 @@
2nd delete should fail
[Documentation] Negative scnenario to delete already deleted event
+ [Tags] CI
${uri} = create a test log
${deluri} = catenate SEPARATOR= ${uri} /action/delete
${resp} = openbmc post request ${deluri} data=${NIL}
@@ -78,6 +86,7 @@
... delete middle log
... middle log should not exist
... time stamp should not match between logs(1st and 3rd)
+ [Tags] CI
${event1}= create a test log
${event2}= create a test log
${event3}= create a test log
@@ -91,6 +100,7 @@
restarting event process retains logs
[Documentation] This is to test events are in place even after the
... event service is restarted.
+ [Tags] CI
${resp} = openbmc get request /org/openbmc/records/events/
${json} = to json ${resp.content}
${logs_pre_restart}= set variable ${json['data']}
@@ -107,6 +117,7 @@
deleting log after obmc-phosphor-event.service restart
[Documentation] This is to test event can be deleted created prior to
... event service is restarted.
+ [Tags] CI
${uri}= create a test log
Open Connection And Log In
@@ -120,6 +131,7 @@
makeing new log after obmc-phosphor-event.service restart
[Documentation] This is for testing event creation after the
... event service is restarted.
+ [Tags] CI
Open Connection And Log In
${uptime}= Execute Command systemctl restart obmc-phosphor-event.service
Sleep ${10}
@@ -129,6 +141,7 @@
deleting new log after obmc-phosphor-event.service restart
[Documentation] This testcase is for testing deleted newly created event
... after event service is restarted.
+ [Tags] CI
Open Connection And Log In
${uptime}= Execute Command systemctl restart obmc-phosphor-event.service
Sleep ${10}
@@ -170,6 +183,7 @@
clearing logs results in no logs
[Documentation] This testcase is for clearning the events when no logs present
+ [Tags] CI
${resp} = openbmc post request /org/openbmc/records/events/action/clear data=${NIL}
should be equal as strings ${resp.status_code} ${HTTP_OK}
${resp} = openbmc get request /org/openbmc/records/events/
diff --git a/tests/test_obmcrest.robot b/tests/test_obmcrest.robot
index a3121e2..92d63f9 100644
--- a/tests/test_obmcrest.robot
+++ b/tests/test_obmcrest.robot
@@ -11,6 +11,7 @@
*** Test Cases ***
Good connection for testing
+ [Tags] CI
${content}= Read Properties /
${c}= get from List ${content} 0
Should Be Equal ${c} /org
@@ -30,6 +31,7 @@
Should Be Equal ${jsondata['data']['description']} The specified property cannot be found: ''is_fru''
get directory listing /
+ [Tags] CI
${resp} = openbmc get request /
should be equal as strings ${resp.status_code} ${HTTP_OK}
${json} = to json ${resp.content}
@@ -37,6 +39,7 @@
should be equal as strings ${json['status']} ok
get directory listing /org/
+ [Tags] CI
${resp} = openbmc get request /org/
should be equal as strings ${resp.status_code} ${HTTP_OK}
${json} = to json ${resp.content}
@@ -44,60 +47,70 @@
should be equal as strings ${json['status']} ok
get invalid directory listing /i/dont/exist/
+ [Tags] CI
${resp} = openbmc get request /i/dont/exist/
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put directory listing /
+ [Tags] CI
${resp} = openbmc put request /
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put directory listing /org/
+ [Tags] CI
${resp} = openbmc put request /org/
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put invalid directory listing /i/dont/exist/
+ [Tags] CI
${resp} = openbmc put request /i/dont/exist/
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
post directory listing /
+ [Tags] CI
${resp} = openbmc post request /
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
post directory listing /org/
+ [Tags] CI
${resp} = openbmc post request /org/
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
post invalid directory listing /i/dont/exist/
+ [Tags] CI
${resp} = openbmc post request /i/dont/exist/
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
delete directory listing /
+ [Tags] CI
${resp} = openbmc delete request /
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
delete directory listing /org/
+ [Tags] CI
${resp} = openbmc delete request /
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
delete invalid directory listing /org/nothere/
+ [Tags] CI
${resp} = openbmc delete request /org/nothere/
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
@@ -118,60 +131,70 @@
should be equal as strings ${json['status']} ok
get invalid list names /i/dont/exist/
+ [Tags] CI
${resp} = openbmc get request /i/dont/exist/list
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put list names /
+ [Tags] CI
${resp} = openbmc put request /list
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put list names /org/
+ [Tags] CI
${resp} = openbmc put request /org/list
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put invalid list names /i/dont/exist/
+ [Tags] CI
${resp} = openbmc put request /i/dont/exist/list
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
post list names /
+ [Tags] CI
${resp} = openbmc post request /list
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
post list names /org/
+ [Tags] CI
${resp} = openbmc post request /org/list
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
post invalid list names /i/dont/exist/
+ [Tags] CI
${resp} = openbmc post request /i/dont/exist/list
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
delete list names /
+ [Tags] CI
${resp} = openbmc delete request /list
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
delete list names /org/
+ [Tags] CI
${resp} = openbmc delete request /list
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
delete invalid list names /org/nothere/
+ [Tags] CI
${resp} = openbmc delete request /org/nothere/list
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
@@ -192,84 +215,98 @@
should be equal as strings ${json['status']} ok
get invalid names /i/dont/exist/
+ [Tags] CI
${resp} = openbmc get request /i/dont/exist/enumerate
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put names /
+ [Tags] CI
${resp} = openbmc put request /enumerate
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put names /org/
+ [Tags] CI
${resp} = openbmc put request /org/enumerate
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put invalid names /i/dont/exist/
+ [Tags] CI
${resp} = openbmc put request /i/dont/exist/enumerate
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
post names /
+ [Tags] CI
${resp} = openbmc post request /enumerate
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
post names /org/
+ [Tags] CI
${resp} = openbmc post request /org/enumerate
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
post invalid names /i/dont/exist/
+ [Tags] CI
${resp} = openbmc post request /i/dont/exist/enumerate
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
delete names /
+ [Tags] CI
${resp} = openbmc delete request /enumerate
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
delete names /org/
+ [Tags] CI
${resp} = openbmc delete request /enumerate
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
delete invalid names /org/nothere/
+ [Tags] CI
${resp} = openbmc delete request /org/nothere/enumerate
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
get method org/openbmc/records/events/action/acceptTestMessage
+ [Tags] CI
${resp} = openbmc get request org/openbmc/records/events/action/acceptTestMessage
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
get invalid method /i/dont/exist/
+ [Tags] CI
${resp} = openbmc get request /i/dont/exist/action/foo
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put method org/openbmc/records/events/action/acceptTestMessage
+ [Tags] CI
${resp} = openbmc put request org/openbmc/records/events/action/acceptTestMessage
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put invalid method /i/dont/exist/
+ [Tags] CI
${resp} = openbmc put request /i/dont/exist/action/foo
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
@@ -284,6 +321,7 @@
should be equal as strings ${json['status']} ok
post method org/openbmc/records/events/action/acceptTestMessage invalid args
+ [Tags] CI
${data} = create dictionary foo=bar
${resp} = openbmc post request org/openbmc/records/events/action/acceptTestMessage data=${data}
should be equal as strings ${resp.status_code} ${HTTP_BAD_REQUEST}
@@ -303,18 +341,21 @@
Should Be Equal ${content} ${COUNT}
delete method org/openbmc/records/events/action/acceptTestMessage
+ [Tags] CI
${resp} = openbmc delete request org/openbmc/records/events/action/acceptTestMessage
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
delete invalid method /org/nothere/
+ [Tags] CI
${resp} = openbmc delete request /org/nothere/action/foomethod
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
post method org/openbmc/records/events/action/acceptTestMessage no args
+ [Tags] CI
${data} = create dictionary data=@{EMPTY}
${resp} = openbmc post request org/openbmc/records/events/action/acceptTestMessage data=${data}
should be equal as strings ${resp.status_code} ${HTTP_OK}
diff --git a/tox.ini b/tox.ini
index 5da646a..dffb8d7 100755
--- a/tox.ini
+++ b/tox.ini
@@ -15,6 +15,7 @@
deps = -r{toxinidir}/requirements.txt
commands =
-mkdir {envtmpdir}
+ bash {toxinidir}/tools/generate_argumentfile.sh
[testenv:custom]
@@ -22,8 +23,7 @@
setenv = {[testenv]setenv}
commands =
{[testenv]commands}
- bash {toxinidir}/tools/generate_argumentfile.sh
- python -m robot.run --exclude reboot_tests --argumentfile {env:ARG_FILE} {posargs}
+ python -m robot.run --argumentfile {env:ARG_FILE} {posargs}
[testenv:barreleye]
@@ -32,7 +32,6 @@
OPENBMC_MODEL=./data/Barreleye.py
commands =
{[testenv]commands}
- bash {toxinidir}/tools/generate_argumentfile.sh
python -m robot.run --exclude reboot_tests --argumentfile {env:ARG_FILE} {posargs}
@@ -42,7 +41,6 @@
OPENBMC_MODEL=./data/Palmetto.py
commands =
{[testenv]commands}
- bash {toxinidir}/tools/generate_argumentfile.sh
python -m robot.run --exclude reboot_tests --argumentfile {env:ARG_FILE} {posargs}
@@ -52,15 +50,22 @@
OPENBMC_MODEL=./data/Firestone.py
commands =
{[testenv]commands}
- bash {toxinidir}/tools/generate_argumentfile.sh
python -m robot.run --exclude reboot_tests --argumentfile {env:ARG_FILE} {posargs}
-[testenv:Garrison]
+[testenv:garrison]
deps = {[testenv]deps}
setenv = {[testenv]setenv}
OPENBMC_MODEL=./data/Garrison.py
commands =
{[testenv]commands}
- bash {toxinidir}/tools/generate_argumentfile.sh
+ python -m robot.run --exclude reboot_tests --argumentfile {env:ARG_FILE} {posargs}
+
+
+[testenv:qemu]
+deps = {[testenv]deps}
+setenv = {[testenv]setenv}
+ OPENBMC_MODEL=./data/Palmetto.py
+commands =
+ {[testenv]commands}
python -m robot.run --exclude reboot_tests --argumentfile {env:ARG_FILE} {posargs}