Add CI approved test cases tag infrastructure
Use Robot Test Framework's use of Tags to create a subset of tests that
can be run on real and simulated hardware for a basic set of validations
against each commit.
Our yocto qemu session has real basic support and can not pass very many
of the robot tests. Not because there is a bug but simply because the
simulation does not support many of the openbmc rest interfaces (i.e.
inventory, sensors). I found some REST and Event Log test suites that
are available in QEMU to start testing now with. When we move the QEMU
simulation to more of a palmetto based model more test cases will get
the CI tag. Eventually we will be able to go to a negative model where
we mark the tests that should not be part of a CI run rather then this
small set of includes.
The tag of 'CI' was chosen over a 'QEMUCI' because I fear the use of
tags could get out of hand (BARRELEYECI vs WEDGECI vs BARRELEYE_SIM_CI,
etc).
Invoking....
To test in QEMU first launch the session, find the IP and then run...
OPENBMC_HOST=<ip> tox -e qemu -- --include CI tests
You can also run the CI suite against real hardware...
OPENBMC_HOST=<ip> tox -e barreleye -- --include CI tests
Change-Id: I2c5fbcf2457d958edf7b89c70d7990f5f97e8b75
Signed-off-by: Chris Austen <austenc@us.ibm.com>
diff --git a/tests/test_obmcrest.robot b/tests/test_obmcrest.robot
index a3121e2..92d63f9 100644
--- a/tests/test_obmcrest.robot
+++ b/tests/test_obmcrest.robot
@@ -11,6 +11,7 @@
*** Test Cases ***
Good connection for testing
+ [Tags] CI
${content}= Read Properties /
${c}= get from List ${content} 0
Should Be Equal ${c} /org
@@ -30,6 +31,7 @@
Should Be Equal ${jsondata['data']['description']} The specified property cannot be found: ''is_fru''
get directory listing /
+ [Tags] CI
${resp} = openbmc get request /
should be equal as strings ${resp.status_code} ${HTTP_OK}
${json} = to json ${resp.content}
@@ -37,6 +39,7 @@
should be equal as strings ${json['status']} ok
get directory listing /org/
+ [Tags] CI
${resp} = openbmc get request /org/
should be equal as strings ${resp.status_code} ${HTTP_OK}
${json} = to json ${resp.content}
@@ -44,60 +47,70 @@
should be equal as strings ${json['status']} ok
get invalid directory listing /i/dont/exist/
+ [Tags] CI
${resp} = openbmc get request /i/dont/exist/
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put directory listing /
+ [Tags] CI
${resp} = openbmc put request /
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put directory listing /org/
+ [Tags] CI
${resp} = openbmc put request /org/
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put invalid directory listing /i/dont/exist/
+ [Tags] CI
${resp} = openbmc put request /i/dont/exist/
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
post directory listing /
+ [Tags] CI
${resp} = openbmc post request /
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
post directory listing /org/
+ [Tags] CI
${resp} = openbmc post request /org/
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
post invalid directory listing /i/dont/exist/
+ [Tags] CI
${resp} = openbmc post request /i/dont/exist/
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
delete directory listing /
+ [Tags] CI
${resp} = openbmc delete request /
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
delete directory listing /org/
+ [Tags] CI
${resp} = openbmc delete request /
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
delete invalid directory listing /org/nothere/
+ [Tags] CI
${resp} = openbmc delete request /org/nothere/
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
@@ -118,60 +131,70 @@
should be equal as strings ${json['status']} ok
get invalid list names /i/dont/exist/
+ [Tags] CI
${resp} = openbmc get request /i/dont/exist/list
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put list names /
+ [Tags] CI
${resp} = openbmc put request /list
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put list names /org/
+ [Tags] CI
${resp} = openbmc put request /org/list
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put invalid list names /i/dont/exist/
+ [Tags] CI
${resp} = openbmc put request /i/dont/exist/list
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
post list names /
+ [Tags] CI
${resp} = openbmc post request /list
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
post list names /org/
+ [Tags] CI
${resp} = openbmc post request /org/list
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
post invalid list names /i/dont/exist/
+ [Tags] CI
${resp} = openbmc post request /i/dont/exist/list
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
delete list names /
+ [Tags] CI
${resp} = openbmc delete request /list
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
delete list names /org/
+ [Tags] CI
${resp} = openbmc delete request /list
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
delete invalid list names /org/nothere/
+ [Tags] CI
${resp} = openbmc delete request /org/nothere/list
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
@@ -192,84 +215,98 @@
should be equal as strings ${json['status']} ok
get invalid names /i/dont/exist/
+ [Tags] CI
${resp} = openbmc get request /i/dont/exist/enumerate
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put names /
+ [Tags] CI
${resp} = openbmc put request /enumerate
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put names /org/
+ [Tags] CI
${resp} = openbmc put request /org/enumerate
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put invalid names /i/dont/exist/
+ [Tags] CI
${resp} = openbmc put request /i/dont/exist/enumerate
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
post names /
+ [Tags] CI
${resp} = openbmc post request /enumerate
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
post names /org/
+ [Tags] CI
${resp} = openbmc post request /org/enumerate
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
post invalid names /i/dont/exist/
+ [Tags] CI
${resp} = openbmc post request /i/dont/exist/enumerate
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
delete names /
+ [Tags] CI
${resp} = openbmc delete request /enumerate
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
delete names /org/
+ [Tags] CI
${resp} = openbmc delete request /enumerate
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
delete invalid names /org/nothere/
+ [Tags] CI
${resp} = openbmc delete request /org/nothere/enumerate
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
get method org/openbmc/records/events/action/acceptTestMessage
+ [Tags] CI
${resp} = openbmc get request org/openbmc/records/events/action/acceptTestMessage
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
get invalid method /i/dont/exist/
+ [Tags] CI
${resp} = openbmc get request /i/dont/exist/action/foo
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put method org/openbmc/records/events/action/acceptTestMessage
+ [Tags] CI
${resp} = openbmc put request org/openbmc/records/events/action/acceptTestMessage
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
put invalid method /i/dont/exist/
+ [Tags] CI
${resp} = openbmc put request /i/dont/exist/action/foo
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
@@ -284,6 +321,7 @@
should be equal as strings ${json['status']} ok
post method org/openbmc/records/events/action/acceptTestMessage invalid args
+ [Tags] CI
${data} = create dictionary foo=bar
${resp} = openbmc post request org/openbmc/records/events/action/acceptTestMessage data=${data}
should be equal as strings ${resp.status_code} ${HTTP_BAD_REQUEST}
@@ -303,18 +341,21 @@
Should Be Equal ${content} ${COUNT}
delete method org/openbmc/records/events/action/acceptTestMessage
+ [Tags] CI
${resp} = openbmc delete request org/openbmc/records/events/action/acceptTestMessage
should be equal as strings ${resp.status_code} ${HTTP_METHOD_NOT_ALLOWED}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
delete invalid method /org/nothere/
+ [Tags] CI
${resp} = openbmc delete request /org/nothere/action/foomethod
should be equal as strings ${resp.status_code} ${HTTP_NOT_FOUND}
${json} = to json ${resp.content}
should be equal as strings ${json['status']} error
post method org/openbmc/records/events/action/acceptTestMessage no args
+ [Tags] CI
${data} = create dictionary data=@{EMPTY}
${resp} = openbmc post request org/openbmc/records/events/action/acceptTestMessage data=${data}
should be equal as strings ${resp.status_code} ${HTTP_OK}