Enhance the syslib library
Enhance syslib/utils_os.robot by
- Adding messages to the "Should Be Equal" keywords.
- Enhancing the "Is HTX Running" keyword to return status
rather than terminating the run.
- Update the systest/* tests to use the revised
"Is HTX Running" keyword.
- Display BMC error logs before termination.
Resolves openbmc/openbmc-test-automation#1304
Change-Id: I17ab8d2deafbe1ef2a232d8356e78740b1b20991
Signed-off-by: Steven Sombar <ssombar@us.ibm.com>
diff --git a/syslib/utils_os.robot b/syslib/utils_os.robot
index a637f61..d18dd20 100755
--- a/syslib/utils_os.robot
+++ b/syslib/utils_os.robot
@@ -39,9 +39,10 @@
# Description of argument(s):
# command Shell command to be executed on OS.
${stdout} ${stderr}= Execute Command ${command} return_stderr=True
- Should Be Empty ${stderr}
+ Should Be Empty ${stderr} msg=Failed running ${command}. stderr=${stderr}
[Return] ${stdout}
+
Login To OS
[Documentation] Login to OS Host.
[Arguments] ${os_host}=${OS_HOST} ${os_username}=${OS_USERNAME}
@@ -52,7 +53,6 @@
# os_username OS Host Login user name.
# os_password OS Host Login passwrd.
# alias_name Default OS SSH session connection alias name.
- # TODO: Generalize alias naming using openbmc/openbmc-test-automation#633
Ping Host ${os_host}
SSHLibrary.Open Connection ${os_host} alias=${alias_name}
@@ -92,10 +92,23 @@
Is HTX Running
- [Documentation] Check if the HTX exerciser is currently running.
+ [Documentation] Return "True" if the HTX is running, "False"
+ ... otherwise.
+
+ # Example usage:
+ # ${status}= Is HTX Running
+ # Run Keyword If '${status}' == 'True' Shutdown HTX Exerciser
+
${status}= Execute Command On OS htxcmdline -status
- Should Not Contain ${status} Daemon state is <IDLE>
+
+ ${match_count_idle}= Count Values In List ${status}
+ ... Daemon state is <IDLE>
+ ${match_count_not_running}= Count Values In List ${status}
+ ... No MDT is currently running
+ ${running}= Evaluate
+ ... not (${match_count_idle} or ${match_count_not_running})
+ [Return] ${running}
Write Log Data To File
@@ -150,6 +163,7 @@
${resp}= Post Request openbmc /upload/image &{data}
Should Be Equal As Strings ${resp.status_code} ${HTTP_OK}
+ ... msg=Openbmc /upload/image failed.
# Take SSH connection to BMC and switch to BMC connection to perform
# the task.
@@ -219,6 +233,7 @@
${dmesg_log}= Execute Command On OS dmesg | egrep '${ERROR_REGEX}'
# To enable multiple string check.
Should Not Contain ${dmesg_log} ${ERROR_DBE_MSG}
+ ... msg=OS dmesg shows ${ERROR_DBE_MSG}.
Collect NVIDIA Log File
@@ -395,6 +410,7 @@
${profile}= Execute Command On OS htxcmdline -createmdt
Rprintn ${profile}
Should Contain ${profile} mdts are created successfully
+ ... msg=Create MDT profile failed. response=${profile}
Run MDT Profile
@@ -405,20 +421,22 @@
... htxcmdline -run -mdt ${HTX_MDT_PROFILE}
Rprintn ${htx_run}
Should Contain ${htx_run} Activated
+ ... msg=htxcmdline run mdt did not return "Activated" status.
Check HTX Run Status
[Documentation] Get HTX exerciser status and check for error.
Rprint Timen Check HTX mdt Status and error.
- ${status}= Execute Command On OS
+ ${htx_status}= Execute Command On OS
... htxcmdline -status -mdt ${HTX_MDT_PROFILE}
- Rprintn ${status}
+ Rprintn ${htx_status}
- ${errlog}= Execute Command On OS htxcmdline -geterrlog
- Rprintn ${errlog}
+ ${htx_errlog}= Execute Command On OS htxcmdline -geterrlog
+ Rprintn ${htx_errlog}
- Should Contain ${errlog} file </tmp/htxerr> is empty
+ Should Contain ${htx_errlog} file </tmp/htxerr> is empty
+ ... msg=HTX geterrorlog was not empty.
Shutdown HTX Exerciser
@@ -437,7 +455,6 @@
... Fail msg=Shutdown command returned unexpected message.
-
Create JSON Inventory File
[Documentation] Create a JSON inventory file, and make a YAML copy.
[Arguments] ${json_file_path}
@@ -450,6 +467,7 @@
# Format to JSON pretty print to file.
Run python -m json.tool ${json_tmp_file_path} > ${json_file_path}
OperatingSystem.File Should Exist ${json_file_path}
+ ... msg=File ${json_file_path} does not exist.
Compile Inventory JSON
@@ -461,6 +479,7 @@
Retrieve HW Info And Write List ${I/O} ${json_tmp_file_path} I/O last
Close New JSON List ${json_tmp_file_path}
+
Write New JSON List
[Documentation] Start a new JSON list element in file.
[Arguments] ${json_tmp_file_path} ${json_field_name}
@@ -469,6 +488,7 @@
# json_field_name Name to give json list element.
Append to File ${json_tmp_file_path} { "${json_field_name}" : [
+
Close New JSON List
[Documentation] Close JSON list element in file.
[Arguments] ${json_tmp_file_path}
@@ -476,6 +496,7 @@
# json_tmp_file_path Path of file to write to.
Append to File ${json_tmp_file_path} ]}
+
Retrieve HW Info And Write
[Documentation] Retrieve and write info, add a comma if not last item.
[Arguments] ${class} ${json_tmp_file_path} ${last}=false
@@ -492,6 +513,7 @@
Run Keyword if '${last}' == 'false'
... Append to File ${json_tmp_file_path} ,
+
Retrieve HW Info And Write List
[Documentation] Does a Retrieve/Write with a list of classes and
... encapsulates them into one large JSON element.
@@ -512,6 +534,7 @@
Run Keyword if '${last}' == 'false'
... Append to File ${json_tmp_file_path} ,
+
Retrieve Hardware Info
[Documentation] Retrieves the lshw output of the device class as JSON.
[Arguments] ${class}
@@ -521,6 +544,7 @@
${output} = Verify JSON string ${output}
[Return] ${output}
+
Verify JSON String
[Documentation] Ensure the JSON string content is separated by commas.
[Arguments] ${unver_string}
@@ -530,6 +554,7 @@
${ver_string} = Replace String Using Regexp ${unver_string} }\\s*{ },{
[Return] ${ver_string}
+
Clean Up String
[Documentation] Remove extra whitespace and trailing commas.
[Arguments] ${dirty_string}
diff --git a/systest/gpu_stress_test.robot b/systest/gpu_stress_test.robot
index f2f9ad8..8788ad5 100755
--- a/systest/gpu_stress_test.robot
+++ b/systest/gpu_stress_test.robot
@@ -57,7 +57,7 @@
# Shutdown HTX if it is already running.
- ${status}= Run Keyword And Return Status Is HTX Running
+ ${status}= Is HTX Running
Run Keyword If '${status}' == 'True'
... Shutdown HTX Exerciser
diff --git a/systest/htx_hardbootme_test.robot b/systest/htx_hardbootme_test.robot
index c1b2fd5..853241c 100755
--- a/systest/htx_hardbootme_test.robot
+++ b/systest/htx_hardbootme_test.robot
@@ -38,8 +38,10 @@
Resource ../syslib/utils_os.robot
Resource ../lib/openbmc_ffdc_utils.robot
+Resource ../lib/logging_utils.robot
Library ../syslib/utils_keywords.py
Library ../lib/utils_files.py
+Library ../lib/logging_utils.py
Suite Setup Run Keyword Start SOL Console Logging
Test Setup Test Setup Execution
@@ -138,7 +140,13 @@
... Do Inventory And Compare ${json_final_file_path}
... ${PREV_INV_FILE_PATH}
- Error Logs Should Not Exist
+ # Terminate run if there are any BMC error logs.
+ ${error_logs}= Get Error Logs
+ ${num_logs}= Get Length ${error_logs}
+ Run Keyword If ${num_logs} != 0 Run Keywords
+ ... Print Error Logs ${error_logs}
+ ... AND Fail msg=Terminating run due to BMC error log(s).
+
Power Off Host
# Close all SSH and REST active sessions.
@@ -183,6 +191,7 @@
... ${file2} ${json_diff_file_path} ${INV_IGNORE_LIST}
Run Keyword If '${diff_rc}' != '${0}'
... Report Inventory Mismatch ${diff_rc} ${json_diff_file_path}
+ ... ELSE Rprint Timen Inventoy check: No differences found.
Report Inventory Mismatch
@@ -217,7 +226,7 @@
Tool Exist htxcmdline
# Shutdown if HTX is running.
- ${status}= Run Keyword And Return Status Is HTX Running
+ ${status}= Is HTX Running
Run Keyword If '${status}' == 'True'
... Shutdown HTX Exerciser
diff --git a/systest/network_stability_test.robot b/systest/network_stability_test.robot
index da09939..5194a71 100755
--- a/systest/network_stability_test.robot
+++ b/systest/network_stability_test.robot
@@ -82,7 +82,7 @@
Tool Exist htxcmdline
# Shutdown if HTX is running.
- ${status}= Run Keyword And Return Status Is HTX Running
+ ${status}= Is HTX Running
Run Keyword If '${status}' == 'True'
... Shutdown HTX Exerciser