Method to split FFDC files by time stamp
There is no major change from the existing logic but more of a
restructuring for maintainability.
The changes introduces the following functionaliies
- Split FFDC into files by failure time stamp
- Remove SCP logic and use execute and create logic flow
- Add method call to directly invoke user define keywords
as part of the default FFDC list
- Method to save test case execution history
Resolves openbmc/openbmc-test-automation#77
Change-Id: Ia16fdde942a5e9cbf6909f1530e9c6f2a1c6ab0a
Signed-off-by: George Keishing <gkeishin@in.ibm.com>
diff --git a/lib/openbmc_ffdc.robot b/lib/openbmc_ffdc.robot
index eb0bd71..0215234 100644
--- a/lib/openbmc_ffdc.robot
+++ b/lib/openbmc_ffdc.robot
@@ -1,156 +1,63 @@
*** Settings ***
-Documentation This module is for collecting data on test case failure
-... for openbmc systems. It will collect the data with a
-... default name openbmc_ffdc_report.txt under directory
-... logs/testSuite/testcaseName/ on failure.
+Documentation This module is for data collection on test case failure
+... for openbmc systems. Collects data with default name
+... ffdc_report.txt under directory logs/testSuite/testName/
+... on failure.
+... FFDC logging sample layout:
+... logs
+... ├── 20160909102538035251_TestWarmreset
+... │ └── 20160909102538035251_TestWarmResetviaREST
+... │ ├── 20160909102538035251_BMC_journalctl.log
+... │ ├── 20160909102538035251_BMC_proc_list
+... │ └── 20160909102538035251_ffdc_report.txt
+... └── test_history.txt
-Library String
-Library DateTime
-Library openbmc_ffdc_list.py
-
-Resource resource.txt
-Resource connection_client.robot
-
-*** Variables ***
-
-${PRINT_LINE} ------------------------------------------------------------------------
-
-${MSG_INTRO} This document contains the following information:
-${MSG_DETAIL} ${\n}\t\t[ Detailed Logs Captured Section ]
-${HEADER_MSG} ${\n}${PRINT_LINE}${\n}\t\tOPEN BMC TEST FAILURE DATA CAPTURE
-... ${\n}\t\t----------------------------------
-... ${\n}${\n}TEST SUITE FILE\t\t: ${SUITE_NAME} ${\n}
-${FOOTER_MSG} ${PRINT_LINE} ${\n}
-
-${FFDC_LOG_PATH} ${EXECDIR}${/}logs${/}
+Resource openbmc_ffdc_methods.robot
+Resource openbmc_ffdc_utils.robot
*** Keywords ***
Log FFDC
[Documentation] Generic FFDC entry point. Place holder to hook in
- ... other data collection
+ ... other data collection methods
+ ... 1. Collect Logs if test fails
+ ... 2. Added Test execution history logging
+ ... By default this will log Test status PASS/FAIL format
+ ... EX: 20160822041250932049:Test:Test case 1:PASS
+ ... 20160822041250969913:Test:Test case 2:FAIL
- Log FFDC If Test Case Failed
+ Run Keyword If '${TEST_STATUS}' == 'FAIL'
+ ... Log FFDC If Test Case Failed
+
+ Log Test Case Status
Log FFDC If Test Case Failed
[Documentation] Main entry point to gather logs on Test case failure
+ ... 1. Set global FFDC time reference for a failure
+ ... 2. Create FFDC work space directory
+ ... 3. Write test info details
+ ... 4. Calls BMC methods to write/collect FFDC data
- # Return from here if the test case is a PASS
- Return From Keyword If '${TEST_STATUS}' != 'FAIL'
+ ${cur_time}= Get Current Time Stamp
+ Set Global Variable ${FFDC_TIME} ${cur_time}
+ Log To Console ${\n}FFDC Collection Started \t: ${cur_time}
- ${cur_time}= get current time stamp
- Log To Console ${\n}FFDC Collection Started \t: ${cur_time}
# Log directory setup
- ${suite_dir}= get strip string ${SUITE_NAME}
- ${testname_dir}= get strip string ${TEST_NAME}
+ ${suitename} ${testname}= Get Test Dir and Name
- Set Suite Variable ${FFDC_DIR_PATH} ${FFDC_LOG_PATH}${suite_dir}${/}${testname_dir}
+ Set Global Variable
+ ... ${FFDC_DIR_PATH} ${FFDC_LOG_PATH}${suitename}${/}${testname}
- # -- FFDC workspace create --
- create ffdc directory
- openbmc header message
+ ${prefix}= Catenate SEPARATOR= ${FFDC_DIR_PATH}${/} ${FFDC_TIME}_
+ Set Global Variable ${LOG_PREFIX} ${prefix}
+
+ Create FFDC Directory
+ Header Message
# -- FFDC processing entry point --
- Execute FFDC command list on BMC
+ Call FFDC Methods
- ${cur_time}= get current time stamp
+ ${cur_time}= Get Current Time Stamp
Log To Console FFDC Collection Completed \t: ${cur_time}
Log ${\n}${FFDC_DIR_PATH}
-
-
-create ffdc directory
- [Documentation] Creates directory and report file
- Create Directory ${FFDC_DIR_PATH}
- create ffdc report file
-
-
-create ffdc report file
- [Documentation] Create a generic file name for ffdc
- Set Suite Variable ${FFDC_FILE_PATH} ${FFDC_DIR_PATH}${/}openbmc_ffdc_report.txt
- Create File ${FFDC_FILE_PATH}
-
-
-write data to file
- [Documentation] Write data to the ffdc report document
- [Arguments] ${data}=""
- Append To File ${FFDC_FILE_PATH} ${data}
-
-
-get current time stamp
- [Documentation] Get the current time stamp data
- ${cur_time}= Get Current Date result_format=%Y-%m-%d %H:%M:%S,%f
- [return] ${cur_time}
-
-openbmc header message
- [Documentation] Write header message to the report document
- ${cur_time}= get current time stamp
- write data to file ${HEADER_MSG}
- write data to file TEST CASE NAME\t\t: ${TEST_NAME}${\n}
- write data to file FAILURE TIME STAMP\t: ${cur_time}${\n}
- write data to file ${\n}${MSG_INTRO}${\n}
-
- # --- FFDC header notes ---
- @{entries}= Get ffdc index
- :FOR ${index} IN @{entries}
- \ write data to file * ${index.upper()}
- \ write data to file ${\n}
-
- write data to file ${FOOTER_MSG}
- write data to file ${MSG_DETAIL}
-
-write cmd output to ffdc file
- [Documentation] Write cmd output data to the report document
- [Arguments] ${data_str}="" ${data_cmd}=""
- write data to file ${\n}${FOOTER_MSG}
- write data to file ${ENTRY_CMD_TYPE.upper()} : ${data_str}\t
- write data to file Executed : ${data_cmd} ${\n}
- write data to file ${FOOTER_MSG}
-
-
-Execute FFDC command list on BMC
- [Documentation] Get the commands, connect to BMC and execute commands
- ${con_status}= Run Keyword And Return Status Open Connection And Log In
- Run Keyword And Return If ${con_status} == ${False} Log Open Connection Failed
-
- @{entries}= Get ffdc index
- :FOR ${index} IN @{entries}
- \ Loop through ffdc dict list and execute ${index}
-
-
-Loop through ffdc dict list and execute
- [Documentation] Feed in key pair list from dictionary to execute
- [Arguments] ${data_str}=
- @{ffdc_default_list}= Get ffdc cmd ${data_str}
-
- Set Suite Variable ${ENTRY_CMD_TYPE} ${data_str}
- :FOR ${cmd} IN @{ffdc_default_list}
- \ Execute command and write to ffdc ${cmd[0]} ${cmd[1]}
-
-
-Execute command and write to ffdc
- [Documentation] Execute command on bmc box and write to ffdc
- [Arguments] ${data_str}="" ${data_cmd}=""
- write cmd output to ffdc file ${data_str} ${data_cmd}
-
- ${stdout} ${stderr}= Execute Command ${data_cmd} return_stderr=True
- # Write stdout data on success and error msg to the file on failure
- Run Keyword If '${stderr}' == '${EMPTY}' write data to file ${stdout} ${\n}
- ... ELSE Run Keyword write data to file ${stderr} ${\n}
- write data to file ${FOOTER_MSG}
-
-
-Offload file list from BMC
- [Documentation] Copy files to current log directory
- ${con_status}= Run Keyword And Return Status Open Connection for SCP
- Run Keyword And Return If ${con_status} == ${False} Log SCP Connection Failed
-
- # --- Files to be copied ---
- @{ffdc_default_list}= Get ffdc file BMC Files
- Set Suite Variable ${ENTRY_CMD_TYPE} BMC Files
-
- :FOR ${cmd} IN @{ffdc_default_list}
- # Get File from server to current test FFDC directory
- \ write cmd output to ffdc file ${cmd[0]} scp file ${cmd[1]}
- \ scp.Get File ${cmd[1]} ${FFDC_DIR_PATH}
-
diff --git a/lib/openbmc_ffdc_list.py b/lib/openbmc_ffdc_list.py
index 3a2950a..15cd86f 100644
--- a/lib/openbmc_ffdc_list.py
+++ b/lib/openbmc_ffdc_list.py
@@ -2,7 +2,6 @@
'''
#############################################################
# @file openbmc_ffdc_list.py
-# @author: George Keishing
#
# @brief List for FFDC ( First failure data capture )
# commands and files to be collected as a part
@@ -17,17 +16,20 @@
#-----------------------------------------------------------------
#Dict Name { Index string : { Key String : Comand string} }
#-----------------------------------------------------------------
-FFDC_CMD = {
+# Add cmd's needed to be part of the ffdc report manifest file
+FFDC_BMC_CMD = {
'DRIVER INFO' :
{
- 'FW Level' : 'cat /etc/os-release',
- 'OS Details' : 'uname -a',
+ #String Name Command
'Build Info' : 'cat /etc/version',
+ 'FW Level' : 'cat /etc/os-release',
},
'BMC DATA' :
{
- 'System journal log' : 'journalctl --no-pager',
- 'Displays processor activity' : 'top -n 1 -b',
+ 'BMC OS' : 'uname -a',
+ 'BMC Uptime' : 'uptime',
+ 'BMC Proc Info' : 'cat /proc/cpuinfo',
+ 'BMC File System Disk Space Usage' : 'df -hT',
},
'APPLICATION DATA' :
{
@@ -35,13 +37,23 @@
},
}
-# add file list needed to be offload from BMC
-FFDC_FILE = {
+# Add file name and correcponding command needed for BMC
+FFDC_BMC_FILE = {
'BMC FILES' :
{
- # Sample example how to add the file that
- # is needed to be offloaded
- #'Release info' : '/etc/os-release',
+ #File Name Command
+ 'BMC_proc_list' : 'top -n 1 -b',
+ 'BMC_journalctl.log' : 'journalctl --no-pager',
+ },
+ }
+
+# Define your keywords in method/utils and call here
+FFDC_METHOD_CALL = {
+ 'BMC LOGS' :
+ {
+ #Description Keyword name
+ 'FFDC Generic Report' : 'BMC FFDC Manifest',
+ 'BMC Specific Files' : 'BMC FFDC Files',
},
}
@@ -52,27 +64,48 @@
class openbmc_ffdc_list():
########################################################################
- # @@brief This method returns the list from the dictionary for cmds
+ # @brief This method returns the list from the dictionary for cmds
# @param i_type: @type string: string index lookup
# @return List of key pair from the dictionary
########################################################################
- def get_ffdc_cmd(self,i_type):
- return FFDC_CMD[i_type].items()
+ def get_ffdc_bmc_cmd(self,i_type):
+ return FFDC_BMC_CMD[i_type].items()
########################################################################
- # @@brief This method returns the list from the dictionary for scp
+ # @brief This method returns the list from the dictionary for scp
# @param i_type: @type string: string index lookup
# @return List of key pair from the dictionary
########################################################################
- def get_ffdc_file(self,i_type):
- return FFDC_FILE[i_type].items()
+ def get_ffdc_bmc_file(self,i_type):
+ return FFDC_BMC_FILE[i_type].items()
########################################################################
- # @@brief This method returns the list index from dictionary
+ # @brief This method returns the list index from dictionary
# @return List of index to the dictionary
########################################################################
- def get_ffdc_index(self):
- return FFDC_CMD.keys()
+ def get_ffdc_cmd_index(self):
+ return FFDC_BMC_CMD.keys()
+
+ ########################################################################
+ # @brief This method returns the list index from dictionary
+ # @return List of index to the dictionary
+ ########################################################################
+ def get_ffdc_file_index(self):
+ return FFDC_BMC_FILE.keys()
+
+ ########################################################################
+ # @brief This method returns the key pair from the dictionary
+ # @return Index of the method dictionary
+ ########################################################################
+ def get_ffdc_method_index(self):
+ return FFDC_METHOD_CALL.keys()
+
+ ########################################################################
+ # @brief This method returns the key pair from the dictionary
+ # @return List of key pair keywords
+ ########################################################################
+ def get_ffdc_method_call(self,i_type):
+ return FFDC_METHOD_CALL[i_type].items()
########################################################################
# @brief Returns the stripped strings
@@ -81,4 +114,3 @@
########################################################################
def get_strip_string(self, i_str):
return ''.join(e for e in i_str if e.isalnum())
-
diff --git a/lib/openbmc_ffdc_methods.robot b/lib/openbmc_ffdc_methods.robot
new file mode 100644
index 0000000..b3f7e5e
--- /dev/null
+++ b/lib/openbmc_ffdc_methods.robot
@@ -0,0 +1,136 @@
+*** Settings ***
+Documentation Methods to execute commands on BMC and collect
+... data to a list of FFDC files
+
+Resource openbmc_ffdc_utils.robot
+
+*** Keywords ***
+
+################################################################
+# Method : Call FFDC Methods #
+# Execute the user define keywords from the FFDC List #
+# Unlike any other keywords this will call into the #
+# list of keywords defined in the FFDC list at one go #
+################################################################
+
+Call FFDC Methods
+ [Documentation] Calls into FFDC Keyword index list
+
+ @{entries}= Get ffdc method index
+ :FOR ${index} IN @{entries}
+ \ Method Call Keyword List ${index}
+
+
+Method Call Keyword List
+ [Documentation] Iterate the list through keyword index
+ [Arguments] ${index}
+
+ @{method_list}= Get ffdc method call ${index}
+ :FOR ${method} IN @{method_list}
+ \ Execute Keyword Method ${method[1]}
+
+
+Execute Keyword Method
+ [Documentation] Calls into BMC method keywords. Don't let one
+ ... failure skips the remaining. Get whatever data
+ ... it could gather at worse case scenario.
+ [Arguments] ${keyword_name}
+
+ Run Keyword And Continue On Failure ${keyword_name}
+
+
+################################################################
+# Method : BMC FFDC Manifest #
+# Execute command on BMC and write to ffdc_report.txt #
+################################################################
+
+BMC FFDC Manifest
+ [Documentation] Get the commands index for the FFDC_BMC_CMD,
+ ... login to BMC and execute commands.
+ Open Connection And Log In
+
+ @{entries}= Get ffdc cmd index
+ :FOR ${index} IN @{entries}
+ \ Iterate BMC Command List Pairs ${index}
+
+
+Iterate BMC Command List Pairs
+ [Documentation] Feed in key pair list from dictionary to execute
+ [Arguments] ${key_index}
+
+ @{cmd_list}= Get ffdc bmc cmd ${key_index}
+ Set Suite Variable ${ENTRY_INDEX} ${key_index}
+ :FOR ${cmd} IN @{cmd_list}
+ \ Execute Command and Write FFDC ${cmd[0]} ${cmd[1]}
+
+
+Execute Command and Write FFDC
+ [Documentation] Execute command on BMC and write to ffdc
+ ... By default to ffdc_report.txt file else to
+ ... specified file path.
+ [Arguments] ${key_index}
+ ... ${cmd}
+ ... ${logpath}=${FFDC_FILE_PATH}
+
+ Run Keyword If '${logpath}' == '${FFDC_FILE_PATH}'
+ ... Write Cmd Output to FFDC File ${key_index} ${cmd}
+
+ ${stdout} ${stderr}=
+ ... Execute Command ${cmd} return_stderr=True
+
+ # Write stdout data on success and error msg to the file on failure
+ Run Keyword If '${stderr}' == '${EMPTY}'
+ ... Write Data to File ${stdout}${\n} ${logpath}
+ ... ELSE Run Keyword Write Data to File ${stderr}${\n} ${logpath}
+
+
+################################################################
+# Method : BMC FFDC Files #
+# Execute command on BMC and write to individual file #
+# based on the file name pre-defined in the list #
+################################################################
+
+BMC FFDC Files
+ [Documentation] Get the command list and iterate
+ Open Connection And Log In
+ @{entries}= Get ffdc file index
+ :FOR ${index} IN @{entries}
+ \ Create File and Write Data ${index}
+
+
+Create File and Write Data
+ [Documentation] Create files to current FFDC log directory,
+ ... executes command and write to corresponding
+ ... file name in the current FFDC directory.
+ [Arguments] ${key_index}
+
+ @{cmd_list}= Get ffdc bmc file ${key_index}
+ :FOR ${cmd} IN @{cmd_list}
+ \ ${logpath}= Catenate SEPARATOR= ${LOG_PREFIX} ${cmd[0]}
+ \ Execute Command and Write FFDC ${cmd[0]} ${cmd[1]} ${logpath}
+
+
+################################################################
+# Method : Log Test Case Status #
+# Creates test result history footprint for reference #
+################################################################
+
+Log Test Case Status
+ [Documentation] Test case execution result history.
+ ... Create once and append to this file
+ ... logs/test_history.txt
+ ... Format Date:Test suite:Test case:Status
+ ... 20160909214053719992:Test Warmreset:Test WarmReset via REST:FAIL
+ Create Directory ${FFDC_LOG_PATH}
+
+ ${exist}= Run Keyword and Return Status
+ ... OperatingSystem.File Should Exist ${TEST_HISTORY}
+
+ Run Keyword If '${exist}' == '${False}'
+ ... Create File ${TEST_HISTORY}
+
+ ${cur_time}= Get Current Time Stamp
+
+ Append To File ${TEST_HISTORY}
+ ... ${cur_time}:${SUITE_NAME}:${TEST_NAME}:${TEST_STATUS}${\n}
+
diff --git a/lib/openbmc_ffdc_utils.robot b/lib/openbmc_ffdc_utils.robot
new file mode 100644
index 0000000..b4890ab
--- /dev/null
+++ b/lib/openbmc_ffdc_utils.robot
@@ -0,0 +1,113 @@
+*** Settings ***
+Documentation Utility keywords for FFDC
+
+Library String
+Library DateTime
+Library openbmc_ffdc_list.py
+Resource resource.txt
+Resource connection_client.robot
+
+*** Variables ***
+
+${PRINT_LINE} ------------------------------------------------------------------------
+
+${MSG_INTRO} This report contains the following information:
+${MSG_DETAIL} ${\n}\t\t[ Detailed Logs Captured Section ]
+${HEADER_MSG} ${\n}\t\t---------------------------
+... ${\n}\t\t FIRST FAILURE DATA CAPTURE
+... ${\n}\t\t---------------------------
+${FOOTER_MSG} ${\n}${PRINT_LINE} ${\n}
+
+${FFDC_LOG_PATH} ${EXECDIR}${/}logs${/}
+${TEST_HISTORY} ${FFDC_LOG_PATH}${/}test_history.txt
+
+*** Keywords ***
+
+Get Test Dir and Name
+ [Documentation] SUITE_NAME and TEST_NAME are automatic variables
+ ... and is populated dynamically by the robot framework
+ ... during execution
+ ${suite_name}= Get strip string ${SUITE_NAME}
+ ${suite_name}= Catenate SEPARATOR= ${FFDC_TIME}_ ${suite_name}
+ ${test_name}= Get strip string ${TEST_NAME}
+ ${test_name}= Catenate SEPARATOR= ${FFDC_TIME}_ ${test_name}
+ [return] ${suite_name} ${test_name}
+
+
+Create FFDC Directory
+ [Documentation] Creates directory and report file
+ Create Directory ${FFDC_DIR_PATH}
+ Create FFDC Report File
+
+
+Create FFDC Report File
+ [Documentation] Create a generic file name for ffdc
+ Set Suite Variable
+ ... ${FFDC_FILE_PATH} ${FFDC_DIR_PATH}${/}${FFDC_TIME}_ffdc_report.txt
+ Create File ${FFDC_FILE_PATH}
+
+
+Write Data to File
+ [Documentation] Write data to the ffdc report document
+ [Arguments] ${data}= ${filepath}=${FFDC_FILE_PATH}
+ Append To File ${filepath} ${data}
+
+
+Get Current Time Stamp
+ [Documentation] Get the current time stamp data
+ ${cur_time}= Get Current Date result_format=%Y-%m-%d %H:%M:%S:%f
+ ${cur_time}= Get strip string ${cur_time}
+ [return] ${cur_time}
+
+
+Header Message
+ [Documentation] Write header message to the report document manifest.
+ ... TEST_NAME, TEST_MESSAGE,SUITE_SOURCE,TEST_DOCUMENTATION
+ ... are auto variables and are populated dynamically by the
+ ... robot framework during execution
+ ... 1. Writes opening statement headers msg
+ ... 2. Add Test setup and config information
+ ... 3. Types of data collection
+
+ Write Data to File ${HEADER_MSG}
+ Write Data to File ${FOOTER_MSG}
+ Write Data to File Test Suite File\t\t: ${SUITE_NAME} ${\n}
+ Write Data to File Test Case Name\t\t: ${TEST_NAME}${\n}
+ Write Data to File Test Source File\t: ${SUITE_SOURCE}${\n}
+ Write Data to File Failure Time Stamp\t: ${FFDC_TIME}${\n}
+ Write Data to File Test Error Message\t: ${TEST_MESSAGE}${\n}
+ Write Data to File Test Documentation\t:${\n}${TEST_DOCUMENTATION}${\n}
+ Write Data to File ${FOOTER_MSG}
+
+ Test Setup Info
+
+ Write Data to File ${\n}${MSG_INTRO}${\n}
+
+ # --- FFDC header notes ---
+ @{entries}= Get ffdc cmd index
+ :FOR ${index} IN @{entries}
+ \ Write Data to File * ${index.upper()}
+ \ Write Data to File ${\n}
+
+ Write Data to File ${MSG_DETAIL}
+
+
+Write Cmd Output to FFDC File
+ [Documentation] Write cmd output data to the report document
+ [Arguments] ${name_str} ${cmd}
+
+ Write Data to File ${FOOTER_MSG}
+ Write Data to File ${ENTRY_INDEX.upper()} : ${name_str}\t
+ Write Data to File Executed : ${cmd}
+ Write Data to File ${FOOTER_MSG}
+
+
+Test Setup Info
+ [Documentation] BMC IP, Model and other information
+
+ Write Data to File ${\n}-----------------------${\n}
+ Write Data to File Test Setup Information:
+ Write Data to File ${\n}-----------------------${\n}
+ Write Data to File OPENBMC HOST \t: ${OPENBMC_HOST}${\n}
+ Write Data to File
+ ... SYSTEM TYPE \t: ${OPENBMC_MODEL.replace('./data/','').replace('.py','')}${\n}${\n}