Changed obmc_boot_test to improve usability by robot programs.

- Created new obmc_boot_test_resource.robot to define parms for
  obmc_boot_test.robot.  This is to make it possible for other robot
  programs to have access to obmc_boot_functionality.
- Created process_pgm_parms function to process program parameters.
  Previously, this was done in the program prolog.
- Added support for new stack_mode parameter.  When stack mode is
  "normal", the program behaves as it always has.  In "skip" mode,
  when each boot test is pulled from the stack, the boot test is
  skipped if the system is already in the desired end state for
  the boot test in question.  This feature allows a user to run
  a boot like "REST Power Off" only if the machine is not already
  powered off.
- Renamed main_py to obmc_boot_test so that other robot files can
  call it and have it be more self documenting.  Example:
  OBMC Boot Test \ REST Power Off
- Added a single alt_boot_stack parm to the new obmc_boot_test
  function.  Again, this allows the user to call and pass a boot_stack
  as shown in the prior example.
- Added support for new boot_fail_threshold parameter.  This allows
  a caller to indicate how many failures are acceptable.  If the
  number of boot failures exceeds boot_fail_threshold, then the
  OBMC Boot Test keyword will fail.
- Switched many gen_robot_print references to gen_print.
- Made many changes to use new run_key function.

Change-Id: I56aad535dac30ab516fdc530866c09dcbca277f3
Signed-off-by: Michael Walsh <micwalsh@us.ibm.com>
diff --git a/extended/obmc_boot_test.robot b/extended/obmc_boot_test.robot
index f2bef39..c6031a0 100644
--- a/extended/obmc_boot_test.robot
+++ b/extended/obmc_boot_test.robot
@@ -1,62 +1,9 @@
 *** Settings ***
 Documentation  Do random repeated boots based on the state of the BMC machine.
 
-Library   state.py
-Library   obmc_boot_test.py
-
-Resource  openbmc_ffdc.robot
+Resource  obmc_boot_test_resource.robot
 
 *** Variables ***
-# Initialize program parameters variables.
-# Create parm_list containing all of our program parameters.  This is used by
-# 'Rqprint Pgm Header'
-@{parm_list}                openbmc_nickname  openbmc_host  openbmc_username
-...  openbmc_password  os_host  os_username  os_password  pdu_host
-...  pdu_username  pdu_password  pdu_slot_no  openbmc_serial_host
-...  openbmc_serial_port  boot_stack  boot_list  max_num_tests
-...  plug_in_dir_paths  status_file_path  openbmc_model  boot_pass  boot_fail
-...  ffdc_dir_path_style  ffdc_check  state_change_timeout  power_on_timeout
-...  power_off_timeout  ffdc_only  test_mode  quiet  debug
-
-# Initialize each program parameter.
-${openbmc_host}             ${EMPTY}
-${openbmc_nickname}         ${openbmc_host}
-${openbmc_username}         root
-${openbmc_password}         0penBmc
-${os_host}                  ${EMPTY}
-${os_username}              root
-${os_password}              P@ssw0rd
-${pdu_host}                 ${EMPTY}
-${pdu_username}             admin
-${pdu_password}             admin
-${pdu_slot_no}              ${EMPTY}
-${openbmc_serial_host}      ${EMPTY}
-${openbmc_serial_port}      ${EMPTY}
-${boot_stack}               ${EMPTY}
-${boot_list}                ${EMPTY}
-${max_num_tests}            0
-${plug_in_dir_paths}        ${EMPTY}
-${status_file_path}         ${EMPTY}
-${openbmc_model}            ${EMPTY}
-# The reason boot_pass and boot_fail are parameters is that it is possible to
-# be called by a program that has already done some tests.  This allows us to
-# keep the grand total.
-${boot_pass}                ${0}
-${boot_fail}                ${0}
-${ffdc_dir_path_style}      ${EMPTY}
-${ffdc_check}               ${EMPTY}
-${state_change_timeout}     3 mins
-${power_on_timeout}         14 mins
-${power_off_timeout}        2 mins
-${ffdc_only}                ${0}
-${test_mode}                0
-${quiet}                    0
-${debug}                    0
-
-# Flag variables.
-# test_really_running is needed by DB_Logging plug-in.
-${test_really_running}      ${1}
-
 *** Test Cases ***
 General Boot Testing
     [Documentation]  Performs repeated boot tests.
@@ -76,6 +23,6 @@
     # just putting the code in the *** Test Cases *** table are:
     # 1) You won't get a green dot in the output every time you run a keyword.
 
-    Main Py
+    OBMC Boot Test
 
 ###############################################################################
diff --git a/extended/obmc_boot_test_resource.robot b/extended/obmc_boot_test_resource.robot
new file mode 100644
index 0000000..da59112
--- /dev/null
+++ b/extended/obmc_boot_test_resource.robot
@@ -0,0 +1,65 @@
+*** Settings ***
+Documentation  This file is resourced by obmc_boot_test.py to set initial
+...            variable values, etc.
+
+Resource  ../lib/openbmc_ffdc.robot
+Library   ../lib/state.py
+
+Library   ../lib/obmc_boot_test.py
+
+*** Variables ***
+# Initialize program parameters variables.
+# Create parm_list containing all of our program parameters.  This is used by
+# 'Rqprint Pgm Header'
+@{parm_list}                openbmc_nickname  openbmc_host  openbmc_username
+...  openbmc_password  os_host  os_username  os_password  pdu_host
+...  pdu_username  pdu_password  pdu_slot_no  openbmc_serial_host
+...  openbmc_serial_port  stack_mode  boot_stack  boot_list  max_num_tests
+...  plug_in_dir_paths  status_file_path  openbmc_model  boot_pass  boot_fail
+...  ffdc_dir_path_style  ffdc_check  state_change_timeout  power_on_timeout
+...  power_off_timeout  ffdc_only  boot_fail_threshold  test_mode  quiet
+...  debug
+
+# Initialize each program parameter.
+${openbmc_host}             ${EMPTY}
+${openbmc_nickname}         ${openbmc_host}
+${openbmc_username}         root
+${openbmc_password}         0penBmc
+${os_host}                  ${EMPTY}
+${os_username}              root
+${os_password}              P@ssw0rd
+${pdu_host}                 ${EMPTY}
+${pdu_username}             admin
+${pdu_password}             admin
+${pdu_slot_no}              ${EMPTY}
+${openbmc_serial_host}      ${EMPTY}
+${openbmc_serial_port}      ${EMPTY}
+${stack_mode}               normal
+${boot_stack}               ${EMPTY}
+${boot_list}                ${EMPTY}
+${max_num_tests}            0
+${plug_in_dir_paths}        ${EMPTY}
+${status_file_path}         ${EMPTY}
+${openbmc_model}            ${EMPTY}
+# The reason boot_pass and boot_fail are parameters is that it is possible to
+# be called by a program that has already done some tests.  This allows us to
+# keep the grand total.
+${boot_pass}                ${0}
+${boot_fail}                ${0}
+${ffdc_dir_path_style}      ${EMPTY}
+${ffdc_check}               ${EMPTY}
+${state_change_timeout}     3 mins
+${power_on_timeout}         14 mins
+${power_off_timeout}        2 mins
+${ffdc_only}                ${0}
+# If the number of boot failures, exceeds boot_fail_threshold, this program
+# returns non-zero.
+${boot_fail_threshold}      ${1000000}
+${test_mode}                0
+${quiet}                    0
+${debug}                    0
+
+# Flag variables.
+# test_really_running is needed by DB_Logging plug-in.
+${test_really_running}      ${1}
+
diff --git a/lib/obmc_boot_test.py b/lib/obmc_boot_test.py
index ee8b442..f43ae7c 100755
--- a/lib/obmc_boot_test.py
+++ b/lib/obmc_boot_test.py
@@ -21,6 +21,7 @@
 import gen_robot_valid as grv
 import gen_misc as gm
 import gen_cmd as gc
+import gen_robot_keyword as grk
 import state as st
 
 base_path = os.path.dirname(os.path.dirname(
@@ -29,54 +30,25 @@
 sys.path.append(base_path + "extended/")
 import run_keyword as rk
 
-# Program parameter processing.
-# Assign all program parms to python variables which are global to this module.
-parm_list = BuiltIn().get_variable_value("${parm_list}")
-int_list = ['max_num_tests', 'boot_pass', 'boot_fail', 'ffdc_only', 'quiet',
-            'test_mode', 'debug']
-for parm in parm_list:
-    if parm in int_list:
-        sub_cmd = "int(BuiltIn().get_variable_value(\"${" + parm +\
-                  "}\", \"0\"))"
-    else:
-        sub_cmd = "BuiltIn().get_variable_value(\"${" + parm + "}\")"
-    cmd_buf = parm + " = " + sub_cmd
-    exec(cmd_buf)
-
-if ffdc_dir_path_style == "":
-    ffdc_dir_path_style = int(os.environ.get('FFDC_DIR_PATH_STYLE', '0'))
-
-# Set up boot data structures.
-boot_table = create_boot_table()
-valid_boot_types = create_valid_boot_list(boot_table)
-
 # Setting master_pid correctly influences the behavior of plug-ins like
 # DB_Logging
 program_pid = os.getpid()
 master_pid = os.environ.get('AUTOBOOT_MASTER_PID', program_pid)
 
-boot_results_file_path = "/tmp/" + openbmc_nickname + ":pid_" +\
-                         str(master_pid) + ":boot_results"
-if os.path.isfile(boot_results_file_path):
-    # We've been called before in this run so we'll load the saved
-    # boot_results object.
-    boot_results = pickle.load(open(boot_results_file_path, 'rb'))
-else:
-    boot_results = boot_results(boot_table, boot_pass, boot_fail)
+# Set up boot data structures.
+boot_table = create_boot_table()
+valid_boot_types = create_valid_boot_list(boot_table)
 
 boot_lists = read_boot_lists()
 last_ten = []
-# Convert these program parms to more useable lists.
-boot_list = filter(None, boot_list.split(":"))
-boot_stack = filter(None, boot_stack.split(":"))
 
 state = st.return_default_state()
 cp_setup_called = 0
 next_boot = ""
 base_tool_dir_path = os.path.normpath(os.environ.get(
     'AUTOBOOT_BASE_TOOL_DIR_PATH', "/tmp")) + os.sep
+
 ffdc_dir_path = os.path.normpath(os.environ.get('FFDC_DIR_PATH', '')) + os.sep
-ffdc_list_file_path = base_tool_dir_path + openbmc_nickname + "/FFDC_FILE_LIST"
 boot_success = 0
 status_dir_path = os.environ.get('STATUS_DIR_PATH', "")
 if status_dir_path != "":
@@ -89,6 +61,62 @@
 
 
 ###############################################################################
+def process_pgm_parms():
+
+    r"""
+    Process the program parameters by assigning them all to corresponding
+    globals.  Also, set some global values that depend on program parameters.
+    """
+
+    # Program parameter processing.
+    # Assign all program parms to python variables which are global to this
+    # module.
+
+    global parm_list
+    parm_list = BuiltIn().get_variable_value("${parm_list}")
+    # The following subset of parms should be processed as integers.
+    int_list = ['max_num_tests', 'boot_pass', 'boot_fail', 'ffdc_only',
+                'boot_fail_threshold', 'quiet', 'test_mode', 'debug']
+    for parm in parm_list:
+        if parm in int_list:
+            sub_cmd = "int(BuiltIn().get_variable_value(\"${" + parm +\
+                      "}\", \"0\"))"
+        else:
+            sub_cmd = "BuiltIn().get_variable_value(\"${" + parm + "}\")"
+        cmd_buf = "global " + parm + " ; " + parm + " = " + sub_cmd
+        exec(cmd_buf)
+
+    global ffdc_dir_path_style
+    global boot_list
+    global boot_stack
+    global boot_results_file_path
+    global boot_results
+    global ffdc_list_file_path
+
+    if ffdc_dir_path_style == "":
+        ffdc_dir_path_style = int(os.environ.get('FFDC_DIR_PATH_STYLE', '0'))
+
+    # Convert these program parms to lists for easier processing..
+    boot_list = filter(None, boot_list.split(":"))
+    boot_stack = filter(None, boot_stack.split(":"))
+
+    boot_results_file_path = "/tmp/" + openbmc_nickname + ":pid_" +\
+                             str(master_pid) + ":boot_results"
+
+    if os.path.isfile(boot_results_file_path):
+        # We've been called before in this run so we'll load the saved
+        # boot_results object.
+        boot_results = pickle.load(open(boot_results_file_path, 'rb'))
+    else:
+        boot_results = boot_results(boot_table, boot_pass, boot_fail)
+
+    ffdc_list_file_path = base_tool_dir_path + openbmc_nickname +\
+        "/FFDC_FILE_LIST"
+
+###############################################################################
+
+
+###############################################################################
 def initial_plug_in_setup():
 
     r"""
@@ -200,16 +228,22 @@
 
     global cp_setup_called
 
-    grp.rqprintn()
+    gp.qprintn()
+
+    shell_rc, out_buf = gc.cmd_fnc_u("which ssh_pw", quiet=1, print_output=0,
+                                     show_err=0)
+    if shell_rc != 0:
+        robot_pgm_dir_path = os.path.dirname(__file__) + os.sep
+        os.environ['PATH'] = robot_pgm_dir_path +\
+            "../bin:" + os.environ.get('PATH', "")
+        os.environ['PYTHONPATH'] = robot_pgm_dir_path +\
+            os.environ.get('PYTHONPATH', "")
 
     validate_parms()
 
     grp.rqprint_pgm_header()
 
-    cmd_buf = ["Set BMC Power Policy", "RESTORE_LAST_STATE"]
-    grp.rpissuing_keyword(cmd_buf, test_mode)
-    if not test_mode:
-        BuiltIn().run_keyword(*cmd_buf)
+    grk.run_key("Set BMC Power Policy  RESTORE_LAST_STATE")
 
     initial_plug_in_setup()
 
@@ -229,8 +263,8 @@
     # FFDC_LOG_PATH is used by "FFDC" keyword.
     BuiltIn().set_global_variable("${FFDC_LOG_PATH}", ffdc_dir_path)
 
-    grp.rdprint_var(boot_table, 1)
-    grp.rdprint_var(boot_lists)
+    gp.dprint_var(boot_table, 1)
+    gp.dprint_var(boot_lists)
 
 ###############################################################################
 
@@ -242,8 +276,11 @@
     Validate all program parameters.
     """
 
-    grp.rqprintn()
+    process_pgm_parms()
 
+    gp.qprintn()
+
+    global openbmc_model
     grv.rvalid_value("openbmc_host")
     grv.rvalid_value("openbmc_username")
     grv.rvalid_value("openbmc_password")
@@ -257,8 +294,13 @@
         grv.rvalid_integer("pdu_slot_no")
     if openbmc_serial_host != "":
         grv.rvalid_integer("openbmc_serial_port")
-    grv.rvalid_integer("max_num_tests")
+    if openbmc_model == "":
+        status, ret_values =\
+            grk.run_key_u("Get BMC System Model")
+        openbmc_model = ret_values
+        BuiltIn().set_global_variable("${openbmc_model}", openbmc_model)
     grv.rvalid_value("openbmc_model")
+    grv.rvalid_integer("max_num_tests")
     grv.rvalid_integer("boot_pass")
     grv.rvalid_integer("boot_fail")
 
@@ -266,6 +308,7 @@
     BuiltIn().set_global_variable("${plug_in_packages_list}",
                                   plug_in_packages_list)
 
+    grv.rvalid_value("stack_mode", valid_values=['normal', 'skip'])
     if len(boot_list) == 0 and len(boot_stack) == 0 and not ffdc_only:
         error_message = "You must provide either a value for either the" +\
             " boot_list or the boot_stack parm.\n"
@@ -300,12 +343,12 @@
 
     req_states = ['epoch_seconds'] + st.default_req_states
 
-    grp.rqprint_timen("Getting system state.")
+    gp.qprint_timen("Getting system state.")
     if test_mode:
         state['epoch_seconds'] = int(time.time())
     else:
-        state = st.get_state(req_states=req_states, quiet=0)
-    grp.rprint_var(state)
+        state = st.get_state(req_states=req_states, quiet=quiet)
+    gp.qprint_var(state)
 
 ###############################################################################
 
@@ -323,27 +366,51 @@
 
     global boot_stack
 
-    grp.rprint_timen("Selecting a boot test.")
+    gp.qprint_timen("Selecting a boot test.")
 
     my_get_state()
 
     stack_popped = 0
     if len(boot_stack) > 0:
         stack_popped = 1
-        grp.rprint_dashes()
-        grp.rprint_var(boot_stack)
-        grp.rprint_dashes()
-        boot_candidate = boot_stack.pop()
+        gp.qprint_dashes()
+        gp.qprint_var(boot_stack)
+        gp.qprint_dashes()
+        skip_boot_printed = 0
+        while len(boot_stack) > 0:
+            boot_candidate = boot_stack.pop()
+            if stack_mode == 'normal':
+                break
+            else:
+                if st.compare_states(state, boot_table[boot_candidate]['end']):
+                    if not skip_boot_printed:
+                        gp.print_var(stack_mode)
+                        gp.printn()
+                        gp.print_timen("Skipping the following boot tests" +
+                                       " which are unnecessary since their" +
+                                       " required end states match the" +
+                                       " current machine state:")
+                        skip_boot_printed = 1
+                    gp.print_var(boot_candidate)
+                    boot_candidate = ""
+        if boot_candidate == "":
+            gp.qprint_dashes()
+            gp.qprint_var(boot_stack)
+            gp.qprint_dashes()
+            return boot_candidate
         if st.compare_states(state, boot_table[boot_candidate]['start']):
-            grp.rprint_timen("The machine state is valid for a '" +
-                             boot_candidate + "' boot test.")
-            grp.rprint_dashes()
-            grp.rprint_var(boot_stack)
-            grp.rprint_dashes()
+            gp.qprint_timen("The machine state is valid for a '" +
+                            boot_candidate + "' boot test.")
+            gp.qprint_dashes()
+            gp.qprint_var(boot_stack)
+            gp.qprint_dashes()
             return boot_candidate
         else:
-            grp.rprint_timen("The machine state is not valid for a '" +
-                             boot_candidate + "' boot test.")
+            gp.qprint_timen("The machine state does not match the required" +
+                            " starting state for a '" + boot_candidate +
+                            "' boot test:")
+            gp.print_varx("boot_table[" + boot_candidate + "][start]",
+                          boot_table[boot_candidate]['start'], 1)
             boot_stack.append(boot_candidate)
             popped_boot = boot_candidate
 
@@ -359,16 +426,16 @@
                 boot_candidates.append(boot_candidate)
 
     if len(boot_candidates) == 0:
-        grp.rprint_timen("The user's boot list contained no boot tests" +
-                         " which are valid for the current machine state.")
+        gp.qprint_timen("The user's boot list contained no boot tests" +
+                        " which are valid for the current machine state.")
         boot_candidate = default_power_on
         if not st.compare_states(state, boot_table[default_power_on]['start']):
             boot_candidate = default_power_off
         boot_candidates.append(boot_candidate)
-        grp.rprint_timen("Using default '" + boot_candidate +
-                         "' boot type to transtion to valid state.")
+        gp.qprint_timen("Using default '" + boot_candidate +
+                        "' boot type to transition to valid state.")
 
-    grp.rdprint_var(boot_candidates)
+    gp.dprint_var(boot_candidates)
 
     # Randomly select a boot from the candidate list.
     boot = random.choice(boot_candidates)
@@ -386,12 +453,12 @@
     """
 
     # indent 0, 90 chars wide, linefeed, char is "="
-    grp.rqprint_dashes(0, 90)
-    grp.rqprintn("Last 10 boots:\n")
+    gp.qprint_dashes(0, 90)
+    gp.qprintn("Last 10 boots:\n")
 
     for boot_entry in last_ten:
         grp.rqprint(boot_entry)
-    grp.rqprint_dashes(0, 90)
+    gp.qprint_dashes(0, 90)
 
 ###############################################################################
 
@@ -403,18 +470,18 @@
     Print a defect report.
     """
 
-    grp.rqprintn()
+    gp.qprintn()
     # indent=0, width=90, linefeed=1, char="="
-    grp.rqprint_dashes(0, 90, 1, "=")
-    grp.rqprintn("Copy this data to the defect:\n")
+    gp.qprint_dashes(0, 90, 1, "=")
+    gp.qprintn("Copy this data to the defect:\n")
 
     grp.rqpvars(*parm_list)
 
-    grp.rqprintn()
+    gp.qprintn()
 
     print_last_boots()
-    grp.rqprintn()
-    grp.rqpvar(state)
+    gp.qprintn()
+    gp.qprint_var(state)
 
     # At some point I'd like to have the 'Call FFDC Methods' return a list
     # of files it has collected.  In that case, the following "ls" command
@@ -430,16 +497,16 @@
     except IOError:
         ffdc_list = ""
 
-    grp.rqprintn()
-    grp.rqprintn("FFDC data files:")
+    gp.qprintn()
+    gp.qprintn("FFDC data files:")
     if status_file_path != "":
-        grp.rqprintn(status_file_path)
+        gp.qprintn(status_file_path)
 
-    grp.rqprintn(output)
-    # grp.rqprintn(ffdc_list)
-    grp.rqprintn()
+    gp.qprintn(output)
+    # gp.qprintn(ffdc_list)
+    gp.qprintn()
 
-    grp.rqprint_dashes(0, 90, 1, "=")
+    gp.qprint_dashes(0, 90, 1, "=")
 
 ###############################################################################
 
@@ -487,7 +554,7 @@
     global last_ten
 
     doing_msg = gp.sprint_timen("Doing \"" + boot_keyword + "\".")
-    grp.rqprint(doing_msg)
+    gp.qprint(doing_msg)
 
     last_ten.append(doing_msg)
 
@@ -527,11 +594,12 @@
         # Assertion:  We trust that the state data was made fresh by the
         # caller.
 
-        grp.rprintn()
+        gp.qprintn()
 
         if boot_table[boot]['method_type'] == "keyword":
             rk.my_run_keywords(boot_table[boot].get('lib_file_path', ''),
-                               boot_table[boot]['method'])
+                               boot_table[boot]['method'],
+                               quiet=quiet)
 
         if boot_table[boot]['bmc_reboot']:
             st.wait_for_comm_cycle(int(state['epoch_seconds']))
@@ -549,7 +617,7 @@
             st.wait_state(match_state, wait_time=state_change_timeout,
                           interval="3 seconds", invert=1)
 
-        grp.rprintn()
+        gp.qprintn()
         if boot_table[boot]['end']['chassis'] == "Off":
             boot_timeout = power_off_timeout
         else:
@@ -583,13 +651,14 @@
     global next_boot
     global boot_success
 
-    grp.rqprintn()
-
-    boot_count += 1
+    gp.qprintn()
 
     next_boot = select_boot()
+    if next_boot == "":
+        return True
 
-    grp.rqprint_timen("Starting boot " + str(boot_count) + ".")
+    boot_count += 1
+    gp.qprint_timen("Starting boot " + str(boot_count) + ".")
 
     # Clear the ffdc_list_file_path file.  Plug-ins may now write to it.
     try:
@@ -600,15 +669,15 @@
     cmd_buf = ["run_boot", next_boot]
     boot_status, msg = BuiltIn().run_keyword_and_ignore_error(*cmd_buf)
     if boot_status == "FAIL":
-        grp.rprint(msg)
+        gp.qprint(msg)
 
-    grp.rqprintn()
+    gp.qprintn()
     if boot_status == "PASS":
         boot_success = 1
-        grp.rqprint_timen("BOOT_SUCCESS: \"" + next_boot + "\" succeeded.")
+        gp.qprint_timen("BOOT_SUCCESS: \"" + next_boot + "\" succeeded.")
     else:
         boot_success = 0
-        grp.rqprint_timen("BOOT_FAILED: \"" + next_boot + "\" failed.")
+        gp.qprint_timen("BOOT_FAILED: \"" + next_boot + "\" failed.")
 
     boot_results.update(next_boot, boot_status)
 
@@ -631,12 +700,10 @@
             gp.print_error("Call to my_ffdc failed.\n")
 
     # We need to purge error logs between boots or they build up.
-    cmd_buf = ["Delete Error logs"]
-    grp.rpissuing_keyword(cmd_buf, test_mode)
-    BuiltIn().run_keyword(*cmd_buf)
+    grk.run_key("Delete Error logs", ignore=1)
 
     boot_results.print_report()
-    grp.rqprint_timen("Finished boot " + str(boot_count) + ".")
+    gp.qprint_timen("Finished boot " + str(boot_count) + ".")
 
     plug_in_setup()
     rc, shell_rc, failed_plug_in_name = grpi.rprocess_plug_in_packages(
@@ -647,9 +714,7 @@
         BuiltIn().fail(error_message)
 
     # This should help prevent ConnectionErrors.
-    cmd_buf = ["Delete All Sessions"]
-    grp.rpissuing_keyword(cmd_buf, test_mode)
-    BuiltIn().run_keyword(*cmd_buf)
+    grk.run_key_u("Delete All Sessions")
 
     return True
 
@@ -669,8 +734,8 @@
             call_point='cleanup', stop_on_plug_in_failure=1)
 
     # Save boot_results object to a file in case it is needed again.
-    grp.rprint_timen("Saving boot_results to the following path.")
-    grp.rprint_var(boot_results_file_path)
+    gp.qprint_timen("Saving boot_results to the following path.")
+    gp.qprint_var(boot_results_file_path)
     pickle.dump(boot_results, open(boot_results_file_path, 'wb'),
                 pickle.HIGHEST_PROTOCOL)
 
@@ -689,16 +754,21 @@
                "A keyword timeout occurred ending this program.\n"]
     BuiltIn().run_keyword_if_timeout_occurred(*cmd_buf)
 
+    grp.rqprint_pgm_footer()
+
 ###############################################################################
 
 
 ###############################################################################
-def main_py():
+def obmc_boot_test(alt_boot_stack=None):
 
     r"""
     Do main program processing.
     """
 
+    if alt_boot_stack is not None:
+        BuiltIn().set_global_variable("${boot_stack}", alt_boot_stack)
+
     setup()
 
     if ffdc_only:
@@ -714,13 +784,21 @@
     while (len(boot_stack) > 0):
         test_loop_body()
 
-    grp.rprint_timen("Finished processing stack.")
+    gp.qprint_timen("Finished processing stack.")
 
     # Process caller's boot_list.
     if len(boot_list) > 0:
         for ix in range(1, max_num_tests + 1):
             test_loop_body()
 
-    grp.rqprint_timen("Completed all requested boot tests.")
+    gp.qprint_timen("Completed all requested boot tests.")
+
+    boot_pass, boot_fail = boot_results.return_total_pass_fail()
+    if boot_fail > boot_fail_threshold:
+        error_message = "Boot failures exceed the boot failure" +\
+                        " threshold:\n" +\
+                        gp.sprint_var(boot_fail) +\
+                        gp.sprint_var(boot_fail_threshold)
+        BuiltIn().fail(gp.sprint_error(error_message))
 
 ###############################################################################
diff --git a/lib/utils.robot b/lib/utils.robot
index c044ac3..989d61b 100755
--- a/lib/utils.robot
+++ b/lib/utils.robot
@@ -38,6 +38,7 @@
 Get BMC System Model
     [Documentation]  Get the BMC model from the device tree.
 
+    Open Connection And Log In
     ${bmc_model}  ${stderr}=  Execute Command
     ...  cat ${devicetree_base} | cut -d " " -f 1  return_stderr=True
     Should Be Empty  ${stderr}
@@ -785,8 +786,9 @@
 Trigger Host Watchdog Error
     [Documentation]  Inject host watchdog error using BMC.
     [Arguments]  ${milliseconds}=1000  ${sleep_time}=5s
-    # Description of arguments:
-    # milliseconds  The time watchdog timer value in milliseconds (e.g. 1000 = 1 second).
+    # Description of argument(s):
+    # milliseconds  The time watchdog timer value in milliseconds (e.g. 1000 =
+    #               1 second).
     # sleep_time    Time delay for host watchdog error to get injected.
     #               Default is 5 seconds.
 
@@ -821,10 +823,11 @@
     ...  Network Mask, default gatway and serial console IP and port
     ...  information which should be provided in command line.
 
-    [Arguments]  ${host}=${OPENBMC_HOST}  ${mask}=${NET_MASK}  ${gw_ip}=${GW_IP}
+    [Arguments]  ${host}=${OPENBMC_HOST}  ${mask}=${NET_MASK}
+    ...          ${gw_ip}=${GW_IP}
 
-    # Open telnet connection and ignore the error, in case telnet session is already
-    # opened by the program calling this keyword.
+    # Open telnet connection and ignore the error, in case telnet session is
+    # already opened by the program calling this keyword.
 
     Run Keyword And Ignore Error  Open Telnet Connection to BMC Serial Console
     Telnet.write  ifconfig eth0 ${host} netmask ${mask}