|  | *** Settings *** | 
|  | Documentation      Keywords for system test. | 
|  |  | 
|  | Library            ../lib/gen_robot_keyword.py | 
|  | Library            ../lib/gen_print.py | 
|  | Library            ../lib/gen_robot_print.py | 
|  | Resource           ../lib/boot_utils.robot | 
|  | Resource           ../extended/obmc_boot_test_resource.robot | 
|  | Resource           ../lib/utils.robot | 
|  | Resource           ../lib/state_manager.robot | 
|  | Resource           ../lib/rest_client.robot | 
|  | Resource           ../lib/htx_resource.robot | 
|  | Library            OperatingSystem | 
|  | Library            DateTime | 
|  |  | 
|  | *** Variables *** | 
|  |  | 
|  | ${htx_log_dir_path}    ${EXECDIR}${/}logs${/} | 
|  |  | 
|  | # Error strings to check from dmesg. | 
|  | ${ERROR_REGEX}         error|GPU|NVRM|nvidia | 
|  |  | 
|  | # GPU specific error message from dmesg. | 
|  | ${ERROR_DBE_MSG}       (DBE) has been detected on GPU | 
|  |  | 
|  | # Inventory - List of I/O devices to collect for Inventory | 
|  | @{I/O}                 communication  disk  display  generic  input  multimedia | 
|  | ...                    network  printer  tape | 
|  |  | 
|  | # Inventory Paths of the JSON and YAML files | 
|  | ${json_tmp_file_path}  ${EXECDIR}/inventory_temp_file.json | 
|  | ${yaml_file_path}      ${EXECDIR}/inventory_temp_file.yaml | 
|  |  | 
|  |  | 
|  |  | 
|  | *** Keywords *** | 
|  |  | 
|  | Login To OS | 
|  | [Documentation]  Login to OS Host. | 
|  | [Arguments]  ${os_host}=${OS_HOST}  ${os_username}=${OS_USERNAME} | 
|  | ...          ${os_password}=${OS_PASSWORD} | 
|  | ...          ${alias_name}=os_connection | 
|  | # Description of argument(s): | 
|  | # os_host      IP address of the OS Host. | 
|  | # os_username  OS Host Login user name. | 
|  | # os_password  OS Host Login passwrd. | 
|  | # alias_name   Default OS SSH session connection alias name. | 
|  |  | 
|  | Ping Host  ${os_host} | 
|  | SSHLibrary.Open Connection  ${os_host}  alias=${alias_name} | 
|  | SSHLibrary.Login  ${os_username}  ${os_password} | 
|  |  | 
|  |  | 
|  | Tool Exist | 
|  | [Documentation]  Check whether given tool is installed on OS. | 
|  | [Arguments]  ${tool_name} | 
|  | # Description of argument(s): | 
|  | # tool_name   Tool name whose existence is to be checked. | 
|  |  | 
|  | ${output}  ${stderr}  ${rc}=  OS Execute Command  which ${tool_name} | 
|  | Should Contain  ${output}  ${tool_name} | 
|  | ...  msg=Please install ${tool_name} tool. | 
|  |  | 
|  |  | 
|  | Boot To OS | 
|  | [Documentation]  Boot host OS. | 
|  | Run Key  OBMC Boot Test \ REST Power On | 
|  |  | 
|  |  | 
|  | Power Off Host | 
|  | [Documentation]  Power off host. | 
|  | Run Key  OBMC Boot Test \ REST Power Off | 
|  |  | 
|  |  | 
|  | File Exist On OS | 
|  | [Documentation]  Check if the given file path exist on OS. | 
|  | [Arguments]  ${file_path} | 
|  | # Description of argument(s): | 
|  | # file_path   Absolute file path. | 
|  |  | 
|  | Login To OS | 
|  | ${out}  ${stderr}  ${rc}=  OS Execute Command  ls ${file_path} | 
|  | Log To Console  \n File Exist: ${out} | 
|  |  | 
|  |  | 
|  | Is HTX Running | 
|  | [Documentation]  Return "True" if the HTX is running, "False" | 
|  | ...  otherwise. | 
|  |  | 
|  | # Example usage: | 
|  | #  ${status}=  Is HTX Running | 
|  | #  Run Keyword If  '${status}' == 'True'  Shutdown HTX Exerciser | 
|  |  | 
|  | ${status}  ${stderr}  ${rc}=  OS Execute Command | 
|  | ...  htxcmdline -getstats  ignore_err=1 | 
|  | # Get HTX state | 
|  | # (idle, currently running, selected_mdt but not running). | 
|  | ${running}=  Set Variable If | 
|  | ...  "Currently running" in """${status}"""  ${True}  ${False} | 
|  |  | 
|  | RETURN  ${running} | 
|  |  | 
|  |  | 
|  | Write Log Data To File | 
|  | [Documentation]  Write log data to the logs directory. | 
|  | [Arguments]  ${data}=  ${log_file_path}= | 
|  | # Description of argument(s): | 
|  | # data            String buffer. | 
|  | # log_file_path   The log file path. | 
|  |  | 
|  | Create File  ${log_file_path}  ${data} | 
|  |  | 
|  |  | 
|  | Collect HTX Log Files | 
|  | [Documentation]  Collect status and error log files. | 
|  | # Collects the following files: | 
|  | # HTX error log file /tmp/htxerr | 
|  | # HTX status log file /tmp/htxstats | 
|  |  | 
|  | # Create logs directory and get current datetime. | 
|  | Create Directory  ${htx_log_dir_path} | 
|  | ${cur_datetime}=  Get Current Date  result_format=%Y%m%d%H%M%S%f | 
|  |  | 
|  | File Exist On OS  /tmp/htxerr | 
|  | ${htx_err}  ${std_err}  ${rc}=  OS Execute Command  cat /tmp/htxerr | 
|  | Write Log Data To File | 
|  | ...  ${htx_err}  ${htx_log_dir_path}/${OS_HOST}${cur_datetime}.htxerr | 
|  |  | 
|  | File Exist On OS  /tmp/htxstats | 
|  | ${htx_stats}  ${std_err}  ${rc}=  OS Execute Command | 
|  | ...  cat /tmp/htxstats | 
|  | Write Log Data To File | 
|  | ...  ${htx_stats}  ${htx_log_dir_path}/${OS_HOST}_${cur_datetime}.htxstats | 
|  |  | 
|  |  | 
|  | REST Upload File To BMC | 
|  | [Documentation]  Upload a file via REST to BMC. | 
|  |  | 
|  | # Generate 32 MB file size | 
|  | Run  dd if=/dev/zero of=dummyfile bs=1 count=0 seek=32MB | 
|  | OperatingSystem.File Should Exist  dummyfile | 
|  |  | 
|  | # Get the content of the file and upload to BMC | 
|  | ${image_data}=  OperatingSystem.Get Binary File  dummyfile | 
|  |  | 
|  | # Get REST session to BMC | 
|  | Initialize OpenBMC | 
|  |  | 
|  | # Create the REST payload headers and data | 
|  | ${data}=  Create Dictionary  data  ${image_data} | 
|  | ${headers}=  Create Dictionary  Content-Type=application/octet-stream | 
|  | ...  Accept=application/octet-stream | 
|  | Set To Dictionary  ${data}  headers  ${headers} | 
|  |  | 
|  | ${resp}=  Post Request  openbmc  /upload/image  &{data} | 
|  | Should Be Equal As Strings  ${resp.status_code}  ${HTTP_BAD_REQUEST} | 
|  | ...  msg=Openbmc /upload/image failed. | 
|  |  | 
|  | # Take SSH connection to BMC and switch to BMC connection to perform | 
|  | # the task. | 
|  | &{bmc_connection_args}=  Create Dictionary  alias=bmc_connection | 
|  | Open Connection And Log In  &{bmc_connection_args} | 
|  |  | 
|  | # Currently OS SSH session is active, switch to BMC connection. | 
|  | Switch Connection  bmc_connection | 
|  |  | 
|  | # Switch back to OS SSH connection. | 
|  | Switch Connection  os_connection | 
|  |  | 
|  |  | 
|  | Get CPU Min Frequency Limit | 
|  | [Documentation]  Get CPU minimum assignable frequency. | 
|  |  | 
|  | # lscpu | grep min  returns | 
|  | # CPU min MHz:           1983.0000 | 
|  |  | 
|  | ${cmd}=  Catenate  lscpu | grep min  | tr -dc '0-9.\n' | 
|  | ${cpu_freq}  ${stderr}  ${rc}=  OS Execute Command  ${cmd} | 
|  | RETURN  ${cpu_freq} | 
|  |  | 
|  |  | 
|  | Get CPU Min Frequency | 
|  | [Documentation]  Get CPU assigned minimum frequency. | 
|  |  | 
|  | # ppc64_cpu --frequency -t 10  returns | 
|  | # min:    3.295 GHz (cpu 143) | 
|  | # max:    3.295 GHz (cpu 0) | 
|  | # avg:    3.295 GHz | 
|  |  | 
|  | ${cmd}=  Catenate  ppc64_cpu --frequency -t 10 | grep min | 
|  | ...  | cut -f 2 | cut -d ' ' -f 1 | tr -dc '0-9\n' | 
|  | ${cpu_freq}  ${stderr}  ${rc}=  OS Execute Command  ${cmd} | 
|  | RETURN  ${cpu_freq} | 
|  |  | 
|  |  | 
|  | Get CPU Max Frequency Limit | 
|  | [Documentation]  Get CPU maximum assignable frequency. | 
|  |  | 
|  | # lscpu | grep max  returns | 
|  | # CPU max MHz:           3300.0000 | 
|  |  | 
|  | ${cmd}=  Catenate  lscpu | grep max  | tr -dc '0-9.\n' | 
|  | ${cpu_freq}  ${stderr}  ${rc}=  OS Execute Command  ${cmd} | 
|  | RETURN  ${cpu_freq} | 
|  |  | 
|  |  | 
|  | Get CPU Max Frequency | 
|  | [Documentation]  Get CPU assigned maximum frequency. | 
|  |  | 
|  | # ppc64_cpu --frequency -t 10  returns | 
|  | # min:    3.295 GHz (cpu 143) | 
|  | # max:    3.295 GHz (cpu 0) | 
|  | # avg:    3.295 GHz | 
|  |  | 
|  | ${cmd}=  Catenate  ppc64_cpu --frequency -t 10 | grep max | 
|  | ...  | cut -f 2 | cut -d ' ' -f 1 | tr -dc '0-9\n' | 
|  | ${cpu_freq}  ${stderr}  ${rc}=  OS Execute Command  ${cmd} | 
|  | RETURN  ${cpu_freq} | 
|  |  | 
|  |  | 
|  | Get CPU Max Temperature | 
|  | [Documentation]  Get the highest CPU Temperature. | 
|  |  | 
|  | ${temperature_objs}=  Read Properties | 
|  | ...  ${SENSORS_URI}temperature/enumerate | 
|  | # Filter the dictionary to get just the CPU temperature info. | 
|  | ${cmd}=  Catenate  {k:v for k,v in $temperature_objs.items() | 
|  | ...  if re.match('${SENSORS_URI}temperature/p.*core.*temp', k)} | 
|  | ${cpu_temperatuture_objs}  Evaluate  ${cmd}  modules=re | 
|  | # Create a list of the CPU temperature values (current). | 
|  | ${cpu_temperatures}=  Evaluate | 
|  | ...  [ x['Value'] for x in $cpu_temperatuture_objs.values() ] | 
|  |  | 
|  | ${cpu_max_temp}  Evaluate  int(max(map(int, $cpu_temperatures))/1000) | 
|  | RETURN  ${cpu_max_temp} | 
|  |  | 
|  |  | 
|  | Get CPU Min Temperature | 
|  | [Documentation]  Get the  CPU Temperature. | 
|  |  | 
|  | ${temperature_objs}=  Read Properties | 
|  | ...  ${SENSORS_URI}temperature/enumerate | 
|  | # Filter the dictionary to get just the CPU temperature info. | 
|  | ${cmd}=  Catenate  {k:v for k,v in $temperature_objs.items() | 
|  | ...  if re.match('${SENSORS_URI}temperature/p.*core.*temp', k)} | 
|  | ${cpu_temperatuture_objs}=  Evaluate  ${cmd}  modules=re | 
|  | # Create a list of the CPU temperature values (current). | 
|  | ${cpu_temperatures}=  Evaluate | 
|  | ...  [ x['Value'] for x in $cpu_temperatuture_objs.values() ] | 
|  |  | 
|  | ${cpu_min_temp}  Evaluate  int(min(map(int, $cpu_temperatures))/1000) | 
|  | RETURN  ${cpu_min_temp} | 
|  |  | 
|  |  | 
|  | Check For Errors On OS Dmesg Log | 
|  | [Documentation]  Check if dmesg has nvidia errors logged. | 
|  |  | 
|  | ${dmesg_log}  ${stderr}  ${rc}=  OS Execute Command | 
|  | ...  dmesg | egrep '${ERROR_REGEX}' | 
|  | # To enable multiple string check. | 
|  | Should Not Contain  ${dmesg_log}  ${ERROR_DBE_MSG} | 
|  | ...  msg=OS dmesg shows ${ERROR_DBE_MSG}. | 
|  |  | 
|  |  | 
|  | Collect NVIDIA Log File | 
|  | [Documentation]  Collect ndivia-smi command output. | 
|  | [Arguments]  ${suffix} | 
|  | # Description of argument(s): | 
|  | # suffix     String name to append. | 
|  |  | 
|  | # Collects the output of ndivia-smi cmd output. | 
|  | # +--------------------------------------------------------------------+ | 
|  | # | NVIDIA-SMI 361.89                 Driver Version: 361.89           | | 
|  | # |-------------------------------+----------------------+-------------+ | 
|  | # | GPU  Name        Persistence-M| Bus-Id        Disp.A | GPU     ECC | | 
|  | # | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | Utiliz  err | | 
|  | # |===============================+======================+=============| | 
|  | # |   0  Tesla P100-SXM2...  On   | 0002:01:00.0     Off |           0 | | 
|  | # | N/A   25C    P0    35W / 300W |    931MiB / 16280MiB | 0%  Default | | 
|  | # +-------------------------------+----------------------+-------------+ | 
|  | # |   1  Tesla P100-SXM2...  On   | 0003:01:00.0     Off |           0 | | 
|  | # | N/A   26C    P0    40W / 300W |   1477MiB / 16280MiB | 0%  Default | | 
|  | # +-------------------------------+----------------------+-------------+ | 
|  | # |   2  Tesla P100-SXM2...  On   | 0006:01:00.0     Off |           0 | | 
|  | # | N/A   25C    P0    35W / 300W |    931MiB / 16280MiB | 0%  Default | | 
|  | # +-------------------------------+----------------------+-------------+ | 
|  | # |   3  Tesla P100-SXM2...  On   | 0007:01:00.0     Off |           0 | | 
|  | # | N/A   44C    P0   290W / 300W |    965MiB / 16280MiB | 0%  Default | | 
|  | # +-------------------------------+----------------------+-------------+ | 
|  | # +--------------------------------------------------------------------+ | 
|  | # | Processes:                                              GPU Memory | | 
|  | # |  GPU       PID  Type  Process name                      Usage      | | 
|  | # |====================================================================| | 
|  | # |    0     28459    C   hxenvidia                             929MiB | | 
|  | # |    1     28460    C   hxenvidia                            1475MiB | | 
|  | # |    2     28461    C   hxenvidia                             929MiB | | 
|  | # |    3     28462    C   hxenvidia                             963MiB | | 
|  | # +--------------------------------------------------------------------+ | 
|  |  | 
|  | # Create logs directory and get current datetime. | 
|  | Create Directory  ${htx_log_dir_path} | 
|  | ${cur_datetime}=  Get Current Date  result_format=%Y%m%d%H%M%S%f | 
|  |  | 
|  | ${nvidia_out}  ${stderr}  ${rc}=  OS Execute Command  nvidia-smi | 
|  | Write Log Data To File | 
|  | ...  ${nvidia_out} | 
|  | ...  ${htx_log_dir_path}/${OS_HOST}_${cur_datetime}.nvidia_${suffix} | 
|  |  | 
|  |  | 
|  | Get GPU Power Limit | 
|  | [Documentation]  Get NVIDIA GPU maximum permitted power draw. | 
|  |  | 
|  | # nvidia-smi --query-gpu=power.limit --format=csv returns | 
|  | # power.limit [W] | 
|  | # 300.00 W | 
|  | # 300.00 W | 
|  | # 300.00 W | 
|  | # 300.00 W | 
|  |  | 
|  | ${cmd}=  Catenate  nvidia-smi --query-gpu=power.limit | 
|  | ...  --format=csv | cut -f 1 -d ' ' | sort -n -u | tail -n 1 | 
|  | ${nvidia_out}  ${stderr}  ${rc}=  OS Execute Command  ${cmd} | 
|  | # Allow for sensor overshoot.  That is, max power reported for | 
|  | # a GPU could be a few watts above the limit. | 
|  | ${power_max}=  Evaluate  ${nvidia_out}+${7.00} | 
|  | RETURN  ${power_max} | 
|  |  | 
|  |  | 
|  | Get GPU Max Power | 
|  | [Documentation]  Get the maximum GPU power dissipation. | 
|  |  | 
|  | # nvidia-smi --query-gpu=power.draw --format=csv returns | 
|  | # power.draw [W] | 
|  | # 34.12 W | 
|  | # 34.40 W | 
|  | # 36.55 W | 
|  | # 36.05 W | 
|  |  | 
|  | ${cmd}=  Catenate  nvidia-smi --query-gpu=power.draw | 
|  | ...  --format=csv | cut -f 1 -d ' ' | sort -n -u | tail -n 1 | 
|  | ${nvidia_out}  ${stderr}  ${rc}=  OS Execute Command  ${cmd} | 
|  | RETURN  ${nvidia_out} | 
|  |  | 
|  |  | 
|  | Get GPU Min Power | 
|  | [Documentation]  Return the minimum GPU power value as record by | 
|  | ...  nvidia-smi. | 
|  |  | 
|  | ${cmd}=  Catenate  nvidia-smi --query-gpu=power.draw --format=csv | | 
|  | ...  grep -v 'power.draw' | cut -f 1 -d ' ' | sort -n -u | head -1 | 
|  | ${gpu_min_power}  ${stderr}  ${rc}=  OS Execute Command  ${cmd} | 
|  | RETURN  ${gpu_min_power} | 
|  |  | 
|  |  | 
|  | Get GPU Temperature Limit | 
|  | [Documentation]  Get NVIDIA GPU maximum permitted temperature. | 
|  |  | 
|  | # nvidia-smi -q -d TEMPERATURE  | grep "GPU Max" returns | 
|  | #    GPU Max Operating Temp      : 83 C | 
|  | #    GPU Max Operating Temp      : 83 C | 
|  | #    GPU Max Operating Temp      : 83 C | 
|  | #    GPU Max Operating Temp      : 83 C | 
|  |  | 
|  | ${cmd}=  Catenate  nvidia-smi -q -d TEMPERATURE  | grep "GPU Max" | 
|  | ...  | cut -f 2 -d ":" |  tr -dc '0-9\n' | sort -n -u | tail -n 1 | 
|  | ${nvidia_out}  ${stderr}  ${rc}=  OS Execute Command  ${cmd} | 
|  | RETURN  ${nvidia_out} | 
|  |  | 
|  |  | 
|  | Get GPU Min Temperature | 
|  | [Documentation]  Get the minimum GPU temperature. | 
|  |  | 
|  | ${cmd}=  Catenate  nvidia-smi --query-gpu=temperature.gpu | 
|  | ...  --format=csv | grep -v 'temperature.gpu' | sort -n -u | head -1 | 
|  | ${nvidia_out}  ${stderr}  ${rc}=  OS Execute Command  ${cmd} | 
|  | RETURN  ${nvidia_out} | 
|  |  | 
|  |  | 
|  | Get GPU Max Temperature | 
|  | [Documentation]  Get the maximum GPU temperature. | 
|  |  | 
|  | # nvidia-smi --query-gpu=temperature.gpu --format=csv returns | 
|  | # 38 | 
|  | # 41 | 
|  | # 38 | 
|  | # 40 | 
|  |  | 
|  | ${cmd}=  Catenate  nvidia-smi --query-gpu=temperature.gpu | 
|  | ...  --format=csv | sort -n -u | tail -n 1 | 
|  | ${nvidia_out}  ${stderr}  ${rc}=  OS Execute Command  ${cmd} | 
|  | RETURN  ${nvidia_out} | 
|  |  | 
|  |  | 
|  | Get GPU Temperature Via REST | 
|  | [Documentation]  Return the temperature in degrees C of the warmest GPU | 
|  | ...  as reportd by REST. | 
|  |  | 
|  | # NOTE: This endpoint path is not defined until system has been powered-on. | 
|  | ${temperature_objs}=  Read Properties  ${SENSORS_URI}temperature/enumerate | 
|  | ...  timeout=30  quiet=1 | 
|  |  | 
|  | ${core_temperatures_list}=  Catenate  {k:v for k,v in $temperature_objs.items() | 
|  | ...  if re.match('${SENSORS_URI}temperature/.*_core_temp', k)} | 
|  | ${gpu_temperature_objs_dict}=  Evaluate  ${core_temperatures_list}  modules=re | 
|  |  | 
|  | # Create a list containing all of the GPU temperatures. | 
|  | ${gpu_temperatures}=  Evaluate | 
|  | ...  [ x['Value'] for x in $gpu_temperature_objs_dict.values() ] | 
|  |  | 
|  | # Find the max temperature value and divide by 1000 to get just the integer | 
|  | # portion. | 
|  | ${max_gpu_temperature}=  Evaluate | 
|  | ...  int(max(map(int, $gpu_temperatures))/1000) | 
|  |  | 
|  | RETURN  ${max_gpu_temperature} | 
|  |  | 
|  |  | 
|  | Get GPU Clock Limit | 
|  | [Documentation]  Get NVIDIA GPU maximum permitted graphics clock. | 
|  |  | 
|  | # nvidia-smi --query-gpu=clocks.max.gr --format=csv  returns | 
|  | # 1530 MHz | 
|  | # 1530 MHz | 
|  | # 1530 MHz | 
|  | # 1530 MHz | 
|  |  | 
|  | ${cmd}=  Catenate  nvidia-smi --query-gpu=clocks.max.gr | 
|  | ...  --format=csv | cut -f 1 -d ' ' |  sort -n -u | tail -n 1 | 
|  | ${nvidia_out}  ${stderr}  ${rc}=  OS Execute Command  ${cmd} | 
|  | RETURN  ${nvidia_out} | 
|  |  | 
|  |  | 
|  | Get GPU Clock | 
|  | [Documentation]  Get the highest assigned value of the GPU graphics clock. | 
|  |  | 
|  | # nvidia-smi --query-gpu=clocks.gr --format=csv  returns | 
|  | # 1230 MHz | 
|  | # 1230 MHz | 
|  | # 135 MHz | 
|  | # 150 MHz | 
|  |  | 
|  | ${cmd}=  Catenate  nvidia-smi --query-gpu=clocks.gr | 
|  | ...  --format=csv | cut -f 1 -d ' ' | sort -n -u | tail -n 1 | 
|  | ${nvidia_out}  ${stderr}  ${rc}=  OS Execute Command  ${cmd} | 
|  | RETURN  ${nvidia_out} | 
|  |  | 
|  |  | 
|  | Count GPUs From BMC | 
|  | [Documentation]  Determine number of GPUs from the BMC.  Hostboot | 
|  | ...  needs to have been run previously because the BMC gets GPU data | 
|  | ...  from Hostboot. | 
|  |  | 
|  | # Example of gv* endpoint data: | 
|  | # "/xyz/openbmc_project/inventory/system/chassis/motherboard/gv100card0": { | 
|  | #     "Functional": 1, | 
|  | #     "Present": 1, | 
|  | #     "PrettyName": "" | 
|  | # }, | 
|  |  | 
|  | ${num_bmc_gpus}=  Set Variable  ${0} | 
|  |  | 
|  | ${gpu_list}=  Get Endpoint Paths | 
|  | ...  ${HOST_INVENTORY_URI}system/chassis/motherboard  gv* | 
|  |  | 
|  | FOR  ${gpu_uri}  IN  @{gpu_list} | 
|  | ${present}=  Read Attribute  ${gpu_uri}  Present | 
|  | ${state}=  Read Attribute  ${gpu_uri}  Functional | 
|  | Rpvars  gpu_uri  present  state | 
|  | ${num_bmc_gpus}=  Run Keyword If  ${present} and ${state} | 
|  | ...  Evaluate  ${num_bmc_gpus}+${1} | 
|  | END | 
|  | RETURN  ${num_bmc_gpus} | 
|  |  | 
|  |  | 
|  | Create Default MDT Profile | 
|  | [Documentation]  Create default mdt.bu profile and run. | 
|  |  | 
|  | Print Timen  Create HTX mdt profile. | 
|  |  | 
|  | ${profile}  ${stderr}  ${rc}=  OS Execute Command | 
|  | ...  htxcmdline -createmdt | 
|  | Printn  ${profile} | 
|  | Should Contain  ${profile}  mdts are created successfully | 
|  | ...  msg=Create MDT profile failed. response=${profile} | 
|  |  | 
|  |  | 
|  | Run MDT Profile | 
|  | [Documentation]  Load user pre-defined MDT profile. | 
|  | [Arguments]  ${HTX_MDT_PROFILE}=${HTX_MDT_PROFILE} | 
|  | # Description of argument(s): | 
|  | # HTX_MDT_PROFILE  MDT profile to be executed (e.g. "mdt.bu"). | 
|  |  | 
|  | Print Timen  Start HTX mdt profile execution. | 
|  | ${htx_run}  ${stderr}  ${rc}=  OS Execute Command | 
|  | ...  htxcmdline -run -mdt ${HTX_MDT_PROFILE} | 
|  | Printn  ${htx_run} | 
|  | Should Contain  ${htx_run}  Activated | 
|  | ...  msg=htxcmdline run mdt did not return "Activated" status. | 
|  |  | 
|  |  | 
|  | Check HTX Run Status | 
|  | [Documentation]  Get HTX exerciser status and check for error. | 
|  | [Arguments]  ${sleep_time}=${0} | 
|  |  | 
|  | # Description of argument(s): | 
|  | # sleep_time  The amount of time to sleep after checking status, | 
|  | #             for example "3s" or "2m". | 
|  |  | 
|  | Print Timen  Check HTX mdt Status and error. | 
|  | ${htx_status}  ${stderr}  ${rc}=  OS Execute Command | 
|  | ...  htxcmdline -status -mdt ${HTX_MDT_PROFILE} | 
|  | Printn  ${htx_status} | 
|  |  | 
|  | ${htx_errlog}  ${stderr}  ${rc}=  OS Execute Command | 
|  | ...  htxcmdline -geterrlog | 
|  | Printn  ${htx_errlog} | 
|  |  | 
|  | Should Contain  ${htx_errlog}  file </tmp/htx/htxerr> is empty | 
|  | ...  msg=HTX geterrorlog was not empty. | 
|  |  | 
|  | Return From Keyword If  "${sleep_time}" == "${0}" | 
|  |  | 
|  | Run Key U  Sleep \ ${sleep_time} | 
|  |  | 
|  |  | 
|  | Shutdown HTX Exerciser | 
|  | [Documentation]  Shut down HTX exerciser run. | 
|  |  | 
|  | Print Timen  Shutdown HTX Run | 
|  | ${shutdown}  ${stderr}  ${rc}=  OS Execute Command | 
|  | ...  htxcmdline -shutdown -mdt ${HTX_MDT_PROFILE} | 
|  |  | 
|  | Printn  ${shutdown} | 
|  |  | 
|  | ${down1}=  Evaluate  'shutdown successfully' in $shutdown | 
|  | Return From Keyword If  '${down1}' == 'True' | 
|  | ${down2}=  Evaluate  'No MDT is currently running' in $shutdown | 
|  | Return From Keyword If  '${down2}' == 'True' | 
|  | Fail  msg=Shutdown returned unexpected message. | 
|  |  | 
|  |  | 
|  | Create JSON Inventory File | 
|  | [Documentation]  Create a JSON inventory file, and make a YAML copy. | 
|  | [Arguments]  ${json_file_path} | 
|  | # Description of argument: | 
|  | # json_file_path  Where the inventory file is written to. | 
|  |  | 
|  | Login To OS | 
|  | Compile Inventory JSON | 
|  | Run  json2yaml ${json_tmp_file_path} ${yaml_file_path} | 
|  | # Format to JSON pretty print to file. | 
|  | Run  python -m json.tool ${json_tmp_file_path} > ${json_file_path} | 
|  | OperatingSystem.File Should Exist  ${json_file_path} | 
|  | ...  msg=File ${json_file_path} does not exist. | 
|  |  | 
|  |  | 
|  | Compile Inventory JSON | 
|  | [Documentation]  Compile the Inventory into a JSON file. | 
|  | Create File  ${json_tmp_file_path} | 
|  | Write New JSON List  ${json_tmp_file_path}  Inventory | 
|  | Retrieve HW Info And Write  processor  ${json_tmp_file_path} | 
|  | Retrieve HW Info And Write  memory  ${json_tmp_file_path} | 
|  | Retrieve HW Info And Write List  ${I/O}  ${json_tmp_file_path}  I/O  last | 
|  | Close New JSON List  ${json_tmp_file_path} | 
|  |  | 
|  |  | 
|  | Write New JSON List | 
|  | [Documentation]  Start a new JSON list element in file. | 
|  | [Arguments]  ${json_tmp_file_path}  ${json_field_name} | 
|  | # Description of argument(s): | 
|  | # json_tmp_file_path   Name of file to write to. | 
|  | # json_field_name      Name to give json list element. | 
|  | Append to File  ${json_tmp_file_path}  { "${json_field_name}" : [ | 
|  |  | 
|  |  | 
|  | Close New JSON List | 
|  | [Documentation]  Close JSON list element in file. | 
|  | [Arguments]  ${json_tmp_file_path} | 
|  | # Description of argument(s): | 
|  | # json_tmp_file_path  Path of file to write to. | 
|  | Append to File  ${json_tmp_file_path}  ]} | 
|  |  | 
|  |  | 
|  | Retrieve HW Info And Write | 
|  | [Documentation]  Retrieve and write info, add a comma if not last item. | 
|  | [Arguments]  ${class}  ${json_tmp_file_path}  ${last}=false | 
|  | # Description of argument(s): | 
|  | # class               Device class to retrieve with lshw. | 
|  | # json_tmp_file_path  Path of file to write to. | 
|  | # last                Is this the last element in the parent JSON? | 
|  | Write New JSON List  ${json_tmp_file_path}  ${class} | 
|  | ${output}=  Retrieve Hardware Info  ${class} | 
|  | ${output}=  Clean Up String  ${output} | 
|  | Run Keyword if  ${output.__class__ is not type(None)} | 
|  | ...  Append To File  ${json_tmp_file_path}  ${output} | 
|  | Close New JSON List  ${json_tmp_file_path} | 
|  | Run Keyword if  '${last}' == 'false' | 
|  | ...  Append to File  ${json_tmp_file_path}  , | 
|  |  | 
|  |  | 
|  | Retrieve HW Info And Write List | 
|  | [Documentation]  Does a Retrieve/Write with a list of classes and | 
|  | ...              encapsulates them into one large JSON element. | 
|  | [Arguments]  ${list}  ${json_tmp_file_path}  ${json_field_name} | 
|  | ...          ${last}=false | 
|  | # Description of argument(s): | 
|  | # list                 The list of devices classes to retrieve with lshw. | 
|  | # json_tmp_file_path   Path of file to write to. | 
|  | # json_field_name      Name of the JSON element to encapsulate this list. | 
|  | # last                 Is this the last element in the parent JSON? | 
|  | Write New JSON List  ${json_tmp_file_path}  ${json_field_name} | 
|  | FOR  ${class}  IN  @{list} | 
|  | ${tail}  Get From List  ${list}  -1 | 
|  | Run Keyword if  '${tail}' == '${class}' | 
|  | ...  Retrieve HW Info And Write  ${class}  ${json_tmp_file_path}  true | 
|  | ...  ELSE  Retrieve HW Info And Write  ${class}  ${json_tmp_file_path} | 
|  | END | 
|  | Close New JSON List  ${json_tmp_file_path} | 
|  | Run Keyword if  '${last}' == 'false' | 
|  | ...  Append to File  ${json_tmp_file_path}  , | 
|  |  | 
|  |  | 
|  | Retrieve Hardware Info | 
|  | [Documentation]  Retrieves the lshw output of the device class as JSON. | 
|  | [Arguments]  ${class} | 
|  | # Description of argument(s): | 
|  | # class  Device class to retrieve with lshw. | 
|  | ${output}  ${stderr}  ${rc}=  OS Execute Command  lshw -c ${class} -json | 
|  | ${output}=  Verify JSON string  ${output} | 
|  | RETURN  ${output} | 
|  |  | 
|  |  | 
|  | Verify JSON String | 
|  | [Documentation]  Ensure the JSON string content is separated by commas. | 
|  | [Arguments]  ${unver_string} | 
|  | # Description of argument(s): | 
|  | # unver_string  JSON String we will be checking for lshw comma errors. | 
|  | ${unver_string}=  Convert to String  ${unver_string} | 
|  | ${ver_string}=  Replace String Using Regexp  ${unver_string}  }\\s*{  },{ | 
|  | RETURN  ${ver_string} | 
|  |  | 
|  |  | 
|  | Clean Up String | 
|  | [Documentation]  Remove extra whitespace and trailing commas. | 
|  | [Arguments]  ${dirty_string} | 
|  | # Description of argument(s): | 
|  | # dirty_string  String that will be space stripped and have comma removed. | 
|  | ${clean_string}=  Strip String  ${dirty_string} | 
|  | ${last_char}=  Get Substring  ${clean_string}  -1 | 
|  | ${trimmed_string}=  Get Substring  ${clean_string}  0  -1 | 
|  | ${clean_string}=  Set Variable If  '${last_char}' == ',' | 
|  | ...  ${trimmed_string}  ${clean_string} | 
|  | RETURN  ${clean_string} | 
|  |  | 
|  |  | 
|  | Get OS Network Interface Names | 
|  | [Documentation]  Return a list of interface names on the OS. | 
|  |  | 
|  | ${stdout}  ${stderr}  ${rc}=  OS Execute Command  ls /sys/class/net | 
|  | @{interface_names}=  Split String  ${stdout} | 
|  | RETURN  @{interface_names} | 
|  |  | 
|  |  | 
|  | Run Soft Bootme | 
|  | [Documentation]  Run a soft bootme for a period of an hour. | 
|  | [Arguments]  ${bootme_period}=3 | 
|  | # Description of argument(s): | 
|  | # bootme_time     Bootme period to be rebooting the system. | 
|  |  | 
|  | ${output}  ${stderr}  ${rc}=  OS Execute Command | 
|  | ...  htxcmdline -bootme on mode:soft period:${bootme_period} | 
|  | Should Contain  ${output}  bootme on is completed successfully | 
|  |  | 
|  |  | 
|  | Shutdown Bootme | 
|  | [Documentation]  Stop the bootme process. | 
|  |  | 
|  | ${output}  ${stderr}  ${rc}=  OS Execute Command  htxcmdline -bootme off | 
|  | Should Contain  ${output}  bootme off is completed successfully |