blob: b5a18f73d5c22acace1963968698408e21d6e23d [file] [log] [blame]
Daniel Gonzalezeb40b752018-03-16 16:38:58 -06001*** Settings ***
2Documentation Verify the EEH recovery on the controllers connected to the
3... PCI. This injects an EEH error to every controller installed on the
4... server.
5
6# Test Parameters:
7
8# TYPE EEH error function to use.
9# OPENBMC_HOST The BMC host name or IP address.
10# OS_HOST The OS host name or IP Address.
11# OS_USERNAME The OS login userid (usually root).
12# OS_PASSWORD The password for the OS login.
13# HTX_DURATION Duration of HTX run (e.g. "8 hours", "30 minutes").
14# HTX_INTERVAL The time delay between consecutive checks of HTX
15# status, for example, 30s.
16# In summary: Run HTX for $HTX_DURATION, checking
17# every $HTX_INTERVAL.
18
19# The injection command:
20# echo <pe_num>:<mode>:<type>:<address>:<mask> > /sys/kernel/debug/powerpc/PCIxxxx/err_injct
21# Input arguments to injection command:
22# pe_num: Determined from the output of
23# 'cat /sys/bus/pci/devices/xxxx:xx:xx.x/eeh_pe_config_addr'
24# where the xxxx.xx.xx.x is the PCI location of the device you want.
25# mode: 0 for 32-bit BARs and 64-bit non-prefetchable BARs
26# 1 for 64-bit prefetchable BARs
27# type: Error function to use:
28# 0 to inject on an MMIO load
29# 4 to inject on a config load--Not recommended for standard
30# injection trials.
31# 6 to inject on an MMIO writes
32# 10 to inject on a CFG write
33# address: If you do not care to specify which endpoint under the
34# slot is to be frozen, use a value of '0'.
35# mask: If you do not care to specify which endpoint under the
36# slot is to be frozen, use a value of '0'.
37# xxxx: The PCI domain location of the device.
38
39# Example:
40# echo 5:0:6:3fe280810000:fffffffffffff800 > /sys/kernel/debug/powerpc/PCI0005/err_injct
41# Expected output:
42# HTX Runs mdt.hdbuster or net.mdt and doesn't log errors after error injections.
43
44# Glossary:
45# EEH:
46# Enhanced I/O Error Handling is an error-recovery mechanism for errors that
47# occur during load and store operations on the PCI bus.
48# MDT:
49# Master device table is a collection of hardware devices on the system for
50# which HTX exercisers can be run.
51# mdt.hdbuster:
52# MDT for disk storage device testing. Uses 0s and FFs for data patterns.
53# net.mdt:
54# MDT used to test network adapters.
55# HTX error log file /tmp/htxerr
56# Records all the errors that occur. If there's no error during the test, it
57# should be empty.
58# TODO: Check if monitoring dmesg entries adds value.
59
60Library SSHLibrary
61Library String
62Library ../lib/bmc_ssh_utils.py
Sandhya Somashekar839a0c22019-01-31 05:05:43 -060063Resource ../lib/resource.robot
Daniel Gonzalezeb40b752018-03-16 16:38:58 -060064Resource ../syslib/utils_os.robot
65
66Suite Setup Suite Setup Execution
67Suite Teardown Collect HTX Log Files
68Test Teardown FFDC On Test Case Fail
69
70*** Variables ***
71${HTX_DURATION} 8 horus
72${HTX_INTERVAL} 15 minutes
73${TYPE} 0
74
75*** Test Cases ***
76Test Integrated IO Adapters EEH
77 [Documentation] Inject EEH errors in every integrated ethernet controller
78 ... and check if errors are logged.
79 [Tags] Test_IO_Adapters_EEH
80
81 # Setting HTX_MDT_PROFILE to be used in 'Run MDT Profile' keyword
82 Set Suite Variable ${HTX_MDT_PROFILE} net.mdt
83 # Setting lspci_cmd to be used by 'Get PCI' keyword.
84 Set Test Variable ${lspci_cmd} lspci -D | grep "Ethernet controller"
85 @{pci_list}= Get PCI
86 @{pci_list}= Split Integrated Or External PCI ${TRUE} ${pci_list}
87 #Preconfigure Net MDT
88 Run MDT Profile
89 Repeat Keyword ${HTX_DURATION} Cycle Through PCIs ${pci_list}
90 Shutdown HTX Exerciser
91
92Test External IO Adapters EEH
93 [Documentation] Inject EEH errors in every external ethernet controller
94 ... and check if errors are logged.
95 [Tags] Test_IO_Adapters_EEH
96
97 # Setting HTX_MDT_PROFILE to be used in 'Run MDT Profile' keyword
98 Set Suite Variable ${HTX_MDT_PROFILE} net.mdt
99 # Setting lspci_cmd to be used by 'Get PCI' keyword.
100 Set Test Variable ${lspci_cmd} lspci -D | grep "Ethernet controller"
101 @{pci_list}= Get PCI
102 @{pci_list}= Split Integrated Or External PCI ${FALSE} ${pci_list}
103 Preconfigure Net MDT
104 Run MDT Profile
105 Repeat Keyword ${HTX_DURATION} Cycle Through PCIs ${pci_list}
106 Shutdown HTX Exerciser
107
108Test IO Storage EEH
109 [Documentation] Inject EEH Errors and check if errors are logged in htx.
110 [Tags] Test_IO_Storage_EEH
111
112 # Setting HTX_MDT_PROFILE to be used in 'Run MDT Profile' keyword
113 Set Suite Variable ${HTX_MDT_PROFILE} mdt.hdbuster children=true
114 # Setting lspci_cmd to be used by 'Get PCI' keyword.
115 Set Test Variable ${lspci_cmd} lspci -D | grep Marvell
116 # Setting pci_list to be used in 'Cycle Through PCIs' keyword
117 @{pci_list}= Get PCI
118 Set Test Variable @{pci_list}
119 Run MDT Profile
120 Repeat Keyword ${HTX_DURATION} Cycle Through PCIs ${pci_list}
121 Shutdown HTX Exerciser
122
123*** Keywords ***
124Cycle Through PCIs
125 [Documentation] Run a cycle to make error injections to every PCI
126 ... in the pci_list.
127 [Arguments] ${pci_list}=${EMPTY}
128 # Descroption of argument(s):
129 # pci_list A list that contains the PCIs selected (integrated/external).
130 # Every object in the list is the PCI address as domain number
131 # (0 to ffff), bus (0 to ff), slot (0 to 1f) and function
132 # (0 to 7). (e.g. "0005:01:00.1").
133
134 :FOR ${pci} IN @{pci_list}
135 \ Inject EEH Error ${pci}
136 \ Check HTX Run Status
137 \ Run Key U sleep \ ${HTX_INTERVAL}
138
139Preconfigure Net MDT
140 [Documentation] Run build_net to preconfigure the interfaces to be
141 ... injected with the EEH error.
142
143 OS Execute Command build_net help y y
144 ${output} ${stderr} ${rc}= OS Execute Command
145 ... pingum | egrep "All networks ping Ok"
146 Should Contain ${output} All networks ping Ok
147
148Inject EEH Error
149 [Documentation] Build and inject the EEH error command.
150 [Arguments] ${pci}=${EMPTY}
151 # Description of argument(s):
152 # pci PCI address as domain number (0 to ffff), bus (0 to ff),
153 # slot (0 to 1f) and function (0 to 7).
154 # (e.g. "0000:00:1f.2").
155
156 ${sub_cmd}= Make Injection Command ${pci}
157 ${pci}= Fetch From Left ${pci} :
158 ${cmd}= Catenate echo ${sub_cmd} >
159 ... /sys/kernel/debug/powerpc/PCI${pci}/err_injct_inboundA
160 ${output} ${stderr} ${rc}= OS Execute Command ${cmd}
161
162Get PCI
163 [Documentation] Obtain the PCI IDs for every ethernet controller.
164 ${output} ${stderr} ${rc}= OS Execute Command
165 ... ${lspci_cmd} | cut -d " " -f1
166 @{pci_list}= Split To Lines ${output}
167 [Return] @{pci_list}
168
169Split Integrated Or External PCI
170 [Documentation] Separate a list of internal or external PCIs according to
171 ... the test case.
172 [Arguments] ${integrated}=${TRUE} ${pci_list}=${EMPTY}
173 # Description of argument(s):
174 # integrated Boolean variable to determine if the adapters are external
175 # or integrated to the system.
176 # tmp_pci_list List of PCIs (e.g. ["0000:00:00.0", "0001:00:00.0"]).
177
178 @{new_pci_list}= Create List
179 :FOR ${pci} IN @{pci_list}
180 \ ${output} ${stderr} ${rc}= OS Execute Command
181 ... lscfg -vl ${pci} | grep "Location Code"
182 \ ${status} ${err_result}= Run Keyword And Ignore Error
183 ... Should Contain ${output} SLOT
184 \ Run Keyword If '${status}' == 'PASS' and '${integrated}' == '${FALSE}'
185 ... Append To List ${new_pci_list} ${pci}
186 ... ELSE IF '${status}' == 'FAIL' and '${integrated}' == '${TRUE}'
187 ... Append To List ${new_pci_list} ${pci}
188 [Return] ${new_pci_list}
189
190Make Injection Command
191 [Documentation] Create the string that will inject the EEH error.
192 [Tags] Create_Injection_Command
193 [Arguments] ${pci}=${EMPTY}
194 # Description of argument(s):
195 # pci PCI address as domain number (0 to ffff), bus (0 to ff),
196 # slot (0 to 1f) and function (0 to 7).
197 # (e.g. "0000:00:1f.2").
198
199 ${pe_num}= Get PE Num ${pci}
200 ${mode}= Get Mode ${pci}
201 ${address}= Get Address ${pci}
202 ${mask}= Get Mask ${pci}
203 ${result}= Catenate SEPARATOR=:
204 ... ${pe_num} ${mode} ${TYPE} ${address} ${mask}
205 [Return] ${result}
206
207Get PE Num
208 [Documentation] Return the PE configuration address of the PCI sent.
209 [Tags] Get_PE_Number
210 [Arguments] ${pci}=${EMPTY}
211 # Description of argument(s):
212 # pci PCI address as domain number (0 to ffff), bus (0 to ff),
213 # slot (0 to 1f) and function (0 to 7).
214 # (e.g. "0000:00:1f.2").
215
216 ${output} ${stderr} ${rc}= OS Execute Command
217 ... cut -d "x" -f2 /sys/bus/pci/devices/${pci}/eeh_pe_config_addr
218 [Return] ${output}
219
220Get Mode
221 [Documentation] Return the "mode" field value.
222 [Tags] Get_Mode
223 [Arguments] ${pci}=${EMPTY}
224 # Description of argument(s):
225 # pci PCI address as domain number (0 to ffff), bus (0 to ff),
226 # slot (0 to 1f) and function (0 to 7).
227 # (e.g. "0000:00:1f.2").
228
229 ${cmd_buf}= Catenate lspci -bvs ${pci} | grep "Memory at" |
230 ... grep "64-bit" | wc -l
231 ${output} ${stderr} ${rc}= OS Execute Command ${cmd_buf}
232 ${mode}= Set Variable If '${output}'=='0' 0 1
233 [Return] ${mode}
234
235Get Address
236 [Documentation] Obtain the PCI address.
237 [Arguments] ${pci}=${EMPTY}
238 # Description of argument(s):
239 # pci PCI address as domain number (0 to ffff), bus (0 to ff),
240 # slot (0 to 1f) and function (0 to 7).
241 # (e.g. "0000:00:1f.2").
242
243 ${cmd_buf}= Catenate lspci -vv -s ${pci} | grep "Memory at"
244 ... | cut -d " " -f5 | head -n 1
245 ${output} ${stderr} ${rc}= OS Execute Command ${cmd_buf}
246 [Return] ${output}
247
248Get Mask
249 [Documentation] Return the selected PCI mask.
250 [Arguments] ${pci}=${EMPTY}
251 # Description of argument(s):
252 # pci PCI address as domain number (0 to ffff), bus (0 to ff),
253 # slot (0 to 1f) and function (0 to 7).
254 # (e.g. "0000:00:1f.2").
255
256 ${cmd_buf}= Catenate lspci -vv -s ${pci} | grep "Memory at"
257 ... | head -n 1 | cut -d "=" -f2
258 ${stdout} ${stderr} ${rc}= OS Execute Command ${cmd_buf}
259 ${sze}= Get Substring ${stdout} -2 -1
260 ${size}= Get Substring ${stdout} 0 -2
261 ${size}= Convert To Integer ${size}
262 ${size}= Run Keyword If '${sze}' == 'M'
263 ... Evaluate ${size}*1024*1024
264 ... ELSE IF '${sze}' == 'K'
265 ... Evaluate ${size}*1024
266 Log To Console ${size}
267 ${size}= Convert To Hex ${size}
268 ${mask}= Evaluate "{:f>16}".format(${size})
269 [Return] ${mask}
270
271Suite Setup Execution
272 [Documentation] Do suite setup tasks.
273
274 Boot To OS
275 Tool Exist lspci
276 Tool Exist htxcmdline
277 Create Default MDT Profile