FICON Express2 Channel Performance Version 1.0 - IBM

April 2005
FICON Express2 Channel
Performance Version 1.0
Cathy Cronin
zSeries I/O Performance
ccronin@us.ibm.com

FICON Express2 Channel Performance Version 1.0
Page 1
Introduction
This white paper was developed to help IBM ® field sales specialists and technical
representatives understand the performance characteristics of FICON ® Express2 channels.
What’s New
FICON Express2 channels are a new generation of FICON channels that offer improved
performance capability over previous generations of FICON Express and FICON channels.
They are being introduced on the IBM eServer zSeries ® 990 (z990) and zSeries 890 (z890).
Overview
IBM has made significant improvements to FICON channels since this product was initially
shipped in 1999. The following chart depicts some of those improvements:
14
12
10
8
6
4
2
0
Figure 1
FICON Express2 Channel Performance
I/Os per second (k)
4k block sizes
Channel 100% utilized
1200
3600
6000
7200
9200
FICON
Express2
13000
300
250
200
150
100
FICON
Express2
2 Gbps
270
Reflected in the left bar chart is the "best can do" capabilities of each of the FICON channels
in native FICON or FC mode measured at a point in time using an I/O driver benchmark
program for 4K byte read hits. 4K bytes is the size of most online database I/O operations.
These are the maximum possible or 100% channel utilization 4K I/O rates for each channel.
50
0
MB/sec throughput (Full Duplex)
Large Sequential R/Ws
17
74
120
170

FICON Express2 Channel Performance Version 1.0
Page 2
Normally customers should keep their channels at 50% or less channel utilization to achieve
good online transaction response times.
Reflected in the right bar chart is the "best can do" capabilities of each of the FICON
channels in native FICON or FC mode measured using an I/O driver benchmark program for
6x27K or 6 half-track reads and writes. This is representative of the type of channel
programs used in disk to tape backup jobs or other highly sequential batch jobs. The original
FICON channels run at a link speed of 1 Gigabit/second. FICON Express and FICON
Express2 channels will auto-negotiate to either 1 Gigabit/s or 2 Gigabit/s, depending on the
capability of the director or control unit port at the other end of the link.
As you can see, the FICON Express2 channel as first introduced on the IBM zSeries z890 and
z990 represents a significant improvement in both 4K I/O per second throughput and
maximum bandwidth capability compared to ESCON ® and previous FICON offerings.
Please remember that this performance data was measured in a controlled environment
running an I/O driver program. The actual throughput or performance that any user will
experience will vary depending upon considerations such as the amount of
multiprogramming in the user’s job stream, the I/O configuration, the storage configuration,
and the workload processed.
This paper assumes that the reader is familiar with the basic benefits of FICON vs. ESCON
technology and will explain in more detail the performance characteristics of FICON
Express2 channels running in FC mode (native FICON) including DASD I/O driver
benchmark results, CTC measurement results, FICON Express2 channel and ESTI-M card
level measurement results.
Please note that FICON Express2 channels do not support FCV (FICON Converter) mode for
attachment to ESCON devices. For an introduction to the basic benefits of FICON vs.
ESCON technology and for info on FCV mode performance, please see version 2 of the
FICON and FICON Express Performance white paper on the zSeries I/O connectivity Web
site at the following URL:
www.ibm.com/servers/eserver/zseries/connectivity/

FICON Express2 Channel Performance Version 1.0
Page 3
Introduction to some terminology used in I/O processing
First I would like to start by explaining some of the basic terms that I will be using in the rest
of this paper.
some resources & terminology involved in
I/O processing
zSeries CP
FICON
channel
card
FICON
channel
pci
bus
sti
link
ssch
adapter
esti-M
card
zSeries
Sap/IOP
esti
link
fc
link director
f-port
Figure 2
io
interrupt
MBA
chip
director
f-port
FICON
channel
processor
store
fetch
fc
link
zSeries
Memory
cu
n-port
As depicted in the top row of Figure 2, an I/O is initiated when a zSeries CP (Central
Processor) executes a SSCH (start subchannel) instruction. This sends a signal to a SAP
(System Assist Processor), which is also called an IOP ( I/O Processor) that there is I/O work
to do. It is the SAP’s job to select which channel path to use to get to the device which is the
target of this I/O. The SAP is also involved in processing the I/O interrupts that are sent back
for most I/O’s at the end of the I/O operation. Some channel programs generate PCI’s
(Programmed Controlled Interrupts) which can occur at designated points in the middle of an
I/O operation.
The second row depicts the path that is followed for any data transfer that occurs during an
I/O operation between the FICON channel card and zSeries memory. For a READ I/O, data
is READ from the device and stored into zSeries memory. For a WRITE I/O, data is fetched
from zSeries memory and written to the device. There are 4 FICON Express2 channels on a
FICON Express2 channel card that share a 1GB/sec STI link connected to an ESTI-M card.
Up to 4 channel cards of any type (ESCON, FICON Express or FICON Express2) can be

FICON Express2 Channel Performance Version 1.0
Page 4
connected to the same ESTI-M card and these would share a single 2GB/sec ESTI link from
the ESTI-M card to the MBA chip.
The third row depicts the path followed by commands and data frames transferred from a
FICON channel to a FICON CU port. Each of the 4 FICON Express2 channels on the FICON
Express2 channel card has its own PCI bus connected to an industry standard Emulex fibre
channel adapter chip which handles the transmitting and receiving of frames across the
2Gbps FC (fibre channel) link. The FC link could be connected point-to-point to a CU port
or through a source and destination Fabric port (f-port) on a director. Both the channel and
the CU ports are called N-ports in the Fabric. If two directors were cascaded together the
ports connecting the two directors would be called E-ports and the link connecting the two
directors is an ISL or inter-switch link. One source of confusion that I have seen very often is
to use the term channel adapter or even just channel for the CU port. In this paper, when I
use the term channel, I mean the chip on the card that is plugged into the zSeries CEC. It is
important to understand that FICON channels and FICON CU ports can have very different
performance capabilities. It is the performance capabilities of FICON Express2 channels that
are presented in this paper.
In general, each of the various resources depicted above are utilized at different levels
depending on the type of I/O that is being processed and the numbers of each resource (CPs,
SAPs, MBA chips, ESTI-M cards, channel cards, director ports and CU ports) that are in the
configuration. For the most part, with small block I/O operations, processors such as the
FICON channel and the CU port are pushed to higher levels of utilization than the buses and
links. In contrast, I/O’s that transfer a lot of data push the buses and links to higher levels of
utilizations than the processors. The resource that gets pushed to the highest utilization will
be the one that limits higher levels of throughput from being achieved.
FICON Express2 benchmark measurement results
To achieve maximum channel capabilities, I/O driver benchmark measurements were
conducted using a configuration with 4 FICON Express2 channels on 4 different channel
cards connected through three 2Gbps directors to 4 ports on each of 6 different control unit
(CU) or storage subsystem boxes as depicted in Figure 3:

FICON Express2 Channel Performance Version 1.0
Page 5
z990
FICON
Express2
Channels
A1,B2,C3,D4
CU box 1
CU box 2
Configuration used for
FICON Express2 channel
benchmark measurements
Director(s)
Figure 3
CU box 6
Please note that the response time results reported in this paper are the average of all of the
LCUs (Logical Control Units) on the storage subsystems or CU boxes used for these
measurements.
Measurements done in a point-to-point topology without directors and/or using control units
that have ports with less I/O per second or MB/sec throughput capabilities than the FICON
Express2 channels will not push the channels to their maximum capability. Furthermore, if
one is interested in determining the maximum capability of a CU port instead of a channel,
then it is recommended that a configuration with multiple channels connected through a
director be used to obtain the best results. An example of this is depicted in Figure 4.

FICON Express2 Channel Performance Version 1.0
Page 6
Recommended configuration for
determining max capability of a CU port for
benchmark testing
multiple FICON
channels
connected through
a FICON director to
a single CU port
Figure 4
The four basic DASD I/O driver benchmark programs used to evaluate the capabilities of the
new FICON Express2 channels are as follows:
1. 4K bytes per I/O: this channel program processes small blocks of I/O and is capable of
achieving high I/O per second rates but much lower MB/sec rates than large block
channel programs. With the appropriate read/write ratios and CU cache hit ratios, this
benchmark is representative of online transaction processing workloads.
2. 6x27K bytes per I/O: this channel program processes 6 large blocks with 27K bytes each
or 6 half-tracks of data and is capable of achieving high MB/sec but much lower I/O per
second than the small block channel programs. It is representative of the type of channel
programs used in disk to tape backup jobs or other highly sequential batch jobs.
3. 27K bytes per I/O: this channel program processes a single half track of data and achieves
both I/O per second and MB/sec that are in between the extremes of the 4K and 6x27K
bytes per I/O benchmarks.
4. 32x4K bytes per I/O: this channel program processes 32 small (4K byte) blocks of I/O and
is representative of some DB2 pre-fetching utilities and other channel programs that
process long chains of short blocks of data.
Figure 5 below shows the average of all of the LCU (Logical Control Unit) response times for
the 4k read hit benchmark measurement plotted with FICON Processor Utilization(FPU) %s .
Response times in milliseconds are on the left y-axis. FPU %s are on the 2nd or right y-axis.
The knee of the response time curve occurs around 10,000 I/O’s per second and just above
70% FICON Processor utilization (FPU) when running this very simple 4K read hit
benchmark workload. But most real production workloads are more complex than this
z990
FICON
Express2
Channels
W1,X2,Y3,Z4
Director
CU box

FICON Express2 Channel Performance Version 1.0
Page 7
simple benchmark and in general, we usually recommend that you keep FPU below 50% to
achieve good online transaction response times. The 50% FPU point occurs between 6000
and 7000 I/O’s per second when running this very simple 4K read hit benchmark workload.
FICON Express2(FEx2) 4K read hit
response times & FICON processor utilization(FPU)
resp time in millisec
5
100
90
4
80
70
3
60
50
2
40
30
1
20
10
0
0
0 2 4 6 8 10 12 14
io/sec
Thousands
Figure 5
Figure 6 shows the breakdown of response time components for this simple 4k read hit
benchmark from 10% FICON Processor Utilization(FPU) through 70% FPU or just before
the knee of the response time curve. Both total response times and the response time
components of IOSQ, PEND, DISC (disconnect) and CONN (connect) time can be found on
the RMF Device Activity report. IOSQ time is the time that an I/O is delayed due to the fact
that another I/O from this system is already using the target device for this I/O. PEND time
starts when the CP sends the I/O request to the SAP and includes the amount of time it takes
for the channel to process the first few CCWs (Channel Command Words) in the channel
program and send the commands to the Control Unit and does not end until the Control Unit
sends a CMR or Command Response back to the channel. Disconnect time is the amount of
time it takes the Control Unit to service a CU cache miss and retrieve the data from the
device. CONNECT time is basically the data transfer time for the I/O. So, in this 4k read hit
benchmark, there is no DISC time since there are no CU cache misses and there is no IOSQ
time since the I/O driver program we use is designed to wait for an I/O to an individual
device to finish before it issues another I/O to that same device. So, all we have is PEND
FICON Processor Util%
FEx2 resp time
FEx2 FPU%

FICON Express2 Channel Performance Version 1.0
Page 8
time + CONN time. Since each I/O only transfers 4K bytes of data, CONN time is the smaller
of the two components and remains relatively constant from low I/O rates to high I/O rates.
On the other hand, PEND time grows as the FPU% grows. If we had used a point-to-point
configuration to do this measurement and if the CU port had less capability than the new
FICON Express2 channel, then the PEND time would have grown faster as a function of the
CU port processor utilization and we would not have been able to push the channel to its
maximum capability.
response time components for 4K read
hits...4K bytes/io, 100% cu cache hit ratio
FICON processor utilization
10% FPU
20% FPU
30% FPU
40% FPU
50% FPU
60% FPU
70% FPU
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
response times in ms
Figure 6
pend
conn
Figure 7 shows the response time components for a more realistic version of an online
transaction processing workload with a mix of reads and writes and a 70 to 80% CU cache hit
ratio. In this case disconnect time is the largest component of the total response time and the
component that grows the most as the activity rate increases. PEND and CONNECT times
are about equal up to the 60% FPU point. There is a more significant increase in total
response time beyond the 50% FPU point than there was with the simpler 4K read
benchmark.

FICON Express2 Channel Performance Version 1.0
Page 9
response time components for 4K bytes/io,
3:1 r/w ratio, 70 to 80% cu cache hit ratio
FICON Processor utilization
10% FPU
20% FPU
30% FPU
40% FPU
50% FPU
60% FPU
70% FPU
0 1 2 3 4 5
response times in ms
Figure 7
pend
disc
conn
Figures 8 and 9 represent two different ways of looking at the results of the 6x27k read hit
benchmark. Figure 8 is a plot of response times and FICON processor utilization for the
6x27K read hit benchmark. Since this workload transfers over 165,000 bytes per I/O using a
block size of 27K bytes, it stresses the links and buses more than it does the FICON channel
processor. The maximum I/Os per second achieved was only 1100 io/sec which only drove
the FICON processor utilization to about 40%. Therefore the channel processor is really not
the resource that prevents this workload from achieving higher throughput. In general, for
workloads that use large block sizes such as the 27K byte half-track size, it makes more sense
to look at MB/sec instead of I/O per second and bus or link utilizations instead of processor
utilizations. Figure 9 shows that this workload achieves 200 MB/sec which is the limit of the
2 Gbps link.

FICON Express2 Channel Performance Version 1.0
Page 10
z990 FEx2 6x27K Read hits
response times & FPU vs io/sec
avg response time in ms
5
100
90
4
80
70
3
60
50
2
40
30
1
20
10
0
0
0 200 400 600 800 1000 1200 1400
io/sec per channel
Figure 8
In Figure 9, we look at the results of this same 6x27K read hit benchmark measurement with
response times and FICON link utilization vs. MB/sec. FICON channel link utilizations are
not directly reported on RMF but can be easily calculated by dividing the READ or WRITE
MB/sec by the link capacity. In this case, with 2 Gpbs links, the capacity is approximately
200MB/sec. Here we see that the 2 Gbps link is the limit to achieving higher throughput
since it is the resource that is being pushed closest to 100% utilization. The “knee of the
response time” curve generally occurs between 70 and 80% link utilizations. If you are more
interested in throughput than response times, then these utilization levels may be acceptable.
If, however, you are running response time sensitive workloads then it might be more
appropriate to keep link utilizations below 50%.
FICON processor util%
FEx2 resp time
FEx2 FPU

FICON Express2 Channel Performance Version 1.0
Page 11
z990 FEx2 6x27K Read hits
response times & link util vs MB/sec
avg response time in ms
5
4
3
2
1
0
0
0 40 80 120 160 200
MB/sec per channel
Figure 9
Figure 10 shows the response time components for the 6x27k read hit benchmark. Since the
data transfer size for this channel program is 162K bytes per I/O and it uses a large 27k block
size, CONNECT time is the dominant part of the total response time. CONNECT time grows
from under 2ms at 20% channel link utilization to over 3ms at very high (90%) link
utilization levels but these CONNECT times are still significantly better than the 10ms
measured for this benchmark a few years ago using ESCON channels. PEND time also grows
a few tenths of a millisecond at high link utilizations. But at the 50% FICON Express2
channel link utilization level, total response times are only a few tenths of a millisecond
higher than the best case response times for this workload.
100
90
80
70
60
50
40
30
20
10
FICON link util%
FEx2 resp time
FEx2 link util%

FICON Express2 Channel Performance Version 1.0
Page 12
response time components for 6x27K read
hits...162K bytes/io, 100% cu cache hit ratio
2Gbps Link Utilization(LU)
20% LU
30% LU
40% LU
50% LU
60% LU
70% LU
80% LU
90% LU
0 1 2 3 4 5
response times in ms
Figure 10
pend
conn
Figure 11 depicts the results of the 6x27K read/write mix benchmark where we achieve
270MB/sec by taking advantage of the full duplex capabilities of FICON links and
simultaneously processing some I/O’s that READ from DASD and some I/O’s that WRITE to
DASD. The 270 MB/sec achieved for this benchmark using FICON Express2 channels is
more than 50% higher than the maximum MB/sec that was achieved with the previous
generation FICON Express channels. The Full Duplex Link Utilizations (FDLU) plotted on
the 2nd y-axis in Figure 11 is calculated by dividing the sum of the READ + WRITE MB/sec
by 400 MB/sec which is the sum of the maximum instantaneous capabilities of the two
directional 2 Gigabit per second (Gbps) links that exist between the FICON Express2 channel
and the director port, where one 2 Gbps link transmits the commands and data frames from
the channel to the director and the other 2 Gbps link transfers the commands and data
frames in the opposite direction from the director to the channel.

FICON Express2 Channel Performance Version 1.0
Page 13
z990 FEx2 6x27K Read/write mix
response times & Link Utilization vs MB/sec
Response Times in ms
10
Figure 11
Figure 12 depicts the response time components for the 6x27k read/write mix benchmark.
CONNECT time is the largest component and increases the most as full duplex link
utilization increases.
Full Duplex Link Utilization(FDLU)
8
6
4
2
0
0
30
response time components for 6x27K read/write
mix, 162K bytes/io, 100% cu cache hit ratio
10% FDLU
20% FDLU
30% FDLU
40% FDLU
50% FDLU
60% FDLU
60
90
120
150
180
210
MB/sec per channel
240
270
0 1 2 3 4 5 6
response times in ms
Figure 12
0
300
100
90
80
70
60
50
40
30
20
10
Full Duplex Link Utilization(FDLU)
Resp time
FEx2 FDLU
pend
conn

FICON Express2 Channel Performance Version 1.0
Page 14
Figure 13 shows the FICON Express2 channel PCI bus utilizations for this same 6x27K
Read/Write mix benchmark. PCI bus utilizations are the bus utilizations reported on the
RMF Channel activity report for FICON Express2 channels but there is another internal
channel bus whose utilization is roughly 1.5 to 2 times the PCI bus utilization. This internal
channel bus is the real resource that limits the 6x27K Read/Write benchmark from achieving
higher than 270MB/sec but it is highly unlikely that any real production workload would
come anywhere near approaching this limit. For real production workloads, the most
relevant resource limits to pay attention to are the channel and the control unit processor and
link limits and these are the limits that I have highlighted for each benchmark measurement
result presented in this paper.
z990 FEx2 6x27K Read/write mix
FICON Bus Utilization(FBU) vs MB/sec
FICON bus utilization (FBU)
100
90
80
70
60
50
40
30
20
10
0
0 30 60 90 120 150 180 210 240 270 300
Total READ + WRITE MB/sec
Figure 13
FEx2 FBU
Figures 14 and 15 represent two different ways of looking at the results of the 27K or
half-track read hit benchmark. The first figure below is a plot of response times and FICON
processor utilizations vs. io/sec. Here we see a sharp increase in response times just over
6000 io/sec and at about 60% FICON processor utilization (FPU). The second 27K read hit
graph plots response times and FICON channel link utilizations vs. MB/sec. The sharp
increase in response times occurs at about 170 MB/sec which is about 85% of the maximum
link capability and indicates that the 2 Gbps link is the limiting resource for this workload.
But processor utilizations are pushed to high levels as well. The 27k read hit benchmark
pushes both processor and link utilizations to high levels with link utilizations slightly higher
than the processor.

FICON Express2 Channel Performance Version 1.0
Page 15
FICON Express2 27K read hit
response times & channel processor util
resp time in millisec
5
4
3
2
1
0
0
0 1 2 3 4 5 6 7
io/sec
Thousands
Figure 14
FICON Express2 27K read hit
response times & link utilizations
resp time in millisec
5
4
3
2
1
0
0
0 40 80 120 160 200
MB/sec
Figure 15
100
80
60
40
20
100
90
80
70
60
50
40
30
20
10
FICON Processor Util%
FICON Link Util%
FEx2 resp time
FEx2 FPU%
FEx2 resp time
FEx2 Link U%

FICON Express2 Channel Performance Version 1.0
Page 16
Figure 16 is the response time components for the 27k read hit benchmark. Because of the
large 27k block size, CONNECT time is the largest component of the total response time. As
link utilizations (LU) increase, CONNECT time increases first by one tenth of a millisecond,
and then PEND time also increases by one tenth of a millisecond. At 80% LU, both
CONNECT time and PEND time are 0.4ms higher than they were at 10% LU, another
indicator that this workload is in the middle of the two extremes defined by processor limited
workloads such as the 4K bytes per I/O benchmarks and bus or link limited workloads such
as the 6x27K bytes per I/O benchmarks.
response time components for 27K read hits...
27K bytes/io, 100% cu cache hit ratio
2Gbps Link Utilization(LU)
10% LU
20% LU
30% LU
40% LU
50% LU
60% LU
70% LU
80% LU
0 0.5 1 1.5 2
response times in ms
Figure 16
pend
conn
The 32x4k read hit benchmark depicted in Figure 17 is another benchmark that pushes both
the processor and the link to high levels of utilization with the processor slightly higher than
the link utilization. Figure 17 shows the response time components for a 32x4K read hit
benchmark which is a long chain of short blocks. Here CONNECT time is the largest
component since the total data transfer size is 128K bytes per I/O even though the block size
for each CCW is only 4K bytes. Since there is a separate CCW for each of the 4K bytes and
the FICON Express2 channel processor works on each CCW separately, the processor gets
pushed to high utilization levels for this workload. With this benchmark, CONNECT time
starts out at a little over 2ms at 10% FICON Processor utilization (FPU) and increases to over
3ms at 60% FPU, when just under 120MB/sec are being transferred, which represents about
60% link utilization as well.

FICON Express2 Channel Performance Version 1.0
Page 17
response time components for 32x4K read hits...
128K bytes/io, 100% cu cache hits
FICON Processor Utilization
10% FPU
20% FPU
30% FPU
40% FPU
50% FPU
60% FPU
0 1 2 3 4
response time in ms
Figure 17
pend
conn
The following table summarizes info from both the RMF Channel Activity and RMF FICON
Director Activity reports at high levels of utilization for these benchmarks measurements:
Channel
program
4K byte read
hits
32x4K byte read
hits
27K byte read
hits
6x27K byte read
hits
FICON
Express2
channel
processor
utilization
91%
83%
62%
39%
FICON
Express2
channel link
utilization
26%
71%
84%
100%
READ MB/sec
52 MB/se
142 MB/sec
168 MB/sec
200 MB/sec
Average
FRAME size in
bytes
843
1,334
1,766
1,967

FICON Express2 Channel Performance Version 1.0
Page 18
The average frame size info from the RMF FICON Director Activity report can be used to
determine if a workload is more likely to be processor or link limited. The following general
rule-of-thumb can be applied:
If the average frame size is less than 1000 bytes, then the workload is most likely
processor limited.
If the average frame size is greater than 1500 bytes, then the workload is most likely
bus or link limited.
For workloads with average frame sizes greater than 1000 bytes and less than 1500
bytes, then both channel processor and bus/link utilizations should be monitored.
More information about how to find these fields on the RMF Channel Activity and
FICON Director Activity reports for your production workload can be found in the
FICON RMF Information section of this paper.
In summary, the performance results of 4 different DASD I/O driver benchmarks run on
FICON Express2 channels were presented here. Response times and utilizations of the most
pertinent channel resources for each benchmark were explained. For all of these
benchmarks, results are significantly better than previous generations of FICON, FICON
Express and especially ESCON channels.

FICON Express2 Channel Performance Version 1.0
Page 19
FICON Express2 CTC performance
For Channel-to-Channel (CTC) applications, the previous generation FICON Express channel
was better than ESCON for all large block transfers. But for customers using CTC as a
transport mechanism for small (1K bytes or less) XCF messages, ESCON CTC previously had
the best response times at low activity rates. Now, as depicted in Figure 18, the new FICON
Express2 CTC response times for short (1K bytes or less) XCF messages are 25 to 35% better
than FICON Express and ESCON CTC response times at low activity rates. Furthermore,
signals per second throughput rates at the 400 usec response time level for short messages
across FICON Express2 CTC are 1.5 to 3 times better than FICON Express CTC and ESCON
CTC link capabilities.
FICON Express2(FEx2) CTC response
times with short XCF messages...
better than ESCON!
i/o resp time in microsec
1000
800
600
400
200
0
0 5 10 15
signals per second
Thousands
Figure 18
ESCON
FEx
FEx2

FICON Express2 Channel Performance Version 1.0
Page 20
FICON Express2 Card level performance
FICON Express2 Channel card configurations...
4 channels per card
4 cards or up to 16 channels per esti
z-series
Memory
MBA
chip
esti
link
esti-M
card
Figure 19
sti
link
sti
link
sti
link
sti
link
FICON Express2
channel card
FICON Express2
channel card
FICON Express2
channel card
FICON Express2
channel card
The 4 FICON Express2 channels on the same physical card all connect to a single 1 GB/sec
STI link. Since 4 times the max capability of a single FICON Express2 channel exceeds 1
GB/sec, measurements were done to determine the max capability of the 1 GB/sec STI link.
In Figure 20, these measurement results are compared to the previous generation FICON
Express channel card which had 2 channels per card connected to a 333 MB/sec STI link.
For the FICON Express2 channel card, a max of 644 READ MB/sec, 651 WRITE MB/sec
and 970 READ+WRITE MB/sec was measured which represents a 2.5 to 3.5 times
improvement compared to the previous generation FICON Express channel card.

FICON Express2 Channel Performance Version 1.0
Page 21
z990 FICON Express2 vs FICON Express
card level MB/sec comparison
MB/sec
1200
1000
800
600
400
200
0
276
265
178
z990 FEx reads
FEx writes
FEx r/w mix
644 651
970
FEx2 reads
FEx2 writes
FEx2 r/w mix
Figure 20
4 FICON Express2
channels per card
with a 1GB/sec STI
vs 2 FICON Express
channels per card
with a 333MB/sec STI
--> 2.5x to 3.5x
improvement at card
level
As depicted in Figure 19, the 4 FICON Express2 channel cards can be connected via an
ESTI-M card to a single 2 GB/sec ESTI link. Since 4 times the max capability of a single
FICON Express2 channel card exceeds 2 GB/sec, measurements were done to determine the
maximum capability of a single 2GB/sec ESTI link. As shown in Figure 21, a maximum of
1551 READ MB/sec, 1587 WRITE MB/sec and 1843 READ + WRITE MB/sec was measured
in this configuration using a 6x27K channel program.

FICON Express2 Channel Performance Version 1.0
Page 22
z990 FICON Express2 single card vs
4 cards per 2GB/sec sti domain
MB/sec
2000
1500
1000
500
0
644 651
single card reads
single card writes
single card r/w mix
970
1587
1551
1843
sti domain reads
sti domain writes
sti domain r/w mix
Figure 21
4 FICON Express2
cards per 2GB/sec STI
domain-->
78% of sti speed for
reads
79% of sti speed for
writes
92% of sti speed for
read/write mix
It has been my experience that the only time customers come close to pushing this level of
I/O bandwidth in a single I/O domain is when running I/O driver benchmarks in a test
environment. It is unlikely to see it in a real customer production environment for several
reasons. Normally the configurator will spread different types of channel cards across multiple
ESTI-M cards so that any one ESTI-M card might have a mixture of 1 ESCON card, 1 FICON
Express channel card and 1 or 2 FICON Express2 channel cards. For many years, we have
recommended that when configuring up to 8 channel paths per LCU that the individual
channels in the path group be selected from different physical channel cards. In this way if a
particular LCU is running a high I/O bandwidth application the MB/sec load will be spread
across multiple channel cards, STI links, MBA chips and books. Furthermore, in any
particular time interval “hot spots” of activity tend to be limited to small groups of channels.
In any case, the maximum ESTI-M card capability is presented here for your awareness. To
determine if you are one of the vast majority of customers that has NO reason to be
concerned about this, you can take the following approach:

FICON Express2 Channel Performance Version 1.0
Page 23
1. As a first pass, you can simply add up the READ and WRITE MB/sec from the RMF
Channel Activity report for all of the channels configured on your system. If that sum is
less than 1.5 GB/sec, you are done.
2. If not, then select the 16 channels with the highest MB/sec and add those up. If that sum
is less than 1.5 GB/sec, you are done.
3. If not, then you need to more carefully determine which channels are plugged in to which
ESTI-M card (the PCHID report can help you do this) and add up the READ + WRITE
MB/sec for those channels.
Again, with the exception of those customers running high I/O bandwidth benchmark tests,
most customers will be able to stop at step 1.

FICON Express2 Channel Performance Version 1.0
Page 24
FICON RMF Information
This section of the white paper will explain the I/O performance information available on the
following RMF reports:
1. Channel Path Activity report
2. Device Activity report
3. FICON Director Activity report
4. I/O Queuing Activity report
The primary RMF report of interest for FICON is the Channel Path Activity report. Figure 22
is an excerpt from this report.
C H A N N E L P A T H A C T I V I T Y
MODE: LPAR CPMF: EXTENDED MODE CSSID: 0
CHANNEL PATH UTILIZATION(%) READ(MB/SEC) WRITE(MB/SEC)
ID TYPE G SHR PART TOTAL BUS PART TOTAL PART TOTAL
95 FC_S 4 Y 61.11 61.11 32.56 119.34 119.34 0.00 0.00
Figure 22
FICON channels can be identified from the TYPE column; their type begins with FC:
type FC indicates a native FICON channel;
type FC_S indicates a native FICON channel connected to a switch or director
type FCV indicates a FICON bridge channel which connects to an ESCON control unit via a
bridge card in a 9032 model 5 ESCON director. FICON Express2 channels do not support
FCV mode.
The ID column is the Channel Path ID or CHPID number. CHPID 95 is displayed in Figure
22.

FICON Express2 Channel Performance Version 1.0
Page 25
The Generation (G) field tells you a combination of which generation FICON channel is
being used and the speed of the fibre channel link for this CHPID at the time the machine
was IPL’d. A “4” appears in the G field for CHPID 95 in Figure 22. This means that this
channel is a FICON Express2 channel with a link speed of 2 Gbps. If this channel was
connected to a 1 Gbps director, then there would be a “3” in the G field. A “2” indicates a
FICON Express channel with a link speed of 2 Gbps and a “1” indicates a FICON Express
channel operating at 1Gbps.
For a given FICON channel there are three possible entries under UTILIZATION (%):
1. PART denotes the FICON processor utilization due to this logical partition.
2. TOTAL denotes the FICON processor utilization for the sum of all the LPARs.
3. BUS denotes the FICON PCI bus utilization for the sum of all the LPARs.
The FICON processor is busy for channel program processing, which includes the processing
of each individual channel command word (CCW) in the channel program and some setup
activity at the beginning of the channel program and cleanup at the end. A very precise
algorithm is used for calculating zSeries FICON Express and FICON Express2 channel
utilizations. This algorithm is based on monitoring the amount of time the channel processor
spends doing various separate functions, and the results of this algorithm give a much more
accurate measure of FICON processor busy time than the original algorithm based on
counting command and data sequences, which is still used for 9672 G5/G6 FICON channels.
The FICON bus is busy for the actual transfer of command and data frames from the FICON
channel chip to the fibre channel adapter chip, which is connected via the fibre channel link
to the director or control unit. For FICON and FICON Express channels, the FICON bus is
also busy when the FICON processor is polling for work to do. This is why one can see
anywhere from 5 to 15% FICON bus utilization on the RMF Channel Activity report during
time intervals when there are no I/Os active on those channels. The new FICON Express2
channels, however, no longer use the bus for polling and therefore the bus utilization should
be less than 1% for these channels when there are no I/O’s active for an entire RMF
reporting interval.
The actual FC channel processor and bus utilizations as reported by RMF will vary by
workload and by channel type. As shown in Figure 22 above, FICON Express2 channels
provide bandwidth information (MB/SEC) not available for ESCON channels. This is
provided separately for READs and WRITEs since the fibre channel link is full duplex, at
both the logical partition level (PART) and the entire system level (TOTAL). Fibre channel
link utilizations are not directly reported on RMF but can be easily calculated by dividing the

FICON Express2 Channel Performance Version 1.0
Page 26
READ or WRITE MB/sec by the link capacity. Several examples of FICON Express2 channel
processor, bus and link utilizations based on I/O driver benchmark measurements are
displayed in Figures 5 through 17 of this paper.
With FICON Express2 channels, customers should continue to analyze their I/O activity by
looking at the DASD or TAPE activity reports, just as they did with FICON, FICON Express
and ESCON channels. An example of a Direct Access Device Activity report is shown in
Figure 23.
device activity report...response times...
benefit of PAVs...
D I R E C T A C C E S S D E V I C E A C T
z/OS V1R6 SYSTEM ID xxxx DATE 01/24/2005
RPT VERSION V1R5 RMF TIME 11.09.28
DEVICE AVG AVG AVG AVG AVG AVG AVG
DEV DEVICE VOLUME PAV LCU ACTIVITY RESP IOSQ CMR DB PEND DISC CONN
NUM TYPE SERIAL RATE TIME TIME DLY DLY TIME TIME TIME
4612 33903 DS3B02 1 0037 54.736 2.2 1.2 0.0 0.0 0.2 0.4 0.4
4613 33903 DS3B03 1 0037 48.996 8.7 5.3 0.0 0.0 0.2 1.8 1.4
4616 33903 DS3B06 1 0037 15.196 8.0 2.5 0.0 0.0 0.2 3.1 2.2
4617 33903 DS3B07 1 0037 20.761 9.7 3.6 0.0 0.0 0.2 3.3 2.6
461C 33903 DS3B0C 1 0037 17.189 13.6 6.6 0.0 0.0 0.2 3.8 2.9
461E 33903 DS3B0E 1 0037 41.288 9.0 4.9 0.0 0.0 0.2 2.3 1.7
LCU 0037 1196.01 3.5 1.7 0.0 0.0 0.2 0.9 0.9
4612 33903 DS3B02 4 0037 55.669 0.5 0.0 0.0 0.0 0.2 0.1 0.2
4613 33903 DS3B03 4 0037 50.145 1.8 0.0 0.0 0.0 0.2 0.8 0.8
4616 33903 DS3B06 4 0037 13.828 8.2 0.0 0.0 0.0 0.2 4.3 3.7
4617 33903 DS3B07 4 0037 20.348 6.4 0.0 0.0 0.0 0.2 3.3 2.9
461C 33903 DS3B0C 4 0037 16.929 8.0 0.0 0.0 0.0 0.2 4.2 3.6
461E 33903 DS3B0E 4 0037 41.106 3.4 0.0 0.0 0.0 0.2 1.7 1.5
LCU 0037 1226.54 1.7 0.0 0.0 0.0 0.2 0.7 0.8
Figure 23
Here one can examine the AVG RESP TIME and various response time components (IOSQ,
PEND, DISC and CONN times) for activity to the LCUs attached to the FICON Express2
channels. If response time is a problem, then the response time components need to be
looked at. If disconnect time is a problem, then an increase in CU cache size might help. If
IOSQ time is a problem, then Parallel Access Volumes might help. Figure 23 shows an
example of the reduction in IOSQ time experienced on an IMS benchmark measurement
when 4 PAVs were defined vs. 1. In this particular case, IOSQ time improved from an
average of 1.7ms to 0ms for this LCU. If PEND or CONNECT times are too high, then one

FICON Express2 Channel Performance Version 1.0
Page 27
can look at the FICON processor, bus and link utilizations. If any one of these utilizations is
above 50% then overuse of the FICON channel could be contributing to additional PEND
and CONNECT time delays. If, on the other hand, PEND and CONNECT times are high and
FICON channel utilizations are less than 50%, then overuse of a FICON director port or
control unit port could be contributing factors. If FICON channels from multiple CECs are
connected to the same director destination port, then one must add up the activity from all
the CECs to determine the total destination port activity. This total activity level should be
less than the “knee of the curve” points depicted in the measurement results that appear in
the white papers for the specific native FICON DASD or TAPE product that is being used.
One of the basic differences between native FICON and ESCON channel performance is the
CONNECT time component of response time. Since an ESCON channel is only capable of
executing one I/O at a time, the amount of time that it takes to execute the protocol + data
transfer components of CONNECT time is relatively constant from one I/O operation to the
next with the same exact channel program. With FICON however, CONNECT time can vary
from one execution of a channel program to another. This is a side effect of the multiplexing
capability of FICON. Since both the channel and the control unit can be concurrently
executing multiple I/O operations, the individual data transfer frames of one I/O operation
might get queued up behind the data transfer frames of another I/O operation. So, the
CONNECT time of an I/O with FICON is dependent upon the number of I/O operations that
are concurrently active on the same FICON channel, link and control unit connection.
Multiplexing also means that the start and end of the CONNECT time for one native FICON
I/O operation can overlap the start and end of the CONNECT time for several other native
FICON I/O operations. But AVG CONN TIME for large block size transfers should be
significantly less for native FICON channels than for the same transfer size on ESCON or
FICON Bridge channels due to the much faster (2 Gbps or 200 MB/sec) link transfer speeds
of native FICON vs. the 20 MB/sec link transfer speed of ESCON. Several examples of
CONNECT times at various levels of FICON Express2 channel processor, bus and link
utilizations are shown in the FICON Express2 benchmark measurement results displayed in
Figures 5 through 17 of this paper.
Little’s Law can be used to estimate the average number of open exchanges or simultaneously
active I/O’s or multiplexing level for both a FICON channel and a control unit port for a
given RMF interval. This formula is essentially a variation of the formula for calculating I/O
intensity levels which has been used for years to identify “hot spots” in an I/O configuration.
I/O intensity levels are calculated by multiplying total response times by activity rates. The
number of I/O’s that are simultaneously active and transferring data between the channel and
the control unit can be determined by multiplying the CONNECT time component of
response time (in units of seconds or milliseconds(ms) times 0.001) by the activity rate (in

FICON Express2 Channel Performance Version 1.0
Page 28
units of I/Os per second). If there is only 1 LCU (logical control unit) connected to a single
set of FICON channels, then the average number of open exchanges can be calculated by
multiplying the activity rate for that LCU by the sum of the “CMR + CONN + DISC” times
for that LCU divided by the number of channels in the path group for that LCU. If there are
multiple LCUs connected to a set of FICON channels, then the results of this calculation
needs to be summed for all these LCUs. Similarly, to determine the average number of
exchanges for a given physical CU port if there are multiple sets of channels from multiple
LPARs on multiple CECs connected to the same set of CU ports, this calculation needs to be
done for each LCU for each LPAR and then summed to get the total for the CU port.
In any case, if the result of this calculation is a higher than normal value for your workload,
then one must look at each of the components of the formula to determine the cause of the
high number of open exchanges. AVG CMR DLY or “command response” delay time is a new
field that has been added to the RMF Device Activity report for FICON. An example of this
is displayed in Figure 23 above. AVG CMR DLY time is a subset of PEND time. As shown in
Figure 24, when a channel opens a new exchange with a control unit by sending the first
command in the channel program to the control unit, the control unit responds with a CMR.
Architecturally, the official end to PEND time (for both FICON and ESCON) is designated by
the time when the channel receives the CMR signal from the control unit.
FICON Command/Data Transfer
CCW=Channel Control Word CE=Channel End DE=Device End
CMR = Command Response
FICON
Express2
Channel
total pend time
ssch
CCW1
CCW2
CCW3
CE/DE
Control Unit
cmr time...subset of pend
CCW1
CMR
Figure 24
CMR
cmd
End
cmd
End
cp ---> sap ---> channel ---> cu port ---> channel
Device
CMR time begins when exchange begins & ends when pend time ends

FICON Express2 Channel Performance Version 1.0
Page 29
If the control unit is excessively busy with other I/O operations or exchanges that are already
active, then this will be reflected in larger than normal AVG CMR DLY times. If DISC time is
high, then the cause of a high number of average open exchanges could be low control unit
cache hit ratios or contention in other internal resources of the control unit involved in
reading or writing data from disk. Synchronous copying of data from primary DASD to
secondary DASD located many kilometers away can also cause high DISCONNECT times.
If CONN time is high then the cause of a high number of open exchanges could be either
high channel utilizations or high control unit port utilizations or director port contention or
long distances between the channel and the control unit or large data transfers or the nature
of the particular channel programs being executed. Channel (processor and bus) utilizations
can be found on the RMF Channel Activity report. Unfortunately, control unit port
utilizations are not reported directly on any RMF report. However, some information about
FICON director ports that are connected to either control unit ports or channels can be found
on the RMF FICON Director Activity report. An example of this report is shown in Figure
25.
RMF FICON Director Activity report
F I C O N D I R E C T O R A C T I V I T Y
z/OS V1R6 SYSTEM ID S08 DATE 12/01/2004
RPT VERSION V1R5 RMF TIME 16.18.00
IODF = 4C NO CREATION INFORMATION AVAILABLE ACT: POR
SWITCH DEVICE: 00C2 SWITCH ID: ** TYPE: 006140 MODEL: 001 MAN: MCD
PORT -CONNECTION- AVG FRAME AVG FRAME SIZE PORT BANDWIDTH (MB/SEC)
ADDR UNIT ID PACING READ WRITE -- READ -- -- WRITE --
note: channel program = 32x4K read
49 CHP-H 95 0 70 1334 2.19 125.21
7A CU ---- 0 1334 70 39.70 0.70
7B CU ---- 0 1334 70 41.65 0.73
83 CU BF00 0 1334 70 41.71 0.73
compare MB/sec at CU port to max CU port capability to approximate CU
port utilization and compare to CU link max MB/sec based on link speed
(100MB/sec for 1Gbps or 200MB/sec for 2Gbps links) to get CU link
utilization
Figure 25

FICON Express2 Channel Performance Version 1.0
Page 30
The first column “PORT ADDR” identifies the switch port address. The 2nd and 3rd
“CONNECTION” columns identify what this switch port is connected to. The “UNIT”
indicates whether is it a channel (CHP-H), a control unit port (CU) or in the case where two
directors are cascaded, another switch port (SWITCH). The “ID” in column 3 is the CHPID
number for the channel or the control unit address for the CU. The values in the “AVG
FRAME PACING” column will be zero most of the time. This column is intended to display
the amount of time that a frame is delayed when there are no more buffer credits available.
The “AVG FRAME SIZE” columns display the average number of bytes per frame being
“READ” into that director port or written out from that director port. These columns can be
used to help understand if your workload is a processor or bus/link limited workload. The
maximum frame size is 2K bytes. If your workload is transferring a small amount of data
using small block sizes, such as the 4K bytes per I/O typically found in online transaction
processing, then the average frame size will most likely be less than 1000 bytes and your
workload will most likely be channel processor or control unit port processor limited. On the
other hand, if your workload transfers a lot of data using large block sizes, then the average
frame size will most likely be in the 1500 to 2000 byte range and your workload will most
likely be channel or control unit bus or link limited. Figure 25 is an example of a workload
that is in between these two extremes and has an average frame size of 1334 bytes. In this
case, both processor and bus/link utilizations should be monitored.
The last two columns on this report, the “PORT BANDWIDTH (MB/SEC)” “READ” and
“WRITE” columns contain the MB/sec that are being “READ” into that director port or
written out from that director port. Please note that for an RMF interval where 10 MB/sec of
data is being “READ” from a device on a control unit that the 10 MB/sec value will appear
on the line for the director port connected to the control unit in the “READ” column but in
the “WRITE” column for the director port connected to the channel in the RMF FICON
Director Activity Report and in the “READ(MB/SEC)” column of the channel in the RMF
Channel Activity Report. The “READs” and “WRITEs” on the FICON Director Activity
report are from the perspective of the port, whereas the “READs” and “WRITEs” on the
Channel Activity report are from the perspective of the higher level application. Figure 25 is
an example of a benchmark measurement where about 40 MB/sec was “READ” from each of
3 different control unit ports and over 120 MB/sec was written to a single channel, CHPID
#95.
To convert control unit port MB/sec data into control unit port utilizations, you also need to
know what the maximum capability of the control unit port is for both small and large block
sizes and whether your workload is a small or large block size workload. If a control unit
vendor tells you or you run your own test to determine that the maximum capability of a
single port on their box for 4k byte READs is 5000 I/Os per second, then this is the same as

FICON Express2 Channel Performance Version 1.0
Page 31
seeing 20MB/sec in the READ MB/sec column and less than 1000 bytes in the AVG READ
FRAME SIZE column for the CU port line on the RMF FICON Director Activity report. If
your workload is reporting more than 10 READ MB/sec with an AVG READ FRAME SIZE
less than 1000 bytes, then your workload is driving this CU port to greater than 50%
utilization. Similarly, if a control unit vendor tells you or you run your own test to determine
that the maximum capability of a single port on their box for half-track or 27K byte READs is
about 2500 I/Os per second or about 70 MB/sec, then this is the same as seeing 70 MB/sec
in the READ MB/sec column and greater than 1500 bytes in the AVG READ FRAME SIZE
column for the CU port line on the RMF FICON Director Activity report. If your workload is
reporting more than 35 READ MB/sec with an AVG READ FRAME SIZE greater than 1500
bytes, then your workload is driving this CU port to greater than 50% utilization. Driving a
CU port to greater than 50% utilization could be the cause of higher than normal CONN
times which could result in higher than normal average open exchanges for that CU port or
for any of the channels connected to that CU port.
If you were to ask the question, “what is an appropriate value for average open exchanges in
an RMF interval for my workload?”, the answer, of course, would be “it depends on the
characteristics of the workload”. The following example should illustrate this point.
ACTIVITY
5,634.1
5,634.1
5,634.1
5,634.1
5,634.1
5,634.1
5,634.1
RESP
1.8
2.7
3.7
4.7
5.7
6.7
7.7
PEND
0.3
0.3
0.3
0.3
0.3
0.3
0.3
CMR
0.2
0.2
0.2
0.2
0.2
0.2
0.2
DISC
1.2
2
3
4
5
6
7
CONN
0.4
0.4
0.4
0.4
0.4
0.4
0.4
OPEX
2.6
3.7
5.1
6.5
7.9
9.3
10.7
CU H/R
88%
80%
70%
60%
50%
40%
30%
The first row of this table is taken from the RMF reports for a 15 minute interval of the LSPR
OLTP-T workload measurement.
ACTIVITY = I/Os per second rate.
RESP = total response time for each I/O in ms.
PEND = pend time.
CMR = command response time, which is a subset of PEND time.
DISC = disconnect time.
CONN = connect time.
OPEX = average number of open exchanges per channel. In this configuration, there were 4
channels per LCU.
CU H/R = control unit cache hit ratio.

FICON Express2 Channel Performance Version 1.0
Page 32
This workload has a control unit cache hit ratio of 88% and a disconnect time of 1.2ms
which implies that it takes an average of about 10ms to resolve each CU cache miss. The rest
of the rows in the above table illustrate what the response time and average open exchanges
per channel would be if instead of a CU H/R of 88%, this workload had an 80%, 70%, 60%,
50%, 40% or 30% control unit cache hit ratio. For each 10% drop in CU H/R, disconnect
time and total response times increase by 1ms. Average open exchanges per channel increase
from 2.6 with a CU H/R of 88% to 10.7 with a CU H/R of 30%. So, if the nature of your
workload is such that it has a poor CU cache hit ratio, then it is acceptable to have higher
average open exchange values for this workload compared to a workload with much better
CU cache hit ratios. Furthermore, adding additional channel paths to a workload with poor
CU cache hit ratios is not the appropriate action to take. For this workload the channels are
only 18% busy. To improve the performance of a workload with high disconnect times,
attention needs to be paid to actions that will either improve the CU cache hit ratio or reduce
the amount of time that it takes to resolve each CU cache miss.
This is just one example of how values for average open exchanges can vary based on
workload characteristics. In general, an acceptable average open exchange value should be
determined for each workload based on experiences of when bottom line workload
performance is acceptable or not.
With ESCON, the additional queuing delays caused by having multiple I/Os concurrently
active appear in the PEND or DISC time component of response time. If the same workload
with the same activity rate and the same level of I/O concurrency is run on native FICON
channels instead of ESCON channels, then one could see the PEND and DISC time
components of response time decrease and the CONNECT time component increase for small
data transfer sizes. For large data transfers, the improved CONNECT time due to the 100
MB/sec or 200 MB/sec link transfer speed will most likely offset any increased CONNECT
time due to multiplexing queuing delays. Figure 26 illustrates the type of improvement in
CONNECT time experienced on the z900 FICON and for FICON Express as compared with
ESCON. The exact CONNECT time will, of course, vary depending on the details of the I/O
configuration (type of storage system, number of devices, workload intensity, etc.). Figures 5
through 17 of this paper show several examples of CONNECT times at various utilization
levels for the new z990 FICON Express2 channels.

FICON Express2 Channel Performance Version 1.0
Page 33
Connect times in ms
Sample FICON vs ESCON connect
times for large data transfer sizes
12
10
8
6
4
2
0
Figure 26
In addition to the RMF Channel Activity, Device Activity and FICON Director Activity
reports, the RMF I/O Queuing Activity report also provides information about your I/O
configuration. Starting with z/OS V1R2 and RMF Release 12, several new fields were added
to the I/O Queuing Activity report. Figures 27, 28 and 29 are examples of excerpts from this
report. The “Initiative Queue” section of the report is the same as it has been for several
years. The “IOP UTILIZATION” and the “RETRIES/SSCH” sections were added with z/OS
V1R2. The “% IOP BUSY” column is the SAP utilization. The “I/O START RATE” column
is the number of SSCHs per second sent from a CP to a particular SAP. The “INTERRUPT
RATE” column is the number of I/O interrupts per second processed by each SAP. In
general, if the channel programs being executed do not have the PCI (Programmed
Controlled Interrupt) flag set, the total number of interrupts per second processed will be
equal to the total number of SSCHs per second processed. The “RETRIES/SSCH” section
indicates the average number of times per SSCH that the SAP encountered a busy signal in
the process of doing its path selection work for this I/O operation. There are four types of
busies reported:
1. CP busy = channel path busy,
2. DP busy = director port busy
3. CU busy = control unit port busy
4. DV busy = device busy
ESCON 27K
FICON 27K
FICON Express 27K
ESCON 6x27K
FICON 6x27K
FICON Express 6x27K

FICON Express2 Channel Performance Version 1.0
Page 34
Each time a SAP encounters a busy and has to retry another path for a SSCH, additional SAP
cycles are consumed and the %IOP busy or SAP utilization will increase. One of the benefits
of native FICON is that it makes SAPs or IOPs more productive due to the reduction in
busies or RETRIES/SSCH. For the same activity rate, one should see less IOP utilization %
busy with native FICON, FICON Express and FICON Express2 channels than with ESCON
channels. One must be careful not to misinterpret IOP utilization %’s however. High IOP
utilization %’s are usually an indicator of contention especially with ESCON channels,
directors and control units. Adding additional IOPs will NOT help reduce channel
configuration contention. One must identify the source of the configuration contention and
fix it. Migrating from ESCON to native FICON configurations is a natural solution to this
problem. Figures 27 and 28 represent a dramatic example of this. Figure 27 is from a z900
ESCON configuration with a lot of contention. Specifically, for the time interval reported,
there were a total of 4.73 retries per SSCH. 4.19 of these were channel path busies. This
means that when the SAP tried to start a new I/O operation on an ESCON channel, that
channel was already busy processing another I/O and the SAP had to try to find another
ESCON channel path that was available for this I/O and on the average it did this 4.19 times
per SSCH. This means that either the ESCON channels were operating at high utilizations or
there were not enough paths per LCU defined to handle the number of I/O operations that
were being issued simultaneously to the total number of LCU’s that shared the same set of
ESCON paths. This can happen when there is a burst of activity during a subset of the total
RMF interval, e.g. for a few minutes out of a 30 minute or longer interval. There was also an
average of 0.54 director port busies per SSCH during this time interval. This means that 2 or
more ESCON channel paths most likely from multiple CECs in the same sysplex were trying
to connect to the same director port and control unit port at the same time and with ESCON
only 1 I/O operation to a given director port or CU port can be active at once. The other
I/O’s that attempt to use the same destination port will get DP busy signals. The rules of
thumb available for these statistics are:
1. keep SAP utilization or %IOP BUSY below 70%,
2. AVG Q LNGTH should be less than 1 and
3. Total RETRIES/SSCH should be less than 2 with the sum of DP, CU and DV busies per
SSCH less than 1.

FICON Express2 Channel Performance Version 1.0
Page 35
I/O Q U E U I N G A C T I V I T Y
RPT VERSION V1R2 RMF
Figure 27
from a z900 ESCON
configuration with a lot of
contention
- INITIATIVE QUEUE - ------- IOP UTILIZATION -------
IOP ACTIVITY AVG Q % IOP I/O START INTERRUPT
RATE LNGTH BUSY RATE RATE
00 2745.205 0.77 68.02 2745.181 3684.715
01 3236.994 0.11 53.70 3236.990 3566.626
02 3067.562 0.82 73.73 3067.292 3262.451
SYS 9049.758 0.55 65.15 9049.461 10513.79
IOP
00
01
02
SYS
-------- RETRIES / SSCH ---------
CP DP CU DV
ALL BUSY BUSY BUSY BUSY
4.80 4.17 0.62 0.00 0.00
2.92 2.60 0.31 0.00 0.00
6.58 5.88 0.69 0.00 0.00
4.73 4.19 0.54 0.00 0.00
rules of thumb:
avg q lngth < 1,
%IOP busy < 70%,
retries/ssch < 1 or 2
Figure 28 shows the dramatic improvements in effective SAP capacity after the migration
from the z900 with all ESCON channels to the z990 with most of the I/O activity occurring
on FICON channels. The number of RETRIES/SSCH went from 4.73 to 0.21 and average
SAP utilizations dropped from over 65% to under 20%, resulting in a 3x improvement in
effective SAP capacity. Improvements like this are not typical, however, and would be much
less dramatic if the original ESCON configuration was much better tuned to the point where
RETRIES/SSCH was less than 1.

FICON Express2 Channel Performance Version 1.0
Page 36
- - INITIATIVE QUEUE - ------- IOP UTILIZATION -------
IOP ACTIVITY AVG Q % IOP I/O START INTERRUPT
RATE LNGTH BUSY RATE RATE
00 3424.947 0.01 13.53 3424.922 3374.204
01 1969.652 0.00 5.02 1969.651 1921.234
02 401.022 0.00 2.75 400.995 591.365
03 4950.215 0.02 35.58 4950.211 5147.980
SYS 10745.84 0.01 14.22 10745.78 11034.79
significant
reduction in
retries...3x
improvement
in effective
Sap capacity
Figure 28
after migration to z990
and FICON (+ESCON)
-------- RETRIES / SSCH ---------
CP DP CU DV
ALL BUSY BUSY BUSY BUSY
0.15 0.14 0.00 0.01 0.00
0.25 0.24 0.00 0.01 0.00
0.23 0.22 0.00 0.01 0.00
0.24 0.16 0.07 0.01 0.00
0.21 0.17 0.03 0.01 0.00
Figures 27 and 28 show the average RETRIES/SSCH at the overall I/O configuration level.
To identify which part of the overall I/O configuration is experiencing contention, one needs
to look at the LCU section of the I/O Queuing activity report. An example of this is displayed
in Figure 29. The first column is the LCU id and the 2nd column is the CU id. In the 3rd
column is a list of the channel path ids for this LCU. Up to 8 channel paths can be defined
per LCU. In Figure 29, 6 channel paths are defined for LCU 0222. The “CHPID TAKEN”
column is the equivalent of an activity rate. It is the number of SSCHs per second that were
executed on the channel paths defined for this LCU. The %DP BUSY column is the % of
times that the SAP encountered a busy signal at an ESCON director port when attempting to
select this path for a new SSCH. %DP BUSY will be 0 for native FICON due to the
elimination of destination port busy signals with native FICON packet-switched directors.
%CU BUSY should also be 0 for native FICON in most customer production environments.
CU busies will only occur with native FICON when an individual CU port is being overloaded
with work from many different FICON channels simultaneously. The high % CU BUSY (15%
for path 03 & 14% for path 06) in Figure 29 is an example of FICON CU port contention.
Further evidence of this contention is the high AVG CMR DLY times for these channel paths
and the low CHPID TAKEN values or activity rates for channel paths 03 & 06 in comparison
to the other channel paths defined for this LCU. The AVG CMR DLY of 203ms for channel
path 03 and 207ms for channel path 06 indicates that the CU ports that are connected to
these channel paths are taking a very long time to respond to the new SSCH work that the
channel is trying to send to them. In contrast, the CU ports that are connected to channel

FICON Express2 Channel Performance Version 1.0
Page 37
paths 01, 02, 04 and 05 are responding on average in about 0.5ms. In this case, if no errors
were made in the IOCDS, then some “tuning” of the configuration is necessary to reduce
these CU busies and achieve better response time results. Contention due to CU busies
results in higher than normal PEND times and contributes to a higher than normal average
number of open exchanges for this LCU. The solution to this is to identify the source of
contention at the CU ports connected to channel paths 03 and 06 in this example and fix it.
I/O Q U E U I N G A C T I V I T Y from a configuration
with FICON CU port
contention causing
high pend times
LCU CONTROL UNITS
0222 1000
AVG AVG
CHAN CHPID % DP % CU CUB CMR
PATHS TAKEN BUSY BUSY DLY DLY
01 89.256 0.00 0.00 0.0 0.5
02 86.348 0.00 0.00 0.0 0.5
03 1.908 0.00 15.08 0.0 203
04 89.644 0.00 0.00 0.0 0.5
05 86.055 0.00 0.00 0.0 0.5
06 2.132 0.00 13.95 0.0 207
* 355.34 0.00 0.19 0.0 2.9
note: CU busies and high CMR delay times is NOT
normal for native FICON...indicates CU port contention
Figure 29
For FICON channels it is also possible to estimate the average number of bytes transferred
per SSCH by dividing the MB/sec of a FICON channel from the Channel Path Activity report
by the total SSCH/sec processed by a FICON channel from the I/O Queuing Activity report.
The total SSCH/sec processed by a FICON channel can be determined by adding up all of the
“CHPID taken” fields on the I/O Queuing Activity report for each LCU that a single FICON
channel is connected to. If the average data transfer sizes of your channel programs are
greater than 27K bytes, then your workload is most likely pushing the channel and CU port
buses and links to higher levels of utilization than other resources and you should focus on
the MB/sec fields on your RMF Channel Activity and FICON Director Activity Reports and
compare these to the maximum capability of the FICON channels, CU ports and links used in
your configuration.
In summary, the basics of performance analysis do not change with a FICON configuration
versus an ESCON configuration. In both environments, an appropriate technique to use is to
first calculate I/O intensities, which equals I/O rate multiplied by response times. This
analysis can be done at a device volume level, an LCU level, a physical CU box level or for a
group of channels. The parts of the total I/O configuration that have the highest I/O

FICON Express2 Channel Performance Version 1.0
Page 38
intensities are the “hot spots” of the configuration. These are the areas where configuration
tuning has the potential for yielding the highest benefit. As explained above, the individual
components of response time (IOSQ, DISC, PEND and CONN) will tell you where you should
focus your efforts. The average open exchange calculation is a subset of the I/O intensity
calculation that uses the DISC + CONN + CMR components of response time. Except in
cases of extremely low control unit cache hit ratios, the open exchange limit is not the cause
of high values of average open exchanges. Instead high values for average open exchanges
are most likely the result of driving either the channels or the control unit to high levels of
utilization. Tuning efforts need to be focused on the appropriate areas based on the DISC,
CONN and CMR components of workload response times. If the FICON channel processor
and bus utilizations as reported on the RMF Channel Activity report and link utilizations
calculated from the MB/sec info are less than 50%, then the tuning efforts need to focus on
the control units in the configuration.
The basic architecture and design differences between FICON and ESCON resulted in many
changes to the performance data that appear on RMF reports. Additional information in the
form of FICON processor and bus utilizations, READ and WRITE MB/sec, AVG FRAME
SIZE and AVG CMR DLY is provided to help analyze the multiplexing capability of FICON.
Since ESCON is only capable of executing one I/O operation at a time, RMF reports the time
that the entire CHPID path is busy for ESCON channel utilization. With FICON, we must
consider the individual components of the total CHPID path such as the FICON channel
processor and bus, the fibre link, the director destination port and the control unit port
adapter microprocessor, bus and link. The charts and examples provided in this paper
should help guide you in assessing the maximum capability of FICON Express2 channels for
your workload.

FICON Express2 Channel Performance Version 1.0
Page 39
Conclusion
The zSeries FICON Express2 channels available on the z990 and z890 offer many benefits
over ESCON and previous generations of FICON channels. The increased throughput and
bandwidth capabilities of these channels offer the opportunity for improved performance with
simpler configurations and reduced infrastructure over longer distances to meet the needs of
future datacenter growth including backup and disaster recovery requirements. The total
native FICON solution – DASD, TAPE and Printer attachments, directors and the new and
improved FICON Express2 channels – are available and ready for your installation.
Additional FICON product information is available on the IBM System Sales Web site and
the zSeries I/O connectivity Web site at
www.ibm.com/servers/eserver/zseries/connectivity/.
Acknowledgements
The data presented in this paper is based upon measurements carried out over several years
using a mixture of IBM internal tools and non-IBM I/O driver programs, specifically Version
13 of the PAI/O Driver for z/OS. I would like to thank all of the reviewers of this paper for
their helpful comments. Special thanks go to Mario Borelli for his continued support on this
effort.

FICON Express2 Channel Performance Version 1.0
Page 40
Copyright IBM Corporation 2005
IBM Corporation
Marketing Communications, Server Group
Route 100
Somers, NY 10589
U.S.A.
Produced in the United States of America
04/05
All Rights Reserved
IBM, IBM eServer, IBM logo, ESCON, FICON, RMF, and zSeries are trademarks or registered trademarks of
International Business Machines Corporation of the United States, other countries or both.
Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc. in the United States,
other countries or both.
Linux is a registered trademark of Linus Torvalds
ON (LOGO) DEMAND BUSINESS is a trademark of International Business Machines Corporation.
PAI/O is a trademark of Performance Associates, Inc.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Intel is a trademark of Intel Corporation in the United States, other countries or both.
Other company, product and service names may be trademarks or service marks of others.
Information concerning non-IBM products was obtained from the suppliers of their products or their published
announcements. Questions on the capabilities of the non-IBM products should be addressed with the suppliers.
IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our
warranty terms apply.
IBM may not offer the products, services or features discussed in this document in other countries, and the
information may be subject to change without notice. Consult your local IBM business contact for information on
the product or services available in your area.
All statements regarding IBM’s future direction and intent are subject to change or withdrawal without notice, and
represent goals and objectives only.
Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard
IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary
depending upon considerations such as the amount of multiprogramming in the user’s job stream, the
I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given
that an individual user will achieve throughput improvements equivalent to the performance ratios
stated here.
GM13-0702-00