The BreakingPoint Resiliency Score - Fortinet
The BreakingPoint Resiliency Score - Fortinet
The BreakingPoint Resiliency Score - Fortinet
Create successful ePaper yourself
Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.
FortiGate-3950B <strong>Score</strong>s 95/100 on <strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong><br />
(Security, Performance, & Stability)<br />
Overview<br />
<strong>Fortinet</strong>® FortiGate®-3950B enterprise consolidated security appliance has<br />
achieved a <strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong> of 95/100 - the highest<br />
published score on record. <strong>The</strong> <strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong> is a test based<br />
on industry standards for performance, security and stability of network and<br />
security devices. <strong>The</strong> <strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong> is presented as a<br />
numeric grade from 1 to 100. Networks and devices may receive no score if<br />
they fail to pass traffic at any point or they degrade to an unacceptable<br />
performance level.<br />
<strong>The</strong> <strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong> establishes standards against which<br />
network and security devices are measured. It provides automated,<br />
standardized and deterministic methods for evaluating and ensuring the<br />
resiliency of networks, network equipment and data centers. <strong>The</strong><br />
<strong>BreakingPoint</strong> Cyber Tomography Machines (CTM) provides a standard<br />
measurement of the performance, security and stability of networks and data<br />
centers using real-world application traffic, real-time security attacks, extreme<br />
user load and application fuzzing.<br />
<strong>The</strong> FortiGate-3950B appliance, running FortiOS 4.0 MR3 and utilizing<br />
its two built-in 10-GbE interfaces was tested. <strong>The</strong> evaluation was<br />
performed using one <strong>BreakingPoint</strong> Storm CTM containing one four-port<br />
10-GbE card.<br />
<strong>The</strong> <strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong> validates the performance of <strong>Fortinet</strong>’s network security solutions using metric-based,<br />
rigorous real-world testing. <strong>Fortinet</strong> customers can deploy products with the utmost confidence, as these standards-based<br />
evaluations enable enterprises to easily determine the right fit for their environments. As demonstrated by the <strong>BreakingPoint</strong><br />
<strong>Resiliency</strong> <strong>Score</strong> testing, the FortiGate-3950B appliance continues to impress with its security capabilities, scalability, flexibility<br />
and stability.<br />
For more information on the <strong>Fortinet</strong> FortiGate-3950B series please visit<br />
http://www.fortinet.com/products/fortigate/3950series.html<br />
FortiGate-3950B: Modular Security Platform<br />
Download the <strong>BreakingPoint</strong> whitepaper, “A Six-Step Plan for Competitive Device Evaluations” here<br />
http://www.breakingpointsystems.com/resources/white-papers/six-step-plan-competitive-device-evaluation-bakeoffs/
Firewall <strong>Resiliency</strong> Report<br />
1. Firewall Report<br />
Product: <strong>Fortinet</strong> FortiGate-3950B<br />
<strong>Resiliency</strong> <strong>Score</strong>: 95<br />
2. Synopsis<br />
Throughput 64: 93.75<br />
1518: 98.43<br />
Sessions Count: 100.00<br />
Rate: 79.94<br />
Rate: 100.00<br />
Robustness IP: pass<br />
UDP: pass<br />
TCP: pass<br />
Lab Real Session Stress Session Rate Stress<br />
Rate: 93.28<br />
Count: 100.00<br />
Security pass pass pass pass<br />
Overall <strong>Score</strong> 95.056404<br />
Throughput<br />
<br />
Sessions<br />
Measures a device’s ability to handle large numbers of TCP sessions and the rate at which it handles them.<br />
Robustness<br />
<br />
Security<br />
<br />
Overall <strong>Score</strong><br />
A blended average of all sub-test. This number represents the over score relative to<br />
expected performance for the resiliency test.
2.1. <strong>Score</strong> Calculation<br />
Overall <strong>Score</strong><br />
Calculation:(A(93) + B(98) + C(93) + D(100) + E(100) + F(79) + G(100)) / 7<br />
Overall <strong>Score</strong> = 95.06<br />
Throughput<br />
A) I EEE Throughput measurement of 64 byte frames A = (100 X Throughput Achieved[w/64]) / Max Wireline<br />
Throughput = (100 X 9375) / 10000 <strong>Score</strong> = 93<br />
B) IEEE Throughput measurement of 1518 byte frames B = (100 X Throughput Achieved[w/1518]) / Max Wireline<br />
Throughput = (100 X 9843) / 10000 <strong>Score</strong> = 98<br />
C) Throughput, Simulated Real World Conditions C = (100 X Application Frames Rate) / Max Application Frames<br />
Rate = (100 X 1399140) / 1500000 <strong>Score</strong> = 93<br />
Sessions<br />
D) Concurrent IETF 793 TCP Connections D = (100 X Number Flows) / Max Number Flows = (100 X 10000000) /<br />
10000000 <strong>Score</strong> = 100<br />
E) Concurrent IETF 2581 TCP and IETF 768 UDP Connections E =(100 X Number Flows) / Max Number Flows =<br />
(100 X 10069955) / 10000000 <strong>Score</strong> = 100<br />
F) IETF 793 TCP Connections/sec F = (100 X Flow Rate / Max Flow Rate = (100 X 119908) / 150000) <strong>Score</strong> = 79<br />
G) IETF 2581 TCP and IETF 768 UDP Connections/sec G = (100 X Flow Rate) / Max Flow Rate = (100 X 121114)<br />
/ 100000 <strong>Score</strong> = 100<br />
Robustness<br />
H) IETF 7 91 IP Stack Stability H = Dropped Pings <strong>Score</strong> = pass<br />
I) IETF 768 UDP Stack Stability I = Dropped Pings <strong>Score</strong> = pass<br />
J) IETF 79 3 TCP Stack Stability J = Dropped Pings <strong>Score</strong> = pass<br />
Security<br />
K) CVE Security F ault Injection, Independent K = Dropped Pings <strong>Score</strong> = pass<br />
L) CVE Security Fault Injection, Benign L = Dropped Pings <strong>Score</strong> = pass<br />
M) CVE Security Fault Injection, Concurrent Sessions Stress M = Dropped Pings <strong>Score</strong> = pass<br />
N) CVE Security Fault Injection, Session Rate Stress N = Dropped Pings <strong>Score</strong> = pass
2.2. Throughput<br />
<br />
for the device.<br />
Network Packet Stress<br />
This test will be repeated once with 64 byte and again with 1518 byte packets. <strong>The</strong>se two packet sizes represent the<br />
smallest and larget valid packet size for a single network frame. <strong>The</strong> test begins by transmitting packets at half of<br />
the theoretical maximum rate for the given packet size. Any dropped or corrupted packets result in a failed iteration.<br />
Testing continues iterating in a binary search pattern;; for each iteration testing at a rate halfway between the last<br />
passed test and the last failed test, until a maximum successful rate is found.<br />
Benign Realistic Network Packets<br />
<br />
bandwidth. <strong>The</strong> test begins by transmitting packets at half of the theoretical maximum rate for the given packet size.<br />
Any dropped or corrupted packets result in a failed iteration. Testing continues iterating in a binary search pattern;;<br />
for each iteration testing at a rate halfway between the last passed test and the last failed test, until a maximum<br />
successful rate is found.<br />
2.3. Sessions<br />
<br />
TCP Sessions Stress<br />
<br />
<br />
test begins by opening a single TCP session. Every 5 seconds, 1500 additional TCP sessions are attempted, up to a<br />
maximum of 10,000,000 concurrent TCP sessions. Once completed, an analysis is made to determine the achieved<br />
concurrent sessions based on the measured number of active concurrent TCP sessions.<br />
Benign Realistic Network Sessions<br />
This test begins by opening a single TCP session, which remains open for the duration of the test. Every 5 seconds<br />
1500 additional TCP sessions are attempted, up to a maximum of 10,000,000 concurrent TCP sessions. Once<br />
completed, an analysis is made to determine the achieved concurrent sessions based on the measured number of<br />
active concurrent TCP sessions.
2.4. Robustness<br />
<br />
IP Robustness<br />
<br />
<br />
Packets will have random payload ranging in size from 46 to 1500 bytes, and will be transmitted at a rate between<br />
2000 and 2500 packets per second. Data will be transmitted for at least one hour, for a minimum of 5,000,000 distinct<br />
<br />
the Urgent pointer, and the IP checksum.<br />
UDP Robustness<br />
<br />
<br />
of the UDP header in addition to the IP header. Packets will have random payload ranging in size from 46 to 1500<br />
bytes, and will be transmitted at a rate between 2000 and 2500 packets per second. Data will be transmitted for at<br />
<br />
TCP Robustness<br />
This test limits the scope of random testing to Layer 4 by randomizing portions of the TCP header in addition to the<br />
IP header. Packets will have random payload ranging in size from 46 to 1500 bytes, and will be transmitted at a<br />
rate between 2000 and 2500 packets per second. Data will be transmitted for at least one hour, for a minimum of<br />
<br />
2.5. Security<br />
<br />
Security - Laboratory conditions<br />
<br />
payloads are blocked or neutered.<br />
Security - Benign Realistic Conditions<br />
<br />
<br />
<br />
Security - Concurrent Sessions Stress Conditions<br />
<br />
<br />
<br />
as measured in the performance baseline.<br />
Security - Session Open Rate Stress Conditions<br />
<br />
<br />
<br />
performance baseline.
2.6. Settings<br />
Setting Value<br />
Speed 10.00 Gigabits<br />
Device Type Firewall<br />
Run Type Full<br />
Session Rate yes<br />
Robustness yes<br />
Throughput yes<br />
Security yes<br />
<br />
Client Routing Server Routing <br />
Network:192.168.50.0/<br />
24 Min:192.168.50.3<br />
Max:192.168.50.254<br />
DUT Address:192.168.50.1<br />
Network:192.168.50.0/ 24<br />
Gateway:192.168.50.2<br />
- End of Document -<br />
DUT Address:192.168.51.1<br />
Network:192.168.51.0/ 24<br />
Gateway:192.168.51.2<br />
Network:10.0.0.0/8 Min:10.0.0.1<br />
Max:10.255.255.254
<strong>The</strong> <strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong><br />
www.breakingpoint.com<br />
© 2005 - 2010. <strong>BreakingPoint</strong> Systems, Inc. All rights reserved. <strong>The</strong> <strong>BreakingPoint</strong> logo is a trademark of <strong>BreakingPoint</strong> Systems, Inc.<br />
All other trademarks are the property of their respective owners.<br />
<strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong><br />
An automated, standardized, and deterministic measure of the performance, security, and stability of network<br />
and application infrastructure devices and systems<br />
1
Table of Contents<br />
www.breakingpoint.com<br />
© 2005 - 2010. <strong>BreakingPoint</strong> Systems, Inc. All rights reserved. <strong>The</strong> <strong>BreakingPoint</strong> logo is a trademark of <strong>BreakingPoint</strong> Systems, Inc.<br />
All other trademarks are the property of their respective owners.<br />
<strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong><br />
Introduction .......................................................................................................................................................................3<br />
<strong>The</strong> <strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong> ................................................................................................................................................................ 3<br />
<strong>Resiliency</strong> <strong>Score</strong> Assessment Elements .......................................................................................................................................................... 3<br />
Prescribed Conguration .................................................................................................................................................................................... 3<br />
Duration .................................................................................................................................................................................................................... 4<br />
Results ........................................................................................................................................................................................................................ 4<br />
<strong>Resiliency</strong> <strong>Score</strong> Phases .....................................................................................................................................................4<br />
Phase 1: Throughput ............................................................................................................................................................................................. 4<br />
64-Byte Packet ................................................................................................................................................................................................. 5<br />
1518-Byte Packet Measurement ............................................................................................................................................................... 5<br />
Real Bandwidth Measurement .................................................................................................................................................................. 6<br />
Phase 2: Sessions .................................................................................................................................................................................................... 6<br />
Basic Concurrent Sessions ........................................................................................................................................................................... 6<br />
Real Concurrent Sessions ............................................................................................................................................................................. 7<br />
Basic Connections per Second................................................................................................................................................................... 8<br />
Real Connections per Second .................................................................................................................................................................... 9<br />
Phase 3: Robustness.............................................................................................................................................................................................. 9<br />
IP Stability .......................................................................................................................................................................................................... 9<br />
UDP Stability ..................................................................................................................................................................................................... 9<br />
TCP Stability ...................................................................................................................................................................................................... 9<br />
Phase 4: Security .................................................................................................................................................................................................... 10<br />
Independent Security ................................................................................................................................................................................... 10<br />
Benign Trac Security .................................................................................................................................................................................. 10<br />
Concurrent Sessions Security ..................................................................................................................................................................... 10<br />
Session Open Rate Security ........................................................................................................................................................................ 10<br />
2
www.breakingpoint.com<br />
© 2005 - 2010. <strong>BreakingPoint</strong> Systems, Inc. All rights reserved. <strong>The</strong> <strong>BreakingPoint</strong> logo is a trademark of <strong>BreakingPoint</strong> Systems, Inc.<br />
All other trademarks are the property of their respective owners.<br />
<strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong><br />
<strong>Resiliency</strong> <strong>Score</strong> Setup .......................................................................................................................................................11<br />
Physical Connection ............................................................................................................................................................................................. 11<br />
Network Conguration ........................................................................................................................................................................................ 11<br />
Switch ................................................................................................................................................................................................................. 11<br />
Router ................................................................................................................................................................................................................. 12<br />
Load Balancer ................................................................................................................................................................................................... 12<br />
Proxy .................................................................................................................................................................................................................... 13<br />
Firewall ............................................................................................................................................................................................................... 13<br />
Intrusion Prevention System ...................................................................................................................................................................... 14<br />
Unied Threat Manager ................................................................................................................................................................................ 14<br />
Summary .............................................................................................................................................................................14<br />
About <strong>BreakingPoint</strong> Storm CTM .....................................................................................................................................15<br />
3
Introduction<br />
www.breakingpoint.com<br />
© 2005 - 2010. <strong>BreakingPoint</strong> Systems, Inc. All rights reserved. <strong>The</strong> <strong>BreakingPoint</strong> logo is a trademark of <strong>BreakingPoint</strong> Systems, Inc.<br />
All other trademarks are the property of their respective owners.<br />
<strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong><br />
Organizations want measurable answers, not assurances, when it comes to network and application performance, security, and stability.<br />
We have come to expect evaluation and certication of product performance for everything from our phones to our automobiles—yet<br />
network and application infrastructures and the equipment upon which they rely have no standardized certication for performance and<br />
security. Instead, buyers must trust statements made in product marketing literature, which are based on best-case scenarios, not real-world<br />
evidence.<br />
<strong>The</strong> <strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong> puts an end to this problem by establishing a standard against which networks and security devices<br />
are measured. This document describes a step-by-step approach for determining a <strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong>, as implemented in<br />
version 2.0 of the <strong>BreakingPoint</strong> Storm Cyber Tomography Machine (CTM) rmware.<br />
<strong>The</strong> <strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong><br />
<strong>The</strong> <strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong> provides an automated, standardized, and deterministic method for evaluating and ensuring the<br />
resiliency of networks and network equipment. This feature of the <strong>BreakingPoint</strong> Storm CTM provides a standard measurement of the<br />
performance, security, and stability of every component of the network and data center using real-world application trac, real-time<br />
security attacks, extreme user load, and application fuzzing.<br />
A <strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong> is calculated using standards by organizations such as<br />
US–CERT, IEEE, and the IETF, as well as real-world trac mixes from the world’s largest<br />
service providers. A <strong>BreakingPoint</strong> Storm CTM user simply selects the network or device<br />
for evaluation and the speed at which the device or system is required to perform for an<br />
automated measurement of resiliency. <strong>The</strong> <strong>BreakingPoint</strong> Storm CTM then subjects the<br />
device to a battery of simulations using a blended mix of application trac and malicious<br />
attacks, including obfuscations. <strong>The</strong> <strong>BreakingPoint</strong> Storm CTM delivers this measurement in<br />
the form of a <strong>Resiliency</strong> <strong>Score</strong>—much like an Underwriters Laboratories (UL) certication.<br />
<strong>The</strong> <strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong> is presented as a numeric grade from 1 to 100. Networks<br />
and devices may receive no score if they fail to pass trac at any point or degrade to an<br />
unacceptable performance level. <strong>The</strong> <strong>Resiliency</strong> <strong>Score</strong> takes the guesswork and subjectivity<br />
out of validation and allows administrators to quickly understand the degree to which system security will be impacted under load and<br />
under the threat of newly evolved cyber attacks.<br />
<strong>Resiliency</strong> <strong>Score</strong> Assessment Elements<br />
<strong>The</strong>re are four phases of the <strong>Resiliency</strong> Scoring process—each with a set of associated measurements—that target specic aspects of<br />
behavior:<br />
• Throughput Phase<br />
• Sessions Phase<br />
• Robustness Phase<br />
• Security Phase<br />
Prescribed Conguration<br />
Because the <strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong> is a standardized measurement designed to maintain the consistency of scores across vendors,<br />
the <strong>BreakingPoint</strong> Storm CTM requires little or no conguration. <strong>The</strong> user simply chooses a class of device from a set list of options; that<br />
choice determines the network conguration that will be used. <strong>The</strong> user congures their own device to support that network conguration,<br />
then selects a bandwidth limit for the device, in 100 Mbps increments.<br />
4
Duration<br />
www.breakingpoint.com<br />
© 2005 - 2010. <strong>BreakingPoint</strong> Systems, Inc. All rights reserved. <strong>The</strong> <strong>BreakingPoint</strong> logo is a trademark of <strong>BreakingPoint</strong> Systems, Inc.<br />
All other trademarks are the property of their respective owners.<br />
<strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong><br />
<strong>The</strong> <strong>Resiliency</strong> <strong>Score</strong> process can be run in either “Full” or “Quick” mode. <strong>The</strong> “Quick” option allows the user to work through initial issues<br />
without waiting through the turnaround time required for a full resiliency assessment. However, only the “Full” option provides an actual<br />
<strong>Resiliency</strong> <strong>Score</strong>.<br />
Results<br />
Once a full assessment encompassing all elements is complete, resiliency is scored from 0 to 100 by calculating the results from individual<br />
phases of the <strong>Resiliency</strong> <strong>Score</strong> process. If the network or device fails any part of the assessment, no score is given, and the assessment is<br />
declared “Failed.”<br />
<strong>Resiliency</strong> <strong>Score</strong> Phases<br />
<strong>The</strong>re are four phases of the process for producing a <strong>Resiliency</strong> <strong>Score</strong> for a component of a network or data center:<br />
Phase 1: Throughput<br />
<strong>The</strong> throughput phase targets the device’s basic packet forwarding ability. It consists of three parts:<br />
64-Byte Packet<br />
To determine the maximum achievable bandwidth of a device when forwarding 64-byte packets only, the <strong>BreakingPoint</strong> Storm CTM<br />
executes a series of simulations in a binary search pattern. For each one, success or failure is determined based on the following criteria:<br />
• <strong>The</strong> number of received frames must equal the number of transmitted frames to ensure that there were no dropped packets.<br />
• No corrupted frames should be received.<br />
Initially, trac is generated at 50 percent of the user-specied rate of the device. If the device passes this assessment, the next rate attempt<br />
is halfway between the initial rate and the maximum rate, or 75 percent of the user-specied rate. If the device fails, the next rate attempted<br />
is 25 percent. <strong>The</strong> process continues for seven iterations to determine the maximum achieved rate within a 1 percent margin of error.<br />
<strong>The</strong> individual score for this portion of the overall <strong>Resiliency</strong> <strong>Score</strong> is based on the measured bandwidth of the device and is expressed as<br />
the percentage of the measured speed versus the user-congured speed. <strong>The</strong> results of the 64-byte packet assessment are shown in the<br />
following way in section 2.1 of the report:<br />
A) IEEE Throughput measurement of 64 byte frames<br />
A = max(100,100 * Speed(999)<br />
/ MaxSpeed(1000))<br />
=100.00<br />
1518-Byte Packet Measurement<br />
Next, a second binary search is run with the frame size set to 1518. Aside from the frame size, this component runs identically to the 64-byte<br />
packet assessment and is scored the same way. <strong>The</strong> results below are provided in the nal report:<br />
B) IEEE Throughput measurement of 1518 byte frames<br />
B = max(100,100 * Speed(999)<br />
/ MaxSpeed(1000))<br />
=100.00<br />
5
Real Bandwidth Measurement<br />
www.breakingpoint.com<br />
© 2005 - 2010. <strong>BreakingPoint</strong> Systems, Inc. All rights reserved. <strong>The</strong> <strong>BreakingPoint</strong> logo is a trademark of <strong>BreakingPoint</strong> Systems, Inc.<br />
All other trademarks are the property of their respective owners.<br />
<strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong><br />
For the nal part of the Throughput Phase, the device’s throughput is measured when presented with a realistic blend of application<br />
trac. <strong>The</strong> <strong>BreakingPoint</strong> Application Simulator transmits at a bandwidth no higher than the user-congured maximum, with maximum<br />
concurrent sessions set to 10,000,000, and a load prole set to control the session open rate.<br />
For a full <strong>Resiliency</strong> <strong>Score</strong>, the load prole begins by opening sessions at a rate of 100 sessions per second. Every 5 seconds, the attempted<br />
open rate is increased by another 100 sessions per second, until 750,000 sessions per second is reached. <strong>The</strong> quick version of the assessment<br />
proceeds in a similar way, except the initial rate is 10,000, and the attempt rate is increased by 10,000 every ve seconds.<br />
<strong>The</strong> bandwidth achieved is determined by searching for the maximum measured received frame rate that was sustained for at least three<br />
seconds. This bandwidth achieved is factored into the score as a percentage of the congured bandwidth and is also used as a basis to<br />
determine the rate of some subsequent processes, as detailed below.<br />
C) Throughput, Real<br />
C= max(100,100 * Application Frame Rate(1420140)<br />
/ Max Application Frame Rate(150000))<br />
=100.00<br />
Phase 2: Sessions<br />
<strong>The</strong> Sessions Phase of the <strong>Resiliency</strong> <strong>Score</strong> focuses on the device’s ability to support UDP and TCP sessions, in terms of both rate and<br />
number of concurrent sessions.<br />
Basic Concurrent Sessions<br />
During the Sessions Phase, the ability of the device to process straightforward TCP sessions is determined using <strong>BreakingPoint</strong> Session<br />
Sender. <strong>The</strong> bandwidth limit is set to the user-congured speed for the device. <strong>The</strong> steady-state behavior for the TCP sessions is set to “Hold,”<br />
so that sessions stay open for the duration of the assessment.<br />
A load prole is generated to control the maximum number of concurrent sessions. For a full <strong>Resiliency</strong> <strong>Score</strong>, 5,000 sessions are opened<br />
at the beginning. Every ve seconds, 5,000 more sessions are opened, up to a maximum of 10,000,000 concurrent sessions. Once the<br />
6
www.breakingpoint.com<br />
© 2005 - 2010. <strong>BreakingPoint</strong> Systems, Inc. All rights reserved. <strong>The</strong> <strong>BreakingPoint</strong> logo is a trademark of <strong>BreakingPoint</strong> Systems, Inc.<br />
All other trademarks are the property of their respective owners.<br />
<strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong><br />
maximum is reached, the sessions are held open for an additional 30 seconds before wrapping up the assessment. A quick <strong>Resiliency</strong> <strong>Score</strong><br />
follows similar behavior, except that it starts with 500,000 sessions and adds another 500,000 every ve seconds.<br />
For scoring, the maximum achieved number of concurrent sessions that was sustained for at least three seconds becomes the measured<br />
maximum. This is applied to the score as a percentage of the 10-million-session maximum.<br />
D) Concurrent IETF 793 TCP Connections<br />
D = max(100,100 * Number Flows(10000000)<br />
/ Max Number Flows(10000000))<br />
=100.00<br />
Real Concurrent Sessions<br />
<strong>The</strong> Sessions Phase also measures the number of concurrent TCP sessions supported by the device using the <strong>BreakingPoint</strong> Application<br />
Simulator and the <strong>BreakingPoint</strong> Enterprise application prole. <strong>The</strong> bandwidth limit is set to the user-congured speed for the device.<br />
Again, the steady-state behavior for the TCP sessions is “Hold,” so that they stay open for the duration. <strong>The</strong> operation is identical to the<br />
<strong>BreakingPoint</strong> Session Sender assessment.<br />
E) Concurrent IETF 2581 TCP and IETF 768 UDP Connections<br />
E = max(100,100 * Number Flows(10066931)<br />
/ Max Number Flows(10000000))<br />
=100.00<br />
7
Basic Connections per Second<br />
www.breakingpoint.com<br />
© 2005 - 2010. <strong>BreakingPoint</strong> Systems, Inc. All rights reserved. <strong>The</strong> <strong>BreakingPoint</strong> logo is a trademark of <strong>BreakingPoint</strong> Systems, Inc.<br />
All other trademarks are the property of their respective owners.<br />
<strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong><br />
<strong>The</strong> basic connections per second measurement uses <strong>BreakingPoint</strong> Session Sender to generate the simple TCP connections required to<br />
measure the maximum rate at which the device can open new TCP connections. <strong>The</strong> bandwidth limit for <strong>BreakingPoint</strong> Session Sender is set<br />
to the user-specied rate for the device, and the maximum number of cumulative sessions is set to 10,000,000.<br />
When performing a full <strong>Resiliency</strong> <strong>Score</strong>, this process begins by opening TCP sessions at a rate of 100 per second. Every ve seconds, the<br />
attempt rate is increased by another 100 sessions per second. This continues until the attempted rate is 750,000 connections per second.<br />
Once the maximum is reached, the process continues at that rate for 30 more seconds.<br />
<strong>The</strong> “Quick” resiliency assessment is conducted in a similar way, except that the rate starts at 10,000 and is increased by another 10,000<br />
every ve seconds.<br />
For scoring purposes, the <strong>BreakingPoint</strong> Storm CTM evaluates the maximum achieved connection rate that was held for at least three<br />
seconds. <strong>The</strong>n a rate goal is calculated as follows:<br />
• For a user-specied bandwidth of more than 1Gbps, the goal is 150,000.<br />
• For user-specied bandwidths of 1Gbps or less, the goal is the bandwidth in Gbps times 150,000. For example, for a user-specied<br />
bandwidth of 500Mbps, the goal would be half of 150,000, or 75,000.<br />
<strong>The</strong> maximum measured rate is then incorporated into the score as a percentage of the determined goal:<br />
F) IETF 793 TCP Connections/sec<br />
F = max(100,100 * Number Flows(181390)<br />
/ Max Number Flows(150000))<br />
=100.00<br />
8
Real Connections per Second<br />
www.breakingpoint.com<br />
© 2005 - 2010. <strong>BreakingPoint</strong> Systems, Inc. All rights reserved. <strong>The</strong> <strong>BreakingPoint</strong> logo is a trademark of <strong>BreakingPoint</strong> Systems, Inc.<br />
All other trademarks are the property of their respective owners.<br />
<strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong><br />
<strong>The</strong> data from the throughput “Real Bandwidth” measurement is reused to evaluate real connections per second. <strong>The</strong> same process is run<br />
when either throughput or sessions is measured; when both are measured, the process is run only once, and the data is used for both.<br />
<strong>The</strong> connection rate is scored in a very similar manner to the basic session rate assessment. For scoring purposes, the maximum achieved<br />
connection rate that was held for at least three seconds is determined. <strong>The</strong>n a rate goal is calculated as follows:<br />
• For a user-specied bandwidth of more than 1Gbps, the goal is 100,000.<br />
• For user-specied bandwidths of 1Gbps or less, the goal is the bandwidth in Gbps times 100,000. In other words, for a user-specied<br />
bandwidth of 500Mbps, the goal would be half of 100,000, or 50,000.<br />
<strong>The</strong> maximum measured rate is then incorporated into the score as a percentage of the determined goal.<br />
G) IETF 2581 TCP and IETF 768 UDP Connections/sec<br />
G = max(100,100 * Number Flows(120213)<br />
/ Max Number Flows(100000))<br />
=100.00<br />
Phase 3: Robustness<br />
Three processes contribute to the robustness score. Each of them is structured very similarly, using a <strong>BreakingPoint</strong> Stack Scrambler to<br />
target a dierent layer of the network stack.<br />
IP Stability<br />
To evaluate IP stability, <strong>BreakingPoint</strong> Stack Scrambler is congured to send corruptions at 12 Mbps, including bad IP version elds, bad IP<br />
options, bad “urgent” pointers, and bad checksums. A diagnostic ICMP ping request is sent every second. <strong>The</strong> ICMP packet is structured to<br />
distinguish it from any other packets, and a count is kept of how many of these are received on the other side.<br />
After this process is complete, the count of received pings is compared to the count of transmitted pings. Up to two pings not received are<br />
allowed with no penalty. A third missed ping results in one point being subtracted from the overall score. Four pings dropped result in a<br />
10-points deduction, and ve or more dropped pings result in a failing <strong>Resiliency</strong> <strong>Score</strong>.<br />
H) IETF 791 IP Stack Stability<br />
H = Dropped Pings<br />
= pass<br />
UDP Stability<br />
<strong>The</strong> IP stability process is followed by an identical UDP stability evaluation. For this process, however, the target stack is set to UDP.<br />
I) IETF 768 UDP Stack Stability<br />
I = Dropped Pings<br />
= pass<br />
TCP Stability<br />
For the nal robustness assessment, the same process is used to target the TCP stack. This assessment is also scored the same way.<br />
J) IETF 793 TCP Stack Stability<br />
J = Dropped Pings<br />
= pass<br />
9
Phase 4: Security<br />
www.breakingpoint.com<br />
© 2005 - 2010. <strong>BreakingPoint</strong> Systems, Inc. All rights reserved. <strong>The</strong> <strong>BreakingPoint</strong> logo is a trademark of <strong>BreakingPoint</strong> Systems, Inc.<br />
All other trademarks are the property of their respective owners.<br />
<strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong><br />
<strong>The</strong> Security Phase measures the device’s response to attack using the <strong>BreakingPoint</strong> Strike Set repeatedly under dierent network<br />
background conditions. For a full <strong>Resiliency</strong> <strong>Score</strong>, the strike set “<strong>Resiliency</strong>—All CVE Strikes” is used. This Strike Set includes all Strikes that<br />
are documented as exploits in the CVE database. A “Quick” resiliency assessment uses the Strike set “Backdoor Strikes.”<br />
Each Security Phase assessment includes a <strong>BreakingPoint</strong> Stack Scrambler component targeting the TCP layer. This produces diagnostic<br />
pings used to measure security capabilities in the same way that robustness is scored: Up to two pings not received are allowed with no<br />
penalty; a third missed ping subtracts one point from the overall score; four pings dropped result in 10 points o; and ve or more dropped<br />
lead to a failing <strong>Resiliency</strong> <strong>Score</strong>.<br />
Independent Security<br />
This measurement is initially conducted using only the <strong>BreakingPoint</strong> Security component with the <strong>BreakingPoint</strong> Stack Scrambler in active<br />
mode; in other words, there is no background trac.<br />
K) CVE Security Fault Injection, Independent<br />
K = Dropped Pings<br />
= pass<br />
Benign Trac Security<br />
Next, the same process is executed with the addition of <strong>BreakingPoint</strong> Application Simulator and the <strong>BreakingPoint</strong> Enterprise application<br />
prole. Maximum attempted sessions per second, maximum number of concurrent sessions, and throughput are all congured to a value<br />
that is half of the maximum measured in the “Real Bandwidth” / “Real Sessions per Second” process.<br />
L) CVE Security Fault Injection, Benign<br />
L = Dropped Pings<br />
= pass<br />
Concurrent Sessions Security<br />
<strong>The</strong> previous process is run again, but this time with <strong>BreakingPoint</strong> Security and Stack Scrambler components, as well as <strong>BreakingPoint</strong><br />
Application Simulator with the maximum attempted sessions per second and the throughput both congured to a value half of the<br />
maximum measured in the “Real Bandwidth” / “Real Sessions per Second” process. <strong>The</strong> number of concurrent sessions for Application<br />
Simulator is set to 10,000,000.<br />
M) CVE Security Fault Injection, Concurrent Sessions Stress<br />
M = Dropped Pings<br />
= pass<br />
Session Open Rate Security<br />
Finally, the process is executed once more with <strong>BreakingPoint</strong> Security and Stack Scrambler components, as well as <strong>BreakingPoint</strong><br />
Application Simulator with the maximum number of concurrent sessions and the throughput both congured to a value half of the<br />
maximum measured in the “Real Bandwidth” / “Real Sessions per Second” process. <strong>The</strong> maximum session open rate is congured to 750,000<br />
sessions per second.<br />
N) CVE Security Fault Injection, Session Rate Stress<br />
N = Dropped Pings<br />
= pass<br />
10
<strong>Resiliency</strong> <strong>Score</strong> Setup<br />
Physical Connection<br />
www.breakingpoint.com<br />
© 2005 - 2010. <strong>BreakingPoint</strong> Systems, Inc. All rights reserved. <strong>The</strong> <strong>BreakingPoint</strong> logo is a trademark of <strong>BreakingPoint</strong> Systems, Inc.<br />
All other trademarks are the property of their respective owners.<br />
<strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong><br />
<strong>The</strong> device under evaluation should be connected via two of its interfaces to the <strong>BreakingPoint</strong> Storm CTM. <strong>The</strong>se connections will be<br />
referred to as Physical Interface 1 and Physical Interface 2, based on the interface reservation on the <strong>BreakingPoint</strong> Storm CTM.<br />
Network Conguration<br />
Figure 1 - Physical Connection to the <strong>BreakingPoint</strong> Storm CTM<br />
In preparation for the <strong>Resiliency</strong> <strong>Score</strong> process, the device that will be evaluated must be congured to support the following network<br />
conguration.<br />
Switch<br />
<strong>The</strong> network conguration used to evaluate a device classied as a switch will be made up of two separate IP ranges within the same<br />
subnet, both directly attached to the device’s network, as illustrated below:<br />
Figure 2 - Logical Network, Switch<br />
11
Router<br />
www.breakingpoint.com<br />
© 2005 - 2010. <strong>BreakingPoint</strong> Systems, Inc. All rights reserved. <strong>The</strong> <strong>BreakingPoint</strong> logo is a trademark of <strong>BreakingPoint</strong> Systems, Inc.<br />
All other trademarks are the property of their respective owners.<br />
<strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong><br />
<strong>The</strong> network conguration used to evaluate a device that is classied as a Router is made up of two ranges of hosts in nonlocal networks.<br />
Each will arrive at the device via a router attached to a separate local subnet of the device. Trac should be routed through gateway IPs on<br />
the device, as follows:<br />
Load Balancer<br />
Figure 3 – Logical Network, Router<br />
<strong>The</strong> network conguration used to evaluate a device that is classied as a Load Balancer includes:<br />
• Trac that originates from a network of client addresses in a nonlocal subnet.<br />
• Client addresses that reach the device being scored via a router on a local subnet of interface 1.<br />
• Trac from the clients addressed to a single public IP address on interface 1 of the device being assessed.<br />
• A device being assessed that is expected to translate the destination address to one from a pool of addresses within a local subnet on<br />
interface 2 of the device being assessed.<br />
• Requests within the destination address that are handled by a set of hosts that are simulated as a multi-homed host. That is, the set of<br />
IP addresses all originate from a single MAC address, avoiding the possibility of overowing MAC tables on the device.<br />
Figure 4 - Logical Network, Load Balancer<br />
12
Proxy<br />
<strong>The</strong> network conguration used to evaluate a device that is classied as a Proxy includes:<br />
• Trac that originates from a network of client addresses in a local subnet.<br />
www.breakingpoint.com<br />
© 2005 - 2010. <strong>BreakingPoint</strong> Systems, Inc. All rights reserved. <strong>The</strong> <strong>BreakingPoint</strong> logo is a trademark of <strong>BreakingPoint</strong> Systems, Inc.<br />
All other trademarks are the property of their respective owners.<br />
<strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong><br />
• Client requests that are handled by a set of hosts simulated as a multi-homed host. <strong>The</strong> set of IP addresses will all originate from a<br />
single MAC address, avoiding the possibility of overowing MAC tables on the device.<br />
• Server hosts listening on a set of hosts on a nonlocal subnet.<br />
• Server addresses that are reachable by the device being assessed via a router on a local subnet of interface 2.<br />
• A device being scored that is expected to translate the source address to one from an unspecied pool, which must be reachable by<br />
Firewall<br />
the server hosts. <strong>The</strong> specic client addresses are learned as they are observed.<br />
Figure 5 – Logical Network, Proxy<br />
<strong>The</strong> network conguration used to measure the resiliency of a device classied as a Firewall includes:<br />
• Trac that originates from a network of client addresses in a local subnet<br />
• Client requests that are handled by a set of hosts that are simulated as a multi-homed host. <strong>The</strong> set of IP addresses all originate from a<br />
single MAC address, avoiding the possibility of overowing MAC tables on the device.<br />
• Server hosts that will be listening on a set of hosts on a nonlocal subnet.<br />
• Server addresses that are reachable by the device being assessed via a router on a local subnet of interface 2.<br />
• A device being assessed that is expected to translate the source address to one from an unspecied pool, which must be reachable by<br />
the server hosts. <strong>The</strong> specic client addresses are learned as they are observed.<br />
Figure 6 – Logical Network, Firewall<br />
13
Intrusion Prevention System<br />
www.breakingpoint.com<br />
© 2005 - 2010. <strong>BreakingPoint</strong> Systems, Inc. All rights reserved. <strong>The</strong> <strong>BreakingPoint</strong> logo is a trademark of <strong>BreakingPoint</strong> Systems, Inc.<br />
All other trademarks are the property of their respective owners.<br />
<strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong><br />
<strong>The</strong> network conguration used to measure the resiliency of an Intrusion Prevention System includes two separate IP ranges within the<br />
same subnet, both directly attached to the device’s network as illustrated below:<br />
Unied Threat Manager<br />
Figure 7 - Logical Network, Intrusion Prevention System<br />
<strong>The</strong> network conguration used to evaluate a device classied as a Unied Threat Manager will be made up of two ranges of hosts in<br />
nonlocal networks. Each will arrive at the device being assessed via a router attached to a separate local subnet of the device being<br />
assessed. Trac should be routed through gateway IPs on the device, as follows:<br />
Summary<br />
Figure 8 – Logical Network, Unied Threat Manager<br />
Based on insights into global network trac proles and using standards from US–CERT, IEEE, and the IETF, the <strong>BreakingPoint</strong> <strong>Resiliency</strong><br />
<strong>Score</strong> provides an easy-to-produce and defensible measurement of the resiliency of networks and network devices and servers. A<br />
<strong>Resiliency</strong> <strong>Score</strong> can be produced only with the <strong>BreakingPoint</strong> Storm CTM, as it is the only product capable of producing the blended mix<br />
of application trac, malicious attacks, user load, and application fuzzing required to understand how devices will perform in today’s global<br />
networks.<br />
14
About <strong>BreakingPoint</strong> Storm CTM<br />
<strong>BreakingPoint</strong> has pioneered the rst Cyber Tomography Machine with<br />
the introduction of the <strong>BreakingPoint</strong> Storm CTM. This innovative product<br />
enables users to see for the rst time the virtual stress fractures lurking<br />
within their cyber infrastructure through the simulation of crippling attacks,<br />
high-stress trac load, and millions of users. <strong>The</strong> <strong>BreakingPoint</strong> Storm<br />
CTM exposes previously impossible-to-detect vulnerabilities within cyber<br />
infrastructure components before they are exploited to destroy what is most<br />
important to you—your customer data, your assets, your reputation, and<br />
even national security. <strong>BreakingPoint</strong> Storm CTM is delivered in a three-slot<br />
chassis that provides the equivalent performance and simulation of hundreds<br />
of racks of servers:<br />
• 40 Gigabits per second of blended stateful application trac<br />
• 30 million concurrent TCP sessions<br />
• 1.5 million TCP sessions per second<br />
• 600,000+ complete TCP sessions per second<br />
• 80,000+ SSL sessions per second<br />
• 130+ stateful applications<br />
• 4,500+ live security strikes<br />
www.breakingpoint.com<br />
© 2005 - 2010. <strong>BreakingPoint</strong> Systems, Inc. All rights reserved. <strong>The</strong> <strong>BreakingPoint</strong> logo is a trademark of <strong>BreakingPoint</strong> Systems, Inc.<br />
All other trademarks are the property of their respective owners.<br />
Contact <strong>BreakingPoint</strong><br />
<strong>BreakingPoint</strong> <strong>Resiliency</strong> <strong>Score</strong><br />
Learn more about <strong>BreakingPoint</strong><br />
products and services by contacting a<br />
representative in your area.<br />
1.866.352.6691 U.S. Toll Free<br />
www.breakingpoint.com<br />
<strong>BreakingPoint</strong> Global Headquarters<br />
3900 North Capital of Texas Highway<br />
Austin, TX 78746<br />
email: salesinfo@breakingpoint.com<br />
tel: 512.821.6000<br />
toll-free: 866.352.6691<br />
<strong>BreakingPoint</strong> EMEA Sales Oce<br />
Paris, France<br />
email: emeasales@breakingpoint.com<br />
tel: + 33 6 08 40 43 93<br />
<strong>BreakingPoint</strong> APAC Sales Oce<br />
Suite 2901, Building #5, Wanda Plaza<br />
No. 93 Jianguo Road<br />
Chaoyang District, Beijing, 100022, China<br />
email: asiasales@breakingpoint.com<br />
tel: + 86 10 5960 3162<br />
15