TECHNOLOGY REPORT - AUGUST 2005
Anti-Virus Firewall Solutions
An Independent Technology Report produced by
Product Testing, Evaluation
and Certification Services
West Coast Labs provides a superior
quality testing and certification service for
infosecurity technology developers and has
established independent industry-accepted
standards on product effectiveness and
performance for the benefit of corporate
end-users and decision-makers alike.
Through its global reach, West Coast Labs
brings technology developers and corporate
end-users together, creating a meaningful link
between what the market needs and what
technology developers are offering.
West Coast Labs Services
■ Advanced product testing and
■ Product feature and performance
■ Product-design review and
■ Beta testing and evaluation
■ Custom testing
■ Marketing your technology
message to a global buying market
For full details of West Coast Labs' product testing, evaluation and certification
services contact Mark Thomas, Sales Manager: firstname.lastname@example.org
TECHNOLOGY REPORT SUPPLEMENT FROM 3
Blended threats need to be
addressed by a unified
response for greater security
Simulating realistic business
processes is essential when
testing product capabilities
Welcome to the
second of West
Technology Reports. The
primary focus of this issue is
Anti-virus Firewall technology.
Part of the Haymarket
Publishing Services Group,
West Coast Labs’ is a wellestablished
CTO, West Coast Labs
independent testing facility for
information security products and services.
Working with over 60 of the world’s leading technology
developers, it has a reputation for high standards of
testing and objective judgement of the effectiveness of
product performance and functionality
West Coast Labs provides leading edge testing,
evaluation and certification services and, based on its
tests of leading IT security technologies, it is able to
offer up quantitative data upon which sound
management and purchasing decisions can be made.
The westcoastlabs.org knowledge base of Technology
Reports, White Papers, Custom Test Reports,
Certification Results receives over 500,000 hits a
month, a clear indication of the value that security
professionals put on this resource and the regard they
have for it as a research tool.
Unified Threat Management
In recent years, the rise of UTM solutions has
transformed the security market. Instead of single
function appliances and services, developers are
increasingly offering products which deploy multiple
security features in a single solution, allowing users to
achieve high levels of security with flexible, easily
Late last year, the Yankee Group reported that
“…firewalls combined with Anti Virus are the two most
highly valued security solutions”, helping customers
thwart blended threats by offering a variety of
functionality and performance benefits.
This AV Firewall Technology report looks at
appliances from Juniper Networks and Equiinet, plus a
managed service provider - SecurePipe.
The full test results are available online at
The overall objective of this test was to evaluate
each AV firewall product in a controlled
environment. Throughout the test period, each
product had internet access and was configured to
update online as recommended.
The testing environment mirrored that of a small to
medium sized business or branch office: the internal
interface of the firewall was connected to a 100Mbs
network, and traffic loads were set accordingly.
The products were tested in accordance with the
functionality criteria set out below, which form part of
the Checkmark certification programs for Firewall Level
1 and Anti-Virus Level 1. See www.check-mark.com.
The White Paper test reports online address three
specific areas: firewall competency, AV detection
functionality, and performance testing.
Outline test specifications
A range of tests were carried out using a variety of
firewall scanning tools. These were configured with full
knowledge of both the firewall and network
Tests were conducted to confirm that:
■ All specified outbound services (and no others)
were available from internal clients.
■ All specified inbound services (and no others)
were available to external clients.
■ The firewall management console was not
available to any users unless authenticated.
■ The firewall was resistant to a range of known
denial-of-service (DoS) tests.
■ The firewall did not allow uncontrolled access to
either the internal or demilitarized zone (DMZ)
West Coast Labs Testing Team
All West Coast Labs tests are carried out by fully
trained content and perimeter security test
engineers under the direction of the CTO Jon
Stearn, an acknowledged technical authority
among his peers, who has over 25 years
experience in the IT and security industries.
Particular thanks go to Michael Parsons, Matt
Garrad, Richard Thomas and Mike McMenamin.
4 TECHNOLOGY REPORT SUPPLEMENT FROM
■ The underlying operating system was hardened
and not vulnerable to known OS-specific attacks.
Tests were repeated as follows:
■ Probe the internal network from the Internet.
■ Probe the DMZ from the Internet.
■ Probe the firewall from the Internet.
■ Probe the external network from the internal
network (test security policy).
■ Probe the DMZ from the internal network.
■ Probe the firewall from the internal network.
Management of the firewall was evaluated using the
■ The local console must be secure.
■ The management console should not be open to
the external network.
■ The firewall configuration should be fully protected
and tamper-proof (except from an authorized
■ Authentication should be required for local
■ Authentication and an encrypted link should be
available for remote administration.
■ All attacks should be logged with date and time.
AV detection functionality
The testing reported on the following virus detection
■ Products were tested in accordance with
Checkmark AV Level 1 to determine their ability to
detect viruses. (West Coast Labs uses live viruses.
It does not use simulators.)
■ Multi-part viruses were reproduced in their various
manifestations and must be detected in each place
in which an infection may occur.
■ Polymorphic viruses were replicated to a minimum
(usually) of 250 iterations and must be detected in
The following performance tests were carried out on the
firewall technology, details of the results can be found in
the White Papers online.
■ Throughput measured the maximum transmission
rate at which the firewall can forward IP traffic
without frame loss.
■ Frame loss measured the percentage of frames
lost from flows and groups sent through the firewall
that should have been forwarded.
■ Tests also determined under what load (number of
packets per second and size of packets) the
firewall began to drop packets.
■ Latency calculated the minimum, maximum, and
average latency of received frames in flows and
groups of flows sent through the firewall.
■ Maximum connection rate measured the maximum
rate of connection requests that the product could
service without dropping connections.
■ In addition a range of tests were run to evaluate
overall performance with a typical mix of
background traffic (HTTP/SMTP/FTP) consistent
with the deployment profile. Testing aimed to
assess the appliance’s ability to perform under
significant load: syn flood, udp flood, and other
■ Tests were also be carried out to assess the
product’s ability to continue performing under
sustained worm/virus attack with multiple
simultaneous attacks on the external interface.
Both the firewall and the AV performance were
taken into consideration during these tests.
■ It was expected that all attacks would be blocked
Find the full results online
The analysis and full test results for each solution, which include
both functionality and performance data, are online at
www.westcoastlabs.org along with white papers, buyer’s guides
and other product information.
TECHNOLOGY REPORT SUPPLEMENT FROM 5
Juniper Networks NetScreen-5GT
DEVELOPER’S STATEMENT: For IT managers who need an
advanced security appliance with superior price/performance and
manageability to protect against all manner of network attacks. Ideal for
remote offices, retail outlets and fixed telecommuters.
Product: Juniper Networks NetScreen-5GT
Manufacturer: Juniper Networks
Contact details: www.juniper.net
Full white paper: www.westcoastlabs.org
5GT from Juniper
certified to Anti
Virus Level 1 and
Firewall Level 1
The Juniper Networks NetScreen-5GT
incorporates a stateful firewall with deep
inspection, VPN, anti-virus and web filtering
capabilities in one enterprise-class, all-in-one
Management can be carried out at the command
line,(locally or across the network), or using the
attractive and intuitive web interface.
The WUI allows for quick navigation and detailed
control over the entire operation of the appliance. By
default, access is restricted to HTTPS on the internal
network, but, like most other features of the appliance,
this can be reconfigured by the administrator as
The main menu in the web interface consists of clearly
marked sections and subsections: options are easy to
find and well grouped.
Testing of the firewall technology was conducted
within the framework of the Checkmark Firewall Level 1
certification test criteria.
The default configuration of the NetScreen-5GT
allows common outbound services : Telnet, FTP, HTTP,
SSL/HTTPS, SMTP and DNS. All inbound traffic is
Probes of both the internal network and the appliance
from the internet confirmed no ports open, as expected.
All attempts to pass traffic to prohibited ports failed.
Probes conducted from the internal network confirmed
that the full specified range of services were available.
There were no open ports on the appliance other than
those used for management. During a denial-of-service
attack, the NetScreen-5GT continued to allow normal
traffic flow whilst repelling all attempted break ins.
The initial configuration does not include a
demilitarized zone (DMZ) set up, but it is possible to
change the organization of the five network ports on the
rear of the device to provide several different
configuration options, including DMZ functionality if
In order to test the DMZ functionality, the NetScreen
5GT was reconfigured so that the test machines in the
DMZ had full outbound access to the internet and the
internal networks, and had access to the Telnet and
HTTP/HTTPS ports on the firewall.
Probes launched from the DMZ confirmed that
designated services were available on both the internal
and external networks, and on the NetScreen-5GT.
Attacks launched from both the internal and external
networks against hosts on the DMZ were unable to
access any services other than those specified in the
Probes from the external connection to both the
internal connection and the device itself, showed that
no inbound services had been allowed through by the
reconfiguration. Probes from the internal network also
showed that there had been no change in the outbound
Throughout all reconfigurations employed, the
NetScreen-5GT performed exactly in accordance with
the behaviour set down in the firewall rulebase.
Logging is thorough, and configurable. Logs may be
viewed in the web interface and exported in plain text
for analysis. All relevant information was correctly
logged throughout the course of testing.
AV functionality is provided by a Trend Micro engine
and provides scanning of HTTP, FTP, POP3 and SMTP
traffic. There is a wide range of configuration options
In testing over a number of months against the
Checkmark AV Level 1 certification criteria the product
achieved a 100 percent detection rate.
The NetScreen-5GT has the smallest footprint of the
devices on test, but as the results of the firewall and AV
detection results show, its size belies the ease of use of
the interface and the ability of this appliance toperform
well under pressure.
Juniper Networks NetScreen-5GT is a fully
featured security appliance with a small footprint.
The configurable firewall provides excellent
performance. The proven anti virus screening
covers a wide range of network protocols. It
should be considered by any
administrator who wants full
control over perimeter security.
6 TECHNOLOGY REPORT SUPPLEMENT FROM
Equiinet NetPilot Plus
DEVELOPER’S STATEMENT: Equiinet specialises in the manufacture
of multi-functional smart unified threat management appliances that
provide secure Internet access for small and medium sized enterprises.
Equiinet has over 30,000 of its products installed in the U.K.
Product: Equiinet NetPilot Plus
Contact details: www.equiinet.com/netpilot/
Full white paper: www.westcoastlabs.org
The NetPilot Plus
from Equiinet is
certified to Anti
Virus Level 1,
Firewall Level 1
The NetPilot Plus has been designed and
developed to provide an all-in-one solution to the
problems of network security. It offers a range of
unified threat management features including firewall,
antivirus, anti-spam, and intrusion detection/prevention
capabilities, together with VPN capabilities. For the
purposes of this AV firewall report, only the virus
detection and firewall technologies were tested.
The appliance can be managed via a text interface
using an attached monitor and keyboard, or by the
simple but attractive web interface
User options on the web interface are well grouped
and clearly labelled. They provide quick configuration
for the normal day-to-day operations of a small to
medium sized office. The presentation of the traffic data
in a graphical form helps to highlight any intrusion
attempts and adds to the overall ease of use, especially
for those users who find it difficult to interpret network
activity from purely text based logs.
Prior to testing of the firewall, the NetPilot Plus was
reset to factory defaults. The IP addresses of the
internal and external interfaces were set at the text
interface using an attached keyboard, and all further
configuration performed using the web interface.
The default rules allow only HTTP and SSL/HTTPS
traffic as outbound services and only inbound SMTP
traffic, directed to the device itself, which hosts an
easily configurable mail server. No inbound traffic was
allowed through to the internal network.
Probes of the internal network from the internet
revealed nothing, whilst a probe of the appliance itself
confirmed the only allowed inbound service to be
All attempts to pass traffic to non-allowed services,
including DNS, failed.
Probes from the internal network showed that only
HTTP and SSL/HTTPS traffic was allowed to pass to
external hosts, and attempts to connect to other
During a directed denial-of-service attack the NetPilot
Plus allowed legitimate traffic out from the internal
network whilst stopping any attempted incursions from
violating the firewall.
After reconfiguration of the appliance to provide a
wider range of outbound services, probes from the
internal network against the external connection
showed that the full range of services were available, as
was to be expected. Conversely, only those inbound
services that had been specifically allowed on the
NetPilot Plus were accessible from the internet. Any
attempt to access prohibited services on the internal
network was still not allowed.
Even when under extended attack the NetPilot Plus
logged all attacks and dropped packets with time and
date as well as other relevant data, including source
and destination IP addresses and ports, and MAC
addresses. The web management interface also
provides a useful visual graphical representation, which
gives an indication of network traffic and can act as a
warning light to point an administrator to look at the
detailed text logs.
NetPilot Plus successfully achieved the standard
required for Checkmark the Firewall Level1
An additional license must be purchased to enable the
AntiVirus functionality of the NetPilot Plus The
appliance’s Sophos engine provides scanning of web
and email traffic. In tests, the NetPilot Plus successfully
detected 100 percent of the viruses in the May 2005
virus collection over both HTTP and SMTP protocols.
As the results of the firewall and AV detection results
show, the NetPilot Plus appliance is straightforward to
set up, configure and use and, as the performance test
data in the online White Paper shows, it is an effective
and efficient appliance.
The Equiinet NetPilot Plus has a simple
interface for a complex security device. Default
settings are well chosen and allow quick and
easy deployment, with the potential for more
complex configuration. Proven AV protection for
HTTP and SMTP is included.
TECHNOLOGY REPORT SUPPLEMENT FROM 7
SecurePipe Managed Network Security
DEVELOPER’S STATEMENT: SecurePipe delivers managed network
security services to organizations impacted by regulatory requirements.
Via 24x7x365 monitoring and management and dynamic reporting, it
helps clients strengthen security, reduce costs and improve compliance.
Product: SecurePipe Managed Network
Contact details: www.securepipe.com
Full white paper: www.westcoastlabs.org
certified to Anti
Virus Level 1,VPN
Firewall Level 1
The approach here is somewhat different to the
other products on test, as SecurePipe provides a
managed service. Customers specify exactly
what services they require and SecurePipe configures
and maintains the firewall accordingly.
All requests for changes to the firewall configuration
are made via SecurePipe’s Security Console website
and are implemented by their technical team. The
website is secured with SSL and access is restricted by
username and password.
The initial set up requested for the SecurePipe
solution included no inbound servicesand restricted
outbound services. The appliance was shipped ready to
plug in to the test network with this configuration
already set up.
Probes of both the internal network and the device
itself from the external network revealed nothing and all
attempts to pass prohibited traffic or bypass the
Throughout a directed denial-of-service (DoS) attack
the SecurePipe solution allowed legitimate traffic out
from the internal network while stopping any rogue
packets from getting through to the internal network.
To test the demilitarized zone (DMZ) functionality the
SecurePipe box was reconfigured by technical support
to allow a DMZ on one of the ports via an expansion
card. The configuration allowed access to specific ports
on specific hosts on the DMZ: HTTP on one, FTP on
another, and so on. Probes confirmed that any attempts
to access other services were completely blocked,
exactly as requested. The internal network remained
completely protected, and outbound services were
The box was then reconfigured again to disable
access to some outbound services. The availability of
those services was confirmed by test from the external
network; however any attempt to connect to those
services from the internal network was met with an
error message saying that the connection had been
blocked at the proxy level.
This gives the rather interesting option of being able to
see that services exist on remote servers while being
unable to connect to them.
The SecurePipe web interface can provide detailed
text logs of dropped and blocked traffic along with some
system logs. These packet logs provide date and time
as well as other relevant data. Information is also
provided about network attacks and Securepipe will
also send emails to administrators of domains which
have been the source of attacks: these too can be
SecurePipe can provide a managed anti-virus solution
for SMTP. This requires amendment of the MX records
for the domain to be protected to pass all mail through
the Securepipe service. Functionality was tested
against the requirements of the Checkmark certification
for AV Level 1.
When tested against the May 2005 virus collections
the SecurePipe service intercepted all viruses sent by
email to the target domain.
The SecurePipe managed service does not remove
responsibility for a secure network from the local
administrator. It allows the administrator to specify what
security policy they wish to have in place, but leaves the
implementation and monitoring of that policy to security
The online support request form was not used during
testing of the service, but the technical support proved
to be knowledgeable and effective.
The SecurePipe managed service allows an
administrator to specify their security policy
and leave the implementation of that policy to
external security professionals. AV support is
available for SMTP. An excellent solution for any
hard pressed IT department.
In the dark
when it comes to
choosing the right
Anti Virus, Trojan,
and VPN solution?
Check for the Checkmark
The Checkmark System independently tests and certifies that security products genuinely
achieve internationally recognised standards. West Coast Labs’ independent testing
laboratories have a worldwide reputation for accuracy and reliability. The Checkmark Systems
tests products regularly, in some cases as frequently as every six weeks, to ensure that the
product maintains compliance with the international standards.
If the product your using doesn’t have one, maybe you should ask why.
To find out more about the Checkmark visit our website at www.check-mark.com
The following companies have products tested and certified under the Checkmark system:
AhnLab • Aladdin • Alcatel • Blackspider • Cat • Command • Computer Associates • Cybersoft
• Equiinet • ESET • F-Secure • GData • GFI • Grisoft • Hauri • ISS • Juniper Networks •
Kaspersky • McAfee • Microworld • NGS Software • Norman • Panda • Preventon • Rapid 7 •
SecurePipe • Softwin • Sophos • Symantec • Trend Micro • VirusBuster • Wanadoo