ICSA Labs Product Assurance Report - Verizon Business

More documents

Recommendations

Info

ICSA Labs Product Assurance ReportIn 1998, ICSA Labs began researching andsoon launched a program to test IntrusionDetection Systems (IDS). The program ranfor several years until a shift occurred withinthe industry away from detection alone toreal-time prevention (the reasons behind thisshift are a story unto themselves). PreviousIDS vendors began promoting IPS insteadand ICSA Labs discontinued IDS testing andbegan incorporating lessons learned into a newprogram to certify IPS.Based on input from enterprise users andproduct developers, ICSA Labs built thenetwork IPS testing program to reflect realworld conditions as much as possible. Whilesome testing labs settled for artificial networktraffic from commercial traffic generation tools,ICSA Labs was researching what a suitablemix of legitimate business traffic ought to be.The answer was simple in theory but less soin application. Why not use actual traffic froma typical enterprise network? After modifyingover 40% of the code from an open source tool,ICSA Labs was soon able to replay once livetraffic through devices at rates up to at least10Gbps. While building the infrastructure toreplay realistic traffic in the background, ICSALabs also conducted extensive research todetermine which exploits should be run in theforeground. Here again, analysts opted for real,working exploits to test the effectiveness of IPS.One of the unexpected, initial dilemmas forthe new IPS program involved throughput.At issue was the fact that tested devices wereperforming at 40-60% less than what theyadvertised in their marketing material. ManyIPS posted dramatically different numbers inrelatively homogenous simulated networktraffic than in our mixed traffic taken fromactual networks. Though ICSA Labs providedmembers with histograms of the legitimatenetwork traffic that included detailed protocolbreakdowns, most could not significantly- Continued on next page -Historically, logging is a weakness of any kind of firewall product;almost every network or web application firewall tested at ICSALabs has at least one logging violation. Logging is relatively newto the certification criteria for IPSec (and so not much historicaldata exist) but is a frequent cause of violations within the SSL VPNprogram. This mostly involves not logging the right information.Within the IPS program, logging issues usually manifestthemselves in the form of reporting deficiencies.Another interesting observation with respect to logging violationsis that they can be particularly difficult to fix. Quite often, anadequate solution requires a total reengineering of the product.This has a lot to do with the product architecture (i.e., whetherlogging is handled by the firewall engine or not) and seemsto be getting worse as hardware sophistication, throughputrequirements, and reliance on third-party software increase.The widespread nature of logging violations is made moredistressing when overlaid with findings from the Verizon BusinessData Breach Investigations Report. The report makes it plainthat the only time many organizations learn they lack the rightinformation is after something has gone wrong. Though logsusually contain some evidence of an incident, investigatorsoften find critical information to be missing or incomplete. Theview held by many (vendors and users) that logging is a nuisanceand something merely done to “put a check in the box” surelycontributes to the inability of organizations to detect and respondto incidents in a timely manner.Product SecurityDo security products have security problems? According toour test results, over 40% of them do indeed. Security issuesexposed in testing range from vulnerabilities that compromisethe confidentiality or integrity of the system to seemingly randombehavior that affects availability.One of the more ironic examples we’ve ever come acrosswas a web application firewall that turned up numerousvulnerabilities within its web administration interface. Crosssitescripting, SQL injection, and buffer overflow vulnerabilitiesand unencrypted admin interfaces are some of the commonsecurity issues identified within the Custom Testing engagements,Web Application Firewalls, and Network Firewalls programs.Vulnerability testing is not a major component of the AV programbut they are sometimes found nonetheless. AV products, especiallythe all-in-one varieties, do struggle with performance and stabilityproblems that negatively impact the integrity and/or availabilityPage 12 of 19
ICSA Labs Product Assurance Reportimprove their performance numbers. Thisfinding exposed several important things aboutIPS devices at the time.First, there is no standard set of network trafficto use when determining IPS throughputand no standard method to create that mix.As a result, developers often test differentlywhen they produce throughput numbers fortheir marketing materials. Second, there are anumber of factors that strongly affect achievedthroughput including the policy being enforcedon the device and the properties of the trafficused in testing (e.g., the mix of protocols, theaverage frame size, etc.). Finally, because ofthis disparity, users should be cautious whencomparing published throughput values .and evaluating how IPS products will affecttheir networks.of the system on which they’re running. Many of the securitytroubles for SSL VPNs can be traced to the OpenSSL toolkit (whichroughly 80% of them use). In these instances, codefixes are hard,usually requiring a major release. By comparison, IPSec VPNs testedhave significantly fewer security violations.DocumentationIt may seem odd that certification criteria address documentationbut its importance has plenty of historical precedent. Plato, forinstance, recognized this and called for anyone leaving behind awritten manual to make sure it was accurate and understandable 5 .Unfortunately, many vendors today give little import to productdocumentation. Sometimes the problem is with quantity (toolittle documentation), other times it’s quality (unhelpful and/ormisleading), or even currency (doesn’t match the current productversion). Either way, poor documentation is at best unhelpful andat times, dangerously misleading.Because ICSA Labs configures and tests so many products, ouranalysts spend more time trying to figure out instructions thanthe average IT staffer. It would be difficult to tally the numberof late nights, frustrated outbursts, and grey hairs attributed todocumentation over the last 20 years but Table 4 can at least attestto its frequent occurrence. Though frustrating in the lab, the realdanger from poor documentation is when it results in deploymentmistakes and misconfigurations that erode enterprise security.Documentation violations arise more often in newer products(regrettably a period of time in which accuracy would be mosthelpful). This accounts for the high percentage in Table 4 forthe Custom Testing program, which regularly tests newertechnologies. As products mature and features stabilize, sotoo does the accompanying documentation. We also seedocumentation problems vary based on product type. Forinstance, though prevailing opinion holds that SSL VPNs are easierto implement than IPSec, we find that SSL products consistentlyfail documentation quality tests, while IPSec VPNs have few issues.The main issue behind documentation problems seems to be oneof prioritization. Documentation is sometimes written in a rushat the last minute by people who know little about the product.We find that the writers and product designers often work indifferent locations and never talk. Sometimes the documentationwas obviously written by someone in marketing who was moreconcerned with declaring the wonders of the product thanimparting technical know-how.5“...Then anyone who leaves behind him a written manual, and likewise anyone who receives it, in the belief that such writing will be clearand certain, must be exceedingly simple-minded....” – Plato, Phaedrus.Page 13 of 19
Page 1: ICSA Labs Product Assurance ReportA
Page 8: ICSA Labs Product Assurance ReportO
Page 11: ICSA Labs Product Assurance ReportC
Page 15 and 16: ICSA Labs Product Assurance ReportI
Page 17 and 18: ICSA Labs Product Assurance ReportV
Page 19 and 20: ICSA Labs Product Assurance Report

ICSA Labs Product Assurance Report - Verizon Business

Create successful ePaper yourself

Delete template?

Save as template?