Metrics and Analysis in Security Management - Ohlhausen Research

ohlhausen.com

Metrics and Analysis in Security Management - Ohlhausen Research

WHITEPAPERMETRICS AND ANALYSISIN SECURITY MANAGEMENTBy Brian McIlravey, CPPand Peter Ohlhausen


About the Authors:Brian McIlravey, CPP, is Co-CEO of PPM 2000 Inc. (www.ppm2000.com) and isresponsible for driving strategic planning and product direction. He is a member ofASIS International’s Information Security Technology Council and has experience in bothcorporate security and public law enforcement.Peter Ohlhausen is president of Ohlhausen Research, Inc. (www.ohlhausen.com), whichfor more than 20 years has provided research and consulting to the security, technology,and criminal justice fields. He formerly served as editor of Security Management, themonthly magazine of ASIS International.Published by PPM 2000 Inc.www.ppm2000.comFor over twenty years, PPM has worked with organizations around the world—using theirknowledge of risk management, security management and loss prevention—to providehigh quality subject matter expertise in the design and application of Incident Reportingand Investigation Management software. Thousands of organizations have implemented aPPM solution, and the company’s clients span all industries and the Fortune 1000. PPMis recognized by Microsoft as a Gold Independent Software Vendor.From incident reporting, to investigation management, to actionable businessintelligence, PPM offers end-to-end Incident Management solutions for—and from—security professionals. For more information on Perspective by PPM 2000, contact PPMtoll-free at 1-888-776-9776 or email information@ppm2000.com.Copyright © 2012 PPM 2000 Inc.


ContentsExecutive Summary 7The Power and Importance of Metrics and Analysis 8Fortified Decision Making 11Metrics as a Security Operations Tool 12Metrics as Marketing for the Security Program 14Developing Specific Metrics 17Essential Ingredient: Data 20From Data to Information: Analyzing Metrics 21Getting Started 23References 25PPM 2000 Inc. 10088 ‐ 102 Avenue, Suite 1307 • Edmonton, Alberta • T5J 2Z11‐888‐776‐9776 • information@ppm2000.com • www.ppm2000.com


Executive SummaryThe use of metrics and analysis (MA) is a sophisticated practicein security management that takes advantage of data to produceusable, objective information and insights that guide decisions.In addition, MA provides chief security officers (CSOs) with clearevidence of their operations’ value, expressed in the language oftop management.As Carnegie Mellon University notes, “metrics are quantifiablemeasurements of some aspect of a system or enterprise…Security metrics focus on the actions (and results of thoseactions) that organizations take to reduce and manage the risksof loss of reputation, theft of information or money, and businessdiscontinuities that arise when security defenses are breached.”Through MA, a CSO or other security professional can betterunderstand risks and losses, discern trends and manageperformance. He or she can also report clearly and accurately toexecutive management. These uses of MA all work to support theorganization’s strategic goals.Software designed specifically for the security field can make thegathering of security and risk-significant data orderly, convenientand accurate—and hold the data in a format that facilitatesanalysis. Security and risk-focused incident management softwareoffers both the standardization and consolidation of data. Suchsoftware also automates the task of analysis through trending andpredictive analysis and the generation of customized statisticalreports.This paper synthesizes the current MA literature in the securitymanagement field. It describes the use of metrics and analysis to:• Improve decision making;• Strengthen security operations; and• Gain support for the security and risk management operation.It then describes the process of developing specific metrics,collecting and managing data and performing useful analyses withsecurity risk-focused software.Metrics and Analysis in Security Management7


The Power and Importance ofMetrics and AnalysisThis paper examines key themes and thinking in the field ofmetrics and analysis (MA), focusing on applications in thedomain of security management. The aim is to inform securityprofessionals about a powerful practice that is becomingincreasingly essential in competitive business environments—and,in fact, is often demanded by executive management.The use of MA is part of a serious approach to securitymanagement. In contrast to more casual, gut-oriented approachesto security decision making, MA takes advantage of data toproduce usable, objective information and insights that guidedecisions. In addition, MA provides CSOs with clear evidenceof their operations’ value, expressed in the language of topmanagement.The Systems Security Engineering Capability Maturity Model,developed by a team headed by Carnegie Mellon University toadvance security engineering, provides an especially clear view ofmetrics:WHYUSEMETRICS?Metrics and analysisprovides CSOs withclear evidence of theiroperations’ value, expressedin the language of topmanagement.“What’s the benefit of usingmetrics? Basically, toimprove overall security andreduce costs.”At a high level, metrics are quantifiable measurements of someaspect of a system or enterprise. For an entity (system, product, orother) for which security is a meaningful concept, there are someidentifiable attributes that collectively characterize the security ofthat entity. Further, a security metric (or combination of securitymetrics) is a quantitative measure of how much of that attributethe entity possesses…Raymond Musser, CPPVice President, SecurityGeneral Dynamics(Musser, 2011)Security metrics focus on the actions (and results of thoseactions) that organizations take to reduce and manage the risksof loss of reputation, theft of information or money, and businessdiscontinuities that arise when security defenses are breached.They are useful to senior management, decision makers, users,administrators, or other stakeholders who face a difficult andcomplex set of questions regarding security, such as:• How much money/resources should be spent on security?• Which system components or other aspects should betargeted first?• How can the system be effectively configured?• How much improvement is gained by securityexpenditures, including improvements to securityprocesses?8 Metrics and Analysis in Security Management


• How do we measure the improvements?• Are we reducing our exposure?The MA approach results in business intelligence, which has beendefined as (PPM 2000 Webinar, 2009):ALIGNSTRATEGY ANDPERFORMANCEThe collection, integration, analysis, interpretation andpresentation of business information to provide historical, currentand predictive views of business operations, [and] the use of thisinformation through extraction, analysis and reporting to supportbetter business decision making.The insights and findings a CSO gains through MA can supportactivities both inside and outside the corporate securitydepartment. Inside the department, the CSO can betterunderstand risks and losses, discern trends and manageperformance based on actual measurements. Outside thedepartment, the CSO can report clearly and accurately toexecutive management. Both the internal and external uses of MAwork to support the organization’s strategic goals.The related concept of benchmarking—comparing one’sorganization with others in the same industry—relies in parton using metrics. That comparison relies first of all on anunderstanding of one’s own organization, and that understandingmust be developed through MA. According to Hayes and Kotwica(2011),Business leaders recognize benchmarking as a proven businesspractice that can identify competitive strengths and vulnerabilitiesas well as opportunities for improvement… But while the demandfor performance measures has trickled down to the securityfunction, the appreciation for them hasn’t always come along forthe ride. Too many security leaders create or find benchmarks forthe sole purpose of appeasing their bosses rather than from anearnest desire to use these tools to explore what others are doing,address potential gaps and add value.[C]orporate performancemetrics… [was] the topictackled by the most recentBlue Ribbon Commission atthe National Association ofCorporate Directors (NACD).Why corporate performancemetrics? Because theylink corporate strategy andcorporate performance…Strategy is about the future,performance is about the pastand metrics align the two.Financial Executive(Daly, 2011)It is important to remember that MA consists of both metricsand analysis. Hayes and Kotwica emphasize that point withthe example of benchmarking on corporate ethics hotlines. Thebenchmark report may suggest that the average organization ofa certain size and industry receives eight to nine calls to thecorporate ethics hotline per thousand employees. If a particularcompany receives only three calls per thousand employees,analysis is warranted. Does the company have fewer ethicsproblems than its peers? Are employees intimidated into notreporting their concerns? Is the hotline underpublicized?Metrics and Analysis in Security Management9


may mean that security is being left out of the mainstream of theorganization... [S]ome security managers don’t know what metricsare or how they should gather or report metrics, and that willrequire some training and education. [O]ther security managersfeel that collecting metrics is more work than they want to do, [butif] your management has an interest or develops an interest in thisarea, you’d better be ready to respond.The practice of MA is more advanced in the field of informationtechnology security than in the field of corporate security as awhole. Although much of the research conducted so far on MAhas been focused on IT, a growing interest in studying MA’sapplication to security management is evident in an expandingfocus on the subject in security conferences and publications.This paper synthesizes the current MA literature primarily inthe security management field and also adds insights from morefoundational IT MA sources.The sections that follow address six key aspects of thismanagement tool:• Fortified Decision Making• Metrics as a Security Operations Tool• Metrics as Marketing for the Security Program• Developing Specific Metrics• Essential Ingredient: Data• From Data to Information: Analyzing MetricsThe paper then presents recommendations on how to startemploying metrics and analysis in security. A list of sources foradditional information concludes the paper.Fortified Decision MakingHow can security managers make decisions that are more likely tolead to success? What, specifically, leads to better decisions? Inthe Harvard Business Review, Davenport and Harris (2010) reportresults from their study of 400 companies in 35 countries and 19industry sectors.They found that “better decisions emerge when companiessystematically:How can security managersmake decisions that aremore likely to lead tosuccess? What, specifically,leads to better decisions?In the Harvard BusinessReview, Davenport andHarris (2010) report resultsfrom their study of 400companies in 35 countriesand 19 industry sectors.They found that “betterdecisions emerge whencompanies systematically:• Identify their criticaldecisions.• Inventory thosedecisions that requireanalytical help.• Intervene whereneeded.• Institutionalize whatwas learned.”• Identify their critical decisions.• Inventory those decisions that require analytical help.Metrics and Analysis in Security Management11


• Intervene where needed.• Institutionalize what was learned.”Emphasizing the “analytical help” mentioned in the second step,the authors note that “those who view analytics as just reportingon past performance don’t understand the full scope and value ofanalytics.”Analytics, they explain, has descriptive, predictive andprescriptive properties. Descriptive analytics describe pastperformance. Predictive and prescriptive analytics examine datato determine significance:Predictive analytics—which include forecasting, predictivemodeling, and optimization—are focused on the future. The useof predictive analytics takes an organization to a higher degree ofintelligence and can yield competitive advantage.Thus, analytics based on metrics, which this paper refers to asMA, appears to be an essential, foundational step in optimaldecision making.Metrics as a SecurityOperations Tool“Predictive analytics—which include forecasting,modeling, andoptimization—are focusedon the future. The use ofpredictive analytics takesan organization to a higherdegree of intelligenceand can yield competitiveadvantage.“Harvard Business Review(Davenport & Harris, 2010)Metrics and analysis (MA) can guide decisions regarding securityoperations in both specific and general ways. For example, atDelta Air Lines, MA is used to guide policy making. Accordingto Kim Hodgkin, Delta’s Manager of Security Administration,the company tracks compliance issues, accidents, medicalemergencies, financial crimes and other losses. He notes,“Based on our metrics and analysis, we make recommendationsto security leadership and other divisions.” Changes suggestedby MA include improved employee training, changes toscreening methods, security awareness messages, and targetedinvestigations (Hodgkin, 2011).Similarly, Treece and Freadman (2010) describe the use ofmetrics and analysis at the Massachusetts Port Authority(Massport) to solve the specific problem of security door alarms.They report that Massport greatly reduced such alarms throughthe analysis of alarm metrics. That analysis helped securitymanagement “determine the cause of each type of alarm anddevelop solutions to eliminate or reduce them.” Analysis ofdetailed door transaction data, including video, showed thecauses of alarms. That understanding led to a variety of corrective12 Metrics and Analysis in Security Management


actions, including maintenance and user training. The authorsreport, “The result has been fewer false alarm police dispatches,which results in a more efficient use of this valuable lawenforcement and security resource.”MA can be used not only to identify security problems but alsoto gauge the effectiveness of various security measures usedto counteract those problems. For example, the chart belowshows the correlation between the execution of various securitycountermeasures and the number of thefts from vehicles:Effect of Security Measures on Thefts from Vehicles20151050Began video surveillanceImproved lightingBegan employee awareness effortsShared data with policeArrests madeJan Feb Mar Apr May Jun Jul Aug Sep Oct Nov DecReinforced employee awarenessMA can also be used to guide more general decisions andanswer big-picture questions for security management, executivemanagement and others. Campbell (2006a) identifies severalquestions that can be answered with metrics:• How much money/resources should be spent on security?• Which system components or other aspects should be targetedfirst?• How can the system be effectively configured?• How much improvement is gained by security expenditures,including improvements to security processes?CSOS NEEDMETRICS FORCSOSMetrics are measures thatmatter, providing evidenceof performance… That’swhy CSOs [chief securityofficers] are hungry forthem… Security executiveswant to understand howtheir operations are workingand how they can improve.CEOs want to know how thesecurity function is faring bylooking at the department’sdata. And metrics canprovide the hard numbersand context on theperformance of the securityfunction, proving thatnothing happening was thedirect result of an effectivesecurity managementprogram.CSO(Wailgum, 2005)• How do we measure the improvements?• Are we reducing our exposure?Metrics and Analysis in Security Management13


Even before major problems occur, MA can be used to watchindicators—signs of risk—that may suggest a need for differentsecurity measures. Campbell (2006b) notes that these indicatorsor metrics “become the earliest prompts for more in-depth analysisof trend dynamics,” allowing CSOs to “look at the root causes ofproblems, not just the symptoms.” He lists several trends thatmetrics may help identify:METRICSPROVIDEANSWERS• More frequent or more severe accidents, crimes or policyinfractions;• Increased downtime of critical equipment;• Rise in negative background investigations;• Changes in security response times;• Reduction in building evacuation exercises; and• Rise in misconduct cases within a business unit.With careful analysis of the right metrics, a security professionalcan devise appropriate strategies to reduce risks. Expanding onthe example of increased misconduct cases, Campbell (2006b)suggests that further investigation might show poor supervisionof employees in that unit, as well as little employee awareness ofcompany policies on business conduct. Solutions would requireefforts by the security, human resources and legal departments.MA can also be used for external comparisons—that is, comparingone organization’s security-relevant metrics to those of otherorganizations. This process of benchmarking depends on theavailability of metrics and, of course, the underlying data thatmust be collected to produce those metrics.Metrics as Marketing for theSecurity ProgramIn their definition of security metrics management, Kovacichand Halibozek (2005) emphasize both the operational aspects ofmetrics (discussed above) and the business aspects of metrics. Onthe business side, they note:Consider this: Does yourmanagement want to beable to clearly see whetheryou are conforming withcorporate values andpolicies? Would they like tohave a visual representationof the state of the company’srisk—desirable orundesirable? Would theylike to have measurementsand data at hand that showwhether the company is incompliance with applicablelaws and regulations? Dothey want to know whetherpast and current securityinvestments have resultedin decreased risk or fewerincidents, so they canmore easily determinethe direction of futureinvestment?Through the use of metrics, the security cost versus benefitanalysis becomes more quantitative and easier to understand andcommunicate in common business terms. Metrics help the securityprofessional and others better understand the efficiency andeffectiveness (value) of an assets protection program.(Campbell & Blades, 2009)14 Metrics and Analysis in Security Management


That definition hints at the power of metrics and analysis (MA)to demonstrate a security department’s contribution to overallcorporate success. The message continues to be emphasized inprominent security forums. For example, in a presentation titled“The Security Metrics Challenge” at the 2011 ASIS InternationalSeminar and Exhibits, speakers stressed the indispensability ofmetrics:• “It’s about the value proposition. If you can’t show valuein industry, then you are not going to go very far… Are youproducing something that’s going to increase or sustain thevalue of the company? This is very important in the boss’sview.”James Shamess, CPP, Senior Adviser for Security Policyand Oversight, Office of the Administrative Assistant to theSecretary of the Air Force• “When you talk to senior executives, you have to talk inlanguage they understand: money, what’s the return oninvestment, and what’s the benefit to me?... Securityprofessionals, to have a seat at the table, need to be seen asvalue-added and cost-effective. You need to be able to reportmeaningful, intelligent, risk-based performance metrics tobuild confidence in your executive teams… Use those metricsto create a business case and measure program success. Youhave to show success in measurement. You can’t just providemetrics for the sake of metrics.”Klaus Heerwig, Director of Security, SRA InternationalWhen Security Management magazine gathered five leadingsecurity professionals to discuss challenges and trends in thesecurity field, the topic of return on investment, or ROI, came upquickly. When asked how to make the business case for security,Chad Callaghan, CPP, Vice President, Enterprise Loss Prevention,Marriott International, Inc., replied (Harowitz, n.d.):When you talk to seniorexecutives, you have totalk in language theyunderstand: money,what’s the return oninvestment, and what’s thebenefit to me?... Securityprofessionals, to have aseat at the table, need tobe seen as value-addedand cost-effective. Youneed to be able to reportmeaningful, intelligent, riskbasedperformance metricsto build confidence in yourexecutive teams… Usethose metrics to create abusiness case and measureprogram success. Youhave to show success inmeasurement. You can’t justprovide metrics for the sakeof metrics.For us, it’s metrics. Having something you can measure, that youcan show improvement in year over year that attaches to somethingthat has intrinsic value to your company. Because we do safetyand security both and because we are part of risk management,we’re able to measure total losses to the company and that has ahuge impact. It is one of the key metrics used, and it gets a lot ofattention.Klaus Heerwig,Director of Security,SRA InternationalBrian Tuskan, Microsoft Corporation’s Senior Director of GlobalSecurity Technology & Investigations, notes that by using security-Metrics and Analysis in Security Management15


focused incident management software (Perspective by PPM 2000)to track and analyze laptop computer thefts, his organization hasbeen able to identify trends, change its security approach and cuttheft losses in half. Use of such software has helped Microsoftthwart or prevent major theft trends, with savings running in thehigh hundreds of thousands of dollars to millions of dollars (PPM2000, 2012).The ROI is substantial. The investment in software is modest, andthe returns include:• Loss reduction (e.g., fewer laptops stolen);• Labor savings (more efficient data collection and reportmanagement);• Increased efficiency;• Intelligent resource allocation; and• Unknown numbers of prevented high-impact incidents.What might a detailed metrics-based ROI calculation look like?Bonnie Michelman, CPP, Director of Police, Security & OutsideServices, Massachusetts General Hospital, and 2001 President ofASIS International, describes a case study of a small corporationthat conducted 63 investigations one year (Michelman, 2011).Investigators recovered $1 million, and they estimate theyprevented $5.5 million in future losses. Looking at recoveriesalone (leaving out the value of future losses prevented), with aninvestigator cost of $250,000, the investigations provided an ROIof 300 percent.Risk management is theprocess of identifying andunderstanding applicablerisks and taking informedactions to reduce potentialfailure, achieve businessobjectives and decreasebusiness performanceuncertainty.Security’s valuable rolein risk management canbe demonstrated mosteffectively through metrics.CSOs can use MA to reduce a range of corporate risks. Campbell(2006a) identifies four categories of risks that businesses face:strategic, organizational, financial, and operational risks. Heobserves:It is only because there are unacceptable risks that the cost of asecurity program is tolerated. Risk management is the process ofidentifying and understanding applicable risks and taking informedactions to reduce potential failure, achieve business objectives anddecrease business performance uncertainty.The connection between security and risk management should bemade clear when CSOs inform executive management about theirprograms. Security’s valuable role in risk management can bedemonstrated most effectively through metrics.16 Metrics and Analysis in Security Management


Developing Specific MetricsWhat, exactly, CSOs should measure in their metrics and analysis(MA) efforts is not obvious. Choosing metrics requires carefulconsideration, as the metrics must be relevant to the particularorganization and its vulnerabilities. For example, in the context ofbusiness continuity planning, Heerwig (2011) observes:When you design metrics for your organization and present themto executive management, make sure you’re doing it as it appliesto your company. Don’t just go out and say, ‘This is the greatestthreat to industry today.’ Find out what affects your organization.PRODUCTSOFANALYSISBy analyzing collectedmetrics, a securityprofessional can:Typically, CSOs practicing MA collect and analyze a wide rangeof metrics. Doing so helps in developing an accurate pictureof what is actually happening in the organization. As Heerwigstates, “How do you measure ‘what if’?... It’s hard to reportmeasurements when sometimes the true measurement is thatnothing happened.” Using numerous indicators makes it possibleto compensate for the challenge of proving that security actionsprevented future losses.To create a list of metrics to collect and analyze, Treece andFreadman (2010) suggest that CSOs “list… core securitymissions and then determine what activities are involved ingetting those things done.”• Illustrate findings.• Issue reports.• Plan preventivemeasures.• Create correctiveaction summaries.Campbell (2006a) lists several hundred possible security metricsthat may be relevant to a company’s cost, risk, ROI, legal, policyand life safety issues. The following are just a few examples:• Make knowledge-based decisions.Security cost per square footLosses per square footSecurity cost per companyemployeeSecurity cost as percentage oftotal revenueIncrease or decrease ininsurance cost due tosafeguards or lossesTotal lossesNumber of employees involvedas subjects of investigationsas percentage of employeepopulationNumber of internalinvestigation subjects whoindicate a lack of knowledge ofthe policy they are accused ofviolatingNumber of hostile workplaceincidents in specificorganizational unitsInvestigative case aging• Demonstrate ROI.Metrics and Analysis in Security Management17


Cost of downtime in criticalbusiness processesInformation security violationsInventory shrinkageFines paid for false alarmsIncident response timesPercentage of personnel withknown derogatory backgroundissues hired versus not hiredNumber of personnel notbackground-investigatedbefore hireBackground investigation costper caseRate of unfavorablebackground investigationsPercentage of positivepreemployment drug testsNumber of businessrelationships establishedwithout due diligenceinvestigationDerogatory findings from postcontractaward examinationTerminations for cause asa percentage of employeepopulationInvestigations per investigatorRecoveries per investigatorNumber of nuisance alarmsSecurity personnel: hours ofpre- and post-assignmenttrainingNumber of securityvulnerabilities reported bypatrol officersNumber of workplace violenceincidents per X number ofemployeesPercentage of inactivecomputer user accountsthat have been disabled inaccordance with policyPercentage of mobileinformation devices withautomatic protectionPercentage of securityincidents that exploitedexisting vulnerabilities thathave known solutionsNumber of security incidentsthat should have been butwere not reported to securitydepartmentNumber of safety hazardsproactively identified andeliminated annuallyCampbell furtherrecommends that CSOsmaintain a “dashboard”of the most importantmetrics, a quick-view meansof gauging some of themost important securityconcerns in the company.These might be half adozen “survival metrics”—metrics that are vital tothe organization’s successor of special concern tomanagement.Campbell further recommends that CSOs maintain a “dashboard”of the most important metrics, a quick-view means of gaugingsome of the most important security concerns in the company.These might be half a dozen “survival metrics”—metrics thatare vital to the organization’s success or of special concern tomanagement. For example, a financial services company “mightbe particularly attuned to the number of business units with datedcontingency plans and inadequate software patch administration,internal misconduct or numbers of people hired with known18 Metrics and Analysis in Security Management


derogatory backgrounds.” The key to a good dashboard is to “selecta few key metrics we should watch because they are the things thatkeep us awake at night.” It may even make sense to maintain twodashboards—one for internal security use and one for monitoringissues that executive management cares about the most.An important distinction to understand when developing specificmetrics is the difference between leading, coincident and laggingindicators. A leading indicator suggests that the particularmetric will be followed by a particular (but different) condition;a coincident metric suggests, for example, that if one metric ishigh, another condition (perhaps not directly measured) is highat the same time. A lagging indicator may confirm that a certaincorrelated condition existed in the past (near or far) and may stillexist—the condition may not be easily measured, but the laggingindicator would prove that the condition did or does exist. In areport for the National Institute of Standards and Technology,Jansen (2009) writes:Analogous to economic indicators, security metrics may bepotentially leading, coincident, or lagging indicators of the actualsecurity state of the system. The distinction is significant… If alagging indicator is treated as a leading or coincident indicator, theconsequences due to misinterpretation and reaction can be serious.The longer the latency period is for a lagging indicator, the greaterthe likelihood for problems. That is, a lagging security metric witha short latency period or lag time is preferred over one with a longlatency period, since any needed response to an observed changecan take place earlier.Examples of leading and lagging indicators in security include thefollowing (Campbell, 2009):• Unresolved nuisance alarms: leading indicator of future risk.• Reduced false and nuisance alarm rates: lagging indicator ofefforts to improve alarm system reliability.• Hiring despite unfavorable background investigation: leadingindicator of integrity issue.• Reduction in number of security responders: possible leadingindicator of excessive response times.LEADING:A leading indicator suggeststhat the particular metric willbe followed by a particular(but different) condition.COINCIDENT:A coincident metricsuggests, for example,that if one metric is high,another condition (perhapsnot directly measured) ishigh at the same time.LAGGING:A lagging indicator mayconfirm that a certaincorrelated condition existedin the past (near or far)and may still exist—thecondition may not be easilymeasured, but the laggingindicator would prove thatthe condition did or doesexist.Of course, leading indicators—those “measurable factors thatchange before the risk starts to follow a particular pattern or trend”(Campbell, 2009)—do not justify an automatic response but mustbe analyzed carefully.Metrics and Analysis in Security Management19


Essential Ingredient: DataThe practice of metrics and analysis (MA) requires, as its basicingredient, data concerning security-significant issues. CSOs who arealready collecting data possess a valuable resource that they can mineto guide their decision making and gain support for their programs.For CSOs who are not already collecting data, or who are doing so inways that do not facilitate analysis, incident management softwareis one of the foundations for effective MA and an inherent part ofthe risk management cycle. Programs designed specifically for thesecurity field can make the gathering of security-significant dataorderly, convenient, and accurate—and hold the data in a formatthat facilitates analysis.For example, Perspective by PPM 2000 is security-focused incidentmanagement software that incorporates activity tracking, incidentreporting, investigation management and case management. Bypresenting users with carefully designed input forms and selectionfields, the program collects essential data uniformly and completely.In addition to being designed for the security field, it can be customconfiguredto a company’s particular needs and terminology based onindustry standards, legislation (such as Sarbanes-Oxley or the CleryAct) or corporate direction. In addition to having security personnelcollect data, the corporate security department may opt to enlistother divisions (such as health and safety, human resources, audit orIT) in collecting and tracking data. Inviting non-security employeesinto the process—as part of an enterprise-wide MA approach orsecurity awareness program—enables employees, either anonymouslyor not, from numerous sites to input incidents and other data,thereby casting a wider net in the data collection process.Programs designedspecifically for the securityfield can make the gatheringof security-significant dataorderly, convenient, andaccurate—and hold the datain a format that facilitatesanalysis.Security-focused incidentmanagement software offersboth the standardization andconsolidation of data. Bothof those features are vital forlater analysis and reporting.For example, Brian Tuskan, Microsoft Corporation’s Senior Directorof Global Security Technology & Investigations, notes (PPM 2000,2012):Perspective features a Web-based module (e-Reporting) sothat non-security employees can file incident reports online, bythemselves. We have a total workforce of over 90,000 full-timeemployees and thousands more that come and go through theMicrosoft campuses. With such large numbers, there are inevitablylosses, thefts and suspicious circumstances. With Perspective, all ofthese people have become part of the security reporting process.Security-focused incident management software offers both thestandardization and consolidation of data. Both of those featuresare vital for later analysis and reporting.20 Metrics and Analysis in Security Management


Such software can also automate the task of analysis, whichis addressed in the next section. For example, Perspective cangenerate customized statistical reports, trending insights andpredictive analysis.From Data to Information:Analyzing MetricsThe second part of a metrics and analysis (MA) program is,obviously, analysis, which is the stage that leads directly toimportant understandings that might not otherwise be possible.Treece and Freadman (2010) specify three reasons analysis isimportant:• It shows the accomplishments of the security mission.• It leads to an understanding of why specific metrics may bedifferent from one period to another. For example, metricsmay show that security guard overtime ran high in the previousquarter, but analysis may show that a large, aggressive politicalrally near the site necessitated extra security.• It provides the CSO with figures that can be used to gainsupport, resources or recognition for exemplary performance bysecurity staff.With the right analytical software, such as the business intelligencecomponents of Perspective, a CSO can easily and constantly analyzethe data on security activities, losses and investigations, viewing graphsand charts that are generated automatically. Statistics that could takedays or weeks to prepare using conventional database queries areavailable instantly, as all the formulas and queries are built in.For example, by using analytical software, a CSO can automaticallyretrieve core business statistics that answer questions like these:Three reasons why analysisis important:• It shows theaccomplishments ofthe security mission.• It leads to anunderstanding of whyspecific metrics maybe different from oneperiod to another.• It provides the CSOwith figures that canbe used to gainsupport, resourcesor recognition forexemplaryperformance bysecurity staff.• What types of incidents are occurring the most, and how muchare they costing the company?• Is the company on track to reduce incidents by 30 percent fromlast year?• Are losses for a particular site up, down or steady?• Where are incidents and losses occurring with the highestfrequency and greatest impact?• On what days and times might more security officers be required?• Have countermeasures instituted in the previous quarter takeneffect yet?Metrics and Analysis in Security Management21


The idea is to strategically analyze the metrics collected todevelop business intelligence.An especially valuable product of automated metrics analysis istrend spotting. Even with the right metrics on hand, identifyingtrends with the naked eye is not necessarily easy. Analyticalsoftware can bring trends to the foreground, helping a CSOidentify problems accurately and then implement suitablesecurity responses.For example, analysis of key metrics might tell the CSO thatlaptop thefts have been rising at one corporate site but notothers. Without analysis, a simple count of laptop thefts acrossthe corporation might not suggest any particular countermeasure.In fact, total corporate laptop thefts could be down, even thoughthefts at one site were up. Once the analysis pinpoints thespecific trend—that thefts are up at only one particular site—the CSO can intelligently choose the right steps to address theproblem, such as investigation, employee security awarenessbriefings or improved locking methods.Using the same example, analysis software can also point outcorrelations between the thefts and certain days of the weekor times of day, employee turnover or other factors that mightsuggest appropriate, tailored security measures.An especially valuableproduct of automatedmetrics analysis is trendspotting. Even with the rightmetrics on hand, identifyingtrends with the naked eyeis not necessarily easy.Analytical software can bringtrends to the foreground,helping a CSO identifyproblems accurately andthen implement suitablesecurity responses.Analysis software points out trends and correlations thatstrengthen decision making within the security operation. It alsoproduces charts and reports that are useful in demonstrating thevalue of the security program to others. Treece and Freadman(2010), describing metrics and analysis at the MassachusettsPort Authority, note:[O]ur metrics over the years... have grown to include some 229information line items that cover everything from direct andindirect security program costs to the uptime percentages ofkey security equipment, like surveillance cameras... [W]e use aquarterly Security Scorecard to show what we are getting for our(currently) $68 million annual investment in security... We havefound that collecting monthly and reporting quarterly allows us tohave month to month data in case we need it, while only having toproduce the reports four times a year.22 Metrics and Analysis in Security Management


Getting StartedThe case for using metrics and analysis (MA) in securitymanagement appears strong, yet it may be hard to overcomeinertia and start the process. Campbell and Blades (2009)identify two barriers to getting started: (1) no request fromexecutive management; and (2) budget concerns.The first potential hurdle—that management has not made aspecific request that the corporate security department initiatea metrics and analysis program—should be irrelevant to aprofessional CSO. Executive management may not know thatMA is the best way to obtain good results, but, as Campbell andBlades note,It does not matter why management is not asking us for metrics.We should be providing them. As the security experts, it is ourjob to manage risk and to inform management on our status. Weshould be taking metrics to them— we should not have to wait tobe asked.A second potential hurdle, the cost of undertaking the metricsand analysis approach, is also less of a challenge than it mightfirst appear. As Campbell and Blades (2009) observe,The first potential hurdle—that management has notmade a specific requestthat the corporate securitydepartment initiate a metricsand analysis program—should be irrelevant to aprofessional CSO.A second potential hurdle,the cost of undertakingthe metrics and analysisapproach, is also less of achallenge than it might firstappear.If you conduct afteraction reviews, if you speak to your peersabout trends and best practices, if you assess your risk on aregular basis, if you track project status or log incidents, youalready have the necessary data.Security-focused MA software can make the process efficientenough to obviate any need for a dedicated metrics-producingemployee. With the right software, data can be collected andinput by numerous staff members, leaving CSOs time to conductthe necessary analysis.Once the hurdles are recognized as irrelevant or insignificant,a CSO can take steps to start the MA program. Campbelland Blades (2009) list five key steps in implementing sucha program. The first step is to “identify the business driversand objectives for the security metrics program.” That meansconsidering the organization’s “goals, needs, values andpolicies.” This step also includes identifying the metricsprogram’s objectives, which could include reducing risk exposureor demonstrating the security department’s conformity withbusiness goals or its value or cost-effectiveness.Metrics and Analysis in Security Management23


The second step is to identify the various audiences for themetrics, as well as their business goals. Doing so can help increating specific metrics. For example,A metric that demonstrates a business unit’s inaction to correct aknown, reported vulnerability could be presented to the businessunit manager (to encourage [him or her] to correct the issue) or toan internal audit committee (to pre-emptively show that Securityreported the problem for correction).The third step is to list the types of data that the metrics andanalysis program will require. The objective is to line up data thatwill lead to actionable metrics—in other words, metrics that onecan analyze to “provide direction for decisions, affirm actionstaken, or provide clarity for next steps. Non-actionable metricssimply count things and have little value for influencing or findingcauses of risk.”The fourth step is to develop metrics that demonstrate thesecurity department’s contribution to enterprise risk management,the company’s overall strategy and objectives or, ideally, both.Risk-related metrics show how the security department reducesrisks to the business. Metrics focused on overall businessobjectives might show how new technology has reducedsecurity officer costs or how other security measures “remove avulnerability that could impact brand reputation and compromisecustomer confidence in [the company’s] products or services.”The fifth and final step is to treat the data carefully, ensuringits integrity and protecting its confidentiality. Security-focusedmetrics and analysis software is a significant aid in this step,organizing, protecting and ensuring consistency of collected data.Steps to starting a metricsand analysis program:1. Identify the businessdrivers and objectives.2. Identify the variousaudiences for the metrics,as well as their businessgoals.3. List the types of data thatthe metrics and analysisprogram will require.4. Develop metrics thatdemonstrate the securitydepartment’s contribution toenterprise risk management.5. Treat the data carefully,ensuring its integrity andprotecting its confidentiality.24 Metrics and Analysis in Security Management


ReferencesCampbell, George. (2006a). Measures and Metrics in CorporateSecurity: Communicating Business Value. Framingham, MA:CSO Executive Council.Campbell, George. (2006b). “How to Use Metrics.” CSO Online,August 1, 2006. http://www.csoonline.com/article/220980/how-to-use-metricsCampbell, George. (2009). “Metrics for Success: TrackingLeading and Lagging Indicators,” Security TechnologyExecutive, November 2009.Campbell, George, & Blades, Marleah. (2009). “Building aMetrics Program that Matters.” SecurityInfoWatch.com.http://www.securityinfowatch.com/Security+Executive+Council/1310623Carnegie Mellon University. (1995). “Security Metrics,” inSystems Security Engineering - Capability Maturity Model.http://www.sse-cmm.org/metric/metric.aspDaly, Ken. (2011). “Corporate Performance Metrics to Top BoardAgendas,” Financial Executive, January/February.Davenport, Thomas. (2009). “Make Better Decisions,” HarvardBusiness Review, November.Davenport, Tom, & Harris, Jeanne. (2010). “Analytics and theBottom Line: How Organizations Build Success.” Key LearningSummary published by Harvard Business Review.Harowitz, Sherry. (n.d.). “Challenges and Trends.” SecurityManagement online. http://www.securitymanagement.com/article/challenges-and-trendsHayes, Bob, & Kotwica, Kathy. (2011). “Benchmarks Aren’tMagic, They’re Tools,” Security, September.Heerwig, Klaus. (2011). Presentation within “The SecurityMetrics Challenge” at the 2011 ASIS International Seminarand Exhibits, September 20, 2011, Orlando, FL.Hodgin, Kim. (2011, July 26). Interview with author PeterOhlhausen.Metrics and Analysis in Security Management25


Jansen, Wayne. (2009). Directions in Security Metrics Research.Gaithersburg, MD: National Institute of Standards andTechnology, NISTIR 7564.Kohl, Geoff. (2009). “Measuring the Business Value of Security.”SecurityInfoWatch.com. http://www.securityinfowatch.com/root+level/1286197Kovacich, Gerald, & Halibozek, Edward. (2005). Security MetricsManagement: How to Manage the Costs of an Assets ProtectionProgram. Waltham, MA: Butterworth-Heinemann.Michelman Bonnie, CPP. (2011). “Business Case for Security:Creative Ways to Show Security’s Proposition and Profitability,”a presentation at the 2011 ASIS International Seminar andExhibits, September 19, 2011, Orlando, FL.National Institute of Standards and Technology. (2008).Information Security. NIST: Gaithersburg, MD.Musser, Raymond. (2011). Presentation within “The SecurityMetrics Challenge” at the 2011 ASIS International Seminar andExhibits, September 20, 2011, Orlando, FL.Payne, Shirley. (2006). “A Guide to Security Metrics.” SANSInstitute. http://www.sans.org/reading_room/whitepapers/auditing/guide-security-metrics_55PPM 2000. (2009). Webinar: INSTANT Business Intelligence andStatistical Reporting. http://www.ppm2000.com/downloads/Webinar-recording/Webinar-Focal-Point-Aug09.wmvPPM 2000. (2010). Webinar: Security’s ROI: How to DeliverMetrics That Reveal Trends and Justify Budgets. http://www.ppm2000.com/downloads/LiveSeminar/SecuritysROI.wmvPPM 2000. (2012). “Optimizing Resources, Shrinking Losses—Perspective at Microsoft’s Global Security Operations Centers.”http://www.ppm2000.com/downloads/Case_Studies/resources/case-studies/microsoft.aspShamess, James. (2011). Presentation within “The SecurityMetrics Challenge” at the 2011 ASIS International Seminar andExhibits, September 20, 2011, Orlando, FL.Treece, Dennis, & Freadman, Michele. (2010). “Metrics Is Not aFour-Letter Word,” Security, November.Wailgum, Tom. (2005). “Metrics for Corporate and PhysicalSecurity Programs,” CSO, February.Metrics and Analysis in Security Management26


27 Metrics and Analysis in Security Management


PPM 2000 Inc.10088 ‐ 102 Avenue, Suite 1307Edmonton, Alberta T5J 2Z11‐888‐776‐9776information@ppm2000.comwww.ppm2000.comCopyright © 2012 PPM 2000 Inc. All rights reserved.PPM 2000, the PPM 2000 logo and DispatchLog are registered trademarks of PPM 2000 Inc. Perspective by PPM 2000, the Perspective by PPM 2000 logo, Perspectivee-Reporting, Perspective Focal Point, Perspective Mobile, Perspective Visual Analysis and Perspective Workflow are trademarks of PPM 2000 Inc. Microsoft and the MicrosoftGold Independent Software Vendor (ISV) Partner logo are trademarks or registered trademarks of Microsoft Corporation in the United States and other countries. All other brands,names or trademarks mentioned may be trademarks of their respective owners. Printed in Canada 03/12.

Similar magazines