Leveraging Data Analytics in Federal Organizations - AGA

Executive SummaryData analytics is a powerful tool thatcan help government agencies reducefraud, waste and abuse. The commercialsector has used data analyticsfor years to improve decision making,achieve better financial outcomes andimprove customer service. The use ofdata analytics is growing at a rapid rate.The International Data Corporation, aprovider of market intelligence in theinformation technology field, estimatesthat the business analytics market forsoftware, hardware and consultingservices is expected to grow at an 8percent rate worldwide, reaching nearly$33 billion in 2012.AGA set out to determine howthe federal government is using dataanalytics and what it is doing with theresulting information. We interviewedeight agencies and surveyed a broadspectrum of federal financial officials.From this, we learned that some federalagencies have embraced data analyticsand have demonstrated the benefitsof integrating analytics tools into theiroperations. As a result, the federalgovernment is in a position to build onthe analytic advances it has alreadymade. Some organizations are poisedto share their capabilities with otherfederal organizations and possiblywith other levels of government thatimplement federally funded programs.However, there is no clear plan toleverage the government’s investmentin data analytics.From our agency interviews,we found:• The Food and Nutrition Service(FNS) within the U.S. Departmentof Agriculture reduced the rate offood stamp trafficking from about2.5 percent of food stamp benefitsto about 1 percent by using dataanalytics. (Trafficking is the buyingor selling of food stamp benefits forcash.) Based on benefit levels of $75.6billion in FY 2011, we estimate thatthis would translate into $1.1 billionof benefits that were not traffickedlast year. Its data analytics systemhas allowed FNS to quickly identifymerchants who traffic in food stampsand remove them from the program.The reduction in food stamp benefittrafficking helps ensure that funds arespent on their intended purpose.• In a matter of months, theRecovery Operation Center (ROC)at the Recovery Accountability andTransparency Board implementeda powerful data analytics systemto screen those who received muchof the $800 billion appropriatedin stimulus funds. Using its highpoweredsystem, the ROC identifiedrecipients who had previous brusheswith the law or who were receivingmultiple awards.• The Center for Program Integrity(CPI) at the Centers for Medicare& Medicaid Services developeda predictive analytics system forMedicare fee-for-service paymentsthat screens $450 million in Medicareclaims each day. The system is justbeginning to bear fruit but shouldhave a major impact on identifyingand reducing Medicare fraud.We also surveyed federal financialofficials on the development and use ofdata analytics in their operations.Two-thirds of the respondents reportedthe use of data analytics in operations,with nearly all systems focused onfinancial performance, improper paymentsand identifying high-risk investigativetargets. This is not surprising, asthe federal government has launcheda major initiative to reduce the annualrate of improper payments, currentlyestimated at $125 billion annually. Theremaining third of respondents, whosaid they had not implemented a dataanalytics system, identified multiplereasons for the delay. Sixty-sevenpercent of respondents to this questioncited a lack of budget, 53 percent citeda lack of staff and 33 percent said theywere unsure of how to start developinga data analytics system. (Respondentswere able to provide more than oneresponse.)4AGA Corporate Partner Advisory Group Research

Objectives, Scope andMethodology for the StudyThe objective of this study is toprovide the government accountabilitycommunity with information on thesuccessful development of data analyticssystems within a sample of federalagencies. We sought to identify commonfactors and characteristics of agenciesand programs that were successfulin establishing and implementing a dataanalytics project. Our study focused onmanagement and organizational issues,including agency or program leadership,funding, human capital for acquiring andsustaining the project, and the need forprogram and process reengineering.We also asked about the procurementmethods used, if any, for both softwareand consulting contractors. Other areasof inquiry included sources of data,change management issues, the basicdata analytics techniques employed bythe systems, how system results wereused and the success of each program.As part of our research, we interviewedofficials from eight offices withinsix federal agencies. We questionedofficials on a variety of topics includingthe factors that led them to concludethey should develop a system, the basicprocess they employed to develop thesystem, the contributors to their successand lessons learned. We also interviewedseveral firms that had conducteddata analytics work for federal agencies.We conducted interviews with thefollowing federal organizations:• Recovery Accountability andTransparency Board, RecoveryOperations Center (ROC)• U.S. Department of Agriculture, Foodand Nutrition Service (FNS)• U.S. Department of Defense, DefenseFinance and Accounting Service(DFAS)• U.S. Department of Defense,Defense Logistics Agency, Office ofOperations Research and ResourceAnalysis (DORRA)• U.S. Department of Defense,United States Navy, Naval SeaSystems Command, Office ofFraud Deterrence and Detection(NAVSEA-OFDD)• U.S. Department of Education, Officeof Inspector General(ED-OIG)• U.S. Department of Health andHuman Services, Centers forMedicare and Medicaid, Center forProgram Integrity (CPI)• U.S. Postal Service, Office ofInspector General (USPS-OIG)We also surveyed federal financialofficials including chief financial officers,inspectors general and deputy chieffinancial officers to gain information ontheir activities related to integrating dataanalytics into their operations.Our work did not concentrate on thetechnical aspects of the systems thatwere developed. We did not ask whetherthe systems were using a particularanalytical software package, were webaccessible and/or used predictive datamodeling of a certain type. Rather, wedealt with the overall processes usedto develop the system and attemptedto identify the common factors thatappeared to contribute to successfuldevelopment.Leveraging Data Analytics in Federal Organizations 7

Part One: Work Has Started in Federal AgenciesSurvey of FederalFinancial OfficialsWe also surveyed federal agencyofficials in February and March 2012 toascertain the extent and focus of dataanalytics activities in their agencies. Oursurvey was sent to the offices of inspectorsgeneral, chief financial officers andother federal financial officials. Resultsfrom the 39 responses we receivedhave been incorporated throughoutthis report. Our survey asked aboutthe extent to which data analytics hadbeen implemented in their operations,views on leadership support for theuse of data analytics and the levels ofexpertise in developing and using dataanalytics in the agency. We also askedquestions about any barriers that mighthave slowed the development of dataanalytics in agencies.The first survey question asked if anagency had implemented data analyticsto improve decision making. As shown inFigure 1, two-thirds of the federal officialsresponded that their agencies had.This response can be viewed asa positive sign for data analytics ingovernment. We also asked respondents,who could select more thanone answer, to identify their areas offocus. The answers indicate that dataanalytics have been implemented mostfrequently in four key areas:1. The detection and prevention ofimproper payments for IPERA2. Information on financialperformance and/or budgetingdecisions3. The identification of vendors orcontractors for further audit andinvestigation4. Staff deploymentHowever, it is equally importantto note that one third of the respondentsreported that their agency hadnot implemented data analytics toimprove decision making. We askedthese respondents what factors haveprevented them from proceeding withdata analytics. (Respondents were ableto provide more than one answer toour question.) The top three answersprovided were:1. A lack of budget resources (67%)2. A lack of appropriate staff (53%)3. An uncertainty as to how to developa data analytics system (33%)With budget limitations, a lack ofstaff resources and an uncertainty as thetop three factors preventing progresson data analytics, it makes sense toleverage the experience gained by thosewho have implemented data analyticssystems to help other organizationsovercome these obstacles.The continued integration of dataanalytics into all facets and levels ofgovernment has the potential to makea marked difference in governmentalaccountability and performance. Withthe effective use of data analytics, governmentswill be better able to demonstratethat they are monitoring funds in aprudent manner and are achieving effectiveresults. Increasingly, taxpayers wantto know that governments are spendingfunds wisely and that small problems arebeing identified before they become bigproblems. Today, improvements in datacollection, increases in Internet speedand decreases in data storage costs haveallowed information to be collected andstored closer to the time of its generation.These advancements will allowmanagers to have more timely informationand data on events and operationsto make better decisions.Figure 1: Has your federal agencyimplemented data analytics toimprove decision making?Yes67%No33%10AGA Corporate Partner Advisory Group Research

Part Two: What Is Data Analytics?The goal of the data analysis process(or data analytics) is to support decisionmakers by identifying patternsand trends and by highlighting usefulinformation. Data analytics can help toidentify and/or develop information thatwould otherwise not be discernible bysimply examining raw data.A variety of data analytics methodsare available; no one standard methodapplies to all situations. Rather, thecorrect data analytics method mustbe selected from among a variety ofpossibilities. Identifying the best optioncan be a complex task, and it may beuseful to use a team approach in makinga determination.In some cases, off-the-shelf software,like spreadsheet programs, mightbe sufficient. For example, a governmentmight determine which vendorsto audit by arraying the total dollarvalue of payments to each vendor fromhigh to low. The auditors could thenreview the list and select the top 10 vendorsfor an audit of payment accuracyand service quality.Data matching could also beconsidered another form of data analysis.Data matching involves finding similarentities across disparate databasesand matching the information to showexceptions. For example, a governmententity that pays for medical services forgovernment employees might match theclaims it receives for dental services to afile of licensed dental providers before itauthorizes payment. Unlicensed dentistsor mismatches would be subject to auditfollow-up or rejection. Governmentagencies have successfully used datamatching techniques for many years.Figure 2: Basic Steps in Data AnalysisDataCollectionDataCleanupInitial DataAnalysisRevisionof AnalysisBased onInitial ResultsOutput fromData AnalysisData analytics is a buzz word or catch phrase that has come to be associated with the field of data analysis.It generally refers to a variety of processes and techniques all focused on improving the value of informationfor decision makers. The basic process of analyzing data is depicted in the graph above.Leveraging Data Analytics in Federal Organizations 11

Part Two: What Is Data Analytics?However, these techniques can alsohave a downside by producing morefalse positives than the agency caneffectively handle. (A false positivewould be an exception that appears tobe an error based on an initial analysis,but is found not to be an error wheninvestigated.) If there are too many falsepositives, an agency’s staff will expendvaluable resources reviewing proper andcorrect transactions.False positives can be reduced byusing more advanced forms of dataanalytics. This requires better data,better systems and a more highlytrained staff. Shown here is a list of dataanalytics techniques Elder Research,Inc., has used in providing training forFigure 3:AnalyticTechniquesDescriptive1. Standard Reporting2. Custom Reporting or“Slicing and Dicing”the Data (Excel)3. Queries/Drilldowns(SQL, OLAP)4. Dashboards/Alerts(Business Intelligence)5. Statistical Analysis6. Clustering (UnsupervisedLearning)Predictive7. Predictive Modeling8. Optimization and Simulation9. Next Generation Analytics:Text Mining and Link Analysis 3AGA. (Elder Research officials said thatthe list was based on the Eight Levels ofAnalytics identified by the SAS InstituteInc.) The list classifies the data analyticstechniques as either descriptiveor predictive. Descriptive techniquesare those that describe the populationbeing examined. Predictive techniquesuse patterns discovered by examininghistorical data to identify previouslyunidentified risks and opportunities.The list arranges data analyticstechniques from the most elementarylevel, number 1, to the more advancedlevel, number 9. Standard Reporting,listed as number 1, includes any reportsgenerated by a system intended formanagement use in monitoring operations.For example, a standard reportfrom an accounts receivable systemwould be a list of overdue accountsreceivable. Management would use thistype of standard report to determinewhether overdue accounts receivableis growing or remaining stable and toidentify which accounts should be sentto collection. This would constitutedata analytics because it providesmanagement with information formaking decisions based on insightsthat are not discernible by simplylooking at individual transactions. Thestandard report is just the first step indata analytics. As the analysis becomesmore sophisticated, management maydevelop special queries or drill-downs toproduce more useful information, suchas the geographic location that producesthe highest level of overdue accountsreceivable.Predictive techniques such as predictivemodeling, link analysis and textmining are more advanced techniquesthat are designed to predict outcomes.Predictive data analytics methods arecommonly used in the computation ofcredit scores. A credit score assigns anumerical rating to an individual thatwill predict the risk of default associatedwith loaning the person money.Data mining, another term frequentlyassociated with the field ofdata analytics, is both a descriptive andpredictive data analytics technique.Data mining is defined as the examinationof large sets of data, using a varietyof techniques to discover patternsin data that were unknown to users.For instance, a clustering data miningtechnique would be used to discovertransactions that appear to fall outsideof the population’s norms. Clusteringtechniques assign data into clustersso that the objects are grouped similarlyby a number of characteristics.This same analysis would identifyobjects that fall outside of the clusters.Researchers then use the informationlearned from investigating the results tofurther refine the data analysis. Figure 4illustrates the typical data mining cycle.12AGA Corporate Partner Advisory Group Research

Part Two: What Is Data Analytics?Figure 4: Data Mining CycleBusinessUnderstandingDataUnderstandingData PreparationDataDeploymentEvaluationModelingAs depicted in the graphic above, data mining work starts with the organization gaining an understanding of its businessand the associated risks. The next phase involves understanding the data and its qualities, quantities and sources. Datapreparation for analysis includes cleaning to identify and correct errors. Data analysis follows, and exceptions (or outliers)are identified. Exceptions are evaluated, frequently in some form of investigation or audit, to confirm that they are trueexceptions. The results of the evaluation phase are then factored back into the process to further refine the results and themodels.Leveraging Data Analytics in Federal Organizations 13

Part Three: Key Questionsfor ImplementersThis part examines key questionsthat organizations should considerwhen designing, developing andimplementing a data analytics system.The questions, which are based on lessonslearned from our interviews andthrough our survey, are:• What is the goal of the data analyticsproject, and is leadership committedto the project and its goals?• What data will be needed for thedata analytics work, and is thisdata available? In addition, whatchallenges must be overcomebefore the data can be effectivelyanalyzed?• Does our staff have the knowledgeand skill sets needed to design,develop and implement our analyticsprocesses and systems?• Has the agency considered the needfor cultural and organizational skills?• Does the organization have systemsand software to implement asolution, or will they need to beacquired?We begin this part by providingdetail on the oldest system among thosewe examined — the ALERT systemdeveloped by the Food and NutritionService (FNS) of the U.S. Department ofAgriculture. We chose ALERT as an illustrativeexample because it is a highlymature system. Based on the ratingsbelow, the ALERT system operates at ahigh level with respect to both its levelof maturity and the degree to which FNSrelies on its results.Level of System MaturityHigh level — Analytics are wellestablished, and the results are trustedby decision makers and integrated intooperations.Medium level — Analytics are stillbeing refined, results are still subject tocaution and integration into operationsis still in process.Low level — Analytics have beendeveloped, testing is in process andintegration is at a low level.The ALERT System atFNS: An Early DataAnalytics ProjectThe Supplemental NutritionAssistance Program (SNAP), formerlyknown as the Food Stamp Program,has existed in various forms since theearly 1960s. 4 To understand the issuesthat organizations typically weigh whendeveloping data analytics processes andsystems, we began by examining howFNS developed its Anti-Fraud Locatorusing the EBT Retailer Transactions(ALERT) system. While data analyticsmay be used for a variety of purposes,most agencies’ first systems are generallygeared toward detecting fraud orreducing erroneous payments — paymentsmade in the wrong amount, to thewrong person or at the wrong time.In the 1980s, FNS began testing ofan electronic benefit transfer system(EBT) to replace the antiquated papercoupons for distributing benefits. EBTwas developed to use the infrastructurealready being developed by thecommercial sector for processingdebit and credit card transactions atvarious retail outlets. The EBT systemwas deployed on a state-by-state basisacross the United States. Marylandwas the first state to fully implementan EBT card system, going operationalin 1993. Deployment continued acrossthe country with full deployment by2004, when California came online.EBT was designed in part to eliminatethe stigma attached to receiving SNAPbenefits and to reduce administrativecosts associated with printing, storing,distributing and accounting for thepaper coupons. It was estimated thatSNAP was handling 1 billion pieces ofpaper annually; consequently, backroomoperations were costly to operateand provided very little information onwhere benefits were redeemed.When states began issuing EBTcards, recipients no longer had to standin line each month to obtain their benefits.Like a debit card, an EBT card issecured by a PIN; benefits are automaticallyavailable via the EBT card each14AGA Corporate Partner Advisory Group Research

Part Three: Key Questions for Implementersmonth. EBT is a savings for governmentbecause coupons — which were like asecond form of currency that was usedonce and then destroyed — did nothave to be delivered to distribution sitesvia armed guard and secured in vaults.Merchants like the system because theydo not have to count and deposit thecoupons.Along with the development of theEBT system, FNS modernized the systemfor enrolling merchants in SNAP.This new enrollment system improvedthe agency’s ability to screen prospectiveSNAP merchants, facilitating thedetection of merchants who had priordifficulties with the program. The newenrollment system also provided betterdemographic information on each merchant,the merchant’s location, numberof outlets, volume of sales, etc. Thismerchant enrollment system, known asthe Strategic Tracking and RedemptionSystem (STARS), combined withthe EBT system to provide FNS withtransaction-specific data including, butnot limited to, a merchant’s identificationnumber, transaction date and time,and amount of the transaction. FNSobtains transaction information dailyfrom the three contractors that operatethe states’ EBT systems.During development of the EBTsystem, FNS personnel at the regionaland central offices recognized that thisnew system would provide informationthat could be valuable in monitoring theprogram. FNS monitors a number ofrisks associated with SNAP benefits. Themost basic challenge for the program isensuring that recipients are eligible. Thistask is handled by the states that enrollrecipients and monitor their continuingeligibility. However, other risks involveeligible recipients using benefits topurchase illegal items such as tobacco,and benefit trafficking whereby papercoupons are sold by benefit recipientsto individuals or merchants for cash andnot exchanged for eligible food items.FNS has been successful with implementingits ALERT system in large part becauseit had the support of mid-level managerswho understood the power inherent intransaction-level data.Before the advent of EBT, it was nosecret that trafficking was a problem.Investigators and the press reportedthat it was not unusual for coupons tobe sold by some recipients for 50 centson the dollar. Trafficking was moredifficult to detect in the paper couponsystem because information was notavailable for investigators or auditorsto analyze. With the advent of EBTbenefits, the methods for traffickingchanged because benefits were providedelectronically. What FNS neededwas a system to analyze the data andprovide this information for investigationby FNS staff in various regionaloffices. This resulted in a project thatwould develop the ALERT system.ALERT took more than two years todevelop and the process was evolutionary.FNS used a continuous feedbackloop in developing the system, wherebymodels were updated based on theresults of investigations. This helpedthem improve the system overall. Forthe first time, people were able to seewhere the money was spent, when itwas spent, and how much was spent ona comprehensive basis. In short, ALERTfacilitated “market research” by FNSpersonnel.The system was developed througha team of in-house business expertsfrom FNS regional offices, investigativestaff and outside contractors. FNS stafffrom regional offices and the centraloffice combined to provide the businesstalent for the ALERT developmentteam. FNS staff had knowledge of thetypes of problems they were seekingto identify, what patterns and trends inthe data could indicate that a vendormight be trafficking in benefits, andwhich business rules, if violated,would be problematic. This knowledgewas translated by the FNS staff andcontractors into automated systemsthat could screen and analyze the EBTdata. The system looked for variouspatterns, computed results for allEBT merchants, and provided a list ofexceptions for follow-up. Initially, thecontractor provided a wide range ofservices for FNS, from developing thecomputer programs to performing theanalyses to hosting ALERT on its owncomputer system. As it has evolved,FNS now operates ALERT on its ownsystems. The contractor continuesto provide system maintenance andenhancements.Our discussion with FNS personnelrevealed several factors that appearcritical to the initial and sustained successof ALERT. FNS has been successfulwith implementing its ALERT system inlarge part because it had the support ofmid-level managers who understoodthe power inherent in transactionleveldata. With the advent of EBT, theagency had access to a large amount ofdata, and because management understoodits value, they were supportive ofbuilding a system to analyze the dataand reengineering processes to betterwork with this new information. FNSLeveraging Data Analytics in Federal Organizations 15

Part Three: Key Questions for ImplementersMining Group, the inspector general ledthe efforts at their office and remainsactively involved in the system’sdevelopment. He served as a championfor the project — monitoring the work ata high level and providing a clear visionfor the system. He also reinforced theneed for RADR throughout his organization,and assisted in overcomingproblems and promoting collaborationwith other USPS-OIG departments.U.S. Department of Education,Office of Inspector GeneralThe Department of Education’sOffice of the Inspector General (ED-OIG)also had a clear goal and champion forits project at the top of the organization.The project’s goal is to providesupport for the audit and investigativeoperations by using predictive modelsto identify audit and investigativetargets. One part of the system is usedto identify higher education studentaid fraud. It analyzes a wide variety ofinformation and ranks transactions forinvestigative follow-up. The secondpart of the system analyzes informationrelated to federal funds given to localeducation agencies. The analysis resultsin rankings that are used to select localagencies for audit. ED-OIG officials saidchampions of the data analytics projectinclude Inspector General Kathleen S.Tighe and Assistant Inspector Generalfor Information Technology Audits andComputer Crime Investigations CharlesCoe. Mr. Coe, recognizing that analyticalprojects require a multiyear effort,secured support from the top levels ofthe OIG, to ensure that these projectscontinue to move forward.Centers for Medicare and MedicaidServices, Center for Program IntegrityAt the Centers for Medicare &Medicaid Services (CMS), a clear goal forthe project was set by the passage of theSmall Business Jobs Act of 2010 (SBJA),and strong support for the work was providedby CMS leaders. The SBJA, signedinto law in September 2010, contained aFigure 5: Excerpt —Small Business Jobs Act of 2010(a) USE IN THE MEDICARE FEE-FOR-SERVICE PROGRAM.—The Secretary shall use predictive modeling and other analytics technologies(In this section referred to as ‘‘predictive analytics technologies’’)To identify improper claims for reimbursement and to Prevent the payment ofsuch claims under the Medicare fee-for service Program.(b) PREDICTIVE ANALYTICS TECHNOLOGIES REQUIREMENTS.—The Predictive analytics technologies used by the Secretary shall—(1) capture Medicare provider and Medicare beneficiary activities across theMedicare fee-for-service program to provide a comprehensive view across allproviders, beneficiaries, and geographies within such program in order to—(A) identify and analyze Medicare provider networks, provider billingpatterns, and beneficiary utilization patterns; and(B) identify and detect any such patterns and networks that represent a highrisk of fraudulent activity;(2) be integrated into the existing Medicare fee-for-service program claimsflow with minimal effort and maximum efficiency;(3) be able to—(A) analyze large data sets for unusual or suspicious patterns or anomaliesor contain other factors that are linked to the occurrence of waste, fraud,or abuse;(B) undertake such analysis before payment is made; and(C) prioritize such identified transactions for additional review beforepayment is made in terms of the likelihood of potential waste, fraud, andabuse to more efficiently utilize investigative resources;(4) capture outcome information on adjudicated claims for reimbursementto allow for refinement and enhancement of the predictive analyticstechnologies on the basis of such outcome information, including postpaymentinformation about the eventual status of a claim; and(5) prevent the payment of claims for reimbursement that have been identifiedas potentially wasteful, fraudulent, or abusive until such time as the claimshave been verified as valid.specific provision, Part II, Section 4241,titled “Use of Predictive Modeling andOther Analytics Technologies to Identifyand Prevent Waste, Fraud and Abuse inthe Medicare Fee-for-service Program.”This provision provided CMS with a cleardirection to develop a predictive analyticsystem that would prevent the paymentof potentially wasteful, fraudulent orabusive claims. The law clarified that thesystem was to implement real-time, prepaymentclaims analysis that would allowthe agency to identify fraudulent claimsprior to payment. The SBJA formallydirected CMS’s transition from a systemthat pays Medicare claims and thenseeks to recover improper payments,often referred to as “pay and chase,” toa system that enables CMS to preventpayment of bad claims by screening andLeveraging Data Analytics in Federal Organizations 17

Part Three: Key Questions for Implementersanalyzing all Medicare Fee-for-serviceclaims prior to releasing payment. Thelaw further mandated that CMS implementthis system to screen Medicareclaims for 10 states by July 1, 2011.CPI officials said HHS and CMSleadership provided strong support forthe project. Dr. Peter Budetti, DeputyAdministrator and Director for ProgramIntegrity, ensured that resources werededicated to this critical objective. Hewas personally and visibly involvedwhile providing strategic directionthroughout the implementation. Heencouraged all components withinCPI to work together and participatedin business process re-engineeringsessions to develop appropriateresponses for working with the resultsof this new transforming technology. Hislead-by-example approach supportedCMS’s strategy to realign the internalorganizational structure the previousyear, consolidating the Medicare andMedicaid program integrity groupsunder a unified CPI. In a series ofCongressional hearings in the spring of2011, Dr. Budetti continually stressed theimportance of these new data analyticsprograms, stating that “[g]iven thechanging landscape of health-care fraud,any successful technology will needto be nimble and flexible, identifyingand adjusting to new schemes as theyappear.” Dr. Budetti also testified shortlyafter the implementation of the new dataanalytics tools on July 12, 2011, stating“[t]he new authorities given to us byCongress and the experience of privatesector industries in combating fraudhave greatly enhanced our capacity tocarry out this task.”The implementation of new dataanalytics technology has receivedsignificant support from the highestlevels of the Department of Healthand Human Services. HHS SecretaryKathleen Sebelius announced the launchof the new technology at a joint fraudsummit with the Department of Justice.The Secretary announced that “[a]s longas we continue to aggressively put thesetools to work preventing and prosecutingfraud, we can continue to protect andstrengthen Medicare’s future.” Then-CMS Administrator Donald Berwick, MD,also noted that “[t]his new technologywill help us better identify and preventfraud and abuse before it happens andhelps to ensure the solvency of theMedicare Trust Fund.”Recovery Accountability andTransparency BoardThe American Recovery andReinvestment Act of 2009 (ARRA)created the Recovery Accountabilityand Transparency Board (RATB) withtwo goals in mind: to provide transparencyof recovery-related funds, and todetect and prevent fraud, waste andmismanagement. The board originallyincluded 13 members, with a chairpersonappointed by the president and 12inspectors general. ARRA containedambitious transparency and reportingrequirements. Reporting was doneonline and published by the RATB on itspublicly accessible website, Recovery.gov. The RATB, under the leadership ofthe chairperson, formed the RecoveryOperations Center (ROC) and tasked itwith quickly developing an advanceddata analytics system to aid in theprevention and detection of ARRA fundfraud, waste and mismanagement. Theproject required the RATB to develop ina very short time frame a data analyticssystem that could screen the recipientsof ARRA funds for past problems as wellas identify new, potential problems withrecipients of ARRA funds. ROC officialssaid the chairperson’s leadership wasinvaluable to the project’s success. Thechairperson remained actively involvedin the ROC’s development and laid outthe vision for the ROC’s work.Naval Sea Systems CommandThe U.S. Navy’s Naval Sea SystemsCommand (NAVSEA) is beginning anambitious project to develop a data analyticssystem to detect fraud and abuse.High-level leaders in NAVSEA havemade the data analytics project a highpriority and clearly communicated thisto the entire command. The high priorityof the system has been demonstratedthrough the creation of the Office ofFraud Deterrence and Detection (OFDD)within NAVSEA. One of OFDD’s principaltasks is to develop a data analyticssystem that will help NAVSEA identifypotential procurement fraud for furtherinvestigation.Our interview revealed that leadershipsupport has been a significant factorin the project’s progress. NAVSEA’sleadership has made the implementationof the data analytics program oneof the top three priorities in its plans forFY 2012 and 2013. It received this highranking because OFDD’s data analyticsproject is in alignment with NAVSEA’sCORE goals of safeguarding taxpayerfunds and maintaining good stewardshipof its resources. NAVSEA’s leadershiphas also supported the project byimplementing a mandatory contractsreview process that has reshapedNAVSEA’s procurement practices for thebetter.This reshaping allows OFDD toobtain critical procurement insight andaccess to the data needed to performdata analytics. Because OFDD, NavalCriminal Investigative Services and NavyAudit all investigate and audit NAVSEAoperations, they have developed a cooperativeagreement to avoid conflicts intheir respective roles and the duplicationof effort. The agreement has fostered astrong, cooperative working relationshipamong the three units.Survey Results on LeadershipAll of the federal officials we interviewedemphasized the value of settinggoals and the need for leadership to seta positive tone at the top. They notedthat leadership, whether coming froma vision at the top, a strategic planningprocess or mid-management, wasessential. Discussions with firms that18AGA Corporate Partner Advisory Group Research

Part Three: Key Questions for Implementershave worked on data analytics projectsin both the private and the public sectorsconfirmed the need for strong leadershipthat will champion the project andarticulate its vision and goals. However,less than one-fourth of our surveyrespondents indicated that leadershipsupport for using data analytics washigh. While it appears most governmentleaders have a general understanding ofdata analytics, they are not yet championsof the process.Over two-thirds of our respondentsrated leadership’s support for using dataanalytics at the medium level, meaningthat they exhibit a general understandingof what analytics mean and howthey can assist an agency. It would seemthat federal leaders must continue to beeducated on the benefits, the uses of,and the actions needed to implementdata analytics work in their agency.How would you rate youragency in the area of leadershipsupport for usingdata analytics?High level —Champions ofanalytic processes orsystems, with deepanalytical experienceand background.Medium level —General understandingof what analyticsmeans and how theycan assist the agency.Low level — No clearunderstanding ofwhat analytics meanand how they can beused.23%68%9%While it appears most government leaders havea general understanding of data analytics, theyare not yet champions of the process.Question: What data will be needed for the dataanalytics work, and is this data available? In addition,what challenges must be overcome before the data canbe effectively analyzed?The development of a data analyticsprocess or system begins with a clearvision of the desired goals, and progressesto an assessment of what datais available for analysis and whetherthis data will be sufficient to meet theagency’s goals. If it is determined thatsufficient data is not available, agenciesmust then determine where they willgenerate or obtain it (another agencyor a commercial vendor might have theneeded information). In addition, datamust be cleansed to ensure it is useful.(See Figure 2 on page 11.)U.S. Department of Agriculture,Food and Nutrition ServiceThe ultimate design of ALERT wasinfluenced by the availability of dataalready, or soon to be, in the possessionof FNS. FNS was able to balance thesystem’s goals with the informationreadily available. Data collected fromthe new EBT system and the new vendorenrollment system, STARS, provided thedata for analysis in ALERT. This allowedFNS to proceed in development with thecertainty that sufficient data would beavailable. For example, hypotheticallyFNS might have sought informationfrom the Internal Revenue Service thatcould assist in its analysis; yet access tothis information is highly restricted andmay have been impossible to obtain,which could have delayed the project.FNS is not alone in data selfsufficiency.Our interviews with officialsin other agencies disclosed that almostall of them were able to develop theirsystems solely using internal data. Fewhad to obtain information from anothergovernment agency or a commercialvendor. The chief exception to dataself-sufficiency was the need to obtaincorporate information, like the nine-digitDun & Bradstreet D-U-N-S number thatuniquely identifies an individual business.Data self-sufficiency saves timeby allowing development work to moveahead without having to negotiate withother government agencies, seek legislativeauthorization to access governmentinformation or purchase the informationfrom a commercial vendor.U.S. Department of Education,Office of Inspector GeneralThe ED-OIG collects a great deal ofinformation and is an excellent exampleof an organization that can primarilyrely on data from within its agency. TheED-OIG collects information from eightdifferent Department of Education datasources including systems for processingstudent aid applications and studentloan information, as well as paymentinformation from the accounting systems.The ED-OIG has an experiencedanalytic team and understands the valueof information available internally. Thisawareness allows them to focus on theLeveraging Data Analytics in Federal Organizations 19

Part Three: Key Questions for Implementersmechanics of collecting and preparingthe data for analysis.The complexity of ED-OIG’s datasources are identified in a chart providedby Edward Slevin, director ofthe Computer Assisted AssessmentTechniques Division at the ED-OIG.Combining information from a largenumber of data sources into a singledatabase for analysis can presentsome unique challenges in the datacleanup phase of a project. Some ofthe information in ED-OIG’s system isnon-numeric, such as street addresses.If one part of the system’s analysis is tomatch addresses across records, thiscan be problematic. For example, anaddress could be identified on recordsranging from such representations as1st Street, First Street or Furst Street inthe database. The ED-OIG solved thisproblem by employing a solution fromthe U.S. Postal Service to clean up andstandardize addresses. Another usefulaspect that was built into this system isinvestigator email alerts. For example,if an investigator in San Franciscoconducted a series of analytical querieswith regard to a particular lead, andmonths later another investigator —say in Miami — conducted a series ofanalytical queries surrounding the sameset of data, the ED-OIG’s system wouldgenerate an email alert to the originalinvestigator advising of this mutualarea of interest. This has proven to beextremely valuable in terms of increasingthe effective use of limited investigationresources.Figure 6: DATA SOURCES FOR ED-OIG’S SYSTEMAuditorsinvestigatorsState LocalRisk ModelState LocalSingle AuditG5PostSecondaryRisk ModelFraud RingsDataAnalyticsK thru 12Data MartSchoolSummarySystemTitle IVData MartStudentLookupSystemState LocalE-FraudData MartODASWarehouseARRA PEPS G5 NSLDS EDEN AARTSSingleAudit D & B PINServiceProviders20AGA Corporate Partner Advisory Group Research

Part Three: Key Questions for ImplementersU.S. Postal Service,Office of Inspector GeneralUSPS-OIG began developing theRADR system in 2009 with a similarapproach. The RADR system is designedto examine transactions related to healthcare, contract, mail and financial transactionswith the aim of identifying thosethat have a high probability of fraud orabuse. USPS-OIG developed the RADRsystem to analyze information alreadyavailable to USPS, including informationthat was made available by the U.S.Department of Labor.Defense Logistics AgencyIn a similar manner, the DefenseLogistics Agency (DLA) Office ofOperations Research and ResourceAnalysis (DORRA) established a dataanalytics system to assist DLA leadershipin managing operations using informationthat was available from its existingsystems such as the Enterprise ResourceSystem. DLA provides a full spectrumof logistics, acquisition and technicalservices to the military, providing nearlyall of the consumable items used bymilitary forces. This includes such thingsas food, fuel, uniforms, medical supplies,construction equipment and spare parts.DLA established a data analytics systemto aid in managing its operations. Thesystem provides performance metrics ona routine basis to agency managementand leadership. The number of metricsand the detail provided to managementor leadership varies depending upontheir responsibility. DORRA officialssaid some managers or leaders receiveinformation on as many as 50 metrics touse in managing operations.Centers for Medicare and MedicaidServices, Center for Program IntegrityCPI’s systems to review Medicarefee-for-service payments and providerenrollment applications also hadsufficient data for the data analyticsproject. The Fraud Prevention Systemthat analyzes fee-for-service paymentclaims, deployed nationwide on June30, 2011, uses information from theMedicare payment processing systemand other CMS systems for performingthe analysis. CMS officials said obtainingdata for analysis was not a problem,but that handling the volume of thedata presented challenges becausethey receive about 4.5 million Medicarefee-for-service claims each day.Defense Finance andAccounting ServiceSimilarly, the Defense Finance andAccounting Service (DFAS) had informationreadily available for data analysisbut lacked the means to draw theinformation together before it implementedthe Business Activity Monitoring(BAM) system in 2008. This system wasdesigned to focus on reducing improperpayments through data matching andanalysis. BAM is a commercial off-theshelf(COTS) system that interacts withexisting payment systems to integrateinformation, analyze it, and createexceptions for follow-up and correctionbefore payments are made. BAM alsoallows DFAS personnel to modify andcustomize the various analysis routines.Question: Does our staffhave the knowledgeand skill sets neededto design, develop andimplement our analyticsprocesses and systems?It is critical for agencies to assesstheir in-house talent. Our interviewsrevealed that a variety of talent isneeded to develop, design and implementa data analytics system. Agenciesshould anticipate that they will needboth those with business expertise andthose with analytical expertise.Business expertise was generallydefined as expertise in the programsand areas upon which the system willfocus. These individuals need to havean expert knowledge of the business(program) being analyzed, the rules thatgovern the program, past problemsnoted in the program, any risks to theprogram and knowledge of the dataavailable for the project.Analytical expertise was generallydefined as expertise in the field of dataanalysis. This included individuals withexpert knowledge in fields such asstatistics, data analysis and data modeling.This also included expertise in usingsoftware or developing computer codeto analyze data.Needed expertise loosely breaksdown as follows:Business expertise means experience,knowledge or training in:• The agency and the programbeing analyzed• Regulations governing the program• Past problems and risks associatedwith the program• The type of data being collectedby the agency or is available fromother sources• The quality of the data beingcollectedAnalytical expertise meansexperience, knowledge or training in:• Statistics, mathematics and dataanalysis• Software design• Analyzing data and developingpredictive models• Developing computer codeIn nearly all cases, the federalagencies interviewed for this researchprovided the business expertise forthe project and hired a contractor toprovide most, if not all, of the analyticalexpertise. Generally, expert staff fromunits within the agency were assignedto work on the projects with contractorpersonnel. The contractor provided staffthat were trained and had experience instatistics, data mining, data analyticaltechniques and the various softwarepackages that are used to analyze andLeveraging Data Analytics in Federal Organizations 21

Part Three: Key Questions for Implementersdisplay data. The result is a collaborativeapproach with each group bringing inthe talent that is needed for the work.Our survey of federal officials alsoconfirmed that contractors were used toassist in the development of data analytics,with 50 percent of the respondentsstating that contractors assisted them inthe development of data analytics.U.S. Department of Agriculture,Food and Nutrition ServiceFor example, at FNS, expert staff fromthe regional offices and the headquarterswere assigned to the ALERT project towork with a vendor hired to develop thesystem. FNS staff provided insights intowhat types of transactions or patterns oftransactions could be indicative of traffickingin SNAP benefits. The contractorstaff then developed computer routinesthat searched for more than five differentpatterns in the EBT data. The resultingdata was accumulated by merchant andranked using a risk weighting systemdeveloped by FNS staff. The results wereprovided to FNS field staff for investigationand validation, with the actualinvestigation results used to furtherrefine the ALERT system.U.S. Department of Education,Office of Inspector GeneralSimilar methods were used in theED-OIG. Business talent was providedby the ED-OIG staff with experience ininvestigations and audit, which providedinsights into what types of issues andproblems they were seeking to detect.They also assisted in reviewing andrefining the data analytics system toprovide better results.Centers for Medicare and MedicaidServices, Center for Program IntegrityWe found that the CMS Center forProgram Integrity used a somewhatdifferent approach to procuring thetalent it needed to complete its project.CMS created the CPI to serve as thefocal point for all programs that dealwith the prevention and detection offraud and abuse in the Medicare andMedicaid programs. CMS had developeda strategic plan to reorganize and begindevelopment on data analytics, workthat was switched into high gear inSeptember 2010, when President Obamasigned into law the Small Business JobsAct. The law required that CMS issue arequest for proposals and contract withprivate companies to conduct predictivemodeling and other analytics technologiesto identify and prevent paymentof improper claims submitted underMedicare Parts A and B. The deploymentwas to encompass the claims from atleast 10 states and be online by July 1,2011, a 10-month time frame.CPI began development of twodata analytics systems, one to reviewMedicare fee-for-service claims calledthe Fraud Prevention System (FPS) andanother to automate the process forscreening Medicare provider enrollmentFigure 7: cpi roadmapVendor EducationVendor Submissionof CapabilityStatementsReview SolutionsVendorPresentationsVendor Day:Overview of CPIrequirementsVendors submit capabilitystatement for a specificarea1. Screening2. Predictive modeling3. Case management4. Data integration5. OtherCPI reviews and selectscapabilities statements ineach areaOne-on-one meetings,vendor presentation anddiscussion of capabilitiesstatementNext stepsDetermine Acquisition Strategy:F & O Competition8(a) set aside existing IDIQ contracts, etc.22AGA Corporate Partner Advisory Group Research

Part Three: Key Questions for Implementersapplications. The fee-for-service systemfor Medicare claims would representa major change for CMS processing ofMedicare claims. In 2010, many of theadvanced controls over Medicare claimswere oriented toward a “pay and chase”system. The law mandated that CMSwould implement predictive analyticsand place more emphasis on the preventionof problems while still providinginformation for the pay-and-chasesystem controls. The CPI data analyticssystem was developed in a short periodof time by CPI using an approach thatcombined outside business and dataanalytics talent with in-house talent inthe same areas.With a tight time frame, CPI wasforced to think creatively in acquiringtalent and skills for the work. Througha two-phase “Industry Day” initiative,CPI engaged contractors to assist inthe development of the data analyticssystems. Figure 7 from the CPIIndustry Day presentation illustratesthe approach to securing contractors toassist in system development.As shown in Figure 7, CPI providedinformation to vendors on CPI’s requirementsand asked interested vendors tosubmit a statement through a Requestfor Information process as to their firm’sability to meet CPI’s needs in any oneof five areas. CPI staff reviewed theproposed solutions and subsequentlyused these capabilities to inform therequirements. CPI acquired both businesstalent and analytical talent withthe contracts. CPI officials said that thecontractors brought in not only expertsin analyzing data and developing predictivemodels but also experts in the fieldof health care and health-care fraud towork on the project. CPI also includedCMS expert staff in the project, includingexperts in the areas of health care,Medicare and Medicare fraud.CMS deployed the Medicare fee-forservicesystem nationwide on June 30,2011, three years ahead of schedule.This exceeded the requirements ofthe Small Business Jobs Act, whichmandated deployment in only 10 statesby July 1, 2011. CPI officials emphasizedthe importance of the Industry Dayapproach to their success. They feltthat Industry Day allowed them tosecure well-qualified contractors thatunderstood their priorities and visionfor the future, which in turn facilitatedcontractor interaction in the “CommandCenter” with inside experts to producea system quickly and effectively.Survey Results on Knowledgeand Skill SetsOur survey of federal officialsconfirmed that assistance is neededin data analytics, as the skills of theagency’s current work force were notrated at a high level in most instances. Inresponse to the question, “How wouldyou rate the data analytics skills of theorganization?”, only 18 percent of therespondents rated the analytical skillsof their organization at a high level, withmost rating the skills at a medium or lowlevel in their organization:High level — Highlyskilled work force,data analyticstechniques are18%understood acrossthe agency ordepartment.Medium level —Pockets of skilledanalytical professionals,focused on 50%individual businessunits or programsLow level — Someskilled analyticalprofessionals across 32%the organization.Based on our survey results, mostfederal agencies will need to acquiresome amount of analytical expertisefrom contractors to implement dataanalytics solutions in their operations.Question: Has theagency considered theneed for cultural andorganizational skills?Firms that work in the area of dataanalytics and analysis also discussed theneed for organizations to consider culturaland organizational issues when integratingdata analytics into an organization.One firm in the area of data analyticsdiscusses the need for achieving “culturalalignment” to drive high performanceand sustained results. 5 “Cultural alignment”for this firm constitutes a numberof considerations. This firm speaks aboutcreating a culture in the organization thatvalues analytics and respects the data,and combines this with a pervasive curiosityfor information. If this is not done,the pockets of existing analytical talentquickly grow disillusioned and, becausethey are not integrated into the businessas a whole, fail to deliver much strategicvalue. A culture of analytics can be builtby focusing on five elements:• Respect for data: Organizations withanalytical cultures demonstrate aprofound respect for data and factbaseddecision-making.• Pragmatic decision-making: Onthe other hand, organizations withanalytical cultures know the limits ofdata and do not get stuck in “analysisparalysis.”• Drive to optimize: An analyticalorganization is fundamentally curious— about what others are doing in themarket, about performance patternsand root causes, and about new andbetter ways to do things.• Collaboration and transparency:An analytical culture is markedby collaboration and informationsharing across organizationalboundaries.• Rewards for analytics: Individualsare recognized and rewarded fortheir analytical capability, includingnot only the quality of analyses andLeveraging Data Analytics in Federal Organizations 23

Part Three: Key Questions for Implementersinsights, but also the breakthroughbusiness results achieved by puttingthem into action.Other experts make similar points.One not only spoke of the need fora champion, but also stressed theimportance of making data analytics anintegral part of the organization’s work.U.S. Department of Agriculture,Food and Nutrition ServiceEvidence of a cultural realignmentwithin FNS is the ALERT system’slong-term acceptance and integrationinto FNS operations. An example of thecultural alignment is in the proceduralchange for removal of merchantsdeemed to be trafficking in SNAPbenefits. All retailers who participatein SNAP must be licensed by FNS andmay be banned from the program foroffenses including trafficking or failureto comply with program rules. At thestart of ALERT, retailers could not bedisbarred from the program, no matterhow compelling the analytical information,without an actual field investigation.Today, retailers can be barred fromthe SNAP program based solely uponinformation provided by the ALERTsystem. The effectiveness of ALERT hasprompted confidence in the system andhas resulted in a cultural realignment insupport of data analytics within FNS.Centers for Medicare and MedicaidServices, Center for Program IntegrityCPI officials said they are addressingcultural alignment in their work by usingthe “Command Center” to integrateCPI and CMS staff into the work. TheCommand Center was launched afterthe FPS system went live in June 2011.The Command Center is used to sharetechnical details and expectations withinvestigators and analysts, work collaborativelyon the results from the system,refine and improve the analytic modelsand associated business processes,and create consistent investigativeapproaches and appropriate actions.CPI has established a system to rotatekey staff from other CMS units throughthe Command Center to work on thenew systems. Officials say they plan tomaintain this approach to develop newsystems and improve and expand thecurrent system.U.S. Postal Service,Office of the Inspector GeneralUSPS-OIG addressed the issue ofcultural alignment by placing the unitdeveloping the data analytics systeminto the investigative unit that woulduse the results of the analysis. Duringthe initial phase of RADR development,the system was housed in a supportunit outside of the investigative unit thatwould use the system’s output. Duringthe course of system development,the RADR system and its developmentteam were moved to the investigativeunit. USPS-OIG officials felt that thisinstilled ownership by the investigativestaff and better aligned the investigativeunit’s work with the new system. As thesystem has matured and shown results,the RADR team has expanded thesystem’s role to work with the audit sideof USPS-OIG in addition to the investigativeunit. To facilitate this change, theRADR team’s place in the organizationhas been moved to a support unit thatreports to the chief technology officerat USPS-OIG. It was felt that this changewould facilitate RADR development asthe development team will be supportingoperations in two units rather than one.Survey Results on Cultural andOrganizational SkillsIn our survey of federal officials weasked, “How would you rate the integrationof data analytics processes andsystems into the agency’s operations?”High level —Data analyticstechniques are fullyintegrated into management,budgeting andplanning functions.8%Medium level — Dataanalytics techniquesare used inconsistently;there is some linkage tomanagement, budgeting,and planningfunctions.Low level —Data analyticsprocesses areconducted in silos, withlittle consistency orstandardization.46%46%As shown, only 8 percent of respondentsthink that data analytics are fullyintegrated into key functions. Thisindicates that data analytics has not yetachieved the acceptance and integrationthat will be needed to achieve lastingorganizational success.Question: Does theorganization have systemsand software to implementa solution, or will theyneed to be acquired?Starting data analytics processes andsystems may present issues related todata storage and the selection of analytictools. These issues were addressedin different ways by the organizationswe interviewed. Data storage was commonlyaddressed either by establishinga data warehouse or by storing data infiles on servers. In determining how tohandle the information, agencies need toconsider the volume of data, the diversityof the data and the kinds of analysisthey plan to perform. Larger solutionsmay require extensive investments indata warehouses and software, whereassmaller solutions may not require muchup-front investment. Two differentapproaches are highlighted below.U.S. Department of Education,Office of Inspector GeneralThe ED-OIG used a planned approachin developing its data analytics systemthat called for the development of adata warehouse and the acquisition of a24AGA Corporate Partner Advisory Group Research

Part Three: Key Questions for Implementerscommercial software suite designed fordata analytics work. 6 The ED-OIG determinedthat this approach best met itsneeds and would provide the most effectiveplatform for future development. Aspresented in the report, the ED-OIG usesinformation from eight different systemsin its various analysis components.U.S. Postal Service, Office ofthe Inspector General and U.S.Department of Agriculture,Food and Nutrition ServiceIn developing the RADR system,USPS-OIG decided not to establish adata warehouse in the first phase of thedevelopment process. Instead, stafffocused on collecting data and constructingmodels for the health-care fraud partof the RADR system. USPS-OIG felt that,given its level of resources, this approachwould allow construction of data analysismodels to begin sooner than if a datawarehouse was created during the firstphase. However, USPS-OIG recognizedthe need for a data warehouse and theefficiencies it could provide, and is currentlyconstructing one for the RADR system.USPS-OIG also opted not to procurea software suite for its systems. Instead,a decision was made to construct modelsand interfaces using software productsthat were already licensed to the largerUSPS enterprise. FNS also used thisapproach in the ALERT system, usingexisting software licensed to FNS toconstruct its routines. Both approacheswere successful and have merit.Summary of Organizations Reviewedand Stage of DeploymentThe following table captureshighlights of the eight organizationsinterviewed for this report. We havesummarized each data analytics system’spurpose, the stage of development andthe extent to which it has been deployedwithin the organization.Figure 8: Summary of Systems by IntervieweeOrganizationsInterviewedRecovery Accountability and TransparencyBoardRecovery Operations Center (ROC)U.S. Department of Agriculture, Food andNutrition ServiceAnti-Fraud Locator using EBT RetailerTransactions (ALERT)U.S. Department of Defense, DefenseFinance and Accounting Service (DFAS)Business Activity Monitoring System (BAM)U.S. Department of Defense, DefenseLogistics Agency, Office of OperationsResearch and Resource Analysis (DORRA)Enterprise Business System (EBS)U.S. Department of Defense, United StatesNavy, Naval Sea Systems Command,Office of Fraud Deterrence and Detection(NAVSEA-OFDD)U.S. Department of Education, Office ofInspector General (ED-OIG)E-Fraud Analytical Model (EFAM) & Stateand Local Education Agencies Risk Model(SLRM) SystemsU.S. Department of Health and HumanServices, Centers for Medicare andMedicaid, Center for Program Integrity (CPI)Fraud Prevention System (FPS)U.S. Postal Service, Office of InspectorGeneral (USPS-OIG)Risk Assessment Data Repository (RADR)Purpose of SystemROC – Identifies potential fraudulent transactionsby analyzing ARRA funds informationusing 22 different data sets and analyticsIdentifies food stamp trafficking by analyzingEBT transactions using pattern identificationand risk rankingsBAM – Identifies improper payments beforethey are made by matching data acrosssystems, looking for patterns of paymentsindicative of fraud from past transactions, etc.EBS – Provides the DLA leadership withinformation on performance metrics basedupon information from the agency’s systemsPlanned system will be used to identifycases of potential fraud and abuse related tocontract procurementEFAM – Identifies fraud in higher educationassistance programs by using data miningtechniquesSLRM – Analyzes information on federalfunds provided to state and local educationagencies to develop a risk model prototypeFPS – Analyzes Medicare payments, priorto payment, to identify potential fraudand abuse. FPS uses predictive modelingtechniques.RADR – Analyzes activity in four areas toidentify potential fraud and abuse by usingdata mining and predictive analytics. Resultsreferred to investigators.Stage ofDevelopmentDevelopment CompleteDevelopment CompleteDevelopment CompleteDevelopment CompleteIn Planning StagesEFAM – System in TestingSLRM – DevelopmentUnder WayDevelopment CompleteDevelopment CompleteExtent ofDeploymentDeployed across all ARRAfunding, pilot testing againstremaining federal fundsDeployed and fully integratedinto FNS operationsDeployed across nearly allDFAS payment systems.Results are integrated intoDFAS operationsDeployed across DLANot applicableDeployment started.Deployment across allMedicare fee-for-servicepayments; system integrationhas been ongoing forone year.Deployed and fullyintegrated into USPS-OIGoperationsLeveraging Data Analytics in Federal Organizations 25

Part Four: How is Success ofthe Systems Measured?No standard approach exists formeasuring the success of data analyticssolutions. Most agencies measurethe outputs from their systems andthe increase or decrease in the inputsthat are required to achieve results.For example, a data analytics systemimplemented by an OIG investigativeunit may measure the change in thenumber of fraud cases investigated andprosecuted before and after systemimplementation as a way of measuringsuccess. It might combine this numberwith the cost of staff, or combine staffresources with the number of fraudcases identified in investigations, tocompute a return on investment. CPIofficials indicated that more than 510new investigations have been startedas a result of these results. In addition,336 existing investigations are nowbeing supported by real-time FPS data.FPS data has also led to more than 400direct interviews with providers whomay be participating in potentiallyfraudulent activity. The bottom line isthat output measures are necessaryand needed, but long-term results mustalso be measured.Often an agency may measure theeffectiveness of its system by looking atsuch things as the decline in the amountof waste, fraud and abuse uncoveredby the system. While this may be aneffective measure, it might also indicatethat the system has lost effectiveness.Fraud and abuse schemes change rapidly,and the methods that are used todetect them must change as well. Otherfactors might also be in play, such asdeclines in the quality of data or theeffectiveness of the investigative staff.ED-OIG officials said the issues mightbe mitigated by a well-established andreliable line of communication betweenthe respective system owner and theanalytical team to ensure that all significantsystem enhancements are shared.Otherwise it is quite possible thatcritical aspects of previously successfulanalytical engines rely on now-“stale”segments of data.One method to measure the effectsis to periodically examine the programon a wide scale to see if a certainelement is declining, remaining stableor increasing. This is done at USDA forthe SNAP program. USDA implementedthe ALERT system to detect traffickingin SNAP benefits. (Trafficking is definedas the illegal sale of SNAP benefits forcash or other non-allowable items.) Thefirst assessment in 1993 determinedthat about $811 million in programbenefits, or 3.8 percent of the benefitsredeemed, were lost due to trafficking.Since then, FNS has completed fouradditional periodic studies and is currentlycompleting a new study to estimatethe rate at which benefits are lostdue to trafficking. The most recentlycompleted study for 2006 – 2008concluded that the trafficking rate was1.0 percent, or $330 million. Based onbenefit levels of $75.6 billion in FY 2011,we estimate that this would translateinto $1.1 billion of benefits that were nottrafficked last year. This method doesnot measure system outputs but ratheroutcomes of FNS work on the foodstamp program.Malcolm Sparrow, PhD, Professorof the Practice of Public Managementat Harvard’s Kennedy School ofGovernment, has studied and writtenextensively on health-care fraud. Inhis publications, Dr. Sparrow writesabout the lack of solid measurementsto back up the estimates on the extentof fraud in the health-care industry. Headvocates that accurate information onthe extent of fraud can only be achievedby a series of random, rigorous auditsof health-care providers. 7 In this way,studies can estimate the extent of fraudin order to measure future progress.Absent a system such as thatadvocated by Dr. Sparrow, it will bechallenging for federal agencies tomeasure the effectiveness of dataanalytics systems that are designed todetect and prevent fraud. Measuringdirect outcomes will only provide someof the answers regarding data analyticssystem effectiveness. The authorsof an AGA research report, “UsingPerformance Information to DrivePerformance Improvement,” identified26AGA Corporate Partner Advisory Group Research

Part Four: How is Success of the System Measured?a similar concern when reviewingthe use of performance measures infederal agencies. The report found thatmost measures used by agencies relateto program activities and outputs,and it concluded that measures usedto manage a program day-to-day cancause difficulty when no outcomemeasures linked to these programsexist. People want to know programeffectiveness as well as how muchmoney has been saved. People want toknow if fraud detection programs arealso reducing fraud.Leveraging Data Analytics in Federal Organizations 27

Part Five: The Case forGovernment-wide LeadershipOur survey asked federal officials,“If you have not begun using dataanalytics in your operations, whatare the principal reasons for this?” Afrequent response was that they wereawaiting government-wide solutionsin some areas. Our research noted thatdata analytics solutions have beendeveloped, which many agencies coulduse for their operations. Deploymentof such systems would save time andreduce costs while allowing developmentto proceed on additional systems.Two prime examples of this were foundin the work by USPS-OIG and the RTABRecovery Operations Center.An official at ED-OIG with extensiveexperience in data analytics systemsdevelopment said he believed thefuture for many aspects of data analyticswould be in the cloud environment,managed by experienced individualswho would be able to meet the needsof organizations in a cost-effectivemanner. Officials at USPS-OIG alsoexpressed similar sentiments. Oneindividual said that many of the inspectorgeneral operations throughout thefederal government are not very large,and developing data analytics systemscan be beyond their budgets. This officialadvocated sharing existing systemsas a means to reduce costs and provideimmediate results.Possibilities forGovernment-wideSolutionsTwo solutions follow that couldbe quickly deployed in organizationsacross government with concertedaction by federal officials.U.S. Postal Service,Office of Inspector GeneralRADR is used by USPS-OIG toidentify instances of potential fraudfor further investigation, resulting inbetter utilization, a greater return oninvestment and the reduction of waste,fraud and abuse of USPS resources. TheFigure 9:HIGH-LEVEL MODEL ARCHITECTUREContractDataPaymentsDataEmployeeDataExternalData(EPLS, D & B)CF ModelRisk Scores28AGA Corporate Partner Advisory Group Research

Part Five: The Case for Government-wide LeadershipUSPS-OIG RADR system has four components,two of which address issuesunique to USPS operations. However,the remaining two RADR componentsare designed to address issues that arecommon across many federal agencies,namely worker compensation fraud andcontract fraud. It would seem that thesecomponents of RADR could be deployedacross other governmental agencies in acost-effective manner.The RADR system for contract fraudworks in the general manner illustratedin Figure 9. Data from a number ofin-house USPS information systems arecollected and passed through a contractfraudmodel, which merges the datawith information from commerciallyavailable sources to rank the contracts interms of risk for fraud. RADR uses morethan 30 risk ranking criteria to score andrank each contract. Users can accessthe system online, and data is displayedthrough a visual interface.USPS-OIG officials believe that thevisual display makes information moreaccessible and improves how effectivelythe results of the system’s analysis canbe communicated. The RADR systemdisplays the results of its work in a visualmap of the Unites States, using red dotsto indicate “hot spots” and yellow dotsto indicate “warm spots” for contractfraud. Red spots indicate a high numberof contracts that have tripped many ofthe system’s triggers and are thus morelikely to indicate fraud. The size of the dotalso indicates the dollar value of the riskin the area. Users can further refine theresults by filtering information throughthe visual display. The user can also clickon a spot to access the details of theRADR analysis and examine the contractdetails associated with the risk rating.The RADR system uses a similarapproach to reviewing worker compensationpayments. Investigators are ableto display a map of potential workercompensation fraud hot spots and canrank and review all associated casesto determine if further investigation iswarranted. The risk rating system willconsider such factors as the length ofpayments, the frequency of treatmentand the severity of the injury in decidingwhich payments to investigate.USPS-OIG is using this rating systemto change its organization from a reactiveto a proactive organization. For contractfraud, the RADR system has started totransform the work of USPS-OIG investigatorsin a number of ways. USPS-OIG isnow analyzing every contract, instead oftaking a random sample or waiting for alead to initiate an investigation. USPS-OIG is also moving its investigative unitfrom a subjective assessment of risk to amore unified approach.USPS-OIG officials have demonstratedthe system for and shared theirknowledge with other federal agencies.They believe the RADR system could beFigure 10: System InterfaceLeveraging Data Analytics in Federal Organizations 29

Part Five: The Case for Government-wide Leadershipmodified and deployed in a cost-effectivemanner across federal inspectorsgeneral and chief financial officers’ communitiesto assist in the identification ofcontract and worker compensation fraudand abuse.The work of the Recovery OperationsCenter shows another example of costeffectivedeployment solutions.Recovery Accountability andTransparency BoardThe Recovery Accountability andTransparency Board (RATB) was createdby the American Recovery andReinvestment Act of 2009 (ARRA) withtwo goals: to facilitate transparency onthe use of recovery-related funds andto detect and prevent fraud, waste andmismanagement. ARRA was enactedwith provisions that required transparencyand quarterly reporting of fund useand activities. Recipients were requiredto report information on their spendingand other activities via the federalreporting.govwebsite. The RATB publishedthis information on Recovery.gov.To assist in the prevention and detectionof fraud, waste and mismanagementin ARRA funds, the RATB formedthe Recovery Operations Center (ROC).The ROC developed a data analyticssystem that is used to screen recipientsof ARRA funds. In its screening process,the system uses 22 separate outsidedata sets from other governmentagencies and commercial vendors.ARRA recipients are screened to detectanomalies, which are used to identifypotential high risk entities and unlikelycircumstances. Risky organizations mayrange from those with a bad recordfor accountability in past governmentprograms to entities that have re-establishedthemselves with a new identityso that they can continue nefariousoperations. One example of unlikely circumstanceis where several companieshave the same fax number and address,but are not in the same type of business.Information that is obtained by the ROCthrough new or updated data sets areprocessed against the historical data todetermine if transactions that seemedbenign are now potentially problematic.Potential problems are screened bydata analysts using information at theROC’s disposal. After screening, theanalysts provide the potentially problematictransactions to the appropriateinspector general, U.S. Attorney, orother law enforcement or audit office forfollow-up and review. The ROC has concludedinitiatives that include reviews ofprovider enrollment data, veteran’s disabilitypayments and Service DisabledVeteran Owned Small Businessesassistance. The ROC has begun a pilotprogram to allow inspectors generaland program officials access to theROC data sets and tools through a webportal called FederalAccountability.gov. Until recently, the ROC‘s systemswere limited to working only on ARRAfunds; however, changes to the RATB’sappropriations law now allows for thetesting and development of the ROCsystems outside of ARRA.The CurrentGovernment-wideApproach: GovernmentAccountability andTransparency BoardSimilar questions regarding government-widesolutions were addressedby the Government Accountability andTransparency Board (GATB). The GATB,formed in June 2011 by Executive Order(E.O.) 13576, Delivering an Efficient,Effective and Accountable Government,has as its mission to identify implementationguidelines for integrating systemsthat support the collection and displayof government spending data, ensuringthe reliability of those data and broadeningthe deployment of fraud detectiontechnologies, including those provensuccessful during the implementation ofARRA. Specifically, E.O. 13576 directedthe GATB to work with the RATB toextend its successes and lessonslearned to all federal spending.The GATB issued an interim reportin December 2011 with recommendationsin three broad areas that arefoundational to cross-government workin the field of data analytics. The firstrecommendation dealt with the adoptionof a cohesive, centralized accountabilityframework to track and oversee spending.The report concluded that such auniversal framework would have manybenefits in the efforts to fight fraud,including increased collaboration, rapidinnovation, less costly developmentefforts and data sharing.The second recommendationadvised that government must rationalizethe way it collects and displaysspending data by consolidating andstreamlining technology platforms. Thereport states the existing universe offederal reporting systems and applications,developed over a long periodby the federal government, is large,complex and costly. According to theGATB, federal reporting systems andapplications must be integrated in amanner similar to that employed bythe RATB. The RATB uses a limited setof data elements and established datastandards and has developed highlyscalable systems to accommodatechanging reporting and display requirements.The RATB has also implementedan extremely aggressive reportingschedule for recipients and has migratedits solution to a cloud computing environment,which significantly increasesefficiency while reducing operationand maintenance costs. In addition,the RATB has developed close workingrelationships with stakeholders.Third, the GATB recommendsthat the government migrate to auniversal identification system forall federal awards. The GATB foundthat no requirement exists for the30AGA Corporate Partner Advisory Group Research

Part Five: The Case for Government-wide Leadershipstandardization of award IDs acrossthe federal government and that someagencies do not have universal IDs.Performing analytics on inaccurate andnon-standardized data often leads toerroneous conclusions and costs thegovernment untold resources (in wastedmoney and human capital) that could beused more efficiently and effectively.The GATB has a number of recommendationsand observations specificto adopting and implementing improvements.For instance, the report recommendsthat the federal governmentidentify and revise legal authorities thatinhibit data matching and data analysis.The report also cites the work of theRATB and the ROC as a basic model fortransparency and accountability. Thisincludes FederalAccountability.gov,a web-based, secure portal throughwhich both program and enforcementauthorities gain access to its forensicand analytical capabilities. The GATBbelieves that FederalAccountability.govhas applicability beyond the RecoveryAct for identifying and mitigating fraud,waste and abuse related to federal funds.Many of the GATB’s recommendationsdeal with transparency andaccountability. However, other shorttermsolutions have been developedthat already apply to the work of manyfederal agencies and departments. Oneexample discussed earlier is the RADRsystem, which has components that dealwith contract and workers compensationfraud. Work has also started to expandthe reach of the RATB’s ROC, which hasdeveloped a large number of valuabledata analytics routines that can be usedacross the federal government.One OIG official observed that,because many OIG operations are notvery large, developing data analyticssystems might not be possible. Hesuggested some type of shared serviceapproach across OIG communities.Another OIG official suggested thatservices offered in a cloud environmentoffering broad accessibility at a reducedcost. The question: How should theefforts be organized and, if offered insome consolidated manner, where theywould be housed?If fully implemented, recommendationsin the GATB’s December 2011 reportrelated to transparency and accountabilityacross the government and thetracking of federal funds could facilitatedata matching between agencies.Leveraging Data Analytics in Federal Organizations 31

Part Six: Recommendations• Build on Success: The federalgovernment should strive to leveragethe experience gained by those whohave implemented data analyticssystems to help other organizationsovercome deficiencies in budget,staffing and experience. Rather thanassuming that each agency shoulddevelop its own data analyticssolution from cradle to grave, thefederal government should considershared service arrangementsenabling agencies with successfulsystems to build upon the workof others. Prime examples of thisare the contract fraud systemsdeveloped by USPS-OIG, the dataanalytics routines developed bythe ROC and the ED-OIG systemfor identifying fraud among federalfund award recipients. Powered bytechnology, these systems could beused across the spectrum of federalagencies, including the inspectorgeneral and CFO communities.• Continue Education: While it appearsmost government leaders havea general understanding of dataanalytics, they are not yet championsof the process. Federal leaders mustcontinue to be educated on thebenefits and uses of data analyticsand the actions needed to implementdata analytics in their agency.• Focus on Performance andOutcomes: Organizations shouldexpand the use of data analytics toinclude performance and outcomes.Only one of the eight organizationsinterviewed was using data analyticsto measure program performance;the others were primarily usingit to prevent and detect improperpayments, obtain information onfinancial performance and identifyinstances of fraud and abuse forinvestigation and audit. Given thecurrent emphasis on reducinggovernment spending, data analyticscould play a vital role in helpingdecision makers determine wherescarce dollars could be mosteffectively used.• Procurement: A guide to procuringdata analytics systems or consultingservices should be developed (ideallyby a neutral third party) based onthe experiences of federal officialsthat have procured systems and thevendors who provide these services.This guide could include informationon the services that can be procured,ways to determine the correct dataanalytics method, provisions thatsignificantly increase or decreasethe cost of a contract and lessonslearned from previous procurements.• Explore IntergovernmentalServices: Federally funded programsimplemented at the state or localgovernment levels should bereviewed to determine whethera collaborative arrangement canbe developed for data analytics. Acentral system administered at thefederal level might be more effectiveand cost efficient than stand-alonesystems throughout the country.32AGA Corporate Partner Advisory Group Research

EndnotesEndnotes1. “Improper Payments: Not Just thePurview of the CFO Anymore?” AGA.September 2011.2. “Using Performance Informationto Drive Performance Improvement.”AGA. December 2011.3. Antonia de Medinaceli, Director ofBusiness Analytics and Fraud Detection,Elder Research, Inc. Presentation duringAGA audio conference, Data Miningto Prioritize Investigations of ContractFraud. March 7, 2012.4. SNAP provided nutrition assistanceto more than 21 million householdsand disbursed more than $71.8billion in benefits in FY 2011. SNAP isadministered with the assistance ofthe state government agencies. Stateofficials determine recipient eligibility,and FNS enrolls retailers in the paymentsystem and remits payments to theretailers through an electronic benefittransfer system.5. Why Leadership and CultureMatter: Helping Turn Analytic Insightsinto Outcomes and Drive HighPerformance. Accenture. May 20, 2011.6. A data warehouse is a databasecreated for reporting and analysispurposes only. Data is stored in either adimensional or a normalized approach,and multiple users can access thedata. A data mart is a subset of a datawarehouse.7. Sparrow, Malcolm K. Licenseto Steal How Fraud Bleeds America’sHealth Care System. Westview Press,2000.Leveraging Data Analytics in Federal Organizations 33

Project Participants:Government AgenciesU.S. Department of AgricultureJeffrey N. CohenDeputy Associate Administrator,Supplemental NutritionAssistance ProgramFood and Nutrition ServiceThomas (Tim) O’ConnorAssociate Administrator, SpecialNutrition ProgramsFood and Nutrition ServiceShelly PierceProgram Analyst,Benefit Redemption DivisionFood and Nutrition ServiceU.S. Department of Defense,Defense Finance andAccounting ServiceRebecca S. BeckDirector, Accounting, ColumbusTerri SchulzeDirector, Site Support Office, ColumbusWilliam McGeeImproper Payments Functional LeadBusiness Activity Monitoring (BAM)U.S. Department of Defense,Defense Logistics AgencyLawrence VadalaOffice of Operations Researchand Resource AnalysisU.S. Department of Defense,Department of the NavyWillie WhiteChief, Anti-Fraud OfficerNAVSEA Office of Fraud Deterrenceand Detection, SEA 00FLaura CrawfordAnti-Fraud Project ManagerNAVSEA Office of Fraud Deterrenceand Detection, SEA 00FSteve DerryBusiness Operations ManagerNAVSEA Office of Fraud Deterrenceand Detection, SEA 00FU.S. Department of Education,Office of the Inspector GeneralEdward SlevinDirectorComputer Assisted AssessmentTechniques DivisionU.S. Department of Health andHuman Services, Center forMedicare & Medicaid ServicesDavid NelsonDirector, Data Analytics andControl GroupCenter for Program IntegrityKelly GentDeputy Director, Data Analytics andControl GroupCenter for Program IntegrityDevin WilliamsDirector of the Command Center DivisionData Analytics and Control GroupCenter for Program IntegrityU.S. Postal Service,Office of the Inspector GeneralBryan JonesDirector, Countermeasures andPerformance EvaluationData Mining GroupRecovery Accountability andTransparency BoardAlan F. BoehmAssistant DirectorRecovery Operations CenterEd MartinDeputy Assistant DirectorRecovery Operations Center34AGA Corporate Partner Advisory Group Research

AGA CPAG Research ReportsNo. 1, March 2005:No. 2, July 2005:No. 3, November 2005:No. 4, April 2006:No. 5, June 2006:No. 6, June 2006:No. 7, February 2007:No. 8, March 2007:No. 9, May 2007:No. 10, April 2007:No. 11, May 2007:No. 12, June 2007:No. 13, June 2007:No. 14, January 2008:No. 15, July 2008:No. 16, September 2008:No. 17, November 2008:No. 18, January 2009:No. 19, February 2009:No. 20, March 2009:No. 21, June 2009:No. 22, September 2009:No. 23, November 2009:No. 24, June 2010:No. 25, July 2010:No. 26, September 2010:No. 27, June 2011:No. 28, September 2011:No. 29, December 2011:Audit Federal Financial Controls: Sooner Rather than Later?Financial Management Shared Services: A Guide for Federal UsersTrends in TechnologyThe Federal Purchase Card: Use, Policy and PracticeChallenges in Performance Auditing: How a State Auditor withIntriguing New Performance Authority is Meeting ThemPAR—The Report We Hate to LoveThe State Purchase Card: Uses, Policies and Best PracticesFederal Real Property Asset ManagementShould State and Local Governments Strengthen Financial Controlsby Applying SOX-Like Requirements?Process-Based Financial ReportingThe State Travel Card—Uses, Policies and Best PracticesTrends in Technology—2007 ReviewThe Federal Travel Card—Uses, Policies and Best Practices21st Century Financial Managers—A New Mix of Skills and Educational Levels?SAS 70 Reports: Are they Useful and Can They Be Improved?XBRL and Public Sector Financial Reporting: Standardized Business Reporting:The Oregon CAFR ProjectCharacteristics of Effective Audit Committees in Federal, State and Local GovernmentsGrants Management: How XBRL Can HelpProcuring Audit Services in Government: A Practical Guide to Making the Right DecisionPerformance-Based ManagementTrends in Technology—2009 ReviewManagerial Cost Accounting in the Federal Government:Providing Useful Information for Decision MakingState and Local Governments’ Use of Performance Measures to Improve Service DeliveryCreating an Interactive Single Audit DatabaseRedefining Accountability: Recovery Act Practices and OpportunitiesThe Maturity of GRC in the Public Sector: Where Are We Today? Where Are We Going?Trends in Technology: 2011 Report, The Information ExplosionImproper Payments: Not Just the Purview of the CFO Anymore?Using Performance Information to Drive Performance ImprovementDownload complimentary reports at www.agacgfm.org/researchpublications.Leveraging Data Analytics in Federal Organizations 35

AdvancingGovernmentAccountabilityAssociationof GovernmentAccountants2208 Mount Vernon AvenueAlexandria, VA 22301703.684.6931800.AGA.7211703.548.9367 (fax)www.agacgfm.orgagamembers@agacgfm.org