Revised Sampling Guidance | ERDF Audit Authority, Ireland - Interact

ERDF Audit Authority, Ireland – Background• Formally designated as ERDF Audit Authority in Ireland throughMinisterial Directive.• Responsible for 6 Operational Programmes in 2000-06• 2007-2013 - the nominated AA for 3 Operational Programmes1. Border Midland & Western Regional OP2. Southern & Eastern Regional OP3. INTERREG IVA Ireland Wales Programme

ERDF Audit Authority – ETC Roles

ETC Allocation within EUs Cohesion Budget• 2007-2013:– 53 Cross Border,– 13 Transnational, and– 4 Inter-regional• €8.7 billion for 2007-13• ETC makes up 2.5% of total Cohesion Policy budget• €9 billion proposed for Commission 2014-2020

New COCOF Guide - 08-0021-03• Guidance Note on Sampling Methods for Audit Authorities• Previous drafts provided to AAs and asked for comment.• Discussed at Technical Group and COCOF meetings during2012 and 2013• FINAL Version issued 04/04/2013• New guidance note to complement COCOF 08/0019/01 –Common Methodology for Assessment of M&C Systems• Not mandatory, but strongly recommended6

Commission’s Aims & Objectives (1)1. To replace the old guidance note (now 5 years old)2. Reduce the audit burden on AAs3. Make the sample selection more statistical & representative.4. Allow for better stratification of sampling populations7

Commission’s Aims Objectives (2)5. Provide a mechanism for interim sampling in 2 semesters(necessary for 2014-20 given the requirement for fasterclosure)6. Refinements to statistical formulae should reduce sample sizes7. Provide a basis for more sub-sampling8. Provide better examples for projection and evaluation of errors& include statistical tables as Appendices.8

Some Early Observations• Many problems with the 2008 guidance note.• Consultation with ERDF and ESF AAs• Pleased to see one of the objectives was to reduce the auditburden. For ERDF in Ireland – Verification rates of 70%-80% inearly years of National Programmes (due to 100% sampling ofhigh value items)• Early drafts, to support the new regulatory framework for 2014-2020. But later and final drafts – To be applied for 2007-2013.9

Sampling of ETC ProgrammesOld Sampling Guidance Note• Old Guidance Note & Article 17.1 of Regulation 1828/06 –Recommended a Random Statistical Sampling Method• However when operations < 800, the use of statistical approachto determine sample sizes not always appropriate• Many ETC programmes – AA decided and agreed withCommission that Statistical Approach not feasible• Elected for the Formal Approach to Non-Statistical Sampling.(Section 6.6)10

Example ETC Non Statistical SampleSample Size Calculation FormNon Statistical Sample for Programmes with Small Numbers of OperationsOperational Programme:Ireland Wales INTERREG IVAPopulation Details: Estimated Total Eligible Expenditure for 20121. Estimated Total Eligible Population €10,000,0002. Planning Materiality @ 2% €200,000 Cut-off Point3. Number of Projects Claiming in Year 404. Number of Project Claiming > Materiality 8 @ 15% = 1.25. Number of Projects Claiming < Materiality 32 @ 10% = 3.2Breakdown of Sample Sizes6. High Value (rounded) 27. Low Value (rounded) 4Combined Sample Size 6

Sampling Arrangements inIreland Wales INTERREG IVA Programme• Random sample - Audited the LP + one PP (selected atrandom)• ERDF AA in Ireland (Irish Partners) / EFAT Wales (WelshPartners)• GoA meets twice a year (Shared audit programme/reporting)• Historically low rates (Average

Changes Impacting ETC Programmesin New Sampling Guidance Note1. ETC covered in Section 6.2 and 6.3 of new Sampling Guide2. Statistical sampling now encouraged for ETC programmes for2014-203. The Lead Partner may not necessarily be audited.4. Section 7.4.1 - “Non-statistical sampling should beavoided by all available means…..13

Non-Statistical Sampling (Section 7.4)2 OptionsOption 1• If a few high value items• Cut-off 2% (not fixed)• Test 100% of high value• ‘Rule of Thumb’ – Sample 10% of residual population(Early draft of this note said 20% !)• The Lead Partner must be audited15

Non-Statistical Sampling (Section 7.4)2 OptionsOption 2• If there are no high value items• ‘Rule of thumb’ 10% to be sampled / use professional judgement• Take into account level of assurance from systems audits• The Lead Partner must be audited16

Statistical Sampling inNational Programmes 2007-2013• Most Popular : MUS – Anticipated Misstatement is Not Zero• Sample Size Formula (Section 6.5.3.2)n =Book Value x Reliability FactorMateriality – (Anticipated Misstatement x Expansion Factor)• Old Guidance Note provided some limited examples of calculation ofsample sizes & on error projection17

Quantitative Assessment ofManagement & Control SystemsCategoryAssessment of Management& Control SystemsReliability ofEntire SystemConfidenceLevelDetectionRiskPoisson‘R’ Factor1Works well, only minorimprovements neededHighNot lessthan 60 %Less or equalto 40%0.922 Works, some improvements needed Average 70 % 30% 1.213Works partially, substantialimprovements neededAverage 80 % 20% 1.614 Essentially does not work Low Not below 90%Not greaterthan 10%2.31

Calculating Sample SizeUsing Stratified MUS – ERDF National OPsAudit Area:BMW Regional AssemblyPopulation Details: Estimated Total Eligible Expenditure for 20111. Population Value €100,000,0002. High Value Items (excluded as >2% materiality) *€45,000,0003. Residual Population (to be tested by random statistical sampling) €55,000,0004. Residual Population €55,000,0005. Materiality (2% of Population Value) €2,000,0006. Anticipated Misstatement 10% €200,0007. Planning Precision 90% €1,620,0008. Reliability Factor (Poisson) 1.219. Average Sampling Interval €1,338,84310. Expected Sample Size 41.08Sample size = Residual Population (4) x Assurance Factor (8)Planning Precision (7)* High Value = 100% of Operations > Materiality to be tested each year.

Operations Audit SamplesERDF National Programmes (2011 Year)Regional OPNumber ofProjects Declaredin 2011No of OperationsSelected forSamplingAudit Coverage ofSamples%S&E Regional 42 25 87.3BMW Regional 1,138 20 74.220

Sampling of ERDFNational Programmes 2011• National OPs declared all expenditure late December 2011.• Operations audit period restricted to 5 months (February 2012 to30 th June 2012)• Presented serious resource problems.• Head of ERDF AA requested an immediate amendment to AuditStrategy21

Solution Agreed with DG Regio1. No stratification for BMW OP (previously 100% of high valueoperations tested)2. Non-statistical option for S&E OP (only 42 operations declared)3. No longer testing 100% of transactions tested in each sample4. Limit of 30 transactions (Top 20 by value + 10 random).5. Granted an extension of audit period to 30th Sept for 1 yearonly.22

Revised Operations Audit Samples- after amendment of sampling methodRegional OPNumber ofProjects Declaredin 2011No of OperationsSelected forSampling(previous)Audit Coverage ofSamples%(previous)S&E RegionalRevised(Original)42 9(25)20(87.3)BMW RegionalRevised(Original)1,138 17(20)25(74.2)23

Changes to MUS Formula• MUS formula when the anticipated misstatement is not zeropresented in the 2008 Sampling Guide no longer recommended.• Now referred to as the “conservative approach”.• “Relegated” to Section 7.3.4 (page 117) of the new guide.24

New Standard MUS FormulaSample size =Confidence Level (from systems audit & related coefficient z from a normaldistribution) x Book Value X Standard Deviation of error rates producedfrom MUS sample__________________________________________________Tolerable Error (2%) – Anticipated Error(all squared)25

Changes from Old MUS Formula• Normal distribution tables now replace Reliability Factors• Standard deviation also required• Complicated further by additional formula for stratification• Additional complicated formulae for 2 semester sampling(necessary in order to meet the new Annual Reporting deadlinesin 2014-2020)26

Total Projected Error• Should be compared to materiality and reported in ACR• Total Projected Error made up of 3 parts :1. Projected Random Errors+2. Any Systemic Errors+3. Any Uncorrected Anomalous Errors27

1. Projected Random Errorsa) A statistical sample is representativeb) Random errors in audit sample = errors in rest of population.c) Errors must therefore be projectedd) Cause many problems in old guide and still remains confusionover the technicalities of how this should be calculated28

2. Systemic Errorsa) Found in sample auditedb) Also have an impact on non-audited populationc) Occurs in well-defined and similar circumstancesd) Usually arise due to weak controlse) Auditor needs to find out the full extent of the area affected.29

3. Anomalous Errorsa) An error not representative of the populationb) Used frequently in 2007-13 by AAs in error evaluation (auditorjudgement)c) New Guide – for statistical sampling, anomalous errors onlyaccepted in very exceptional circumstancesd) If an anomalous error is removed before the calculation ofprojected error, it must be added back to the total projectederror rate.30

Negative Amounts in Population• Population should only consist of positive amounts.• Negative amounts not to be included in the sampling population• Problem if sampling unit is the operation - amount declared includesnegative amounts.• Do AA’s test these negative transactions now ?• Guideline – may test the negative amounts, but only as a separatepopulation.• When projecting errors, only consider errors from positive amounts31

What to do if Material Misstatement ?• Arises when Upper Limit of Error > Tolerable Error• Covered in Section 5.11 of New Guidance Note• Lists 3 options taken from INTOSAI Guideline No 23• However little guidance on which one is best!• Potential problem area as this can become time consuming inwhat will be a shortened audit period.32

EU Budget & Legislative Package 2014-20• Key responsibility of the Irish presidency during its tenure is to agree theLegislative Package for 2014-2020• Negotiations proving difficult on the Multiannual Financial Frameworkbetween Council, Parliament and Commission (Trilogue)• Statistical Sampling on the agenda last week. Council position is thatStatistical & Non-Statistical sampling can be used in 2014-2020, butCommission strongly against this. The European Parliament seem to besiding with the Commission but negotiations ongoing.• Common Provisions Regulation - When agreed will repeal General Reg.1083/2006

Draft Common Provisions Regulation- Art 116 Functions of AA• Previous reference year 1/1/N to 31/12/N now becomes ‘theaccounting year’ 1/7/N-1 to 30/6/N• Requirement to audit the Annual Accounts• Faster Reporting (7 ½ months instead of 12 months)• Audit Opinion on Annual Accounts to cover the functioning of the M&Csystem and the legality and regularity of the underlying transactions by15 February (exceptionally 1 March)• A separate ACR setting out the findings of audits and supporting theaudit opinion by 15 February (exceptionally 1 March)

2014-20 - Timeline for submission,examination & acceptance of AccountsAccounting yearCompulsory interimpayment applicationExamination andacceptance of accountsJulyN -1DecN-130June NDecN15 Feb(1 Mar)N+131 MayN+130 JunN+1Preparation of the accounts,management declaration &accompanying report incl.annual summary, AO + ACR

New Guide - some reflections … …• Commission did take account of comments from AAs• Welcome reduction in sampling burden & sub-sampling• Encouragement to group programmes• 2 semester auditing will become a necessity• Useful Glossary at Appendix 5• Welcome inclusion of Tables:– (i) Reliability Factors– (ii) Values for Normal Distribution36

… … Questions Remaining ?• Sample of a sample – Implications when projecting errors?• For ETC to comply with statistical sampling requirements, do we use theindividual payment claim as the Sampling Unit and not the operation ?• What additional work if Upper Limit for Error > Tolerable Error ?• Could Commission provide pre-prepared spreadsheets for calculatingsamples & evaluating errors? (AA’s simply slot in figures).• Do AAs need to use & understand formulae? (leave to MS statisticians!).• Acceptance of audit strategies for 2014-2020 will be great fun!37

Thank you