Module 4 - Introduction to Performance Audit_4C

4C. Performing a Performance Audit Engagement (20%)
4C. Learning Outcomes
On completion of this Module, students will be better able to:
Explain the requirements for gathering information for a performance audit.
Propose measures for evaluating economy, effectiveness, and efficiency.
Describe the relative reliability of different sources of information that may be used.
Identify potential sources of error in gathering information.
Select appropriate methods for evaluating information.
Generate valid conclusions and recommendations.
4C.1 Auditing Economy, Effectiveness, and Efficiency
The greatest differences in performance audits when compared with other kinds of internal
audit engagements occur in the planning and preparation stages. The processes for
performing, documenting, reporting, and completing follow-up are somewhat similar. The
focus of all stages of a performance engagement is to measure and evaluate performance
according to the audit objectives, questions, scope, criteria, and methodology, and provide
an assessment of efficiency, economy, and effectiveness.
The IPPF covers standards relevant for performing all kinds of engagements, as identified in
section 4A.2 above. Information gathering as it relates to planning and any pre-study is
described in section 4B.5 but this process continues throughout the engagement.
Standard 2300 – Performing the Engagement summarizes the steps involved, as illustrated
below.
Key Stages in Performing an Audit Engagement
The process of gathering information starts with the planning stage and continues through
the fieldwork. The process is iterative, as indicated by the arrows in the diagram above.
Analysis and evaluation may highlight gaps or additional information needs. The process is
guided by achievement of the engagement objectives and information must satisfy the
following requirements:
Internal auditors must identify sufficient, reliable, relevant, and useful information to
achieve the engagement’s objectives.
Interpretation:
43

Sufficient information is factual, adequate, and convincing so that a prudent,
informed person would reach the same conclusions as the auditor.
Reliable information is the best attainable information through the use of
appropriate engagement techniques.
Relevant information supports engagement observations and recommendations
and is consistent with the objectives for the engagement.
Useful information helps the organization meet its goals. 68
Useful
Sufficient
Reliable
Relevant
Requirements for Identifying Information
ISSAI 3000 provides direction on establishing sufficiency of evidence:
106) The auditor shall obtain sufficient and appropriate evidence in order to establish
audit findings, reach conclusions in response to the audit objective(s) and audit
questions and issue recommendations when relevant and allowed by the SAI’s
mandate. 69
Sufficiency refers to the quantity of evidence, being enough to achieve the objectives to
satisfy a “reasonable” person (compared with a “prudent informed person” in The IIA
Standards). Arriving at sufficient evidence requires a professional judgment based on
consideration of audit risk, materiality, and the significance of the finding.
The other three dimensions – relevance, reliability, and usefulness – concern the
appropriateness of the evidence. This in turn relates to the importance of the evidence to the
audit topic, as well as its accuracy, consistency, and verifiability. There is a natural hierarchy
of evidence with respect to its relative reliability which is illustrated below, from greatest to
least.
68
Standard 2310 – Identifying Information, The International Professional Practices
Framework, The IIA, 2016.
69
ISSAI 3000 Performance Audit Standard, INTOSAI, 2019.
44

Direct
•Evidence gathered directly by the auditor through
observation.
Documentary
•Original documentation in preference to copies.
Testimonial
•Multiple interviews from knowledgeable, credible,
unbiased persons, under relaed coniditons, corroborated
in writing, in preference to individual testimonies or those
that are oral only.
Analytical
•Where the auditor has applied analytical techniques to
process availabel information and extract key evidence,
in preference to analysis performed by another party.
Relative Reliability of Evidence
Information gathering techniques can draw evidence from the audited entity as well as from
other sources (“such as clients, experts, civil society organisations, contractors, professional
organisations, research organisations or other government entities”). 70 Common sources
and approaches are listed below.
Direct:
o Direct observation.
o Physical inspection.
o Testing.
Documentary:
o Original or copies, e.g., plans, contracts, policies, case studies, and reports.
o Electronic as well as paper based.
o Text-based, graphical, photographic, audio, and video.
Testimonial:
o Direct (e.g., interviews and conversations conducted in-person, via telephone,
or teleconferencing), conducted individually or in groups (e.g., focus groups,
panels).
o Indirect (via written statements, surveys, recordings, etc.).
Analytical:
o Where the auditor performs analytical techniques to process, aggregate,
summarize, extrapolate, and otherwise to interpret information and identify
patterns, trends, and salient data points. Techniques for making sense of
information gathered, recognizing root causes, and identifying outcomes and
impacts include walk throughs, process mapping, five whys, charting,
regression, and fishtail diagrams.
o Quantitative analysis may be statistical or descriptive. Qualitative analysis
includes thematic, chronological, and topical approaches.
70
Performance Audit ISSAI Implementation Handbook, IDI, 2021.
45

Case studies are listed above as a potential documentary source but may be used as a
research method conducted by the auditor in which a detailed analysis is made of a
particular aspect of the subject matter as a way of discovering useful learning points about
the subject matter as whole.
Measuring Economy, Effectiveness, and Efficiency
The auditor must select appropriate measures to support conclusions regarding the
economy, effectiveness, and efficiency of the subject matter. Measures may be qualitative
and quantitative as well as financial and nonfinancial, and should consider:
Inputs (resources applied).
o Efficiency (i.e., the degree to which use of resource is minimized) can be
measured using input-to-output ratios in terms of costs and volumes or by
input-to-input ratios of different categories of inputs (e.g., staffing costs as a
percentage of total costs).
o Economy (i.e., degree to which resource costs are minimized), including
trends over time and variations between locations, teams, and individuals.
o Cost of quality, including quality assurance processes and the impact of
failures and wastage.
Output – work produced or services delivered.
o Output quantity.
o Output quality.
o Output timeliness.
o Output costs.
Outcomes – results achieved.
o Effectiveness of outputs in achieving the intended purpose. 71
Outcomes relate to the effects beyond the entity on people such as service users and civil
society as well as broader dimensions like the environment, law and order, the economy,
national security, and international relations. These long-term outcomes may be described
as impacts. Effectiveness measures the value added (or benefits) created by the subject
matter. To measure this the auditor should consider factors such as:
Customer satisfaction.
Coverage.
Financial results.
Financial conditions.
Readiness.
In each case, the auditor will need to determine a suitable unit of measure or proxy and a
means of gathering evidence.
In their analysis and evaluation, auditors are likely to need to apply a range of operational,
financial, and statistical techniques, including net present value (NPV), internal rate of return
(IRR), linear programming, regression, and appropriate financial margins and ratios.
Common project appraisal models covered in this Module are described below.
71
See Rauum and Morgan, Performance Auditing: A Measurement Approach, The Internal
Audit Research Foundation, 2009.
46

Payback
The concept of payback is straightforward. The method answers the question: how long will
it take for the net cash inflows (i.e., any additional revenues and cost savings) to be equal to
the investment needed at the beginning? The payback period can be calculated on the basis
of projected or actual results. Once calculated it can be compared with other investment
options to help select the projects to be funded. For projects already completed, the result
can be compared with planned payback or benchmarks.
For example, the following detail relates to a project.
Year
Additional cash
outflows
(000,000 LEK)
Net cash inflows (from additional
revenue and/or saved costs)
(000,000 LEK)
0 158 (setup costs)
1 18
2 41
3 65
4 85
5 85
Year 0 is the theoretical point at which investment is made and the project begins. An
assumption is usually made that cashflows accrue evenly through the year.
The first step is to add a cumulative column for net cash inflows.
Year
Additional cash
outflows
(000,000 LEK)
Net cash inflows (from additional
revenue and/or saved costs)
(000,000 LEK)
Cumulative net
cash inflows
(000,000 LEK)
0 158 (setup costs)
1 18 18
2 41 59
3 65 124
4 85 209
5 85 294
You can see in this case that at some point during year 4 the cumulative total passes the
initial cost of set up 158. The payback period is somewhere between 3 and 4 years. Year 4
begins with a cumulative total of 124. A further 34,000,000 LEK is needed to reach
158,000,000 LEK. If we assume an even rate of cashflows, it will take 34/85 = 0.4 of a year
or 4.8 months.
Therefore, Payback period is 3.4 years or 3 years and 4.8 months.
The method is simple, easy to understand and easy to calculate. However, it ignores the fact
that future cashflows are harder to estimate accurately. It also ignores the time value of
money (i.e., that money that will be received in the future is worth less than money that will
be received sooner.)
Discounted Cashflow (DCF) and Net Present Value (NPV)
47

To account for the time value of money, project appraisal methods can use discounted
cashflows (DCF). Capital has an assumed cost. It must be borrowed or used from reserves
in which the cost is the opportunity cost of not doing something else with it (like investing it).
The further ahead a cashflow occurs (or is predicted to occur), the lower its relative value.
The cost of capital has a cumulative effect.
For example, suppose the assumed cost of capital is 10%. We can calculate discounted
cashflow factors. The formula is:
Discount factor = 1/(1 + discount rate) years
In this case, the discount rate is 10% or 0.1. Discount factors are as follows:
Year
Discount
factor
0 1.000
1 0.909
2 0.826
3 0.751
4 0.683
5 0.621
(Note: These discount factors are available as tables online or in published form, and in an
exam you would not be expected to calculate them.)
We use these factors to discount future cashflows to the present value. 1 LEK today is worth
1 Lek. 1 LEK in a year’s time has the same value as 0.909 LEK today. I could invest 0.909
LEK today for a year and at the end I will have 1 LEK.
We can use these factors with the information given in the Payback example.
Year
Discount
factor
Net cash
inflow/(outflow)
Discounted
cashflow
Cumulative discounted
cashflow
0 1.000 (158) (158) (158)
1 0.909 18 16 (142)
2 0.826 41 34 (108)
3 0.751 65 49 (59)
4 0.683 85 58 (1)
5 0.621 85 53 52
In this case we can see the project does not have a positive cumulative discounted cashflow
until just after the end of year four. Our Payback period on this basis is just over four years. It
is longer than our previous calculation because we have discounted the cashflows, the
further they are in the future, the greater the amount of discount. 85,000,000 LEK received in
five years’ time has the same value as 53,000,000 LEK received today.
Financial ratios
48

Financial ratios may be used in performance auditing to analyse project information. In
particular, the efficiency and profitability ratios studied in Module 3 may be useful for this
purpose.
Efficiency and profitability
Operating Margin Surplus (or deficit) ÷ Revenue x 100 %
Expense Margins Individual Expense ÷ Revenue x 100 %
Return on Assets Surplus (or deficit) ÷ Assets x 100 %
In these calculations, the revenues, expenses and assets relate solely to the project,
programme or entity under review.
As noted in sections 4A.2 and 4A.3, auditors must remain alert to the importance of audit
risk and materiality at all stages of the engagement, while quality assurance arrangements,
including supervision, are also essential throughout. Audit documentation is likewise
completed at all stages and is described below in section 4C.2.
There is an important difference between information and evidence. Information becomes
evidence when it is used in support of a condition. This requires the information to be
analyzed and evaluated using acceptable logical and numerical techniques to ensure
conclusions are valid.
4C.1: Reflection
Which, if any, is the most important dimension of performance: efficiency, economy, or
effectiveness?
Which is the hardest dimension to measure: efficiency, economy, or effectiveness?
As the auditor, how do you know when you have satisfied the requirement for sufficiency
of information?
4C.2 Audit Documentation
Standard 2330 – Documenting Information describes the requirements for internal auditors
to maintain records during an engagement. Documentation must be available to support the
results and conclusions recorded in the report. Much of the standard focuses on access and
retention.
Internal auditors must document sufficient, reliable, relevant, and useful information to
support the engagement results and conclusions.
2330.A1 The chief audit executive must control access to engagement records. The
chief audit executive must obtain the approval of senior management and/or legal
counsel prior to releasing such records to external parties, as appropriate.
2330.A2 The chief audit executive must develop retention requirements for engagement
records, regardless of the medium in which each record is stored. These retention
49

equirements must be consistent with the organization’s guidelines and any pertinent
regulatory or other requirements.
2330.C1 The chief audit executive must develop policies governing the custody and
retention of consulting engagement records, as well as their release to internal and
external parties. These policies must be consistent with the organization’s guidelines and
any pertinent regulatory or other requirements. 72
The auditor must determine what should be documented while avoiding the temptation to
include every piece of information gathered. Data should only be included if it is used as
evidence to support conditions and recommendations.
While evidence may include physical evidence (such as samples) and testimonials from
individuals and groups, generally evidence is prepared in digital or paper-based format that
are records derived from multiple sources. Documentation requirements also include records
of analysis. The steps taken to establish the sufficiency, reliability, relevance, and usefulness
of evidence should be clear from the documentation.
Potential weaknesses in each major category of evidence are considered by Rauum and
Morgan where the auditor should take extra care, as shown below.
Physical Evidence – be alert to evidence that:
Is collected without a specific methodology and may not be representative (i.e., it may
be an isolated case or an outlier).
Is a photograph without a record of observation or memo to the record.
Lacks a clear chain of custody (e.g., police confiscated drugs).
Lacks a test of inventory (i.e., may look at an oil tank and assume it is full of oil when
actually it contains mostly water with a layer of oil on top).
Is not witnessed (i.e., comes from only one observer).
Is collected by an unqualified observer.
Is a “setup” (i.e., contrived to give a false or misleading impression).
Documentary Evidence – be alert to evidence that is:
Taken out of context (pulled from other documents).
From a newspaper or magazine article (which may be biased by personal, political,
religious, or other opinion).
Other auditors’ work (i.e., relying on their work without checking).
From a secondary, not a primary, source.
Outdated.
Unofficial.
Refutable.
Controversial.
Incomplete.
Unclear (unidentified acronyms, etc.).
Inconsistent.
72
Standard 2330 – Documenting Information, The International Professional Practices
Framework, The IIA, 2016.
50

Testimonial Evidence – be alert to evidence that is:
From a source whose knowledge about the topic is in doubt.
From a source whose credibility is questionable.
From a source who is not impartial and/or has a vested interest.
The only evidence used to support a point (for example, if we rely only on statements
from one person – one worker in a group of workers).
Uncorroborated.
Used out of context.
Inconsistent.
Incomplete.
A “snow job” (an attempt to flatter or persuade).
Disguised as documentary evidence (examples: written comments or numbers written
or typed on a blank piece of paper by an official during an interview).
Improperly substituted for documentary evidence (e.g., relying on what an official told
us the law says instead of obtaining the law or using testimony for the date of a
contract was signed instead of using the contact).
Analytical Evidence – be alert to evidence that:
Uses inappropriate analysis.
Lacks methodological rigor.
Is based on erroneous or useless physical, documentary, and/or testimonial evidence.
Is interpreted inappropriately.
Depends on microcomputer work that does not used quality assurance/control
procedures.
Potential Weaknesses in Evidence 73
The process of documenting evidence should be covered by quality assurance procedures.
Care should always be taken to protect personal, confidential, and sensitive information.
To help with the process of audit documentation, some teams use a template or matrix, like
the one given below, to structure the results. Findings are conclusions based on the analysis
of information, should be stated succinctly and precisely, and be supported by a clear
expression of the criteria used, the condition identified, the causes of the condition, and the
effects of the condition (the outcomes and impacts).
Audit objective
Audit question(s)
Condition
Criteria
Finding Evidence,
analysis
Cause(s)
Effect(s)
Developed in the planning stages, linked to the scope
Developed to help expand and investigate the objective
Relevant discoveries during information gathering (a description of
actual performance in terms of economy, effectiveness, and
efficiency)
Expected or desired conditions
Supporting evidence for the condition
Identification of root causes for the condition
Identification of actual or likely outcomes and impacts of the
condition
73
Rauum and Morgan, Performance Auditing: A Measurement Approach, The Internal Audit
Research Foundation, 2009.
51

Sufficiency
Good practices
Conclusions
Recommendations
Confirmation or otherwise of the sufficiency of evidence to support
the conclusions given as well as a description of any remaining
work needed
Examples of good practice or benchmarks that may inform
recommendations for improvement
Summary messages, opinions, and answers to audit questions
Proposals to address deficiencies and make improvements
Audit Findings Matrix 74
Conclusions and recommendations may not be developed for every finding and are often
based on multiple findings.
4C.2: Reflection
How do you decide what information needs to be documented to prevent excessive audit
documentation?
What kind of evidence can be the most problematic for an auditor when determining
reliability (direct, documentary, testimonial, or analytical)?
What standard templates are used by your internal audit function to document results?
What improvements could be made to those templates?
74
Adapted from Performance Audit ISSAI Implementation Handbook, IDI, 2021. For more on
audit design matrix and evidence collection planning, see for example European Court of
Auditors “Performance Audit Manual,”
52