Measuring performance information
Candidates will be familiar with the concept of Key Performance Indicators (KPIs) which are widely used by private sector organisations in relation to non-financial information such as social and environmental reporting; there have been several examination requirements in past P7 exams focussing on this syllabus area. In the public sector the same principles apply in that target KPIs will be established as a performance objective and the organisation’s performance against the target KPIs will be measured.
Performance measures should be measurable and relevant if they are to be effective. Measurability means trying to ensure that there is consistency in how performance information is captured and reported. The measures should be clearly defined and unambiguous, but measurability is sometimes difficult where the subject matter of the performance information is subjective in nature. For example for an ambulance service it would be quite easy to measure the average time taken for an ambulance to respond to an emergency as this is quantifiable, but more difficult to measure the patient’s satisfaction with the service provided as this is based on the patient’s opinion.
An issue linked to measurability is the existence of data to generate the performance information. Much of the work involved in setting up a good system for reporting on performance information is focussed on ensuring the completeness and accuracy of supporting information and that the information is sufficiently robust to withstand scrutiny.
Relevance means that the performance information addresses a valid concern and public sector organisations should consider the specific needs of their stakeholders in developing relevant performance measures. Continuing to using the UK’s NHS as an example, identified stakeholders who regularly review the NHS performance information include:
- The government department responsible for health services
- Medical staff
- NHS management team and non-executive committee members
- Private companies who supply to the NHS
- Academics and students researching the NHS
The NHS therefore has to produce a range of performance measures relevant to the needs of this wide range of stakeholders. Different stakeholders have different needs, for example patients may focus on the effectiveness of a certain medical procedure, whereas management may focus on the cost of providing that procedure. Therefore a very wide range of performance information may be required yet it would be pointless to set targets and produce performance information on an issue which is not relevant to any stakeholder.
The audit of performance information
It is worth reiterating the difference between the audit of performance information and performance auditing as both are likely to occur in the public sector. Candidates are reminded that the audit of performance information is concerned with the audit of reported performance information against predetermined objectives. The auditor’s role here is usually to report on the credibility, usefulness and accuracy of the reported performance. Performance auditing is related to the evaluation of how the public sector body is utilising resources and often focuses on determining how the public sector body is achieving economy, efficiency and effectiveness, sometimes referred to as value for money auditing. It is the former that is the focus of this area of the P7 syllabus.
In some jurisdictions it is part of the audit requirement for public sector organisations that the auditor should report on performance information. In jurisdictions where this is not a requirement, the auditor may be asked to perform a separate engagement to the financial statement audit, the objective of which is to report specifically on the performance information. In either case, the auditor will need to plan procedures in much the same way as in a conventional audit scenario. Candidates are therefore encouraged to apply their existing knowledge of audit planning (risk assessment) and evidence gathering techniques to this type of information. The auditor is still looking to ultimately report on the validity of the information included in this respect. The auditor may find the principles of ISAE 3000 Assurance Engagements other than Audits or Reviews of Historical Financial Information provide a useful framework for planning and performing the work on performance information.
As with any engagement to provide assurance, this would likely start with an understanding of the entity to ensure knowledge of the predetermined performance measures, an evaluation of the systems and controls used to derive and capture the performance information and also performing substantive procedures on the reported measures. The auditor will also need to understand the rationale behind the measures that are being reported on, considering the relevance and suitability of them in terms of the objectives of the public sector organisation in order to help assess the usefulness of the information being provided.
Audit procedures may include:
- Tests of controls on the systems used to generate performance information
- Performing analytical review to evaluate trends and gauge the consistency of the information
- Discussion with management and other relevant individuals, for example those responsible for the reporting process
- Review of minutes of meetings where performance information has been discussed
- Confirmation of performance information to source documentation; this may be performed on a sample basis
- Recalculation of quantitative performance information measures
Of course, the procedures must be specifically tailored to the performance information subject to the audit. Further as in any audit, the working papers must contain a summary of findings and clear conclusions on the procedures that have been performed.
Reporting on performance information
There is no specific format or wording that is prescribed by international regulations for reporting on public sector performance information, though in some jurisdictions the national regulators may issue country-specific requirements.
Generally, the auditor will provide a conclusion on whether the public sector entity has achieved its objectives as shown by the reported performance information and concludes on the information itself. This conclusion may be in the form of a reasonable assurance conclusion – ie an opinion is expressed, or may be in the form of a negative assurance conclusion – ie no opinion is expressed. Essentially, in the absence of any jurisdiction specific requirements, the auditor will agree the type of conclusion with the public sector organisation and usually its regulating body.
Often the performance information will be provided as part of the public sector organisation’s integrated report, in which case the auditor’s conclusion will be included within the integrated report.
The audit of performance information in public sector organisations can be approached in a similar way to the audit of KPIs in private sector organisations, and conventional audit techniques can be employed, though they will need to be tailored to the specific measures that are subject to audit. In approaching scenarios based on this syllabus area, candidates are encouraged to apply their understanding of audit techniques to the specific information in the question and to avoid vague and unfocussed remarks.
Written by a member of the P7 examining team