“Auditing is essential – and as far I am concerned it should be inquisitive auditing. If you read the report of a major incident you will often see that it was preceded by the neglect of signs that all is not well.”
Lord Cullen, Chair of the UK government inquiry into the Piper Alpha disaster, in his keynote, What have we learned from Piper Alpha?, at the 2013 Oil and Gas UK conference.
What is an Audit?
An audit is sometimes narrowly considered to mean a formal examination of an organization's or individual's accounts or financial situation. There are many different types of audits, however, and financial audits are one among many.
For our purposes, we will use the ISO 19011:2018 Guidelines for auditing management systems. ISO19011 defines an audit as a
"systematic, independent and documented process for obtaining audit evidence [records, statements of fact or other information which are relevant and verifiable] and evaluating it objectively to determine the extent to which the audit criteria [a set of policies, procedures or requirements] are fulfilled."
A risk management system audit evaluates a risk management program to determine its conformance to policies, procedures, legal or contractual commitments, and organizational objectives.
Auditing involves a documented activity performed to evaluate, by examination of objective evidence, that any relevant system elements are appropriate and effective and have been developed, documented, and implemented per specified objectives and requirements.
It should establish whether processes function within set boundaries, against documented criteria such as instructions or standards. An audit might also involve checking conformance against commercial or technical specifications, performance criteria, or contractual requirements.
Depending on the scope of the audit, it might also examine the resources (people, equipment, materials) applied to transform inputs into outputs, the environment, the methods (procedures, instructions), and the measures collected to determine process performance. It should also evaluate the adequacy and effectiveness of procedures, work instructions, flowcharts, training, or process specifications.
Terms of Reference
The first requirement for an audit involves specific terms of reference. These should include the following items as a minimum:
Scope
Purpose
Objectives
Audit criteria
Three Pillars of Performance
Management systems, at their essence, involve four elements.
Inputs
Processes that act on those inputs
Outputs
A feedback loop
Figure 1: Management Systems
All four elements can and should be evaluated against the following three pillars.
POLICY - Policies, procedures, protocols, guidance, forms, standards, et cetera that provide the core philosophy and guidance for the management systems. These give the management system's why, what, when, who, where, and how.
ASSURANCE - This involves training, capability, competency, funding, resources, and communication to provide overall confidence that the organization can execute the Policy element.
COMPLIANCE - The feedback loop is provided by audits, inspections, reviews, certifications, system logs, incident reports, and management reports. This element involves validation and monitoring of the policy and assurance elements.
This Policy Assurance Compliance (PAC) model is illustrated below.
Figure 2: PAC Control Effectiveness Model
All three of these should be involved in assessing any individual control. This policy, assurance, compliance (PAC) model is conceptually similar to a three-legged stool. If you remove any of the legs, it is no longer a stool.
The best conceivable policy or procedure will not be effective if staff are not trained to apply it. Even if they are trained, it will still not be fully effective if the resources required to use it are unavailable.
Equally, even a well-trained staff with a fully resourced policy or procedure may have some gaps. Deficiencies might only become apparent over time or may be introduced due to changes in the operating environment. Regular audits and inspections will help ensure that the Policy and Assurance elements remain effective, provided that recommendations are applied promptly.
Evidence and Rigor
All audit findings, observations, and recommendations should be based on and include documented evidence. Evidence can take many forms, befitting the nature and scope of the audit.
For example, evidence might include photographs, declarations, witness reports, policies, procedures, checklists, inspection reports, or incident reports. The 4C Audit Findings and 4A Audit Recommendations provide an established model to facilitate preparation of audit reports.
Control Effectiveness Rating
Controls can be rated against any number of criteria. They can, for example, be ranked in a binary fashion as 'Effective' or 'Ineffective.'
This can however, introduce unneccessary debate. In practice, few controls are ever fully effective or completely ineffective. Using a binary approach can also result in a fierce debate about which category a control should be rated.
More granular effectiveness ratings can provide nuanced insights regarding which controls should be the priority in addressing deficiencies. For example, providing a third level of 'Partially Effective' allows an auditor to identify which controls require priority attention.
Below is an example of a six-point scale for control effectiveness rating. Each rating can be assessed individually against the three pillars of performance (Policy, Assurance, Compliance) to produce three ratings for each control. The lowest of the three would then be selected as the overall rating for that control.
There are many possible control effectiveness ranking models and they should be customized to the relevant context. This is just one example.
Example of a Control Effectiveness Rating Framework
5. FULLY EFFECTIVE
Controls are designed well for the risk and are performing as expected.
Controls are deemed effective, well documented, consistently implemented and are reliable in addressing the source of risk.
There is a high degree of confidence from management in the protection provided by the controls and no further action is required, except to review and monitor the existing controls.
There is possibly the potential that resources could be diverted from this to other controls if required.
4. SUBSTANTIALLY EFFECTIVE
Controls are generally designed well and are sufficient in managing the source of the risk or, controls are well documented, implemented, and reliable in addressing the source of risk.
There is a high degree of confidence from management in the protection provided by the controls.
3. PARTIALLY EFFECTIVE
Controls are in place but may be partially documented or communicated, inconsistently applied, or infrequently tested.
Weaknesses in the controls are minor or moderate and tend to reflect opportunities for improvement, rather than serious deficiencies in systems or practices.
2. LARGELY INEFFECTIVE
Controls are not documented or communicated or are inconsistently implemented in practice.
The controls are not operating as intended and risk is not being managed.
Controls are not in place to address the source of risk.
Action needs to be taken to remediate (i.e. redesign or replace control).
1. NONE OR INEFFECTIVE
Controls are not documented or communicated or are not implemented in practice.
The controls are not operating as intended and risk is not being managed.
Controls are not in place to address the source of risk. Immediate action needs to be taken to remediate (i.e. redesign or replace control).
0. UNKNOWN
The assessors were unable to rate this control due to time, scope, resources, or other limitations.
Alternatively, an organization could modify this table to include text for the PAC elements for each of the six rankings to provide a rating in one pass.
You can download a series of guides and templates (including a more complex control effectiveness rating system that incorporates PAC) from the DOWNLOADS menu at www.srmbok.com
And if you'd like to understand more about the role of auditing, you might like to check out the story of Piper Alpha in my latest book 'Tales from the Edge'.
Last but not least.
If you would like to know more about this concept and how to apply it, you can get in touch via Clarity.fm.
I also cover this and other models in my book on How to Performance Benchmark Your Risk Management and my Control Effectiveness and Auditing training courses.
