Australian Bureau of Statistics
1542.0 - Statistical Quality Incident Response Plan, Jun 2012
Latest ISSUE Released at 11:30 AM (CANBERRA TIME) 26/06/2012 First Issue
|Page tools: Print Page Print All RSS Search this Product|
The QIRP follows a step-by-step process starting from when the quality of the data or process are first called into question, at which point a decision is needed as to whether the quality incident is serious enough to require a contingency plan. At the Australian Bureau of Statistics (ABS), the decision to 'call a QIRP' is often made by the senior manager who is responsible for final approval of the statistics for release to the ABS website. This is done in consultation with the manager of the collection area and any other relevant team members who can provide insight into the situation to help with the decision making. If the decision is made to go ahead with a QIRP, then the initial QIRP meeting needs to be planned and organised. The initial meeting will discuss the issues surrounding the quality incident and establish a plan for investigating possible causes of the quality incident. The initial meeting will also be when the contingency planning begins, including brainstorming the possible consequences of the quality incident and the actions that can be taken to minimise the risks.
A key to the successful resolution of a quality incident is clear documentation of the management of the QIRP. Documentation should clearly define the roles, such as who is organising the meetings, who is following up on the investigations, and who is the main ‘driver’ of the QIRP. Careful consideration also needs to be given to identify the key stakeholders, and whether, how and when the stakeholders should be informed about the quality incident and how it is progressing.
Note that whilst the quality incident response plan involves a number of steps, these steps can be run at the same time. There may be an operational requirement to ensure that the steps occur concurrently in order to adequately manage the quality incident within a short time frame.
INITIAL QIRP MEETING
Once it has been determined that there is a quality incident and a decision is made to use the quality incident response plan, the initial QIRP meeting needs to be organised (usually at short notice), ensuring that all people who may need to be involved in the process are included in the invite.
The meeting's aims are to identify the actual issue or problem, brainstorm possible solutions, and agree on ways to move forward and resolve the issue.
Back to top
The potential consequences of the quality incident will influence who needs to be involved in the initial meeting. For example, if the issue has been identified prior to any data being published or released then the issue may be able to be handled at a more local level, with senior managers being kept informed of the issue but not necessarily directly involved in the response. However, if the issue is only identified when the data are already in the public domain, then the QIRP meetings may immediately need to involve the senior management of the organisation, as the potential consequences to the reputation of the organisation and impact of incorrect data in the public domain will need to be managed quickly.
Who to include in the initial meeting will depend on the nature of quality incident. Initially, it is probably best to include more people rather than less to ensure that all the issues and potential solutions are identified. Participants at the initial meeting would include those people who are directly responsible for producing the statistics and those people who support the production of the statistics. For example, at the ABS not only would the collection area people be involved, but people from the methodology area and time series analysis area (if relevant) would be involved in the meeting. Other participants might include areas of the ABS who are responsible for other aspects of the collection cycle that may impact on the statistics and process in question. Once again, participants at the meeting will depend on the nature of the quality incident and where it has been identified in the collection cycle process.
Other stakeholders who may be kept informed of progress but do not attend the meeting include areas of the organisation who are not directly involved in the production of the statistics but their work is dependent upon them. For example, at the ABS the publishing area who manages the ABS website and its releases would be kept informed of any potential delays to statistical releases in order to manage their workload accordingly.
Along with identifying the participants, expectations about resourcing also need to be stated. Participating stakeholders need to be aware that they may be expected to allocate dedicated resources to the QIRP process until the issue is resolved.
It is important that all the key people hear all the facts first hand. Once the QIRP is underway there is no time for misunderstandings about what is needed, why, or what it might mean at a later date. It helps if everyone involved is aware of the entire process and their part in it so that they can adapt and help each other. Having all the stakeholders together initially is a more effective way of managing the QIRP compared to having multiple bilateral meetings. The identifying of all of the issues, understanding the problem, working out potential solutions, identifying implications and dependencies, and agreeing on an action plan in a very short period of time is facilitated through having all key players together at the first QIRP meeting.
To effectively manage the QIRP, roles for addressing the quality incident need to be defined up front to ensure that stakeholders are aware of their responsibilities and time is not wasted due to further confusion. The roles that need to be established in this step are:
Note that different people are not necessarily required for each of the roles. For example, the driver and meeting co-ordinator may be the same person. However, it is important to have each of these roles identified and allocated so that the underlying responsibilities are not missed.
Back to top
ESTABLISH THE FACTS, DEFINE THE PROBLEM, AND ESTABLISH OBJECTIVES
The objectives of the first QIRP meeting after the quality incident has been discovered are to outline the key facts of the quality incident so far and come up with a definition of the problem, and establish a plan for investigating the problem. The meeting should include a rigorous discussion of the incident, giving the opportunity for everyone involved to present and discuss the key facts surrounding the incident, so that the real issue(s) can be identified and defined. It is critical that detailed notes are taken from this discussion and circulated afterwards so that everyone involved has a clear understanding of what was discussed and decided.
This meeting is also an opportunity to discuss the QIRP approach so all participants understand the process. A QIRP is about working out what the issues are and why they occurred, in order to implement solutions that fix the problem and enable the process to move forward. Solutions may not only apply directly to the process, they may be more fundamental in terms of overall staff capability building across an organisation. The QIRP approach is not about assigning blame; everyone involved should feel free to contribute openly and honestly without fear of retribution. It is the facilitator's role, if one is being used, to draw out all the potentially relevant facts.
CHANGING MINDSET - ASSUME THAT SOMETHING IS NOT RIGHT
It is important that the meeting triggers a shift in mindset to reflect the potentially serious situation. Once a quality incident is suspected, the aim of the meeting should not be to reassure each other that there is no major problem, but rather agree that there is a potentially serious problem. Potential causes and solutions will be more rigorously considered if there is a working assumption that something has gone wrong. From the outset of the meeting, the ability to recognise the changed situation, its significance, and the need to look at the problem differently may be crucial to resolving the quality incident.
Back to top
ESTABLISH A RANGE OF POSSIBLE SOURCES OF THE PROBLEM
The initial QIRP meeting is the time to establish what the possible sources of the quality problem may be and how these will be investigated. There are some different methods and questions that may help to generate a list of potential sources of the problem. The most common of these is to have a brainstorming session that everyone in the meeting is encouraged to participate in. The brain storming session should come up with a list of any potential sources of the problem, which can then be discussed according to whether each idea is feasible or likely. It may be helpful at this point to rank the ideas from most likely to least likely, as this will indicate where the ensuing investigations should start from. The discussion of each possible source should also identify what further information is needed to confirm or deny the idea as a source of the problem.
If there are other standard quality tools in place, such as quality gates, then it is a good idea to check these as they may help to pin point the cause of the problem. Another approach is to examine each step of the statistical process for any problems, inconsistencies or changes, be they to the process, methodology, collection, processing of data, or key people. For any change it is important to understand what has changed, how it was managed, the expected impacts of the change on the process and statistics, and if there were circumstances surrounding the implementation of the change that may have had an effect. For instance, time and resource constraints may have impacted on the implementation of a change, which could mean that the change was not tested to the usual standard.
It is also important to look at whether there are other issues that may be clouding the process and masking the real problems or distracting team members from the main issues. This could include real world issues that are affecting the statistics (e.g. changes in government policy that may impact such as the baby bonus, changes in school leaving age, federal budget decisions, Christmas and New Year public holidays), or methodological adjustments (e.g. treatment of outliers or the seasonal adjustment process).
This process will likely unveil a number of issues that will need to be investigated further. Some documentation will be helpful to keep a track of the roles and responsibilities for investigating each issue. As the initial generation of potential problems is a brainstorming exercise, all suggested causes should be documented. If suggested causes are not followed up then the reasons why should be documented for future reference. It is also important to make a note of any issues that distract from the serious issue.
A list of the possible sources of the cause of the quality incident should be made along with appropriate information regarding who is responsible for investigating, when the investigation needs to be complete and also the priority of the investigation.
An example of how this may be documented is below:
Back to top
CONTINGENCY PLAN AND ACTION
The brainstorming session and discussion from the QIRP meeting may produce a lot of information about the quality issue and the potential sources of the problem, which will need to be investigated. The final part of the meeting should therefore propose a plan for investigating these leads and subsequently dealing with the problem. If the investigations do show the problem, then what is going to be done about it, what needs to happen next, and what will be done later in the plan?
This part of the meeting is about looking ahead to where the data are going, what they will be used for, and anticipating what consequences the quality issue will have further downstream. The ABS looks at these issues differently depending on whether the quality incident is identified before or after data have been released into the public domain. For pre-release issues, time may be an important consideration. If the data are due to be published, a necessary contingency may be to postpone the publication date until the issue has been thoroughly investigated. If the data are due to be sent to another area of the organisation then the other area needs to be informed of the issue and a possible delay. When the data are released, further explanatory notes may be required so that users can interpret the data in the context of the issue.
For post-release quality issues, the focus is on managing the impact of having incorrect data in the public domain. Senior managers should be involved, as they may need to brief the media and key users, clients, and stakeholders of the incident and how the issue is being resolved. For example, the data may need to be amended or re-released, or have additional explanatory notes added. It is important that this process is handled transparently by the organisation. This is to ensure that the steps being taken to fix the current problem and prevent further incidents of that nature occurring are conveyed to the public to assist in their understanding of the data as well as to maintain the public’s trust in the organisation.
Any potential problems as a result of the quality incident and suggested contingency actions should be documented and monitored. Below is a basic template for these contingency actions.
Template and example for recording contingency actions:
At the end of the QIRP meeting, the meeting coordinator should book the next meeting so that the progress of investigations can be checked and discussed, and potential solutions implemented. A meeting with senior managers may also be needed to present the options and to decide on the appropriate way forward. Given the tight time frames for resolving quality incidents, these meetings should be comprehensive but to the point.
Back to top
These documents will be presented in a new window.
This page last updated 25 June 2012