Australian Bureau of Statistics
Celebrating the International Year of Statistics 2013

STAGE 1: INITIATE A QUALITY INCIDENT RESPONSE PLAN

This section includes the following subsections:

• What is a quality incident?
• Identify a quality incident
• When should a quality incident response plan (QIRP) be initiated?

WHAT IS A QUALITY INCIDENT?

A quality incident occurs when the quality of the data are called into question. This could occur because an anomaly has been identified with data already released in the public domain; or a potential error that could have a negative impact on the outputs and agency is identified in data that are not yet in the public domain.

Throughout this paper the term quality incident is used to mean that either the quality of the data or the processes that produce the data are being called into question (or both), as well as covering the situation when a definite error is identified in the data.


IDENTIFY A QUALITY INCIDENT

A quality incident that warrants a QIRP response is one where the consequences of the potential risk being realised are severe for either the organisation or the public. A severe situation signals extreme circumstances and is a signal to 'call a QIRP' immediately, postpone the release of the data if need be, and commence investigations into the quality incident.

In order to initiate a quality incident response plan a quality incident needs to be identified. There could be a variety of triggers that indicate that there is or is about to be a quality incident. Some of the obvious triggers that the Australian Bureau of Statistics (ABS) has noted are:

• Staff or data users identify coherence problems;
  • within the current set of data, or between the latest release and historical data, e.g. an estimate for a data series may show an unexpected growth or decline compared to the previous time it was collected, which is outside the accepted tolerance level;
  • with other sources of similar data, e.g. between the unemployment rate and job vacancies;
  • or with real world events, e.g. if a key statistic does not move in the expected direction given all other data in the economy then there might be an issue with the data.
• The data do not meet expectations of sources both internal and external to the organisation, e.g. economists, experts, senior management within the organisation.
• Other indicators in the statistical process indicate there may be an issue, e.g. a quality measure shows high imputation rates; there has been a delay in the receiving of data at a critical time, which has put pressure on the process.

Each of the above factors by themselves may not be enough to call a quality incident, but rather a combination of some of them may lead to a quality incident being identified and a quality incident response plan being invoked.

The ABS Data Quality Framework is a useful tool that may help when trying to identify a quality incident. The ABS uses the seven dimensions of the ABS Data Quality Framework; institutional environment, relevance, timeliness, accuracy, coherence, interpretability, and accessibility; to assess, compare and declare the quality of final statistical outputs. However, the ABS Data Quality Framework can also be used as a checklist and applied to intermediate outputs at earlier stages of the statistical cycle.

When an anomaly with the data is identified, it may be useful to create a list of questions relating to each dimension of the ABS Data Quality Framework to help determine if the anomaly is actually a quality incident. By doing so, a determination can be made as to whether the quality incident is severe enough to call a QIRP. Some questions that might be useful to ask in order to determine if a quality incident has occurred are the following:

• Institutional Environment
  • Has there been a legislative change that may have impacted on the data?
  • Has the mode in which the data are collected changed? E.g. formerly face to face interviews are now being collected via telephone or using the internet.
  • Are senior managers questioning the data?
• Relevance
  • Has there been any change to the population of interest for the collection?
  • Have any key data items or questions changed their definition or construction since last time?
  • Are external users questioning the data?
• Timeliness
  • Were there time pressures during the process due to other delays?
  • What parts of the process ran late during the statistical cycle?
  • Has there been insufficient scrutiny of the quality of the outputs due to processes running late?
• Accuracy
  • Has there been a significant change in response to certain questions? E.g. a particular question has a large non response.
  • Have any other quality assurance tools flagged potential issues?
  • Have there been any changes to the methodology underpinning the collection?
• Coherence
  • Do the data make real world sense?
  • Are the data comparable with other statistical releases?
  • Are the data comparable with themselves over time?
• Interpretability
  • Have there been any ‘real world’ events, which may have impacted on the data? E.g. floods, global financial crisis.
  • Have there been any changes to classifications or standards associated with the data? E.g. changes to industry classifications or country classifications.
• Accessibility
  • Has there been a change in the systems that produce the data?
  • Has the format changed for the release of the data? E.g. the layout of the data in the spread-sheet has changed from the previous release.

More information on the dimensions of the ABS Data Quality Framework and its uses can be found in the information paper ABS Data Quality Framework, May 2009, (cat.no 1520.0) or alternatively in the Data Quality Online tool on the National Statistical Services website (http://www.nss.gov.au/dataquality/).

Back to top

WHEN SHOULD A QUALITY INCIDENT RESPONSE PLAN BE INITIATED?

It is important that a quality incident response plan is only used when it is really required. A key point of a contingency plan is to evoke a powerful response to a problem that needs to be dealt with before proceeding with business as usual.

Using a contingency plan too often, or when it is not necessary, may decrease the effectiveness of this response, and waste time and resources, as well as risk undermining the self-confidence of team members. However, it is critical that quality issues are not overlooked for too long or worse still, missed altogether. Hence, it is important to be able to identify quality incidents that require a quality incident response plan.

An assessment of the potential quality incident should be made to determine its impact, if it is realised, on the reputation of the organisation and also on the quality of the data. A quality incident that has potentially severe consequences to the organisation or to users of data should signal that a Quality Incident Response Plan is required. The initial assessment of the potential quality incident may provide enough information for senior managers to be satisfied that the data are correct and hence, a quality incident response plan process will not be invoked.

At the Australian Bureau of Statistics, any quality incident that could cause a loss of reputation, credibility, or impact negatively on provider or user confidence due to the publication of misleading data, would instigate the initiation of a quality incident response plan to deal with the issue. For example, if the Labour Force estimates prior to release revealed that one or more of the headline indicators behaved in an unexpected way, given the previous months' trend and everything else that is known about the current economic situation, then a quality incident response plan would be invoked as a way of further quality assuring the data. This is because the consequence of the data being incorrect and released to the public would not only have a detrimental effect on the reputation of the ABS but may also impact negatively on the economy. A quality incident that is discovered after the data have been released publicly would warrant an immediate quality incident response plan.

For those quality incidents that are identified prior to the public release of data, a risk assessment of the quality incident could be undertaken to assist in determining whether a quality incident response plan is required. Each organisation has its own risk assessment matrix to assist in decision making. However, a risk assessment matrix that may be of use is the ABS Statistical Risk Management Framework . This can be found in the appendix of the Quality Management of Statistical Risk Using Quality Gates information paper.

Knowing when a potential quality incident is serious enough that it requires a quality incident response plan to be invoked is an important part in managing statistical risk. A quality incident response plan assists in determining what the causes are of the potential quality incident and how to resolve them.

Back to top