• Summary
• Downloads
• Explanatory Notes
• Related Information
• Past Releases

Statistical Impact Measurement Framework

The ABS is re-engineering many statistical business processes across a large number of statistical lines as part of the Statistical Business Transformation Program. This re-engineering carries with it some risk to the continuity of time series for the statistics that are produced. Of key concern is the potential to produce statistics that are inconsistent with past estimates due to process change.

While assessing the impact of process change on time series is not new, often significant effort is devoted to detecting, measuring and adjusting for an impact on the statistics produced. The Statistical Business Transformation Program is ramping up and the volume of work to assess the statistical impact due to changes to business processes will increase. As such, there is a need to be able to quickly assess the impact on key statistics as transformation is widely undertaken. The Statistical Risk and Quality Assurance (SRQA) section is developing a generic Statistical Impact Measurement Framework (SIMF) that will be able to be applied to all series based on aggregating unit level data, dealing with any change to current statistical business processes. The aim of the framework is to streamline and accelerate the identification and management of statistical risk.

During the development of the SIMF, it will be tested on historical changes that have been implemented at the ABS. For example, the SIMF has already been tested on surveys where web forms were introduced as a new collection option in 2012. However, since the Statistical Business Transformation Program has the potential to bring in new or different opportunities for statistical impact, simulations will also be used to test the SIMF for scenarios that have yet to happen. This will give an indication of the robustness and power of the framework while also identifying its limitations. The development of the framework will draw on the knowledge and experience of the Methodology Advisory Committee, with a paper to be put forward in June 2016. The committee will seek to provide advice on the validity of the framework as well as guidance with respect to its application during the implementation of the Statistical Business Transformation Program.

The general idea of the framework is to compare the units that were in sample both before and after the change. While not all the units may have been included in a trial, an impact can be detected if the units that were not in the trial were distinctly different to the units that were. A general linear model with a stepwise model selection method is used to determine whether the trial group was impacted statistically or not when assessing the response values. Response rate comparison between the trial group and the control group can also be compared using a logistic regression model.

Further Information
For more information, please contact Greg Griffiths (methodology@abs.gov.au)

The ABS Privacy Policy outlines how the ABS will handle any personal information that you provide to us.