• Summary
• Downloads
• Explanatory Notes
• Related Information
• Past Releases

The Quality Challenge Presented by Big Data

The ABS is involved in an international task team to investigate and address quality issues in using big data for official statistics purposes.

The amount of data available in digital form is increasing in volume, variety and speed of access, a phenomenon known as the emergence of 'Big Data'. This presents both opportunities and risks for National Statistical Offices (NSOs): opportunities created by the new data sources for new products and analyses, as well as the risk of being ‘left behind’. An additional risk is that the quality of the new sources of information and the statistical products derived from them might not meet the quality standards required of official statistics.

The High-Level Group for the Modernisation of Statistical Production and Services (HLG; formed by the Conference of European Statisticians Bureau) identified Big Data as a key challenge for official statistics. As a consequence, a project to investigate Big Data for official statistics, The Role of Big Data in the Modernisation of Statistical Production, was undertaken with the United Nations Economic Commission for Europe (UNECE) undertaking the project management and secretariat functions. The project began in July 2014.

The ABS has been playing a key role across a number of activities of this international collaborative project that includes multiple NSO representatives. One area where we have been particularly active is in the project’s Big Data Quality Task Team, with the ABS taking the leadership of this team.

The goals of the team were to assess existing statistical data frameworks as to their suitability for Big Data and then produce a framework that explicitly addressed Big Data if required, test that framework with test datasets produced by other teams within the project, and propose quality indicators that might be useful in a Big Data environment.

This work is almost complete and a summary of the team’s work so far was presented to the HLG meeting in Geneva in November. The final draft paper will be circulated to members of the ABS big data forum for consultation.

The work represents the first steps in formulating a systematic approach to data quality with Big Data for National Statistical Offices.

Further Information
For more information, please contact David Dufty (02 6252 6187, david.dufty@abs.gov.au).

The ABS Privacy Policy outlines how the ABS will handle any personal information that you provide to us.