• Summary
• Downloads
• Explanatory Notes
• Related Information
• Past Releases

BIG DATA AND DATA INTEGRATION

Efficiency and effectiveness considerations, as well as government and community expectations, require the ABS to consider new ways of ingesting Big Data in the production of official statistics. Big Data, also known as ‘found data’, due to the very nature of their creation, generally suffer from coverage bias and measurement errors. Unless corrected, Big Data cannot be used for producing high quality official statistics often used in informing important decisions.

In this research, we considered the practical implementation of a Regression Data Integration (RDI) method, developed jointly by the ABS and a researcher from the Iowa State University, to correct for this bias using existing ABS estimation processes. The study demonstrates that the RDI method can be readily applied within ABS statistical production processes to produce estimates of total and associated sampling errors. The results show that the method has potential to achieve significant gains where little measurement error correction is needed in either dataset. When measurement error needs to be taken into account, the gains may be eroded.

For more information, contact Lyndon Ang Methodology@abs.gov.au

The ABS Privacy Policy outlines how the ABS will handle any personal information that you provide to us.