Use of ABS microdata and impact on research quality

Data confidentiality guide

ABS microdata products in the context of confidentiality controls, and impact of data treatments on analysis

Released
8/11/2021

ABS microdata products

ABS releases microdata products that can be accessed for statistical and research purposes:

  • basic microdata which can be downloaded into a user's own computing environment
  • detailed microdata files, available through the ABS DataLab

ABS also releases TableBuilder, which uses underlying microdata to allow researchers to create their own automatically confidentialised tables, graphs and maps (aggregate output).

Each product is designed to meet a different research requirement. A comparison of these products is shown in Table 1. You can also Compare data services for more detail about a wider range of ABS products.

Table 1: ABS microdata products
Microdata productBasic microdataDetailed microdataTableBuilder

Access via

Utility and suitability

Basic utility:

  • surveys
  • census samples

Very high utility:

  • surveys
  • census data
  • administrative data
  • complex integrated data

High utility:

  • surveys
  • census data
  • administrative data
  • complex integrated data

Purpose

  • Simple modelling
  • Multivariate analysis
  • Exploratory analysis
  • Complex modelling
  • Detailed analysis
  • Small to very large tables
  • Graphs and maps
Confidentiality controls applied

Data treatment

  • Direct identifiers removed
  • Data available at broad levels only (eg state)
  • Variables aggregated (eg 5 or 10 year age groupings)
  • At risk records suppressed/removed
  • Direct identifiers removed
  • At risk records suppressed/removed
  • Direct identifiers removed
  • At risk records suppressed/removed

Context controls

  • User registration
  • Legal undertakings
  • Secure storage of microdata within user's environment
  • approval of people
  • approval of projects
  • secure IT/physical access environment
  • clearance process applied to output to be removed from system
  • approval of people
  • approval of projects
  • secure IT/physical access environment
  • clearance process applied to output to be removed from system

Impact of data treatment on analysis

Treating the data itself may restrict the ability of a researcher to answer a particular question. For example, a major difference between basic microdata and detailed microdata is that data item categories in the former have been collapsed or aggregated to a greater degree, which reduces the level of detail available. In some instances it may therefore be more appropriate to use an Expanded CURF. Similarly, it may be better to conduct research (within the ABS DataLab) using detailed microdata files if the Expanded CURF does not contain enough detail to answer the researcher's question. 

There are, however, situations where data treatment may not adversely affect the quality of the data or the reliability of the research.  For example, a 2010 study used data from the ABS's Survey of Mental Health (2007) to compare results attained from using the Expanded CURF with results attained from using the untreated main-unit-record file (MURF). As Table 2 shows, the results were almost identical. 

Table 2: Hazard ratios* for smoking cessation and incident of anxiety disorders
DisorderHazard ratio
(from treated microdata)
Hazard ratio
(from original microdata)
Standard error of
hazard ratio
Difference between hazard ratios,
as proportion of standard error
No lifetime mental disorder11(reference category) 
Anxiety disorder (type):
Panic disorder0.596780.596150.103350.0102
Agoraphobia0.449360.449360.08420
Social phobia0.553740.553740.079730
Generalised anxiety disorder0.336310.336310.078460
Obsessive-compulsive disorder0.477820.477820.106740
Post-traumatic stress disorder0.632210.632210.078650
Anxiety disorder (severity):
Mild0.749010.748820.114950.00218
Moderate0.589420.58920.082750.00447
Severe0.391960.392020.060160.00249

* Hazard ratios compare the incidence of an event in one group to another group over time
Source: Lawrence, D., Considine, J., Mitrou, F. & Zubrick, S.R. (2010) ‘Anxiety disorders and cigarette smoking: Results from the Australian Survey of Mental Health and Wellbeing’, Australian and New Zealand Journal of Psychiatry. Vol. 44, pp. 521-528.