4727.0.55.002 - Australian Aboriginal and Torres Strait Islander Health Survey: Users' Guide, 2012-13  
ARCHIVED ISSUE Released at 11:30 AM (CANBERRA TIME) 27/11/2013  First Issue
   Page tools: Print Print Page Print all pages in this productPrint All  


Although care was taken to ensure that the results of the 2012-13 National Aboriginal and Torres Strait Islander Health Survey (NATSIHS) and the National Aboriginal and Torres Strait Islander Nutrition and Physical Activity Survey (NATSINPAS) are as accurate as possible, there are certain factors which may affect the reliability of the results and for which no adequate adjustments can be made. One such factor is known as sampling error. Other factors are collectively referred to as non-sampling error. These factors, which are discussed below, should be kept in mind when interpreting results from the survey.

This chapter provides information on:


All sample surveys are subject to error, which can be broadly categorised as either:
  • sampling error
  • non-sampling error.

Sampling error occurs because only a small proportion of the total population is used to produce estimates that represent the whole population. Sampling error can be reliably measured as it is calculated based on the scientific methods used to design surveys.

Non-sampling error may occur in any data collection, whether it is based on a sample or a full count (eg Census). Non-sampling error may occur at any stage throughout the survey process. Examples of non-sampling error include:
  • persons selected for the survey may not respond (non-response)
  • survey questions may not be clearly understood
  • responses may be incorrectly recorded by interviewers
  • errors in coding or processing survey data.

Sampling and non-sampling errors should be considered when interpreting results of the survey. Sampling errors are considered to occur randomly, whereas non-sampling errors may occur randomly and/or systematically.


Since the estimates are based on information obtained from a sample of the population, they are subject to sampling variability (or sampling error), that is, they may differ from the figures that would have been obtained from an enumeration of the entire population, using the same questionnaires and procedures. The magnitude of the sampling error associated with a sample estimate depends on the following factors.

  • Sample design - there are many different methods which could have been used to obtain a sample from which to collect data on health status, health-related actions and health risk factors. The final design attempted to make survey results as representative as possible within cost and operational constraints. (Details of sample design are contained in the Survey Design and Operation chapter under Sample Design and Selection).
  • Sample size - the larger the sample on which the estimate is based, the smaller the associated sampling error.
  • Population variability - the extent to which people differ on the particular characteristic being measured. The smaller the population variability of a particular characteristic, the more likely it is that the population will be well represented by the sample, and therefore the smaller the sampling error. Conversely, the more variable the characteristic, the greater the sampling error.

Measure of sampling variability

One measure of the likely difference is given by the standard error (SE), which indicates the extent to which an estimate might have varied because only a sample of dwellings was included. There are about two chances in three that the sample estimate will differ by less than one SE from the figure that would have been obtained if all dwellings had been included, and about 19 chances in 20 that the difference will be less than two SEs. This is known as the margin of error (MoE) at the 95% confidence level. The margin of error at the 95% confidence level is expressed as 1.96 times the SE. The 95% confidence interval is the estimate +/- MoE i.e. the range from minus 1.96 times the SE to the estimate plus 1.96 times the SE.

Another measure of the likely difference is the relative standard error (RSE), which is obtained by expressing the SE as a percentage of the estimate to which it relates. The RSE is a useful measure in that it provides an immediate indication of the percentage errors likely to have occurred due to sampling, and thus avoids the need to refer also to the size of the estimate. More detail on the calculation of SEs, MoEs and RSEs can be found in the Technical Note.

For proportion estimates, the RSE is a percentage error of a percentage. The MoE has also been published for proportion estimates to show the absolute size of the sampling error. This will assist users in assessing the reliability of these estimates.

Estimates with relative standard errors less than 25% are considered sufficiently reliable for most purposes. However, estimates with relative standard errors of 25% or more are included in Australian Bureau of Statistics (ABS) publications of results from this survey. Estimates with RSEs greater than 25% but less than or equal to 50% are annotated by an asterisk to indicate they are subject to high SEs relative to the size of the estimate and should be used with caution. Estimates with RSEs of greater than 50%, annotated by a double asterisk, are considered too unreliable for most purposes. These estimates can be used to aggregate with other estimates to reduce the overall sampling error.

Relative standard errors for estimates are published in 'direct' form. For 2012-13 Australian Aboriginal and Torres Strait Islander Health Survey (AATSIHS), RSEs for estimates were calculated for each separate estimate and published individually using a replicate weights technique (Jackknife method). This direct calculation of RSEs can result in larger estimates having larger RSEs than smaller ones, since these larger estimates may have more inherent variability.

More information about the replicate weights technique can be found in the Technical Note.

Standard errors of proportions, differences and sums

Proportions formed from the ratio of two estimates are also subject to sampling error. The size of the error depends on the accuracy of both the estimates.

The difference between, or sum of, two survey estimates (of numbers or percentages) is itself an estimate and is therefore also subject to sampling error. The SE of the difference between, or sum of, two survey estimates depends on their SEs and the relationship between them.

The formulas to approximate the RSE for proportions and the SE of the difference between, or sum of, two estimates can be found in the Technical Note.

Testing for statistically significant differences

For comparing estimates between surveys or between populations within a survey it is useful to determine whether apparent differences are 'real' differences between the corresponding population characteristics, or simply the product of differences between the survey samples. One way to examine this is to determine whether the difference between the estimates is statistically significant. This is done by calculating the standard error of the difference between two estimates (x and y) and using that to calculate the test statistic using the formula below:

If the value of the test statistic is greater than 1.96, then we may say that we are 95% certain that there is a statistically significant difference between the two populations with respect to that characteristic. Otherwise, it cannot be stated with confidence that there is a real difference between the populations.


Lack of precision due to sampling variability should not be confused with inaccuracies that may occur for other reasons, such as errors in response and recording. Inaccuracies of this type are referred to as non-sampling error. This type of error is not specific to sample surveys and can occur in a census enumeration. The major sources of non-sampling error are:
  • errors related to scope and coverage
  • response errors due to incorrect interpretation or wording of questions
  • interviewer bias
  • bias due to non-response, because health status, health-related behaviour and other characteristics of non-responding persons may differ from responding persons
  • errors in processing such as mistakes in the recording or coding of the data obtained.

These sources of error are discussed below.

Errors related to scope and coverage

Some dwellings may have been inadvertently included or excluded because, for example, the distinctions between whether they were private or non-private dwellings may have been unclear. All efforts were made to overcome such situations by constant updating of lists both before and during the survey. Also, some persons may have been inadvertently included or excluded because of difficulties in applying the scope rules concerning the identification of usual residents and Indigenous status of those residents.

Response errors

Response errors may have arisen from three main sources:
  • flaws in questionnaire design and methodology
  • flaws in interviewing technique
  • inaccurate reporting by the respondent.

Errors may be caused by misleading or ambiguous questions, inadequate or inconsistent definitions of terminology used, or poor overall survey design (for example, context effects where responses to a question are directly influenced by the preceding questions). In order to overcome problems of this kind, individual questions and, the questionnaire overall, were thoroughly tested before being finalised for use in the survey. Testing for NATSIHS and NATSINPAS took two forms:
  • cognitive interviewing and focus group testing of concepts, terminology, questions and measurement/reporting issues
  • field testing, which involved a pilot test and dress rehearsal.

As a result of both forms of testing, modifications were made to question design, wording, ordering and associated prompt cards, and some changes were made to survey procedures. In considering modifications, it was sometimes necessary to balance better response to a particular item/topic against increased interview time or effects on other parts of the survey. For example, questions for collecting data on usual intake of fruit and vegetables referred to consumption in the form of 'serves', which required the use of a prompt card to define a serve, and a fair amount of recall and calculation on the part of the respondent.

Although every effort was made to minimise response errors due to questionnaire design and content issues, some errors will inevitably have occurred in the final survey enumeration.

As the survey is quite lengthy, reporting errors may also have resulted from interviewer and/or respondent fatigue (i.e. loss of concentration), particularly for those respondents reporting for both themselves and a child. Inaccurate reporting may also occur if respondents provide deliberately incorrect responses. While efforts were made to minimise errors arising from fatigue, or from deliberate misreporting or non-reporting by respondents, through emphasising the importance of the data and checks on consistency within the survey instrument, some instances will have inevitably occurred.

Reference periods used in relation to each topic were selected to suit the nature of the information being sought. In particular to strike the right balance between minimising recall errors and ensuring the period was meaningful and representative (from both respondent and data use perspectives), and would yield sufficient observations in the survey to support reliable estimates. It is possible that the reference periods did not suit every person for every topic and that difficulty with recall may have led to inaccurate reporting in some instances.

Lack of uniformity in interviewing standards may also result in non-sampling errors. Training and retraining programs, and checking of interviewers’ work were methods employed to achieve and maintain uniform interviewing practices and a high level of accuracy in recording answers on the survey questionnaire (see: Interviews within the Data collection page). The operation of the Computer Assisted Instrument (CAI) itself, and the built in checks within it, ensure that data recording standards are maintained. Respondent perception of the personal characteristics of the interviewer can also be a source of error, as the age, sex, appearance or manner of the interviewer may influence the answers obtained.

Non-response bias

Non-response may occur when people cannot or will not cooperate in the survey, or cannot be contacted by interviewers. Non-response can introduce a bias to the results obtained insofar as non-respondents may have different characteristics and behaviour patterns in relation to their health to those persons who did respond. The magnitude of the bias depends on the extent of the differences and the level of non-response.

The 2012-13 NATSIHS and NATSINPAS achieved an overall response rate of 80.2% and 79.2% respectively (fully/adequate responding households, after sample loss). Data to accurately quantify the nature and extent of the differences in health characteristics between respondents in the survey and non-respondents are not available. Under or over-representation of particular demographic groups in the sample are compensated for at the State, section of State, sex and age group levels in the weighting process. Other disparities are not adjusted for.

Households with incomplete interviews were treated as fully responding for estimation purposes where the only questions that were not answered were legitimate 'don't know' or refusal options, or any or all questions on income, or where weight and height were not obtained. These non-response items were coded to 'not stated'.

To improve the sample representativeness of the surveys generally, as well as for the biomedical component (NATSIHMS) and the second day dietary recall and pedometer component (in NATSINPAS), households that contained at least one fully responding person but also contained a selected adult or child who did not respond to the survey, were also retained and considered adequately responding. This process added an additional 371 and 180 households respectively to the NATSIHS and NATSINPAS samples. Note that this is consistent with the approach taken for previous ABS Aboriginal and Torres Strait Islander surveys.

Processing errors

Processing errors may occur at any stage between the initial collection of the data and the final compilation of statistics. These may be due to a failure of computer editing programs to detect errors in the data, or may occur during the manipulation of raw data to produce the final survey data files. For example, in the course of deriving new data items from raw survey data, or during the estimation procedures or weighting of the data file.

To minimise the likelihood of these errors occurring, a number of quality assurance processes were employed.
  • Comprehensive quality assurance procedures, applied to the NATSIHS and NATSINPAS coding of conditions and NATSIHS coding of alcohol data. Within the instruments, trigram coders were used to aid the interviewer with the collection of this data. This was complemented by manual coding of text fields where interviewers could not find an appropriate response in the coder.
  • Computer editing. Edits were devised to ensure that logical sequences were followed in the questionnaires, that necessary items were present, and that specific values lay within certain ranges. These edits were designed to detect reporting and recording errors, incorrect relationships between data items, and missing data items. Many of these edits were triggered during the interview in order to correct errors or confirm responses at the time of interview.
  • Data file checks. At various stages during processing (such as after computer editing and subsequent amendments, weighting of the file, and derivation of new data items), frequency counts and/or tabulations were obtained from the data file showing the distribution of persons for different characteristics. These were used as checks on the contents of the data file, to identify unusual values which might have significantly affected estimates, and illogical relationships not previously identified by edits. Further checks were conducted to ensure consistency between related data items, and between relevant populations.
  • Comparison of data. Where possible, checks of the data were undertaken to ensure consistency of the survey outputs against results of NATSIHS 2004-05, NATSISS 2008, AHS 2011-12 and data available from other sources.


In addition to data quality issues, there are a number of both general and topic-specific factors which should be considered in interpreting the results of this survey. The general factors affect all estimates obtained, but may affect topics to a greater or lesser degree depending on the nature of the topic and the uses to which the estimates are put. This section outlines these general factors. Additional issues relating to the interpretation of individual topics are discussed in the topic descriptions provided in other sections of this Users' Guide.


The scope of the survey defines the boundaries of the population to which the estimates relate.

The most important aspect of the survey scope affecting the interpretation of estimates from this survey is that institutionalised persons (including inpatients of hospitals, nursing homes and other health institutions) and other persons resident in non-private dwellings (e.g. hotels, motels, boarding houses) were excluded from the survey.


The NATSIHS and NATSINPAS includes all geographic areas except migratory. Due to the close timing of the NATSIHS to the National Health Survey (NHS) and the National Nutrition and Physical Activity Survey (NNPAS), the latter two surveys excluded all persons living in very remote areas, discrete Aboriginal and Torres Strait Islander communities, as well as a small number of persons living within SA1s that include Aboriginal and Torres Strait Islander communities. These exclusions were to minimise overlap of the Aboriginal and Torres Strait Islander household sample of both the NHS and NATSIHS and the NNPAS and NATSINPAS.


Undercoverage is one potential source of non-sampling error and is the shortfall between the population represented by the achieved sample and the in-scope population. It can introduce bias into the survey estimates. However, the extent of any bias depends upon the magnitude of the undercoverage and the extent of the difference between the characteristics of those people in the coverage population and those of the in-scope population.

Undercoverage rates can be estimated by calculating the difference between the sum of the initial weights of the sample and the population count. If a survey has no undercoverage, then the sum of the initial weights of the sample would equal the population count (ignoring small variations due to sampling error). For more information on weighting refer to Weighting, benchmarking and estimation procedures.

In the combined NATSIHS and NATSINPAS sample (known as the Core or AATSIHS) there was a relatively large level of undercoverage when compared to other ABS surveys. There was also an increase in undercoverage compared to previous ABS Aboriginal and Torres Strait Islander surveys. The overall AATSIHS coverage rate was approximately 37% of the in-scope population at the national level. This rate varies across the states and territories, as shown in the table below. In comparison, the estimated coverage in the 2004-05 NATSIHS and the 2008 National Aboriginal and Torres Strait Islander Survey was 58% and 47% respectively.

UNDERCOVERAGE, by state or territory

Rates of undercoverage

2012-13 NATSIHS
2012-13 AATSIHS

Of the national rate, 6% is due to planned frame exclusions and overlap with the Monthly Population Survey where analysis has shown that the impact of any bias is minimal. More information on these exclusions is provided below.

Undercoverage may occur due to a number of factors, including:

Each of these factors are outlined in more detail in the following paragraphs. To assist interpretation, a diagrammatical representation of the potential sources of undercoverage is denoted below.

Image: 2012-13 AATSIHS Core - Potential Sourcers of Undercoverage
Frame exclusions

Frame exclusions were incorporated into the AATSIHS to manage the cost of enumerating areas with a small number of Aboriginal and Torres Strait Islander persons. For more information, see the Scope page of this Users' Guide.


Non-response may occur when people cannot or will not cooperate, or cannot be contacted. Unit and item non-response by persons/households selected in the survey can affect both sampling and non-sampling error. The loss of information on persons and/or households (unit non-response) and on particular questions (item non-response) reduces the effective sample and increases both sampling error and the likelihood of incurring response bias.

To reduce the level and impact of non-response, the following methods were adopted in this survey:
  • face-to-face interviews with respondents
  • local Aboriginal and Torres Strait Islander facilitators were employed to assist with interviewing in communities
  • follow-up of respondents if there was initially no response
  • ensuring the weighted file is representative of the population by aligning the estimates with population benchmarks.

In the AATSIHS non-response accounts for a portion of overall undercoverage. The two components of non-response were:
  • non-response to the screening question
  • non-response to the survey after identification of an Aboriginal and Torres Strait Islander household.

Non-identification as being of Aboriginal and/or Torres Strait Islander origin

Non-identification of Aboriginal and Torres Strait Islander households during the screening process may have occurred due to:
  • Aboriginal and Torres Strait Islander people not identifying themselves as being of Aboriginal and/or Torres Strait Islander origin (passive refusals)
  • the household spokesperson being unaware of (or unwilling to provide) the Indigenous status of other residents.
Issues arising in the field

Known undercoverage, due to other issues arising in the field, included sample being excluded due to:
  • overlap with the Monthly Population Survey (MPS) and other select special social surveys
  • occupational, health and safety issues
  • time constraints, particularly in Aboriginal and Torres Strait Islander communities and their associated out-stations.

Biomedical quality control and quality assurance

For participants living in non-remote areas, most of the AATSIHS blood and urine samples were collected at Sonic Healthcare collection clinics or via a home visit using standard operating procedures for phlebotomy collection. In some areas, other pathology service providers were used (including IMVS Pathology for regional areas in South Australia and Northern Territory) however, the same collection procedures were used. For participants living in remote areas, local temporary collection sites were set up in community areas and samples were collected by Aboriginal Health Workers and/or a Sonic Healthcare professional. All samples were transported to a central Sonic Healthcare laboratory at Douglass Hanly Moir (DHM) Pathology in Sydney, Australia. Sample quality was monitored during remote transportation using temperature logging information. All samples were analysed at DHM on machines accredited by the National Association of Testing Authorities (NATA). DHM conducted Internal Quality Control (QC) analysis for all the instruments used to conduct analysis on the AATSIHS blood and urine samples which were reported to the ABS.

Periodic analysis of external Quality Assurance (QA) samples provided by the Royal College of Pathologists Australasia (RCPA) was conducted at DHM, with results independently assessed against set targets. The ABS monitored the analysis and delivery of results through key performance indicators which met contractual agreements with Sonic Healthcare. The results from the QC and QA reports indicate that the accuracy and precision of instruments used to analyse the AATSIHS samples fell within expected limits against set targets.

More information on the quality assurance methods and procedures can be found in the Biomedical Measures chapter of this Users' Guide.

Previous PageNext Page