1504.0 - Methodological News, Mar 2009  
Previous ISSUE Released at 11:30 AM (CANBERRA TIME) 26/03/2009   
   Page tools: Print Print Page Print all pages in this productPrint All RSS Feed RSS Bookmark and Share Search this Product

Study of non-response in the Survey of Mental Health and Well Being

The 2007 Survey of Mental Health and Well Being (SMHWB) was a voluntary survey with a relatively lengthy questionnaire and questions of a sensitive nature. The achieved response rate was 60% which caused some concern over potential bias in survey estimates.

Standard practice in ABS household surveys is to attempt to treat non-response through survey weighting. This may entail modelling response propensities and comparing responding sample distributions with other aggregate data such as Census population counts and other ABS survey estimates. Standard weighting typically involves calibration to age-sex demographic benchmarks. The key element that is not directly quantifiable is the correlation between response propensity and characteristics being measured by the survey. Where there are significant correlations between respondent's characteristics and response status, over and above that described by age and sex, then standard weighting does not completely eliminate bias. This is the heart of the non-response problem and can only be fully addressed by obtaining survey characteristics from non-respondents.

In order to provide a qualitative assessment of the extent of possible non-response bias for the SMHWB, a non-response follow-up (NRFU) study was carried out two months after the main survey. The study was designed as a short questionnaire in order to help promote the likelihood of gaining useful numbers of respondents covering demographic and the Kessler Psychological Distress Scale-10 (K10) topics. In addition, after an initial personal contact, the options of CAPI (computer-assisted personal interview) or CATI (computer-assisted telephone interview) were made available. These are different from the mode of collection for the 2007 SMHWB where only CAPI was used. The study was only conducted in the regions of Sydney and Perth. The sample selection was driven by cost considerations. This resulted in a purposive, rather than a random, sample but the interviewers were reasonably spread across different areas of the two cities. The NRFU interviewers were assigned a total of 401 households (229 in Sydney and 172 in Perth), and achieved 151 fully responding households (77 in Sydney and 74 in Perth).

Distributions of the NRFU sample by basic demographic variables were compared with corresponding SMHWB respondents in Perth and Sydney. The NRFU picked up proportionally more males and fewer older persons than the SMHWB. This is not surprising: in household surveys, comparisons of achieved sample distributions with population benchmark data typically show poorer coverage rates for younger persons, males in particular. The NRFU also picked up higher relative numbers of "never married" persons compared to the SMHWB, also consistent with other data sources such as the Census and in part related to the younger age associated with non-respondents.

The unweighted distribution of respondents to the SMHWB and the NRFU across the Kessler Psychological Distress Scale groups by sex and geography and by age and geography were compared. These show male respondents to the NRFU in Perth tend to have higher Kessler distress levels than other males. Perth females in the NRFU also tend to have higher Kessler distress levels than Perth females in the SMHWB. This indicates that there may be differences in the direction and magnitude of potential non-response bias between various geographical, age and sex domains. The level of confidence varies depending on the magnitude of the difference between the respondents and non-respondents and the size (often small) of the follow-up sample contributing to the comparison.

The findings suggest the following:

    • the magnitude of bias appears to be small at the aggregate level;
    • possible underestimation of prevalence of mental health conditions in Perth;
    • possible underestimation of prevalence of mental health conditions for males; and
    • possible underestimation of prevalence of mental health conditions for young persons.
The conduct of the study will be reviewed with the aim of developing best practice guidelines on the conduct of non-response follow up studies in household survey collections.

For more information, contact John Martin on (02) 62527006 or john.martin@abs.gov.au.