1504.0 - Methodological News, Mar 2009  
ARCHIVED ISSUE Released at 11:30 AM (CANBERRA TIME) 26/03/2009   
   Page tools: Print Print Page Print all pages in this productPrint All

Embedded experiments in repeated and overlapping surveys

Statistical agencies make changes to the data collection methodology of their surveys in order to improve the quality of the data or to improve the efficiency of the collection process. For cost reasons, it may not be possible to reliably estimate the impact of such a change on survey estimates or response rates, without conducting an experiment that is embedded in the survey which involves enumerating some respondents using the new method and some under the existing method.

From October 2003 to August 2004, pen-and-paper interviewing (PAPI) was replaced by computer-assisted interviewing (CAI) in the Monthly Labour Force Survey (LFS). There was concern about the impact that changes to the data collection method would have on LFS estimates. To address this concern, some effort was taken to answer the following methodological questions: how should CAI be phased in; how should the CAI effect be estimated; and how should the uncertainty in the estimated CAI effect be measured? The methodological developments were required to account for the LFS's multi-stage design and its rotation pattern which gives a high degree of sample overlap from month-to-month. These methodological issues needed to be balanced with operational constraints, such as the constraint that an interviewer could not use both CAI or PAPI in a given month, and managerial considerations, such as managing the risk to the LFS series if the CAI effect were large.

Previous embedded experiments in the literature are used for ongoing and overlapping surveys where maintaining a time series is important. However, the experimental designs and the estimation methods that have been developed assume there is only a single time point. The approach developed as part of the above project has a number of advantages over previous approaches: it exploits the correlation between the overlapping samples to improve estimates of data collection effects; data collection effects are allowed to vary over time; estimation is robust against incorrectly rejecting the null hypothesis of no data collection effect; and it allows for a new data collection method to be introduced over time.

A paper describing the methodology and the practical experience gained during the introduction of CAI will soon appear in the Journal of the Royal Statistical Society Series A. For more details, contact James Chipperfield on (02) 6252 7301 or james.chipperfield@abs.gov.au.