MODE SUITABILITY MODEL
While aspects of these standards will be of interest to those outside the ABS, they were developed for internal use. As such, some information contained in these standards will not be applicable to an external audience. ABS staff should refer to the Corporate Manuals database for the most recent version of these documents, as some details (names, phone numbers etc.) have been removed from the online version.
This document provides a list of factors that should be considered when determining a suitable mode to use as the primary mode
of data collection for a survey. Several modes of collection are assessed against each factor, in terms of the mode's suitability for surveys with certain characteristics.
The document will eventually contain the main modes of data collection used at the ABS. As of June 2008, the modes covered in the document are:
It is strongly recommended that people involved in determining an appropriate data collection method for a survey read the Mode Suitability Framework document, for a more detailed discussion of the advantages and disadvantages of common modes, and the types of surveys they are suitable for.
- Computer Assisted Telephone Interviewing (CATI)
- Mail forms
- Fax forms
- Offline electronic forms (e.g. Excel and PDF forms)
The present document is intended to be read in conjunction with the Mode Suitability Framework, providing a succinct summary of the factors to consider, and allowing the suitability of different modes for a survey to be easily compared.
How to use this document
Thirteen factors that are relevant when determining a mode's suitability as the primary data collection method for a survey are listed in the table below. Each mode is assessed against the factors, and is given a suitability rating in terms of how suitable it is for surveys with that characteristic. The rating system uses a "traffic light" approach with each colour representing the following:
- = Against the factor, the mode is suitable
- = Against the factor, the mode may be suitable, but there are modes that are more suitable for surveys with that characteristic.
- = Against the factor, the mode is unsuitable.
- NA = The factor is not relevant for the mode.
For each factor listed in the left hand column of the table, find the relevant characteristic that matches the survey that is being assessed against the particular mode/s listed at the top of the table. For example, if considering an offline electronic form (such as an Excel form) for a survey that is run quarterly, the survey would get a "green light" against factor 1 ("Survey frequency"), indicating that against this factor, an offline form is likely to be suitable. Each survey should obtain only one rating per factor.
To get an overall indication of a mode's suitability for a survey, assess the survey against each factor for the mode being investigated, and then add up the number of green, amber, and red lights obtained for that mode.
Important notes on using this document
- The more green lights , the greater the suitability of the mode for the survey.
- If a mode has more than 4 amber lights , it is likely that another mode will be more suitable, and it is therefore recommended that a different mode is considered.
- As a general guideline, if a mode yields more than 2 red lights , it is highly likely to be unsuitable for the survey, and an alternative mode should be considered.
This document is intended to provide a range of issues to consider when determining an appropriate mode to use as the main mode
of data collection for a survey. The ratings are not intended to apply when considering the mode as part of a mixed mode design. This is because mixing modes introduces greater complexity, and the suitability of each mode would depend on which other modes are being offered. This would make it difficult to provide general suitability guidelines for the mode using the simple rating scale that has been used below. For example, when assessing the suitability of fax forms below, it is assumed this is being considered as the main mode of data collection (e.g. not as an alternative method to a mail survey in a mixed mode design).
The suitability scores below also assume that (a) the target respondents have access to the technology/equipment required to be able to complete the forms (e.g. a fax machine if fax forms are being considered); and (b) the ABS has access to the required frame in order to make contact with respondents.
As surveys may have their own complex and unique issues, the model is not intended to provide an absolute measure of suitability, and is to be used as a guide only. The Mode Suitability Framework
document contains further explanation about the rationale behind the ratings given in this document.
|Size of the survey|
|1. Survey frequency|
The frequency with which a survey is run is an important factor to consider when determining an appropriate mode. The higher the frequency, the less suitable a mail survey is, due to the delays between dispatch and receipt of the form. When converting to a mode where the initial set-up costs are high (e.g. CATI, IVR), surveys that are run more frequently may be more suitable from a cost-benefit perspective. Fax forms may be more suitable for higher-frequency surveys, because respondents are more likely to be anticipating the form's arrival, compared to a one-off or infrequent survey.
2. Sample size
When considering converting a survey to a mode with a high set-up cost (e.g. IVR, CATI), surveys with bigger sample sizes are generally more appropriate from a cost-benefit perspective. The suitability of other modes, such as mail surveys, is less affected by sample size differences.
|10 thousand or more|
|Under 10 to 5 thousand|
|Under 5 to 2 thousand|
|Under 2 to 1 thousand|
|Under 1 thousand|
3. Survey generalisability
The amount of content that a survey has in common with other surveys should be considered. For example, if most of the questions in the survey are from Standard Question Wording (SQW), then CATIs, IVR surveys, web forms etc. will be more easily adapted to the other surveys which use SQW, therefore potentially saving development time. Note that this factor does not include coverage and reference period questions which are common to almost all surveys. Also note that the length of the survey should be considered when assessing the survey's generalisability - it may still be cost effective to develop CATI, electronic forms etc. for short forms that do not have much (or any) content common with other surveys.
|All questions common to many other surveys|
|Most questions common to many other surveys|
|Most questions common to some other surveys|
|Some questions common to some other surveys|
|No questions common to any other survey|
4. Time taken/interview length
This factor refers to the estimated time taken to complete the survey form, either for respondents (for self-administered forms) or for interviewers. Interview length is particularly important to consider for telephone surveys, as this method is unsuitable for long surveys due to provider load concerns. Fax is also unsuitable for long forms, due to the effort/cost required for the respondent to send the form back to the ABS, and possible problems with pages becoming separated during transmission.
|5 minutes or less|
|31 minutes or more|
5. Total number of data items
This factor is included in addition to time taken as a measure of provider load, because self-reported time taken can be imprecise. A data item in this context is roughly interpreted as equalling the potential number of answers (including yes/no questions), or the number of decisions the respondent may have to make. This includes counting every item in a "mark all that apply" question as one, because when conducted in a verbal interview, each item will need to be considered separately. As described in Factor 4, telephone surveys and fax surveys are less suitable for forms with a large number of data items due to the increased burden placed on respondents.
|5 or less|
|41 or more|
6. Question complexity
The complexity of the survey questions is very important to consider, but is a difficult factor to measure. The intention of this factor is to capture the amount of explanatory notes required to understand the questions, as well as the space taken up by complex matrix structures etc. This factor uses the proportion of the form and other material (not including front page) taken up by instructions, which allows for a consideration of text regardless of how tightly cramped the spacing is. Note that forms with complexity at the higher end (above around 60%) may be problematic in terms of provider load, and consideration should be given to simplifying the form.
|Less than 20%|
|20% to less than 40%|
|40% to less than 60% |
|60% to less than 80%|
|80% or more|
7. Amount of sequencing
This factor assesses the amount of the sequencing in the form. Being required to follow sequencing instructions adds to provider load (or interviewer load where interviews are used), and can lead to errors in terms of which questions are/are not completed. Forms with large amounts of sequencing may be better administered electronically, where it is assumed here that sequencing will be automated (e.g. CATI, CAPI, computerised self-administered questionnaires).
8. Proportion of items requiring record checking
This factor distinguishes questions which may be answered off-hand (such as attitudinal and behavioural questions), from questions which require checking of records, or require input from a range of other people. Most questions requiring numerical data will fall into the latter category, and are less suitable for telephone surveys, where the telephone call should be kept as short as possible.
11. Literacy of respondents
This factor considers the literacy of respondents, and is relevant for self-administered forms, where respondents must read and understand the instructions and questions without assistance from an interviewer. According to Biemer and Lyberg (2003), a general rule of thumb is that forms aimed at the general population should be at a fifth-grade reading level. The percentages below represent the proportion of respondents expected to have a reasonable level of literacy.
|74% or below|
12. Geographic location of respondents
This factor takes into account the location of the majority of respondents, and how difficult it would be for an interviewer to reach them. Mail surveys are usually more suitable than face-to-face interviews for reaching respondents in diverse geographic areas, where actually visiting the respondents would be too difficult/costly. Telephone surveys, electronic forms, and fax forms can also be suitable for reaching difficult to contact respondents, provided that respondents have telephone/internet/fax access.
|Metropolitan (inner and outer), with telephone/internet/fax coverage|
|Metropolitan (inner and outer), without telephone/internet/fax coverage|
|Rural, with telephone/internet/fax coverage|
|Rural, without telephone/internet/fax coverage|
This page last updated 22 January 2010