In 2002, the highly topical issue of salinity led to the rapid development of a survey of Land Management Practices by farming businesses in Australia. The mail-out Salinity Survey was followed by a telephone Post Enumeration Survey (PES) to examine the performance of the form's design. The PES entailed interviewing 131 providers who had returned the paper form and asking them questions similar to those on the form, but structured differently, to see how well the two sets of answers matched. A less formal face to face PES was also conducted by visiting respondents.
The concerns with the paper form's design and content were:
- the question wording explicitly mentioned a topic (salinity) which was potentially sensitive for the provider so the telephone PES asked more general questions to see if the provider would mention salinity voluntarily;
- some questions on the paper form had long and complex structures comprised of varying answer field formats, and the question content covered multiple topics. The PES broke these questions into their components and the interviewer asked the provider for one answer at a time;
- some questions on the form began with a YES/NO filter positioned before response categories and other information, which the provider may have needed to read before they ticked YES or NO. The PES restructured these questions so the provider was aware of all relevant information before they could give an answer to each item;
The providers for the telephone PES were selected from groups based on how they responded to the questions of most concern in the paper form. These included providers who apparently responded correctly, obviously responded incorrectly, and those who left questions blank. The results from the PES were analysed and compared with the original survey responses by the selected providers. Taking editing into consideration, the broad totals were reasonably similar across surveys, but differences at the individual provider level were quite marked.
The subject of salinity wasn't as sensitive to providers as anticipated and those interviewed in the PES volunteered the relevant information freely. However, the results indicated the other design and content concerns were justified. False negatives and false positives were relatively common - i.e. the filter structure at the beginning of new and complex questions, which also covered multiple topics, appeared to be the cause of many of the problems.
Recommendations from the PES analysis include:
- other questions on the form had long lists of answer categories where the provider may have forgotten the actual question wording after the first few categories. The PES rephrased these categories into full questions.
- Avoid filters and have a "None of the above" category at the bottom of lists of answer options when asking conceptually complex questions, especially when they are new. Reading all the categories can assist the provider's memory and comprehension, so they include items not initially thought of at the start of the question.
The lessons learnt from this process will feed into the design of the next Salinity Survey as well as similar ABS forms.
For more information, please contact Glenn West on (02) 6252 6382, Emma Farrell on (02) 6252 7316, or Adam Sincock on (02) 6252 6766
E-mail : firstname.lastname@example.org
- Multiple topic questions should be split into smaller questions, which are more consistent, so each group of questions relates to a specific topic and has the same answer field format.