1504.0 - Methodological News, Sep 2003  
ARCHIVED ISSUE Released at 11:30 AM (CANBERRA TIME) 27/02/2004   
   Page tools: Print Print Page Print all pages in this productPrint All

FORMAL ERROR ANALYSIS OF IMAGING AND RECOGNITION TO IMPROVE PROCESSING AND FORM DESIGN

Background

The ABS is increasing the use of imaging and recognition (I&R) technology for capturing data from business surveys. This has resulted in considerable interest in non-sample error associated with I&R, particularly those related to form design and processing, and the opportunities for improvement offered by a planned I&R software upgrade.

To find out the extent of these problems an error analysis was undertaken using data from the Economic Activity Survey (EAS) which had moved to I&R for data capture for the most recent reference period. The main aims of this project were to:

  • determine common errors on the forms, and the extent of the errors, which may cause specific problems with imaging and recognition as the primary method of data capture (e.g. comments outside designated areas, insufficient space in answer boxes, nils, etc);
  • examine any other obvious errors relating to the design of forms (e.g. $,000 vs whole dollar reporting); and
  • identify possible improvements to general form design standards.

Process

Two samples of 120 previously processed Economic Activity Survey forms were selected, one each of a 'long' form (63 questions, some with multiple data items) and a 'short' form. As well as the original collection forms, two data files were obtained for each of the sampled respondents: the original repaired data file (ie after recognition errors and failures identified by the recognition process had been corrected), and the equivalent data file after output editing.

The paper forms were put through the complete imaging, recognition and repair process in a test environment so both the recognised and repaired values could be extracted and confronted. The repaired values were also compared with the original values provided to the collection area, and with the values after editing. All forms were manually inspected and a range of errors and usage patterns recorded.

Response analysis

The analysis gathered specific information on the effects of recognition and processing on data, with most of the issues identified through previous consultations with collection areas. The main finding from the analysis was that most of the commonly reported problems were not as prevalent as we were led to believe. Issues covered included: European 7's; diagonally crossed 0's; writing the word "nil"; brackets; negative values and dashes; non-black pen; white out/tape; spurious marks; crossed out questions and sections; crossed out answers and overwritten answers; obvious whole dollar reporting; answers running over the answer space provided; front of form label changes; and comments outside designated areas.

The three most common recognition errors were caused by spurious marks (22% of errors), use of white out or tape (20%) and crossed out and overwritten answers. In addition there were significant problems with the reporting of nil or negative values and answered spaces being too small or too close together (mainly tick boxes).

Several of these errors can be minimised with improved software (European 7's, diagonally crossed 0's, writing the word "nil"), where others can be addressed through form design (data entry box spacing and size), while crossing out and correction errors may indicate underlying problems with question wording, instructions or formats.

All ABS forms include an optional 'final comments' question, and, because space on survey forms always appears to be at a premium, the usefulness of this question has been a matter of particular interest. Comments were provided by 22% of respondents in this analysis. More than half of these related to data reported and would be useful during editing. A significant number also related to the status of the businesses surveyed and had frame and imputation implications. Under 5% of respondents had complaints.

The project resulted in ten recommendations for further investigation into some areas and identifying solutions to problems through using new I&R software.

For more information, please contact Tracey Rowley on (02) 6252 5905.

Email: tracey.rowley@abs.gov.au.