1800.0 - Australian Marriage Law Postal Survey, 2017  
Latest ISSUE Released at 11:30 AM (CANBERRA TIME) 15/11/2017  First Issue
   Page tools: Print Print Page Print all pages in this productPrint All RSS Feed RSS Bookmark and Share Search this Product

EXTERNAL OBSERVERS

Summary of results
Over the course of the observation program, external observers provided feedback on the coding decision made by the Australian Bureau of Statistics (ABS) on 606,991 survey returns.

Type of survey response Total number of survey returns Survey returns observed Instances where both observers disagreed with ABS coding decision Instances where one or both observers agreed with ABS coding decision
no. Survey returns observed % no. Instances where both observers disagreed with ABS coding decision % no. Instances where one or both observers agreed with ABS coding decision %
Automatically coded Yes or No survey returns 12,090,959 311,318 2.6 1 0.0003 311,317 99.9997
Manually coded Yes or No survey returns 600,275 261,678 43.6 337 0.1 261,341 99.9
Blank survey returns 16,912 15,659 92.6 29 0.2 15,630 99.8
Response Not Clear survey returns 19,774 18,336 92.7 625 3.4 17,711 96.6


Rates of agreement with the ABS decision applied to an individual survey return ranged from 99.9997% of automatically coded forms, to 96.6% of forms coded as Response Not Clear.

These rates of agreement indicate, that if the observers were to have provided feedback on all 12,727,920 survey responses, then observers would have agreed with 99.99% of forms processed, which equates to 9 survey return disagreements per electoral division.

There were comparable numbers of Yes and No disagreements, such that if all of the disagreements were added up, and the observer's recommendation adopted, there would be a net decrease in the survey margin of just 56 survey responses at the national level.

These results demonstrate that the accuracy of the survey return coding was very high, and that there was no bias in the capture of Yes or No responses.

There were additional survey returns observed where only one observer disagreed with the ABS coding decision; these instances are detailed below.

Establishment and processes

Establishment of the observer program
External observers were engaged to provide feedback on a key element of the survey process: the coding of returned paper survey forms as Yes, No or Response Not Clear.

Two Commonwealth parliamentary committees were established solely for the purposes of the survey, one representing the yes position, and one representing the no position. Each committee nominated around 60 external observers each.

The role of observers was different to that of a scrutineer in an election. As the survey was conducted under the authority of the Census and Statistics Act 1905, the rights and obligations of scrutineers under the Commonwealth Electoral Act 1918, the Referendum (Machinery Provisions) Act 1984, or other legislation did not apply.

Survey data capture and coding processes
All paper survey returns were opened and scanned at one of five processing centres across Australia and the digital images of scanned forms were then processed at the processing centre in Sydney.

The digital images were processed by an automatic coding system. Survey forms with only a single response box marked, no amendments made to the survey question or survey answers, and no other marks made in the white spaces to the left and right of the response boxes, were automatically coded as Yes or No.

95% of all returned survey forms were automatically coded as Yes or No.

Forms that could not be automatically coded were manually coded by trained coders in the processing centre in Sydney.

Manual coding was informed by a set of coding guidelines which were developed in consultation with the two parliamentary committees and the Australian Electoral Commission.

4.7% of survey returns were manually coded as Yes or No, while a further 0.3% were manually coded as Blank or Response Not Clear.

Observer process
Observers worked over a six week period, between 29 September and 8 November 2017, at the processing centre in Sydney.

Observers from both committees were shown a sample of digital images of paper survey forms, along with the survey response as coded by the ABS for that form. Survey responses were coded as either Yes, No, Blank or Response Not Clear.

Where one or both observers disagreed with the decision made by the ABS, the disagreement was recorded, along with the observer's suggested response.

The samples of returns selected for observer review was designed to ensure that:
    • nearly all returns coded as Blank or Response Not Clear were viewed by the observers
    • a significant proportion of survey responses that were manually coded (coded by a human operator rather than the automatic coding system) were viewed by the observers.
    • an approximately equal number of returns coded as Yes and No were reviewed by observers so that they would not have any indication of the overall survey result.

Feedback from observers did not change the recorded result for survey forms; it was recorded for reporting purposes only.

Detailed results

Automatically coded responses
There were 12,090,959 survey returns (95% of the total) that were automatically coded as either Yes or No. Observers provided feedback on 311,318 (2.6%) of these survey returns.

Observer feedback indicates that automatic coding was highly accurate. There was only a single instance where observers from both sides disagreed with the ABS code. In this case, both observers considered that the survey return should have been coded as Response Not Clear.

Type of survey response Survey returns observed Instances where both observers disagreed with ABS coding decision Instances where one or both observers agreed with ABS coding decision
no. Instances where both observers disagreed with ABS coding decision % no. Instances where one or both observers agreed with ABS coding decision %
Automatically coded as Yes 155,656 - - 155,656 100
Automatically coded as No 155,662 1 0.0006 155,661 99.9994
Total automatically coded survey returns 311,318 1 0.0003 311,317 99.9997


Manually coded responses
There were 600,275 survey returns (4.7% of the total) that were manually coded as either Yes or No. Observers provided feedback on 261,678 (43.6%) of these survey forms.

Observer feedback indicates that manual coding was very accurate. There were 337 instances where observers from both sides disagreed with the ABS coding decision. This represented 0.1% of all manually coded forms. Of these 337 instances, the ABS coded 236 as Yes and 101 as No.

The ABS developed very conservative rules for the use of automatic coding that included referring any survey returns which contained marks in the spaces to the left of the Yes response box, or the right of the No response box, for manual coding. This was done to ensure that any survey returns that may have had additional comments written on the form, that may have been material to the capture of the survey response, could be reviewed.

A consequence of this approach was that a higher proportion of No responses were referred to manual coding. Ticks made with a long tail in the No box were in some cases identified by the automatic coding system as possibly having additional markings, and were referred for manual coding. This was not the case for Yes responses indicated with a long tailed tick. Excluding these instances, the overall rate of disagreement for Yes and No manually coded responses is very similar.

Type of survey response Survey returns observed Instances where both observers disagreed with ABS coding decision Instances where one or both observers agreed with ABS coding decision
no. Instances where both observers disagreed with ABS coding decision % no. Instances where one or both observers agreed with ABS coding decision %
Manually coded as Yes 131,049 236 0.2 130,813 99.8
Manually coded as No 130,629 101 0.1 130,528 99.9
Total manually coded survey returns 261,6780 337 0.1 261,341 99.9


Response not clear and blank responses
There were 19,774 survey returns (0.2% of the total) that were coded as Response Not Clear. Additionally 16,912 (0.1%) survey forms were returned blank. Observers provided feedback on 33,995, or 92.7% of these survey forms.

Where the intent of the survey respondent could not be determined, the survey response was coded as Response Not Clear. This commonly occurred where the forms were vandalised, where no attempt to answer the question had been made, or where both response boxes were marked.

There were 654 instances where observers from both sides disagreed with the ABS coding of the form as Response Not Clear or Blank. This represented 0.2% of all Blank survey returns, and 3.4% of survey returns coded as Response Not Clear.

Type of survey response Survey returns observed Instances where both observers disagreed with ABS coding decision Instances where one or both observers agreed with ABS coding decision
no. Instances where both observers disagreed with ABS coding decision % no. Instances where one or both observers agreed with ABS coding decision %
Blank 15,659 29 0.2 15,630 99.8
Response Not Clear 18,336 625 3.4 17,711 96.6
Total Blank and Response not clear survey returns 33,995 654 1.9 33,341 98.1


Implications for overall survey results
As observers provided feedback on only a sample of the survey results, to measure the impact on the survey result, the ABShas extrapolated the disagreement rates to cover the additional forms not viewed.

To extrapolate the results of the observer program to cover all returned survey forms, the ABS used the proportion of survey forms where both observers disagreed, and applied that to the total number of survey forms returned.

The estimated total number of disagreements, had the observers viewed all survey returns, is shown below.

Type of survey response Number of instances where both observers disagreed with the ABS coding decision Estimated number of disagreements had observers viewed all survey returns
Automatically coded Yes or No survey returns 1 29
Manually coded Yes or No survey returns 337 636
Blank survey returns 29 31
Response Not Clear survey returns 625 674
Total - 1,370


At the national level, this indicates that there would have been 1,370 survey responses where the both observers would have been expected to have disagreed with the ABS. This equates to 9 survey responses per electoral division.

Where observers raised a disagreement with the decision applied by the ABS, the coding result that the observers thought should have been applied was recorded by the ABS.

The following table shows the observer suggested coding results against the ABS coded response extrapolated across all survey forms in order to see the potential difference to the survey results.

ABS coded response as Yes No Blank Response not clear No agreement between observers/ Insufficient information Total
Yes - 135 - 148 18 301
No 133 - - 218 13 364
Blank 5 23 - 2 1 31
Response Not Clear 266 365 1 - 42 674
Total 404 523 1 368 74 1,370


These figures show that had the ABS applied code proposed by observers, there would have been, at the national level:
    • 30 fewer returns recorded as Blank
    • 306 fewer returns recorded as Response Not Clear
    • 103 additional returns recorded as Yes responses
    • 159 additional returns recorded as No responses.

This would have resulted in a net decrease in the survey margin of 56 responses across Australia.

Instances where only one observer disagreed with the ABS
There were an additional 386 instances where one of the observers disagreed with the coding decision made by the ABS, while the other observer agreed with the decision made by the ABS. These instances are shown below.

Type of survey response Survey returns observed Only one observer disagreed with ABS coding decisions (Yes observer) Only one observer disagreed with ABS coding decision (No observer)
no. Only one observer disagreed with ABS coding decisions (Yes observer)% no. Only one observer disagreed with ABS coding decision (No observer)%
Automatically coded survey returns Yes 155,656 - - - -
No 155,662 - - - -
Manually coded survey returns Yes 131,049 - - 40 0.03
No 130,629 47 0.04 - -
Blank 15,659 2 0.01 3 0.02
Response Not Clear survey returns 18,336 149 0.8 145 0.8





AMLPS_coding_guidelines.pdf