4839.0.55.001 - Health Services: Patient Experiences in Australia, 2009
Latest ISSUE Released at 11:30 AM (CANBERRA TIME) 30/07/2010
Page tools: Print All RSS Search this Product

TECHNICAL NOTE

RELIABILITY OF ESTIMATES

Sample Survey Errors

1 Two types of error are possible in estimates based on a sample survey; sampling error and non-sampling error.

2 Sampling error occurs because only a small proportion of the total population is used to produce estimates that represent the whole population. Sampling error can be reliably measured, as it is calculated based on the scientific methods used to design surveys.

3 Non-sampling error may occur in any data collection, whether it is based on a sample or a full-count (i.e. Census). Non-sampling error may occur at any stage throughout the survey process. Examples include:

• non-response by selected persons;
• questions being misunderstood;
• responses being incorrectly recorded; and
• errors in coding or processing the survey data.

4 Since the estimates in this publication are based on information obtained from occupants of a sample of dwellings, they are subject to sampling variability. That is, they may differ from those estimates that would have been produced if all occupants of all dwellings had been included in the survey. One measure of the likely difference is given by the standard error (SE), which indicates the extent to which an estimate might have varied by chance due to only a sample of dwellings being included. There are about two chances in three (67%) that a sample estimate will differ by less than one SE from the number that would have been obtained if all dwellings had been included, and about 19 chances in 20 (95%) that the difference will be less than two SEs.

Relative Standard Error (RSE)

5 Another measure of the likely difference is the relative standard error (RSE), which is obtained by expressing the SE as a percentage of the estimate.

6 RSEs for patient experience estimates (numbers or percentages) have been calculated using the Jackknife method of variance estimation.

7 RSEs were calculated for each separate estimate and are available to download free of charge as data cubes (Excel spreadsheets) from the ABS website <www.abs.gov.au> as an attachment to this publication.

8 In the tables in this publication, only estimates with RSEs less than 25% are considered sufficiently reliable for most purposes. However, estimates with larger RSEs have been included and are preceded by an asterisk (e.g. *2.2) to indicate they are subject to high SEs relative to their estimate and should be used with caution. Estimates with RSEs greater than 50% are preceded by a double asterisk (e.g. **1.5) to indicate that they are considered too unreliable for general use.

Replicate Weights Technique

9 A class of techniques called 'replication methods' provide a general method of estimating variances for the types of complex sample designs and weighting procedures employed in ABS household surveys.

10 The basic idea behind the replication approach is to select sub-samples repeatedly from the whole sample, for each of which the statistic of interest is calculated. The variance of the full sample statistic is then estimated using the variability among the replicate statistics calculated from these sub-samples. The sub-samples are called 'replicate groups', and the statistics calculated from these replicates are called 'replicate estimates'.

11 There are various ways of creating replicate sub-samples from the full sample. The replicate weights produced for the survey were created under the delete-a-group Jackknife method of replication. There are numerous advantages to using the replicate weighting approach, including the fact that:
• the same procedure is applicable to most statistics such as means, percentages, ratios, correlations, derived statistics and regression coefficients; and
• it is not necessary for the analyst to have detailed survey design information available if the replicate weights are included with the data file.

Derivation of replicate weights

12 Under the delete-a-group Jackknife method of replicate weighting, weights were derived as follows:
• 30 replicate groups were formed for both household and person weights, with each group formed to mirror the overall sample. Units from a cluster of dwellings all belong to the same replicate group, and a unit can belong to only one replicate group.
• For each replicate weight, one replicate group was omitted from the weighting and the remaining records were weighted in the same manner as for the full sample.
• The records in the group that was omitted received a weight of zero.
• This process was repeated for each replicate group (i.e. a total of 30 times).
• Ultimately each record had 30 replicate weights attached to it with one of these being the zero weight.

Application of replicate weights

13 As noted above, replicate weights enable variances of estimates to be calculated relatively simply. They also enable unit record analyses such as chi-square and logistic regression to be conducted, which take into account the sample design.

14 Replicate weights for any variable of interest can be calculated from the 30 replicate groups, giving 30 replicate estimates. The distribution of this set of replicate estimates, in conjunction with the full sample estimate, is then used to approximate the variance of the full sample.

15 This method can also be used when modelling relationships from unit record data, regardless of the modelling technique used. In modelling, the full sample would be used to estimate the parameter being studied (such as a regression coefficient); i.e, the 30 replicate groups would be used to provide 30 replicate estimates of the survey parameter. The variance of the estimate of the parameter from the full sample is then approximated, as above, by the variability of the replicate estimates.

CALCULATION OF STANDARD ERROR

Standard error of an estimate

16 Standard errors can be calculated using the estimate and the corresponding RSEs. For example, Table T1 at the end of this Technical Note shows the estimated number of persons aged 15 years and over who rated their health as excellent, very good or good in the last 12 months is 14,438,500. The RSE table corresponding to the estimate in Table T2 (also below) shows the RSE for this estimate is 0.6%. The SE is calculated by:

17 Therefore, there are about two chances in three that the value that would have been produced if all dwellings had been included in the survey will fall within the range 14,351,900 to 14,525,100 and about 19 chances in 20 that the value will fall within the range 14,265,300 to 14,611,700. This example is illustrated in the diagram below:

RELATIVE STANDARD ERRORS OF COMPARITIVE ESTIMATES

Proportions

18 Proportions formed from the ratio of two estimates are also subject to sampling error. The size of the error depends on the accuracy of both the numerator and the denominator. A formula to approximate the RSE of a proportion is given below. This formula is only valid when x is a subset of y.

19 As an example, using estimates from Table T1, 11,357,800 persons aged 15 years and over who rated their health as excellent, very good or good saw a general practitioner in the last 12 months, representing 78.7% of the 14,438,500 persons aged 15 years and over who rated their health as excellent, very good or good. As can be seen in Table T2, the RSE of the estimated number of persons aged 15 years and over who rated their health as excellent, very good or good and saw a general practitioner in the last 12 months is 1.2%, and the RSE of the estimated number of persons aged 15 years and over who rated their health as excellent, very good or good is 0.6%. Applying the above formula, the RSE of the proportion is:

20 Therefore, the SE for persons aged 15 years and over who rated their health as excellent, very good or good and saw a general practitioner in the last 12 months as a proportion of persons aged 15 years and over who rated their health as excellent, very good or good is 0.8 percentage points (=78.7×(1.0/100)). Therefore, there are about two chances in three that the proportion is between 77.9% and 79.5% and 19 chances in 20 that the proportion is within the range 77.1% to 80.3%.

21 The exact RSEs of various proportions for patient experience can be found in the data cubes in the download tab of this publication.

Sum of or Differences between estimates

22 Published estimates may also be used to calculate the sum of or difference between two survey estimates. Such estimates are also subject to sampling error. The sampling error of the difference between two estimates depends on their SEs and the correlation between them. An approximate SE of the difference between two estimates (x-y) may be calculated by the following formula:

23 The sampling error of the sum of two estimates is calculated in a similar way. An approximate SE of the sum of two estimates (x+y) may be calculated by the following formula:

24 As an example, from paragraph 16, the estimated number of persons aged 15 years and over who rated their health as excellent, very good or good in the last 12 months is 14,438,500 and the SE is 86,600. Performing the appropriate calculations, the estimated number of persons aged 15 years and over who rated their health as fair or poor is 2,308,600 and the SE is 73,900 (rounded to nearest 100). The estimated number of persons aged 15 years and over who provided a self-assessed health status rating is:

25 The SE of the estimated number of persons aged 15 years and over who provided a self-assessed health status rating is:

26 Therefore, there are about two chances in three that the value that would have been produced if all dwellings had been included in the survey will fall within the range 16,633,300 to 16,860,900 and about 19 chances in 20 that the value will fall within the range 16,519,500 to 16,974,700.

27 While these formulae will only be exact for differences between separate and uncorrelated characteristics or subpopulations, it is expected to provide a good approximation for all differences likely to be of interest in this publication.

STATISTICAL SIGNIFICANCE TESTING

28 To determine whether, with a certain level of confidence, there was evidence of a true difference between corresponding population characteristics, a statistical significance test for comparisons between estimates was performed. The standard error of the difference between two corresponding estimates (x and y) can be calculated using the formula in paragraph 22. This standard error is then used to calculate the following test statistic:

29 If the value of the test statistic is greater than 1.96, there is statistical evidence (with 95% confidence) of a difference in the two populations with respect to that characteristic. Otherwise, it cannot be stated with confidence that there is a real difference between the populations.

SELECTED ESTIMATES AND RSES

 TABLE T1 ESTIMATES Self-assessed health status ESTIMATE ('000) Excellent/very good/good Fair/poor Consultations Saw a general practitioner 11357.8 2188.2 Saw 3 or more health professionals for a single condition 2989.6 1151.3 Hospital Been admitted to hospital 1614.4 622.0 Visited hospital emergency department 1575.7 593.0 Other Services Had a pathology test (c) 6628.0 1670.7 Had an imaging test (d) 4136.2 1168.8 Asked pharmacist for advice 3198.3 668.5 Total persons ('000) 14438.5 2308.6 Source: ABS Patient Experience Survey 2009

 TABLE T2 RSES OF ESTIMATES Self-assessed health status RSE of ESTIMATE (%) Excellent/very good/good Fair/poor Consultations Saw a general practitioner 1.2 3.1 Saw 3 or more health professionals for a single condition 2.1 4.2 Hospital Been admitted to hospital 3.5 6.9 Visited hospital emergency department 3.8 6.9 Other Services Had a pathology test (c) 1.6 4.3 Had an imaging test (d) 2.4 4.4 Asked pharmacist for advice 3.6 5.6 Total persons (%) 0.6 3.2 Source: ABS Patient Experience Survey 2009