RELIABILITY OF THE EXPERIMENTAL ESTIMATES


DATA QUALITY

When interpreting the experimental estimates it is important to take into account factors that may effect the reliability of the experimental estimates.

The quality of the experimental estimates is limited by:

• the validity of the assumptions underpinning the modelling; and
• the accuracy of the data used in the production of the experimental estimates.

The assumptions used in the production of the experimental estimates were outlined in Chapter 2. Users should consider the suitability of these assumptions when interpreting the experimental estimates.

Examination of the following quality indicators will also assist users in determining fitness for purpose of the experimental estimates of the manufacturing industry.


DATA USED IN THE CALCULATION OF THE EXPERIMENTAL ESTIMATES

The experimental estimates presented in this publication were obtained using a combination of data directly collected in EAS and Business Activity Statement (BAS) data. Modelling techniques were applied to combine these two data sources in order to produce the experimental estimates at the class level, as described in Chapter 2.

The EAS uses a sample of businesses, rather than full enumeration (i.e. a census) and is subject to sampling error. The resultant estimates obtained from the regression model may be different if survey information was available for all businesses. The experimental estimates presented in this paper therefore have an associated sampling error.

The experimental ANZSIC class estimates also have additional associated sampling error as a result of constraining these experimental estimates to aggregate to ANZSIC subdivision estimates obtained from the EAS and published in Australian Industry. 2007-08 (cat. no. 8155.0).


SAMPLING ERROR

One measure of sampling variability is given by the standard error which indicates the extent to which an estimate might have varied by chance because only a sample of businesses was included. There are about two chances in three that a sample estimate will differ by less than one standard error from the figure that would have been obtained if a census was conducted, and about 19 chances in 20 that the difference will be less than two standard errors.

Sampling variability can also be measured by the relative standard error (RSE) which is obtained by expressing the standard error as a percentage of the estimate to which it refers. The RSE is a useful measure in that it provides an indication of the sampling error in percentage terms, and this avoids the need to refer also to the size of the estimate.

Approximate RSEs for the manufacturing industry experimental estimates have been created using a replicate method. This method uses replicate final estimates created using sub-samples of reported data to estimate the variance of the estimate.


Distribution of Manufacturing ANZSIC Class Experimental Estimates RSEs

Below is a table which compares the distribution of Manufacturing ANZSIC class experimental estimate RSEs for 2006-07 and 2007-08. Note that due to the larger sample for 2006-07, the RSEs for this year are, in general, smaller than 2007-08. The majority of the ANZSIC class RSEs are less than 15%, with the exceptions being 2006-07 Sales and Service Income (4 ANZSIC classes with RSE in the 15-25% range) and 2007-08 IVA (5 ANZSIC classes with RSE in the 15-25% range). No ANZSIC class has a RSE of greater than 25%.

RSEs for individual experimental estimates greater than 10% are footnoted in the table in Appendix: Experimental Estimates.

NON-SAMPLING ERROR

There are a range of other potential errors that are not caused by sampling and can occur in any statistical collection, whether it is modelled based on full enumeration or a sample. Non-sampling error may be due to inadequacies in available sources from which the population frame was compiled, imperfections in reporting by providers, errors made in collections such as recording and coding data, and errors made in processing data. Inaccuracies of this kind may occur in any enumeration, whether a full census or a sample.

Although it is not possible to quantify non-sampling error, every effort is made to reduce it to a minimum. Collection forms are designed to be easy to complete and assist businesses to report accurately. Efficient and effective operating procedures and systems are used to compile the statistics. The ABS compares data from different ABS (and non-ABS) sources relating to the one industry, to ensure consistency and coherence.

If non-sampling error is systematic (not random) then the estimates will be distorted in one direction and therefore will be unrepresentative of the target population. Systematic error results in bias.


MODEL BIAS

The use of a regression model to generate the experimental estimates may introduce bias. This bias arises from imperfections in the relationship between the EAS data and BAS data. While it is not possible to calculate the size of the modelling bias for these experimental estimates, a comparison of 2006-07 experimental ANZSIC class estimates with ANZSIC class estimates published in Manufacturing Industry, Australia, 2006-07 (cat.no. 8221.0 ) did not indicate obvious systematic error or bias.


VALIDATION OF THE METHODOLOGY

For the 2006-07 EAS, the sample size was increased to enable ANZSIC class data to be published in Manufacturing Industry, Australia, 2006-07 (cat. no. 8221.0). These data have provided a valuable source of comparison for the experimental estimates.

To enable an objective assessment of the validity of the methodology, the experimental estimates for 2006-07 were reproduced based on the reduced sample (i.e. disregarding the increase in sample required to publish ANZSIC class data in Manufacturing Industry, Australia, 2006-07). Experimental estimates generated from the reduced sample compared favourably to the Manufacturing Industry, Australia, 2006-07 estimates, further supporting the use of this methodology. Note that to maintain consistency with estimates published in Australian Industry, 2007-08 (cat. no. 8155.0), the full (increased) sample of units was used to generate the 2006-07 experimental estimates contained in this information paper.


REFERENCE PERIOD

The experimental estimates in this publication relate to manufacturing businesses in Australia during each of the years ended June 2007 and June 2008. Experimental estimates included the activity of any business that ceased or commenced operations during the relevant year. Where businesses were unable to supply information via the EAS on this basis, an alternative accounting period was used for which data could be provided. Such businesses made a substantial contribution to some of the experimental estimates presented in this publication. As a result, the experimental estimates can reflect trading conditions that prevailed in periods outside the twelve months ended June in the relevant year. This had the most impact on the manufacturing ANZSIC subdivision 17 Petroleum and Coal Product Manufacturing.