1001.0 - Annual Report - ABS Annual Report, 2005-06  
ARCHIVED ISSUE Released at 11:30 AM (CANBERRA TIME) 11/10/2006   
   Page tools: Print Print Page  
Contents >> Section 5 - Performance Information >> Chapter 12 - Quality and timeliness

Chapter 12 - Quality and timeliness

INTRODUCTION

The quality of ABS outputs is critical to ensuring the ABS achieves its mission of assisting informed decision-making. The ABS strives to maximise the quality of information produced, taking account of budgetary constraints and an assessment of reasonable load to place on the respondents to our surveys.

INTEGRITY

The ABS wants to produce credible statistics, and wants users to be confident of the quality. The integrity of ABS statistical operations is built on professional standards. The ABS aims to exercise the highest professional standards in all aspects of statistical operations. The ABS also recognises that having a quality culture is fundamental to maintaining the trust of the Australian community.

ABS data, analysis, and interpretations are objective, and the ABS always publishes its statistics in ways that explain and inform, without advocating a particular position.

The ABS regularly reviews the methodology used to produce statistics, providing the opportunity to make improvements and to incorporate new approaches where appropriate. For example, the methodology used for the Census Post-Enumeration Survey has been reviewed to ensure that the population estimates produced following the 2006 Census are as accurate as possible (for more information see chapter 5).

The ABS statistical system is open to scrutiny by people outside the organisation. The methodologies and approaches followed are based on sound statistical principles and practices and disseminated widely. The Methodological Advisory Committee consists of professional statisticians external to the ABS, and meets twice a year to provide peer review of methodological developments in the ABS. A range of research papers are published explaining statistical developments and research.

ASPECTS OF QUALITY

To ensure high quality statistics are produced, monitoring of quality is an integral part of the production process. The ABS quality assurance work covers a number of aspects of quality, which are examined in this chapter:

    • relevance – the degree to which information meets the needs of users
    • accuracy – the degree to which the information correctly describes the phenomena being measured
    • timeliness – the delay between the reference period and the release of the information
    • accessibility – the ease with which the information can be referenced
    • interpretability – the availability of supplementary information necessary to interpret the statistical information
    • coherence – the degree to which the information can be brought together with other information and over time.
Addressing the quality of a statistical product will always involve balancing these aspects. For example, often initiatives that could improve the accuracy of the statistics will reduce the timeliness or the relevance. It is also important the ABS manages its budget and finds the right balance between achieving a high quality of statistical information and collecting an appropriate range of statistics. The ABS aims to produce a large and diverse body of statistics with a quality designed to meet the key needs of researchers, policy makers and other users within the Australian community. The ABS also strives to ensure that the users of ABS statistical information are provided with readily accessible information on quality, so that they can make informed decisions on the suitability of the statistics for their intended use.

RELEVANCE

The relevance of statistical information reflects the degree to which it meets the real needs of the users of the information. It is concerned with whether the available information sheds light on the issues most important to researchers and policy-makers. The outputs produced, the concepts and classifications used and the scope of the collection can all affect the relevance of the data.

A detailed understanding of the users of statistical information and their requirements is a very important part of the statistical process within the ABS, and it has a range of mechanisms to achieve this. The Australian Statistics Advisory Council assists the ABS with this. There are also a range of other groups and mechanisms, as described in chapter 9.

For particular surveys, key stakeholders are identified and consulted before and during the development. Further, each survey is regularly evaluated to assess the degree to which it meets user requirements. The gaps between user requirements and statistical output are formally considered by the ABS senior executive each year.

Other specialised reviews are also conducted regularly, for example, adequacy of data outputs from monthly retail trade, quarterly business indicators and new capital expenditure.

ACCURACY

The accuracy of statistical information is the degree to which the information correctly describes the phenomena it was designed to measure. Most statistics produced by the ABS are obtained from a sample of households or businesses. This process results in some uncertainty as to the accuracy of the estimates published. For example, the estimate from the sample may not be the same as would have been obtained if information had been collected from the whole population – this is known as sampling error. There are also other sources of error that potentially cause inaccuracy including the level of non-response, the magnitude of revisions made as additional information is received, and errors from other parts of the collection process (non-sampling error).

INFORMATION ABOUT ACCURACY

Users will want to use statistical information for different purposes so it is important to make information available to enable users to make their own assessment of quality. Descriptions of accuracy, as well as extensive information on the statistical methods used in collections, are routinely provided in concepts, sources and methods publications, the explanatory notes in publications, and at the Statistical Clearing House through the ABS web site.

In addition, major changes to methodology are explained in feature articles or information papers. Some examples include a feature article on a new methodology for deriving counts of Australian exporters in International Trade in Goods and Services, Australia, February 2006 (cat. no. 5368.0), and Information Paper: Improved Methods for Estimating Net Overseas Migration, 2006 (cat. no. 3107.0.55.003).

The ABS has made few significant mistakes in the statistics it has released. On the infrequent occasions when substantial processing errors are found, it is ABS policy to publish corrected data as soon as possible. A serious error was discovered in the production of retail trade estimates in May 2005. The ABS commissioned a statistical expert, Mr John Cornish, to independently review the causes leading to the error. His report and the ABS management response were published on the ABS web site in July 2005. The ABS has been implementing a range of the recommendations from this review to help prevent or minimise the risk of a repeat occurrence of similar errors, especially when methodological or system changes are being made.

More recently, Producer Price Indexes, Australia, March 2006 (cat. no. 6427.0) and International Trade Price Indexes, Australia, March 2006 (cat. no. 6457.0) had to be reissued due to a processing error that resulted in incorrect exchange rates being used in converting prices expressed in foreign currencies (mainly $US prices for imports) into $A. Again, the ABS was open about the error, and will ensure that processes are examined to ensure that any weaknesses are addressed.

NON-SAMPLING ERROR

Non-sampling error is a general term that describes all sources of error other than the error introduced by the sampling process. Sampling error can be measured by using the mathematical properties of the selected sample. Non-sampling error is much harder to measure.

Some sources of non-sampling error that are most relevant to statistical surveys include: non-response error; errors in identifying and contacting the population of interest for a survey; errors introduced by the questionnaire design, such as misunderstanding the question, inadvertently missing questions, or phrasing questions that predispose a respondent to answer in a particular way; and data capture, processing and coding errors.

The ABS minimises the impact of non-sampling errors by use of better practice procedures in questionnaire design, interview procedures, data validation and repair, and processing. Any significant changes to questionnaire wording or data collection methods are carefully trialled and evaluated before they are implemented.

SAMPLING ERROR

The relative standard error (RSE) is a measure of the sampling error associated with an estimate. The magnitude of standard errors varies between collections and between data items within a collection due to factors such as the responding sample size and the nature of the data item. The RSE is a useful indicator for comparing the accuracy of estimates between surveys. Table 12.1 presents a summary view of the estimated RSEs for key statistics from a number of major ABS surveys. Further detailed information is included with each ABS publication, as well as in the concepts, sources and methods publications released by the ABS.

Relative standard errors are affected by the size of the sample used, the sample design used for the survey, and by the underlying variability of the indicator in the population.

Sample size influences the level of accuracy that can be attained. For example, RSEs for estimates of Indigenous Australians will generally be higher than RSEs for estimates of all persons, as there are generally fewer Indigenous Australians contributing to the sample. This is shown in Table 12.1 below – the sample size for the estimate from the National Health Survey was 25,900 and the resulting RSE was 1.0%, while the sample size for the National Aboriginal and Torres Strait Islander Health Survey was 10,400 and the RSE was 3.0%.

ABS sample designs for business surveys use groups of similar businesses (strata) as the basis of sample selection to improve the efficiency of estimation. Information such as employment size or annual sales can be used in this grouping. Many indicators, such as annual turnover or value of building work done, are closely related to the variables used in stratification, allowing these indicators to be estimated with relatively high accuracy. Other variables, such as capital expenditure or job vacancies, are not as closely related, and so cannot be estimated with the same accuracy.

As well as differences between surveys, RSE can also change with time for any given survey. These changes may be due to changes in the way the survey is conducted, for example, changes in the sample size or the method of producing estimates, or may be due to changes in the population being studied, for example, a change in the prevalence of a particular characteristic.

An improvement was made in the accuracy of retail trade turnover for 2004–05 (to 0.8% from 0.9% in 2003–04) due to an improved sample design and estimation methodology even though the overall sample size was reduced. For more information, refer to Information paper: Changes to the Retail Trade Series, July 2004 (cat. no. 8501.0.55.002).

The RSE for job vacancies is relatively large due to the underlying variability. That is, the number of job vacancies can vary considerably from business to business, and for any business it can vary considerably from month to month, and so a very large sample would be required to measure job vacancies with high precision. In the table below, the estimated RSE for job vacancies for 2004–05 of 5.5% represents a decrease from 6.8% for 2003–04. This was not due to an improvement in survey methodology, but rather due to an increase in the number of vacancies, which has made the error smaller in relative terms.

TABLE 12.1: RELATIVE STANDARD ERRORS (RSE) FOR SELECTED INDICATORS (a)

PublicationIndicatorRSE (%)

Economic indicators
Retail Trade, Australia (cat. no. 8501.0)Total turnover for the retail industry, Australia0.8
Private New Capital Expenditure and Expected Expenditure, Australia (cat. no. 5625.0)Actual private new capital expenditure, Australia1.4
Business Indicators, Australia (cat. no. 5676.0)Company gross operating profit, Australia1.9
Building Activity, Australia (cat. no. 8752.0)Value of work done, Australia0.6
Average Weekly Earnings, Australia (cat. no. 6302.0)Full-time adult ordinary time earnings, Australia0.7
Job Vacancies, Australia (cat. no. 6354.0)Job vacancies, Australia5.5
Social indicators
National Health Survey: Summary of Results (cat. no. 4364.0)Number of persons whose self assessed health status is excellent or very good1.0
National Aboriginal and Torres Strait Islander Health Survey (cat. no. 4715.0) Number of Indigenous Australians whose self assessed health status is excellent or very good 3.0
Childcare (cat. no. 6530.0) [Monthly Population Survey]Number of children using childcare1.6


(a) Relative standard errors are presented for surveys conducted in respect of the 2004–05 reference period

REVISIONS TO DATA

One measurable component of statistical accuracy is revisions to data made after initial publication, resulting from additional information becoming available. Revisions are generally measured by their size and frequency over time.

Revisions are applied to statistical series to ensure that there is an appropriate balance between accuracy and timeliness in the release of statistics. Revisions could be avoided – but this would mean that either the release of statistics is substantially delayed, or that any inaccuracies subsequently discovered in released statistics are not corrected. The ABS aims to maximise the overall quality of the released statistics by releasing accurate statistics in a timely manner, while subsequently improving the accuracy through revisions as new data becomes available. It is also ABS policy to inform users of any significant revisions and, where appropriate, to revise past time series and advise users accordingly.

The tables below provide, for two key series, the mean revision and the mean absolute revision for the past six years. The mean revision shows the percentage difference between the first estimate published, and that estimate one year later, averaged over the four quarters for the year. The mean absolute revision shows the average absolute values of the mean revision.

Table 12.2 describes the revisions to quarterly gross domestic product (GDP). In particular, it shows the difference between the first estimate of GDP and that estimate one year later, in terms of the mean revision and the mean absolute revision expressed as percentage points. The figures continue to show that revisions to quarterly GDP in recent years remain relatively small (mean absolute revision). Zero mean revision figures indicate that the revisions to quarterly GDP over the year have been offsetting. Despite the revisions to quarterly GDP being quite small, efforts to further improve the estimates are ongoing.

TABLE 12.2: REVISIONS TO GROSS DOMESTIC PRODUCT, PERCENTAGE CHANGE (a)

 
Difference between first estimate and estimate one year later

Reference year
Mean absolute revision (% points)
Mean revision (% points)

1999-2000
0.1
-
2000–01
0.2
-
2001–02
0.2
-
2002–03
0.1
-
2003–04
0.2
0.2
2004–05(b)
0.1
-


(a) Seasonally adjusted chain volume measure
(b) Figures based on three quarters of GDP data

A decreasing trend in the mean absolute revisions to the quarterly current account transactions since 1999-2000 is shown in Table 12.3. The revisions to the current account deficit are expressed in percentage terms, rather than percentage points as is the case with the revisions to GDP.

TABLE 12.3: REVISIONS TO QUARTERLY CURRENT ACCOUNT TRANSACTIONS (a)

 
Difference between first estimate and estimate one year later

Reference year
Mean absolute revision (%)
Mean revision (%)

1999-2000
2.3
0.3
2000–01
3.4
-0.3
2001–02
2.8
-0.1
2002–03
1.8
-0.5
2003–04
1.2
0.3
2004–05 (b)
1.3
0.5


(a) Seasonally adjusted data
(b) Figures based on three quarters of the data

TIMELINESS

The timeliness of statistical information can be measured by the gap between the reference period (the period the data relates to) and the date of release of results. The ABS continues to adhere to pre-announced release dates and make improvements, where possible, to the timeliness achieved. Tables 12.4 and 12.5 presents information on the timeliness for ABS monthly and quarterly tabular data for main economic indicator statistics, and other general releases. Table 12.6 reports on the timeliness of confidentialised unit record files (CURFs).

There has been little change to the timeliness of statistical tables in 2005–06, apart from the increase in average number of elapsed days for the release of other quarterly data. The increase is due to the re-issue of small area tourist accommodation data for March 2005 in October 2005. The re-issue affected eight separate products (small area tourist accommodation statistics for eight states and territories), and so contributed eight times to the average. The re-issue was planned as part of expanding the scope of the survey to include additional types of accommodation. The original release of March data reflected the old scope. In October 2005, data for the new scope was released for both March and June quarters. This approach was taken to ensure data quality. For more detail, refer to the description of scope in the explanatory notes of Tourist Accommodation, Australia (ABS cat. no. 8635.0).

TABLE 12.4: TIME BETWEEN END OF REFERENCE PERIOD AND RELEASE OF TABULAR DATA (AVERAGE NUMBER OF ELAPSED DAYS) (a)

 
Main economic indicator tabular statistics
Other general tabular releases

Year of release
Monthly
Quarterly
Monthly
Quarterly

2001–02
29
51
34
78
2002–03
28
49
33
74
2003–04
29
51
26
85
2004–05
29
51
25
75
2005–06
30
52
24
87


(a) Where a publication or spreadsheet has been reissued, the reissue date is used in the calculation of average.


TABLE 12.5: TIME BETWEEN END OF REFERENCE PERIOD AND RELEASE OF TABULAR DATA FOR SELECTED PUBLICATIONS

Publication
Frequency
Average number of elapsed days (a)

Retail Trade, Australia (cat. no. 8501.0)
Monthly
33
Building Approvals, Australia (cat. no. 8731.0)
Monthly
33
Labour Force, Australia (cat. no. 6202.0)
Monthly
10
Consumer Price Index, Australia (cat. no. 6401.0)
Quarterly
25
Australian National Accounts: National Income, Expenditure and Product (cat. no. 5206.0)
Quarterly
66
Australian Demographic Statistics (cat. no. 3101.0)
Quarterly
166


(a) Average is taken over the most recent one year's worth of releases, as at 14 July 2006.

The timeliness of release of information depends on a number of factors, including the amount and complexity of information being collected, the source of the data (for example, whether directly collected or sourced from administrative records), and the amount of processing or validation of the information required before release.

Labour force statistics are released very quickly after the end of the reference month. Part of the explanation for this is that the data collection is completed before the end of the reference month. Interviews are generally conducted over a two week period commencing on the Monday between the 6th and 12th of each month. The information collected from each survey respondent is relatively small compared to other surveys conducted by the ABS, and an 'any responsible adult' methodology is used to allow one member of a household to respond on behalf of other household members, so that subsequent appointments with specific respondents can be avoided. In addition, Labour Force Survey processes have been enhanced and refined over time so they are now very efficient and a large collection is processed in a relatively short timeframe. This all helps in the very timely release of statistics.

Demographic statistics report on Australia's population, components of population growth, and distribution among states and territories. The quarterly changes to population statistics are based on a variety of administrative sources, such as registrations of births and deaths, and passenger cards completed at Australia's borders, and modelled estimates of interstate migration (using information from Medicare card registration address changes, delayed by three months as registration often takes place after the actual move). It takes around five months before estimates can be published due to the time needed to acquire and process the administrative data, particularly with the delay of three months for the Medicare card data.

The average number of elapsed days between the end of the reference period and the supply of confidentialised unit record file (CURF) data has improved significantly in recent years. Note that the information for the 2004–05 reference year only includes CURF data made available prior to 1 July 2006.

TABLE 12.6: TIME BETWEEN END OF REFERENCE PERIOD AND RELEASE OF CONFIDENTIALISED UNIT RECORD FILE (CURF)

Reference year
Number of CURFs released
Average number of elapsed days

2002–03
5
724
2003–04
2
548
2004–05 (a)
4
333


(a) As at 30 June 2006 – further microdata from the 2004–05 reference year are expected to be made available in the second half of 2006, which will increase the total number of 2004–05 CURFs released as well as increase the average number of elapsed days.

ACCESSIBILITY

The accessibility of statistical information refers to the ease with which it can be referenced. This includes the ease with which the existence of information can be ascertained, as well as the suitability of the form or medium through which the information can be accessed. The cost of the information may also be an aspect of accessibility for some users.

All statistics on the ABS web site can now be accessed free of charge. The new policy was announced by the Treasurer, The Hon. Peter Costello MP as an ABS centenary tribute to the people of Australia in December 2005. The change means that all publications, spreadsheets and census data on the ABS web site are now available free to any member of the public with Internet access. However, people who require their own paper copies of ABS publications, information on CD-ROM, or information more detailed than that published, will be charged under the ABS pricing policy.

Confidentialised unit record files (CURFs) are a product that allow approved researchers with a valid statistical purpose to access individual survey responses. The data files are confidentialised and access is carefully controlled to ensure that no individual or organisation can be identified. The price of CURF access has been reduced from 1 July 2006 to $1,320 per application in order to improve the accessibility of this information. The ABS has also worked to improve the accessibility of CURFs, including through the ABS Remote Access Data LaboratoryTM (RADLTM). For more information see chapter 13.

CRITICAL EVALUATION OF THE ABS WEB SITE

In February 2006, the ABS went live with its new look web site, featuring improved search and navigation functions. MP Marketing Partners, who had recently completed a critical evaluation of 1750 government sites from 204 countries, were commissioned to evaluate the ABS web site. Objectives were to compare the new ABS site to the old site as well as benchmark it against other government sites, including those of other government statistical agencies.

CRITERIA

To evaluate the ABS site against other government web sites, a set of criteria developed by the ProfNet Institute in Germany was employed. 146 criteria, grouped under the headings of layout, handling, content and interactivity, measured the sites in terms of 'customer friendliness'. A total of 100 points was achievable.

To further enhance the study, MP Marketing Partners customised a new set of criteria, taking into consideration requirements that are unique to users of statistical sites. These included presentation of data and search capabilities.

ABS WEB SITE – NEW VERSUS OLD

Findings suggested that the new site is a significant improvement over the old site (Table 12.7).

TABLE 12.7: COMPARISON OF OLD AND NEW ABS WEBSITE

Website
Layout
Handling
Content
Interactivity
Total

New
11.0
16.5
26.5
10.0
64.0
Old
9.0
12.5
23.5
7.5
52.5
Difference
+ 2.0
+ 4.0
+ 3.0
+ 2.5
+11.5


INTERPRETABILITY

The interpretability of statistical information reflects the availability of the supplementary information and metadata necessary to interpret and utilise it appropriately. This information normally covers the availability and clarity of metadata, including concepts, classifications and measures of accuracy. In addition, interpretability includes the appropriate presentation of data such that it aids in the correct interpretation of the data.

ABS releases are accompanied by extensive explanatory notes to aid the interpretation of statistical information. There is also a range of material available on the ABS web site detailing the methods, classifications, concepts and standards used by the ABS. During 2005–06, the Australian Consumer Price Index: Concepts, Sources and Methods, 2005 (cat. no. 6461.0) was released for the first time and Labour Statistics: Concepts, Sources and Methods, 2006 (cat. no. 6102.0.55.001) was updated.

The ABS is currently working to improve the metadata available for ABS collections. For more information see chapter 15.

COHERENCE

The coherence of statistical information reflects the degree to which it can be successfully brought together with other statistical information within a broad analytic framework and over time. Coherence encompasses the internal consistency of a collection as well as its comparability both over time and with other data sources. The use of standard concepts, classifications and target populations promotes coherence, as does the use of common methodology across surveys.

Coherence of ABS output requires the use of nationally and internationally agreed concepts and classifications. Standard concepts and classifications are not only used extensively within the ABS, but also promoted to other producers of statistical information within Australia. Information on statistical standards, concepts, classifications and methodologies are readily accessible through the ABS web site. For more information refer to the chapter 15.

The Statistical Clearing House (SCH) provides approval to conduct surveys that are directed to 50 or more businesses and that are conducted by or on behalf of any Australian Government agency, to ensure that surveys are necessary, well designed and place minimum burden on business respondents. One of the criteria used by the SCH is the coherence of the statistical information that will be produced. In particular, surveys are assessed on their use of standard methodologies, concepts and classifications, their consistency with past or future surveys, and the extent to which outputs can be compared or jointly used with other sources of data. For more information about the SCH see chapter 11.

Any changes that may impact on the coherence of ABS statistics are detailed in the explanatory notes that accompany each release. Significant changes may lead to series breaks in time series, or adjustments to past data. An example of a series break is the trend adjustment made in the release of Balance of Payments and International Investment Position, Australia, June quarter 2005 (cat. no. 5302.0) to account for significant increases in the prices of coal and iron ore export commodities during this quarter.



Previous PageNext Page