ACCURACY, RELIABILITY AND TIMELINESS
6.1 This chapter discusses the accuracy, reliability and timeliness of ABS GFS output and makes an assessment of the overall quality of the statistics included in each release of data for a given period.
6.2 In any statistical undertaking, there is usually some trade-off between accuracy, reliability and timeliness. The trade-off involves balancing users’ requirements for timely statistics against the time and cost (of the ABS and data suppliers) required to collect and compile statistics of a given degree of accuracy and reliability. Generally, any increase in timeliness comes at the expense of accuracy and reliability.
6.3 In this discussion, accuracy is defined as the closeness of an estimate to the ‘true’ value. Reliability is defined as the stability of an estimate as measured by the size and frequency of revisions made to the estimate over time. These two attributes should always be considered together, as it is possible to have a statistic that is reliable (because it is revised infrequently) but always inaccurate. In general, timeliness refers to the amount of time between the end of the period to which the statistics refer and the date of first release of the statistics to users. However, this definition is not applicable to forward estimates, which users would probably prefer to have before the reference year begins. Because GFS forward estimates are usually not available to the ABS until the end of the current year, timeliness for the forward estimates is defined as the length of time between the beginning of the forward year and the date of release of the statistics.
FACTORS AFFECTING ACCURACY AND RELIABILITY
6.4 There are a number of factors which affect the accuracy and reliability of ABS GFS. They include:
These factors are discussed below.
- the nature of source data;
- data collection timetables;
- estimation errors;
- data processing errors;
- data revision policies.
6.5 The quality of output is influenced by the nature of the source data available during the different phases of the GFS statistical cycle. The use of different ‘versions’ of annual source data (forward estimates, and final data) and quarterly data affects the quality of output at each stage. The factors affecting quality of data at each stage are discussed in the following paragraphs.
6.6 Forward estimates are derived as part of government budget formulation processes. Because the data are based upon expectations relating to government policies (and the measures by which they will be funded), the accuracy of the data are subject to the course taken by subsequent events. For example, unforeseen trends in the economy could mean that levels of government expenditure, revenue and financing will run at higher or lower levels than anticipated during the budget year.
6.7 In general, because governments have direct control over much of their expenditure, they can anticipate final expenditure outcomes fairly well. While governments set revenue targets for the budget year, the extent of control they have over final revenue outcomes is not as strong as that for expenditures. For example, the level of major revenue items such as taxation depends upon the level of economic activity, which is not under the direct control of governments.
6.8 Furthermore, the forward GFS data are not as complete or as detailed as final annual data. This lack of detail in the forward data means that errors and omissions are less likely to be detected.
6.9 It should be noted that the forward estimates are not statistical projections or extrapolations generated by the ABS. The estimates are made by government budget offices based on planned or anticipated government policies. In a small number of cases the ABS may have to rely on estimates reported by individual entities. In a smaller number of cases where the supply of estimates is delayed, the ABS may use its own indicative estimates rather than jeopardise publication deadlines.
6.10 Final data are the complete audited data for any jurisdiction for any given year, and replace the forward estimates for that year. These data generally satisfy the level of detail required. However, some dissections required for national accounting purposes are not normally available in financial statements and audited accounts and these have to be estimated. For example, State-level estimates of Commonwealth Government final consumption expenditure, personal benefit payments and gross fixed capital formation are derived for publication in Australian National Accounts: State Accounts (ABS Cat. no. 5220.0). The estimates are made using distributive factors that are based on data series which vary in terms of quality and timeliness.
6.11 The accuracy and reliability of quarterly data are affected by the use of a degree of sampling in their compilation (see Chapter 4). Consequently, quarterly data include a higher proportion of estimated data than preliminary and final data (see ‘Estimation errors’ below). They are also subject to revision when benchmarks are revised (see ‘Revisions’ below).
DATA COLLECTION TIMETABLES
6.12 Timetables for the collection and processing of GFS quarterly and annual data are necessarily very tight because users require the data as input to their own time-constrained programs. Quarterly production target dates are set mainly to meet the quarterly national accounts timetable, which requires the supply of quarterly GFS data six weeks after the end of the reference period.
6.13 These deadlines affect the accuracy and reliability of GFS through their impact on:
6.14 While some of these processes can be carried out concurrently, only a limited amount of time can be allocated in total to all the tasks involved in order to meet fixed deadlines, so trade-offs between accuracy and timeliness have to be made.
- the quality of data supplied by data providers;
- the amount of data analysis that can be done;
- the quality of data classification;
- the checking and editing of input and output data;
- the amount of estimation and imputation required;
- the number of revisions processed.
6.15 Timeliness of GFS output differs for the different streams of data. Forward and quarterly estimates are the most timely. The final data are usually released within 9 months of the reference period.
6.16 As noted in Chapter 2, not all in-scope enterprises are individually covered in GFS because the cost of collecting data from small units outweighs gains in accuracy and reliability. The way in which individual units are covered in GFS dictates the level of data estimation, which affects the quality of GFS. Most units are ‘directly’ covered while other units are ‘indirectly’ covered. A directly covered unit is one for which data from the unit’s accounts are included in GFS. An indirectly covered unit is one for which economic flows and stocks are deduced from data recorded by the directly covered units with which the indirectly covered unit undertakes transactions.
6.17 Indirect coverage of units is employed where the data of individual units are not readily available, are not available in sufficient time or are of insufficient statistical significance to warrant the cost of direct coverage. The most common example of units which are indirectly covered are public hospitals. Most of the data for the public hospitals in each state and territory can be deduced from data in the records of the relevant jurisdiction’s health department.
6.18 While the detrimental impact of the indirect (partial) coverage of in-scope units on the accuracy and reliability of GFS has not been quantified, the amount of information missed by use of the procedure is considered to be small.
6.19 A small number of in-scope units are deliberately excluded from coverage because the cost of their inclusion outweighs the marginal increase in the accuracy of GFS. No statistical expansion is made to account for this undercoverage.
6.20 The quarterly data are compiled using a mix of full enumeration of larger units and some sampling of smaller units. Non-probability samples of local government authorities are used to produce quarterly estimates for the local government sector. As well, some dissections of quarterly data for other levels of government are estimated using previously recorded ratios. Overall, the use of sampling in Australia’s GFS is relatively minor.
6.21 Estimation errors for individual levels of government arising from the adjustments made for undercoverage built into the quarterly collection cannot be quantified readily. The estimation techniques involve assuming that the relationships between the collected and uncollected data that existed in the last annual benchmark census remain the same in the current quarter. The estimates made represent only a very small proportion of the value recorded for the data items concerned.
DATA PROCESSING ERRORS
6.22 The ABS GFS processing system has been designed to incorporate a series of data checks and edits (see Chapter 4) with the purpose of minimising or eliminating data processing errors. However, data processing errors can go undetected either because there is insufficient time to undertake all the checks and edits, or because there is not a check or edit covering a particular error. Such occurrences affect the accuracy and reliability of GFS output. Undetected errors arising from incomplete editing are part of the tradeoff between accuracy and timeliness. The errors in question are usually small and are usually detected when more complete editing can be undertaken. Errors that are not detected by input editing may be detected in output editing, which is an essential complement to the input editing process.
6.23 Errors may occur when a data supplier either provides an incorrect figure or has to provide an estimate for data that are not readily available from accounting records. Errors can also occur because analysts may misclassify transactions in such a way that the errors are not detected in the editing process.
6.24 It is impossible to quantify the effect of undetected data processing errors. However, the effect of such errors that go undetected for a time but are eventually detected is reflected in revisions, which are quantifiable (see discussion ahead under ‘Assessment of accuracy, reliability and timeliness’).
6.25 Inaccuracies and imbalances may arise during the process of consolidating data. Inaccuracies can arise because accounting records do not enable identification of intrasector flows and stocks or because errors and omissions are made in the allocation of source and destination codes. Such errors will usually give rise to imbalances that will be detected in the consolidation process. As discussed in Chapter 4, every effort is made to resolve such imbalances that are material. When imbalances cannot be resolved in time for publication the data are forced into a balance by adopting a convention (e.g. the record of the ‘higher’ level of government prevails) or making a judgement as to which of the two values should be accepted. Forced balancing does not necessarily give the ‘right’ answer. However, because the data to which forced balancing is applied should not be material, errors arising from this source should not be significant.
6.26 Revisions are amendments made to previously released data. They can occur for a number of reasons. As previously discussed, in GFS a major reason for revisions is the progressive replacement of data over the processing cycle (i.e. the replacement of forward estimates with final audited data). Revisions are also required because errors are detected in data after their initial release. Conceptual and methodological changes also give rise to revisions.
6.27 Revisions to GFS data are not applied immediately, but are applied at specified times that coincide with the release of publications. This means that, at any point of time, the data include estimates that will not be updated until revisions are applied. However, restriction of the application of revisions to particular times is preferable to having a data set that is continually subject to change.
6.28 The times of application of revisions to GFS data are currently dictated by the revisions policy for the Australian System of National Accounts. The policy allows revisions to be applied in the releases for various quarters as required by National Accounts Branch.