The concept of quality
29.1 Given the importance of the national accounts for economic policy formulation and for monitoring the impact of economic policy, it is important that they are of a high quality. The quality of a particular statistic is determined by its 'fitness' for whatever purposes users have for the statistics. As different users use national accounts statistics for different purposes, views on what aspects of quality are most important will vary from user to user. Nonetheless, there are a number of aspects of quality that are relevant in most circumstances. These are:
29.2 This chapter describes each of these aspects of quality, and assesses the national accounts against them. Compilation of the national accounts is a complex task involving many diverse data sources. It is not possible to provide a single, comprehensive measure of the quality of the estimates. Nonetheless, it is possible to gain an insight into their quality by analysing each of the aspects of quality. To obtain an overall picture, all aspects need to be considered together. However, different users may weight each of the aspects differently, and within each aspect what satisfies one user may not satisfy another. Thus, two users may look at the same set of statistics, with one considering them to be of good quality while the other may think that there are quality deficiencies.
29.3 In understanding quality, it is also important to recognise that there are often trade-offs across the aspects of quality. For example, statistics that are very timely are often less accurate than those that are less timely. To overcome this, a sequence of estimates for a particular period of time may be released, with each estimate considered to be more accurate than the one that preceded it. However, the disturbance in estimates that this induces affects the reliability of the statistics. Trade-offs can also occur between accuracy and relevance. It may be more difficult to measure exactly what users require than something that approximates their requirements. It is also important to recognise that, in general, the higher the quality that an estimate is, the more costly it is to compile it. This is particularly so for the accuracy aspect of quality. Nonetheless, with improvements in technology and other aspects of the compilation process it may be possible to improve the quality of statistics without increasing costs. Another important issue that must be considered in assessing quality is the complexity of what is being measured. As complexity increases, as has been the case with the Australian economy, it becomes more difficult to maintain quality, all other things being equal. Again, improvements in the compilation process can have a mitigating impact on this.
29.4 In compiling the national accounts the ABS is mindful of all of these issues. The ABS maintains regular contact with key users and their views on what aspects of quality are most important to determine how the quality trade-offs are dealt with. User views on quality are also given prominence in developing work programs for improving the quality of the national accounts.
- comparability with other statistics
29.5 Accuracy can be defined as the proximity of an estimate to its notional true value. The true value is considered notional because, in practice, most statistics cannot be measured with perfect accuracy. Also, as this true value is generally unknown, it is generally impossible to quantify exactly how far an estimate deviates from its true value. Nonetheless, by being aware of the factors that influence accuracy, judgements can be made as to the extent of error likely to be associated with an estimate. These judgements can be enhanced by comparing estimates that conceptually should be identical, or by comparing estimates where a particular relationship between the estimates could be expected. In other words, the extent to which a set of statistics are coherent can provide a guide to the accuracy of the statistics. However, it should be noted that a coherent set of statistics is not necessarily an accurate set, as the statistics that are being compared may suffer from similar magnitudes of error with the errors being in the same direction.
29.6 The accuracy of statistics is influenced by a number of factors, including:
- data collection errors, which include errors due to the inability of data providers to report on the correct basis, mistakes in the reporting of data, errors due to non-response, and errors introduced during the processing of data;
- methodological errors, which include errors resulting from shortcomings in data sources and estimation methods;
- errors in the coverage of source data collections; and
29.7 The national accounts are compiled within a comprehensive framework, and so it is possible to reduce the impact of these data errors through the confrontation of the various estimates in the national accounts. Data confrontation is built around the conceptual relationships that exist between data items. The best known data confrontation exercise is the compilation of the annual supply and use tables. In the compilation of these tables, estimates of the supply (production) and use (demand) for commodities are compared, with differences in the initial estimates being eliminated. At the same time estimates of the value of production are compared with estimates of incomes attributable to production and differences are eliminated.
- errors attributable to the use of sample surveys, rather than complete enumerations.
Data collection errors
29.8 The most likely source of data collection error, from a national accounts perspective, would result from the inability of data providers to report on the correct basis. The data requirements underlying the national accounts are complex and, although every effort is made to match survey data items with business accounting practices, it is inevitable that some data providers will include in their survey responses items that should not be included, and exclude items that should be included. Sophisticated techniques are used to edit provider responses but not all errors will be recognised and rectified. If patterns of misreporting are identified, steps will be taken to reduce the incidence of misreporting, either by specifically drawing to the attention of data providers what should be included or excluded, or by changing the reporting requirements to better align with business accounting practices. The problem with the latter approach, though, is that some adjustment will be needed to reported data to place it on the correct basis for national accounts compilation. This adjustment process is another source of error for the national accounts estimates themselves.
29.9 As well as inappropriate inclusions and exclusions, data providers can make errors with regard to the timing and valuation of their transactions. Because the national accounts is a closed system, such errors can lead to inconsistencies, affecting the coherency of the accounts. For example, if the import of a capital good, valued at $150 million, was recorded in one quarter and the related capital formation (assuming that the good never entered inventories) was recorded in the subsequent quarter, then the expenditure measure of GDP will be too low by $150 million in the first quarter and too high by the same amount in the next quarter. This, in turn, will cause errors to growth rates for three quarters, the two quarters affected by the misreporting and the following quarter, with the greatest error occurring in growth rates for the middle quarter.
29.10 Another significant data collection problem concerns income tax data from the ATO, which is used to derive annual estimates for aggregates such as gross operating surplus and gross mixed income. It is likely that some businesses under-report their income or over-report their expenses to the ATO in order to avoid income tax. It is difficult to quantify the extent of this misreporting. However, estimates for it are made based on a variety of sources, including ATO business audit information, confrontation with other data and anecdotal evidence.
29.11 Statistical collections have a cost both to the organisation undertaking the collection and the respondents. The more frequent the collection, the greater the number of respondents or the greater the number of questions then the greater is the cost to both the statistical organisation and respondents. It is therefore necessary to adopt a strategy that makes the best possible use of data available from administrative and other sources and that structures statistical collections around these data such that they maximise the benefits for a given cost. This inevitably means that for many areas of economic activity accurate and detailed data are only available annually or less frequently. In these cases, annual or less-frequent benchmarks are established, with more frequent estimates derived using extrapolation and interpolation techniques. For the most part, indicator series are used for this purpose. The indicators vary in the closeness of their relationship to the concept being measured. For example, annual benchmark estimates of private gross fixed capital formation on machinery and equipment, which are derived from the annual Economy Activity Survey and taxation statistics from 1994-95 (and exclusively from taxation statistics for years prior to 1994-95), are projected using indicators from the quarterly Survey of Private New Capital Expenditure. These indicators are considered to be very good for this purpose, although they do have their deficiencies, such as the lack of coverage of non-employing businesses. In the case of the quarterly chain volume estimates of gross value added by industry, with the exception of agriculture, the quarterly indicators are either output or input measures rather than value added itself. In the short term, output measures are likely to be reasonable indicators of changes in value added in volume terms, but input measures preclude changes in productivity. The quarterly estimates for non-farm gross mixed income are not very satisfactory, as they are derived using broad activity indicators such as retail turnover and capital formation on dwellings. For some relatively minor series where no suitable quarterly information is available, annual estimates are extrapolated and interpolated using mathematical techniques.
29.12 Methodological errors can also affect the compilation of chain volume measures. For the most part, the price indexes that are used for this purpose are very appropriate. However, chain volume measures for some aggregates are compiled using proxy price indexes in the absence of price indexes directly pertaining to the aggregates. For example, in the absence of an index of purchasers' prices for capital equipment, chain volume estimates for private gross fixed capital formation on machinery and equipment are derived using various elements of the import and articles produced by manufacturing industries price indexes. Errors will be present in the chain volume measures of these aggregates to the extent that movements in the proxy price indexes are different to those that would be observed in the 'correct' price indexes if such indexes were to be compiled.
Errors in coverage
29.13 Most of the ABS data sources used in the compilation of the national accounts are taken from the ABS's business register. As this register generally only contains businesses that employ staff, the estimates from these data sources generally exclude the economic activity of non-employing businesses.
29.14 Therefore, in compiling the national accounts, estimates of the economic activity of non-employing businesses have to be obtained from other sources, of which the most important is data from the Australian Taxation Office. However, such sources are generally only available annually, and quarterly estimates are generally based on indicators that exclude non-employing businesses (there are exceptions, e.g., quarterly estimates of building activity are derived from a survey based on building approvals, and include building activity of both employing and non-employing businesses). If the economic activity associated with non-employing businesses changes at a different rate to that associated with employing businesses then there will be errors in the quarterly estimates. The extent of error for any particular series will depend, in part, on the contribution of non-employing businesses to the economic activity being measured by that series. At one end of the scale, the problem does not affect estimates of compensation of employees as, by definition, non-employing businesses are excluded from the compilation of this item. However, for quarterly estimates of gross mixed income, the problem is significant as a large proportion of unincorporated businesses have no employees.
29.15 More generally, there are lags between businesses being created and their appearance on the ABS business register. The ABS seeks to overcome this problem by using 'new business provisions', which are estimates for the economic activity of businesses that are not yet on the register. The business register is also known to suffer from 'leakage', which occurs when employing businesses are inadvertently removed from the register. Again, estimates are made for the impact of this.
29.16 Many of the data sources used to compile the national accounts, particularly those from the ABS, are based on sample surveys, rather than complete collections. Sample surveys are subject to a particular type of error, known as sample error. Unlike most other forms of error, the likelihood of sample error can be quantified using mathematical techniques. The most common measure of sample error is the relative standard error (RSE). The true value of any estimate lies within one RSE of the sample estimate about two-thirds of the time, and within two RSEs about nineteen times in twenty. For example, if a sample estimate of $100 million has an RSE of 3%, then there are about two chances in three that the true value lies in the range $97 million to $103 million, and nineteen chances in twenty that it lies in the range $94 million to $106 million.
29.17 The ABS publishes information on the RSEs for its various sample surveys. These can provide an indication of the accuracy of the national accounts components to which they relate. However, because of the transformations of survey data that are made in order to compile the national accounts, it is generally not possible to calculate the exact impact that RSEs have on the various national accounting aggregates.
Some assessments of the accuracy of quarterly national accounts estimates
29.18 While it is generally not possible to provide exact information on the accuracy of national accounts estimates, intuitive assessments of the accuracy of the estimates can be made, based on knowledge of data sources used. The tables below contain such assessments for the initial quarterly estimates of movement for key components of the accounts. Initial quarterly estimates of movement have been chosen as they are generally the most anticipated of the national accounts estimates. Each component is assigned one of the following grades:
29.19 As well as providing an indication of the accuracy for particular components, the tables can be used to assess the relative accuracy across the components of the national accounts.
29.20 In using the tables, it should be noted that the assessments of accuracy relate to the time this publication was printed. Estimates for prior or future periods may vary as the quality of data sources, compilation techniques, etc can change over time.
29.21 The following table contains accuracy ratings for the current price income and expenditure components of GDP, and for the chain volume measures of the expenditure components of GDP. The accuracy of the expenditure chain volume measures is generally a function of the accuracy of the current price estimates and the accuracy of the price indexes that are used to deflate the current price estimates.
Accuracy Ratings -- Expenditure and Income Components of GDP -- Initial Quarterly Estimates of Movement
29.22 The following table contains accuracy ratings for the industry value added chain volume measures. The accuracy of these estimates reflects both the appropriateness of the indicators used and the accuracy of the indicator estimates themselves.
Accuracy Ratings -- Industry Value Added, Chain Volume Measures -- Initial Quarterly Estimates of Movement
29.23 More objective, but limited, measures of accuracy are provided by the various statistical discrepancies contained in the national accounts. The table below shows estimates of the average absolute values of the statistical discrepancies in the three estimates of GDP. As explained in Chapter 4, there are three approaches that can be used to measure GDP: the income, expenditure and production approaches. In concept, each approach should deliver the same estimate; however if the measures are compiled independently using different data sources then different estimates will result. In Australia's national accounts, a single quarterly estimate of GDP is obtained by averaging the three measures, and statistical discrepancies are inserted to 'balance' each measure of GDP to the average.
Average quarterly statistical discrepancy(a) as a percentage of GDP
|Agriculture, forestry and fishing
|Electricity, gas and water supply
|Accommodation, cafes and restaurants
|Transport and storage
|Finance and insurance
|Property and business services
|Government administration and defence
|Health and community services
|Cultural and recreational services
|Personal and other services
|Ownership of dwellings
|Gross value added at basic prices
|Taxes less subsidies on products
(a) Calculated as the average of the absolute current price expenditure, income and production statistical discrepancies. The current price production statistical discrepancy is equal to the chain volume production statistical discrepancy multiplied by the expenditure IPD.
29.24 For years in which the national accounts are benchmarked to supply and use tables, there are no annual statistical discrepancies in the estimates of GDP as these are eliminated in the balancing process. However, for these years there are still quarterly statistical discrepancies as the sources and methods used to interpolate the annual benchmarks are generally independent.
29.25 Statistical discrepancies are also shown in the capital and financial accounts. The former is equal to the difference between the GDP expenditure and income statistical discrepancies. In the financial accounts, a 'net errors and omissions' item is included to reconcile the conceptually identical but in practice divergent estimates of net lending/borrowing and net change in financial position.
29.26 Small or zero statistical discrepancies do not necessarily mean that the aggregates to which they relate are of higher quality than those subject to larger statistical discrepancies. It is possible that there may be offsetting errors, or, in the case of the GDP statistical discrepancies, similar magnitudes of overstatement/understatement in each measure of GDP. It is for these reasons that statistical discrepancies, while they are useful indicators of quality, need to be interpreted with caution.
Past 10 years
Past 5 years
29.27 Economic analysts and policy makers not only require accurate and timely information on the movements in and magnitude of the principal national accounts aggregates, but they must also have confidence that these indicators are unlikely to change significantly as more complete data become available.
29.28 The extent to which statistics are subject to revision is one of the more easily quantifiable aspects of quality. However, these measures do not provide an unambiguous guide to quality. A series may be subject to few revisions, but the series may be highly inaccurate due to poor data sources. Revisions can be reduced by delaying the release of statistics until all or most 'final' data sources are available, but this would mean that the statistics would be less relevant to users. On the other hand, it may be possible to compile timely statistics that are not subject to revision only by placing an unacceptable load on survey respondents or at great cost to the compiler.
29.29 An analysis of revisions can, however, identify the possibility of inaccurate initial data or inefficient compilation methods. If it can be established that revisions are significantly biased (i.e. consistently positive or negative) then it is self-evident that initial estimates are inaccurate. The information on revisions can then be used to improve compilation methods to remove systematic distortions arising from the estimation process.
29.30 However, even if there are no systematic distortions in compilation processes, users may still consider certain statistics to be unreliable because the revisions are significantly dispersed (i.e. the mean absolute values of the revisions are large). Generally, it is only possible to deal with such problems by improving the quality of source data by, for example, increasing initial survey response rates.
29.31 Revisions are a natural consequence of the processes used to compile the national accounts. Initial quarterly estimates are based on survey responses received and processed before a particular cut-off time. Following the cut-off, imputations are made for the non-respondents based on the responses of similar businesses and the responses of the non-respondents in the previous quarter. Subsequently, when the non-respondents finally respond the imputations are replaced and revisions to the estimates result. For many aggregates, quarterly estimates are compiled by applying indicators to annual (or less frequent) benchmarks based on superior data sources. This benchmarking process typically leads to revisions over an extended period of time. Often the first benchmark data to become available are preliminary estimates and are therefore themselves subject to revisions. For the most part, benchmarks are considered 'final' three years after the period to which they relate has passed.
29.32 Another source of revisions is the availability of a major new data source or the development of an improved estimation methodology. Sometimes, the resultant revisions may even effect estimates for periods prior to those for which the benchmark estimates would have otherwise been considered 'final'.
29.33 Seasonally adjusted and trend estimates will usually experience some degree of revision over several years, due to the prolonged period required to finalise the estimation of seasonal adjustment factors.
29.34 The ABS has a comprehensive revisions policy for its national accounts statistics. A copy of this policy is available on request.
29.35 The table below shows the extent to which revisions have affected estimates of the percentage change in quarterly seasonally adjusted GDP (chain volume measure) for recent years. Two measures of revisability are shown, the mean absolute revision and the mean revision. The former is a measure of dispersion and the latter is a measure of bias.
Revisions to quarterly GDP, percentage change(a)
(a) Seasonally adjusted chain volume measure
29.36 For a comprehensive analysis of national accounts revisions, see the article "Revisions to Quarterly Economic Growth Rates 1984 to 1993", published in the July 1998 issue of Australian Economic Indicators (Cat. no. 1350.0).
Difference between first estimate and estimate one year later
Difference between first estimate and estimate published in MQ 2000 issue of 5206.0
Mean absolute revision
Mean absolute revision
29.37 Timeliness refers to the lag between the end of a reference period and the publication of statistics for that period. There is an important trade-off between accuracy and detail on the one hand and timeliness of the release of statistics on the other. The source data that are used to compile the national accounts are available with varying degrees of timeliness, frequency, accuracy and detail. The data sources providing more detailed and accurate (from a national accounts perspective) data tend to be those that are less frequent and/or less timely. Because of this, within a given level of resources, improvements in timeliness can generally only be made at the expense of accuracy and detail. Different users react to this trade-off in different ways. A user interested in undertaking in-depth, long-term, analysis may prefer detailed statistics that are only availably annually. On the other hand, a user interested in understanding current economic conditions may prefer very timely and frequent estimates that are less detailed.
29.38 The trade-off between timeliness, and accuracy and detail, is accommodated by way of the sequence of releases of national accounts publications. The first published quarterly estimates are usually made available about 65 days after the reference period, although this can vary due to transient issues associated with the timeliness of source data. Historically, the detailed annual estimates were released about 350 days after the reference period, but commencing with the 1998-99 issue the timeliness of the annual national accounts publication (Cat. no. 5204.0) has been improved substantially to about 150 days. The most detailed national accounts estimates are those contained in the input-output tables, which typically become available about 3.5 years after the reference period.
29.39 The graphs below show the timeliness of the release of the two main national accounts publications: the quarterly publication 5206.0 and the annual publication 5204.0. The recent deterioration in the timeliness of 5206.0 is temporary and is primarily associated with lags in the receipt of government finance data due to the introduction of accrual accounting by the Commonwealth Government. The early releases of June quarter issues of 5206.0 evident in the early part of the 1990's were associated with August Federal Budgets and the need to release national accounts in conjunction with the Budget estimates. The estimates contained in these releases were of a somewhat poorer quality than those normally provided in the quarterly publication. The aforementioned improvement in the timeliness of the 1998-99 issue of 5204.0 is evident from the timeliness graph for this publication.
29.40 An important aspect of the quality of national accounts statistics is that the concepts, definitions and classifications should be relevant to, and understandable, by users.
29.41 To this end, Australia's national accounts are produced within an integrated statistical framework based on the international statistical standard, SNA93 for national accounts. SNA93 reflects over 50 years of developments in national accounting and it is based on contemporary economic theory. It was developed on the basis of the needs of key users involved in macroeconomic analysis and policy formulation. To ensure that Australia's national accounts are relevant to domestic users, adaptations are made to the SNA framework to accommodate domestic perspectives. However, these are generally done in such a way that Australia's national accounts remain comparable to those of other countries.
29.42 Furthermore, the SNA93 is subject to continuous review to ensure that it is able to accommodate contemporary economic and financial developments. These reviews may occasionally lead to changes in the framework.
29.43 Publications such as this one assist users to understand the conceptual basis underlying Australia's national accounts. In addition, the ABS provides a range of conceptual and methodological information in all of its national accounts publications. Particular focus is placed on explaining contemporary issues affecting the national accounts. Recent articles on Y2K computer expenditure, the Sydney Olympics and the GST provided in 5206.0 are examples of this. Significant changes to the statistics are generally preceded by consultation with users and the publication of Information Papers describing the changes.
29.44 It is important that statistics are accessible to those who need to use them. A highly accurate, relevant and timely set of statistics is useless to someone who cannot obtain them.
29.45 In order to make its national accounts statistics accessible, the ABS releases them in a number of formats. The most well-known is the traditional paper publication, which are available by mail and 'over the counter' in each capital city. Copies of the key national accounts publications are provided to public libraries around Australia for access by the general public. The paper publications are supplemented by a variety of electronic releases, including Ausstats and tailored time-series releases. The main features section of each national accounts publication is available on the ABS's Website. The ABS also makes its national accounts statistics available to secondary providers for inclusion in their products. National accounts statistics are also provided to the media, who in turn provide extensive coverage of the statistics. Users requiring more detail than that provided in the standard national accounts releases can seek information on request and, subject to cost-recovery charges, the information will be provided if it is available.
29.46 An important aspect of accessibility is knowing when statistics are released and because of this release dates of national accounts publications are announced well in advance, both in national accounts publications and in general ABS release advices.
29.47 Another notable hallmark of accessibility is that statistics are made available to all users simultaneously. The ABS places great importance in ensuring that this occurs, by placing an embargo on the release of statistics until 11:30am (Canberra time) on the designated day of release. There are strict security measures to ensure that there is no unauthorised access of statistics prior to their release.
Comparability with other statistics
29.48 Users interested in understanding economic developments rarely rely solely on the national accounts. They supplement the information contained within the accounts with information from other ABS sources and from non-ABS sources. Therefore, it is important that the national accounts be as comparable with these other statistics as possible.
29.49 For the most part, SNA93 (in conjunction with related international standards such as the IMF's Balance of Payments Manual) is used as the over-arching framework for all ABS economic statistics, not just the national accounts. The use of this common framework obviously helps to achieve comparability between the national accounts and other ABS economic statistics. In addition, the ABS has a range of standard economic classifications including those for institutional sector, industry and commodity (product). These classifications are used in the national accounts and in other ABS statistics and, occasionally, in non-ABS statistics (for information on the use of these classifications in the national accounts see Appendix 1 of this publication).
29.50 Another way in which the comparability of the national accounts with other ABS statistics is achieved is through the use of a common statistical units model. Statistics (particularly those pertaining to industry) from two sources are more comparable when both sources use the same statistical unit. However, for a number of reasons it is not always possible to use the same statistical unit in all collections and thus the ABS's units model provides for a number of different, but relatable, types of statistical units (issues associated with statistical units are discussed in Chapters 5 and 6 of this publication). Statistical comparability is also enhanced when the populations for statistical collections are obtained from a common source; the ABS's Business Register is such a source for the ABS's economic collections.
29.51 From the foregoing, it is clear that the ABS's national accounts are likely to be more compatible with other ABS statistics than with non-ABS statistics. Accordingly, users should exercise care when comparing the national accounts with non-ABS statistics and be aware of any conceptual or methodological differences that may impact upon comparability.