|Page tools: Print Page Print All RSS|
15.15. Compilation of balance of payments and international investment statistics is a complex task. Given the variety of data sources and methods used, there is no single comprehensive measure of the quality of these estimates. Nevertheless, each of the quality indicators described below provides a partial insight into the quality of the statistics. To get an overall picture, all measures need to be viewed together while taking account of their limitations. At best, such an assessment can only be subjective.
Measures to monitor quality
Relative standard errors
15.16. Sampling errors, generated from the use of sampling techniques to derive population estimates for a range of items, can be expressed as relative standard errors, which provide an indication of accuracy for particular items. These relative standard errors are kept under regular review and are considered to be acceptable. When sampling errors are considered too high for aggregates or sub-aggregates, the next opportunity will be taken to adjust sample selection (selecting more units in categories showing higher variability) to reduce future sampling error. The ABS publishes these sampling errors from time to time to make users aware of them and to help users to assess data quality.
Net errors and omissions
15.17. An examination of the size and direction of the net errors and omissions item may shed some light on the accuracy of estimates. As noted earlier, the adoption of the double entry accounting system of recording means that, in principle, the net sum of all credit and debit entries should equal zero. In practice such equality rarely exists, and any differences are recorded in the net errors and omissions item. The item reflects the net effect of differences in coverage, timing and valuation, as well as errors and omissions which occur in compiling all the individual component series. Therefore both users and statisticians can focus on the item as an immediate and systematic indicator of the quality of the balance of payments statistics.
Examination of the statistical processes
15.21. The compilation processes involved in business and household surveys and in the final balance of payments and international investment compilation were described in Chapter 5. Obviously the processes are complex, and poor procedures at any one step in the process may lead to an inaccurate result. Therefore, management of these processes involves ensuring that, at each phase, objectives are set, monitored and evaluated. Careful attention is paid to:
15.22. Where data models are used they are examined to ensure that data sources and estimation procedures continue to be appropriate and accurate. Management of these processes gives the statistician a good feel for the quality of the data so that, where obvious errors occur, these are corrected or procedures established to minimise them. Apart from this, reviews are undertaken regularly of collections and procedures to ensure that they are consistent with their objectives. Where there is an assessment that data quality problems cannot be fixed (e.g. concerns about the ability to measure adequately household investment abroad and purchases of real estate by non-resident individuals), these are drawn to the attention of users as appropriate.
Examination of series over time
15.23. Series are analysed to check whether their behaviour over time provides a good explanation of economic activity, or whether they behave in erratic and inexplicable ways. Often, related series are examined to determine whether a ratio formed by two or more series behaves in an appropriate way. Examples of such ratios are: freight rates (comparing freight earned to imports f.o.b.), and investment yields (comparing investment income to the level of international investment, for various types of instruments of investment).
15.24. Data confrontation studies, in which the same or similar data items provided in different collections are compared and reconciled, if possible on an enterprise by enterprise basis, but also for groups of data providers, have proved useful in improving quality. Data confrontation at this detailed level has been very useful, for example, in identifying deficiencies and improving data quality in international investment position and financial accounts statistics. It is also useful in resolving problems with the definitions of units reporting in the different surveys, or where classifications are employed differently. That is, it also helps in identifying errors from statistical compilation as well as reporting differences by providers.
Partner country comparisons
15.25. Another form of data confrontation is to examine Australia’s balance of payments and international investment statistics by partner country with those countries’ corresponding data for Australia. There are many conceptual and practical difficulties in undertaking such comparisons. However, bilateral studies are regularly undertaken comparing Australian international trade statistics with particular countries against those countries’ corresponding data. Such work undertaken in bilateral comparisons of data by region with our trading partners, both within the framework of the Asia Pacific Economic Co-operation (APEC) initiative (see paragraph 18.23) and more generally, has reaffirmed the quality of Australia’s merchandise trade aggregates. For example, several joint studies in bilateral reconciliation between the ABS and the US Bureau of the Census, covering merchandise trade flows (on an international merchandise trade basis) between Australia and the United States of America, demonstrated that a significant part of the asymmetry in the bilateral results came from conceptual factors underlying the compilation of the statistics. Actual data errors proved to be insignificant. The residual (unexplained) discrepancies represented 4.8 per cent and 6.3 per cent, respectively, of Australia’s published merchandise import and export trade with the USA for 1994. Bilateral merchandise trade reconciliations have also been carried out with Japan, New Zealand and the European Union.
Assessment of revisability
15.26. From an analytical perspective, users want to know how much reliance they can place on an estimate. If it is likely to be revised, how much is that revision likely to be and in which direction? Are decisions made on the basis of initial estimates likely to prove ill-founded in the light of later revisions? To help answer these questions, an analysis of the revisability of the various series, measured by the extent to which the estimate published initially is revised as it is republished, may be undertaken in a variety of ways. The two measures on which most focus is placed, namely bias and dispersion, are summarised in box 15.2.
15.27. Users may wish to take a long term view of quality, analysing bias in and magnitude of revision to first-published estimates compared to ‘final’ estimates after a reasonable time. This view enables users to assess the impact of the various short-term quality issues arising from the process of regular revision, and gives them a perspective on the shifts in estimation levels resulting from changes in concepts and methods. Analyses of the direction of revisions from initial to final estimates, and of changes in period-on-period movements based on initial and final estimates, provide a similar long-term perspective on quality.
15.28. A shorter-term perspective on quality may be adopted by other users. These users have a need to factor into their analyses of statistics for the most recent period the expectation of revision over the next year or so as quarterly data sources replace extrapolations, as annual results replace partial coverage quarterly estimates, and as revised annual results replace preliminary annual results. These users have a focus on the ‘sharp end’ of the series; revisions to periods more than twelve months ago are less significant. A longer-term shift in the level of a series (arising from a methodological change) that does not alter current trends is recognised as a quality improvement that does not impact on current analyses. To support this shorter term perspective on quality, it is useful to analyse:
15.29. Several general points need to be kept in mind when considering the revisability of these statistics:
Subjective assessment of accuracy
15.30. Analyses of the statistical processes used in the ABS, observation of the types of error occurring, examination of residuals and of consistency in the behaviour of series, and comparisons with partner country data, together with the revisions history of the series, have led to the subjective assessments of quality shown in table 15.3. The assessments relate to the first published monthly goods and services estimates, the quarterly and annual estimates for the principal balance of payments aggregates, and the first published quarterly and annual estimates for the principal international investment position aggregates. To give an idea of the relative importance of each item, 1996-97 values are also shown. The ratings given are current assessments of the quality of the estimates in terms of (i) the possible discrepancy between the estimated value and the true value, and (ii) the upper bounds in which revisions may occur from time to time.
15.31. The table shows that initial annual estimates are generally assessed to be more accurate than initial quarterly estimates. Generally, as more complete and more thoroughly validated data become available, revisions are made to estimates and their accuracy improves. The overall accuracy of initial estimates for the current account is judged to be better than that of initial estimates for the financial account.