5331.0 - Balance of Payments and International Investment Position, Australia, Concepts, Sources and Methods, 1998  
ARCHIVED ISSUE Released at 11:30 AM (CANBERRA TIME) 22/09/1998   
   Page tools: Print Print Page Print all pages in this productPrint All  
Contents >> Chapter 15. Data quality >> Assessment of quality

15.15. Compilation of balance of payments and international investment statistics is a complex task. Given the variety of data sources and methods used, there is no single comprehensive measure of the quality of these estimates. Nevertheless, each of the quality indicators described below provides a partial insight into the quality of the statistics. To get an overall picture, all measures need to be viewed together while taking account of their limitations. At best, such an assessment can only be subjective.

Measures to monitor quality

Relative standard errors

15.16. Sampling errors, generated from the use of sampling techniques to derive population estimates for a range of items, can be expressed as relative standard errors, which provide an indication of accuracy for particular items. These relative standard errors are kept under regular review and are considered to be acceptable. When sampling errors are considered too high for aggregates or sub-aggregates, the next opportunity will be taken to adjust sample selection (selecting more units in categories showing higher variability) to reduce future sampling error. The ABS publishes these sampling errors from time to time to make users aware of them and to help users to assess data quality.

Net errors and omissions

15.17. An examination of the size and direction of the net errors and omissions item may shed some light on the accuracy of estimates. As noted earlier, the adoption of the double entry accounting system of recording means that, in principle, the net sum of all credit and debit entries should equal zero. In practice such equality rarely exists, and any differences are recorded in the net errors and omissions item. The item reflects the net effect of differences in coverage, timing and valuation, as well as errors and omissions which occur in compiling all the individual component series. Therefore both users and statisticians can focus on the item as an immediate and systematic indicator of the quality of the balance of payments statistics.

15.18. Persistently large figures in one direction (negative or positive) may be taken as an indication of serious and systematic errors. However, a small figure does not necessarily mean that only small errors and omissions have occurred, since large positive and negative errors may be offsetting. Offsetting errors may be either related or unrelated, resulting from a measurement problem affecting either both sides or only one side of a transaction. If positive and negative net errors and omissions tend to offset each other in successive periods, errors may be due to timing differences in data reported by the different sources used to estimate the credit and debit sides of a transaction.

15.19 The fourth edition of the IMF’s Balance of Payments Manual suggested, as an empirical rule of thumb, that a net errors and omissions component in excess of five per cent of the gross sum of merchandise exports and imports was cause for concern. While the relevance of such a measure has diminished since the fourth edition was published in the late 1970s, when financial flows were generally more constrained and services and income flows were of less significance, it still gives some indication of accuracy, especially when compared to broader balance of payments aggregates. Australia’s experience with the relative size of the net errors and omissions item, expressed as a proportion of gross goods credits and debits, and gross current account credits and debits, is given in table 15.1.

15.20. Over the 14-year period from 1945-46 to 1958-59 net errors and omissions, expressed as an annual average percentage, represented 3.0 per cent and 2.4 per cent of gross goods and current account transactions, respectively. The annual average ratio to gross goods remained steady through to the 1980s while the ratio to the gross current account transactions improved slightly to 2.1 per cent. During the 1990s to date, the ratio to gross goods has been reduced by 70 per cent to 0.9 per cent, while the ratio to gross current account transactions has shown significant (60 per cent) improvement.


15.1
. NET ERRORS AND OMISSIONS ITEM AS A PROPORTION OF GROSS GOODS AND GROSS CURRENT ACCOUNT TRANSACTIONS



Credits plus debits

Net errors and omissions item as a proportion of
Net errors and
Current
Current
Year
omissions item
Goods
account
Goods
account
$m
$m
$m
%
%

Annual average (a)
1945-46 to 1958-59
77
2,548
3,268
3.0
2.4
1959-60 to 1979-80
362
11,366
16,266
3.1
2.2
1980-81 to 1989-90
2,018
64,902
97,543
3.1
2.1
1990-91 to 1996-97
1,142
130,804
204,919
0.9
0.6
1990-91
35
102,249
163,359
0.0
0.0
1991-92
-90
106,896
167,828
-0.1
-0.1
1992-93
482
120,538
187,189
0.4
0.3
1993-94
1,786
129,282
200,075
1.4
0.9
1994-95
257
142,418
223,845
0.2
0.1
1995-96
-90
153,875
240,580
-0.1
0.0
1996-97
-1,238
160,372
251,559
-0.8
-0.5

(a)
The annual averages are derived without regard to sign.

Source: Balance of Payments and International Investment Position, Australia, March Quarter 1998 (Cat. No. 5302.0) except for the 1945-46 to 1958-59 averages, which are taken from the 1995-96 annual publication.

Examination of the statistical processes

15.21. The compilation processes involved in business and household surveys and in the final balance of payments and international investment compilation were described in Chapter 5. Obviously the processes are complex, and poor procedures at any one step in the process may lead to an inaccurate result. Therefore, management of these processes involves ensuring that, at each phase, objectives are set, monitored and evaluated. Careful attention is paid to:

      • identifying the population of businesses and households undertaking certain types of international activity;
      • ensuring that sample frames and/or collection frames and estimation procedures are up-to-date;
      • design and testing of questionnaires to ensure that they are understood and can be completed by respondents;
      • obtaining high response rates (95 per cent or more), with complete response from large enterprises;
      • implementing and monitoring standards for data verification and analysis, and publication procedures; and
      • ensuring that overall management procedures and computer systems are efficient and effective.

15.22. Where data models are used they are examined to ensure that data sources and estimation procedures continue to be appropriate and accurate. Management of these processes gives the statistician a good feel for the quality of the data so that, where obvious errors occur, these are corrected or procedures established to minimise them. Apart from this, reviews are undertaken regularly of collections and procedures to ensure that they are consistent with their objectives. Where there is an assessment that data quality problems cannot be fixed (e.g. concerns about the ability to measure adequately household investment abroad and purchases of real estate by non-resident individuals), these are drawn to the attention of users as appropriate.

Examination of series over time

15.23. Series are analysed to check whether their behaviour over time provides a good explanation of economic activity, or whether they behave in erratic and inexplicable ways. Often, related series are examined to determine whether a ratio formed by two or more series behaves in an appropriate way. Examples of such ratios are: freight rates (comparing freight earned to imports f.o.b.), and investment yields (comparing investment income to the level of international investment, for various types of instruments of investment).

Data confrontation

15.24. Data confrontation studies, in which the same or similar data items provided in different collections are compared and reconciled, if possible on an enterprise by enterprise basis, but also for groups of data providers, have proved useful in improving quality. Data confrontation at this detailed level has been very useful, for example, in identifying deficiencies and improving data quality in international investment position and financial accounts statistics. It is also useful in resolving problems with the definitions of units reporting in the different surveys, or where classifications are employed differently. That is, it also helps in identifying errors from statistical compilation as well as reporting differences by providers.

Partner country comparisons

15.25. Another form of data confrontation is to examine Australia’s balance of payments and international investment statistics by partner country with those countries’ corresponding data for Australia. There are many conceptual and practical difficulties in undertaking such comparisons. However, bilateral studies are regularly undertaken comparing Australian international trade statistics with particular countries against those countries’ corresponding data. Such work undertaken in bilateral comparisons of data by region with our trading partners, both within the framework of the Asia Pacific Economic Co-operation (APEC) initiative (see paragraph 18.23) and more generally, has reaffirmed the quality of Australia’s merchandise trade aggregates. For example, several joint studies in bilateral reconciliation between the ABS and the US Bureau of the Census, covering merchandise trade flows (on an international merchandise trade basis) between Australia and the United States of America, demonstrated that a significant part of the asymmetry in the bilateral results came from conceptual factors underlying the compilation of the statistics. Actual data errors proved to be insignificant. The residual (unexplained) discrepancies represented 4.8 per cent and 6.3 per cent, respectively, of Australia’s published merchandise import and export trade with the USA for 1994. Bilateral merchandise trade reconciliations have also been carried out with Japan, New Zealand and the European Union.

Assessment of revisability

15.26. From an analytical perspective, users want to know how much reliance they can place on an estimate. If it is likely to be revised, how much is that revision likely to be and in which direction? Are decisions made on the basis of initial estimates likely to prove ill-founded in the light of later revisions? To help answer these questions, an analysis of the revisability of the various series, measured by the extent to which the estimate published initially is revised as it is republished, may be undertaken in a variety of ways. The two measures on which most focus is placed, namely bias and dispersion, are summarised in box 15.2.


15.2. BIAS AND DISPERSION
Bias is a measure of the extent to which the initial estimate is generally lower or higher than the latest estimate, and indicates the direction of revisions. Dispersion is a measure of the spread of latest estimates about the initial estimates, and indicates the magnitude of revisions. The ABS Information Paper: Quality of Australian Balance of Payments Statistics, 1996 (Cat. no. 5342.0), provides a thorough review of the revisability of these statistics for the period from 1986 to 1994. It concluded that:
  • initial estimates of most items were negatively biased, i.e. the initial estimates were understated, and that generally positive revisions would be required in later estimates;

  • the size of revisions required was highest for income debits and credits, making them the least reliable of all the current account components for all frequencies, which reflected the errors contained in the reinvested earnings (income credits) estimates; and

  • initial estimates of the current account deficit tended to be relatively closely dispersed around the final estimate, while initial estimates of financial assets carried a strong negative bias (implying an understatement) and the behaviour of revisions was erratic. This means that the initial estimates of the net errors and omissions item, and subsequent revisions to it, largely reflect deficiencies in measuring net financial account transactions.


  • 15.27. Users may wish to take a long term view of quality, analysing bias in and magnitude of revision to first-published estimates compared to ‘final’ estimates after a reasonable time. This view enables users to assess the impact of the various short-term quality issues arising from the process of regular revision, and gives them a perspective on the shifts in estimation levels resulting from changes in concepts and methods. Analyses of the direction of revisions from initial to final estimates, and of changes in period-on-period movements based on initial and final estimates, provide a similar long-term perspective on quality.

    15.28. A shorter-term perspective on quality may be adopted by other users. These users have a need to factor into their analyses of statistics for the most recent period the expectation of revision over the next year or so as quarterly data sources replace extrapolations, as annual results replace partial coverage quarterly estimates, and as revised annual results replace preliminary annual results. These users have a focus on the ‘sharp end’ of the series; revisions to periods more than twelve months ago are less significant. A longer-term shift in the level of a series (arising from a methodological change) that does not alter current trends is recognised as a quality improvement that does not impact on current analyses. To support this shorter term perspective on quality, it is useful to analyse:

        • the direction of revisions within the year immediately following initial release of an estimate (will the estimate go up or down as it is revised?);
        • the magnitude of those revisions (are the initial estimates in the right ball park?); and
        • whether those revisions generally move the series in the right direction (will the early revisions provide data of improved quality on which to make decisions?).

    15.29. Several general points need to be kept in mind when considering the revisability of these statistics:
        • revisability analysis inevitably reflects past experience and may not be a good indicator of the likely behaviour of the statistics in the immediate future;
        • the findings for aggregate or net series should be treated with caution as they will reflect the varying impacts of revisions on their component series;
        • substantial change in the volume and size of transactions for an aggregate occurring over a relatively short time will make the effect on the revisability of the series difficult to predict;
        • the latest estimate is assumed to be a better approximation of the notional true value than earlier estimates; and
        • significant changes in a concept or methodology need to be considered carefully in quality assessment. For example, a better approximation of a concept will improve accuracy, but at the cost of revising series and impairing revisability measures unless these impacts can be isolated.

    Subjective assessment of accuracy

    15.30. Analyses of the statistical processes used in the ABS, observation of the types of error occurring, examination of residuals and of consistency in the behaviour of series, and comparisons with partner country data, together with the revisions history of the series, have led to the subjective assessments of quality shown in table 15.3. The assessments relate to the first published monthly goods and services estimates, the quarterly and annual estimates for the principal balance of payments aggregates, and the first published quarterly and annual estimates for the principal international investment position aggregates. To give an idea of the relative importance of each item, 1996-97 values are also shown. The ratings given are current assessments of the quality of the estimates in terms of (i) the possible discrepancy between the estimated value and the true value, and (ii) the upper bounds in which revisions may occur from time to time.


    15.3
    . SUBJECTIVE ACCURACY RATINGS OF PRINCIPAL BALANCE OF PAYMENTS AND INTERNATIONAL INVESTMENT POSITION AGGREGATES (a)
    Accuracy of 1995-96 estimates
    Aggregate
    Value in 1996-97
    ($m)
    Initial monthly estimates in
    Cat. no. 5368.0
    Initial quarterly estimates in
    Cat. no. 5302.0
    Initial annual estimates in
    Cat. no. 5363.0

    CURRENT ACCOUNT
    Goods
    credits
    80,934
    A
    A
    A
    debits
    -79,438
    A
    A
    A
    Services
    credits
    24,384
    B
    B
    A
    debits
    -24,103
    B
    B
    A
    Income
    credits
    8,319
    . .
    C
    B
    debits
    -27,753
    . .
    B
    A
    Current transfers
    credits
    3,377
    . .
    C
    C
    debits
    -3,251
    . .
    C
    C
    Balance on current account
    -17,531
    . .
    B
    B

    CAPITAL ACCOUNT

    1,318

    . .

    C

    B

    FINANCIAL ACCOUNT
    Direct investment
    abroad
    -5,908
    . .
    C
    B
    in Australia
    11,282
    . .
    C
    B
    Portfolio investment
    assets
    -3,548
    . .
    C
    C
    liabilities
    16,927
    . .
    D
    C
    Other investment
    assets
    -3,558
    . .
    D
    C
    liabilities
    7,480
    . .
    D
    C
    Reserve assets
    -5,224
    . .
    A
    A
    Balance on financial account
    -17,451
    . .
    D
    C

    INTERNATIONAL INVESTMENT POSITION
    Australian investment abroad
    Direct investment abroad
    -67,766
    . .
    B
    B
    Portfolio investment assets
    -69,441
    . .
    C
    C
    Other investment assets
    -38,656
    . .
    C
    C
    Reserve assets
    -22,790
    . .
    A
    A
    Total foreign assets
    -198,653
    . .
    C
    B

    Foreign investment in Australia
    Direct investment in Australia
    151,115
    . .
    B
    B
    Portfolio investment liabilities
    295,816
    . .
    C
    B
    Other investment liabilities
    63,363
    . .
    C
    B

    Total foreign liabilities

    510,295

    . .

    C

    B

    Net international investment position

    311,642

    . .

    C

    B

    (a) The accuracy ratings used relate to the following approximate margins of error:
         A  less than 5 per cent;
         B 5 per cent to less than 10 per cent;
         C 10 per cent to less than 15 per cent;
         D 15 per cent and greater.

    Source: Balance of Payments and International Investment Position, Australia, March quarter 1998 (Cat. no. 5302.0).

    15.31. The table shows that initial annual estimates are generally assessed to be more accurate than initial quarterly estimates. Generally, as more complete and more thoroughly validated data become available, revisions are made to estimates and their accuracy improves. The overall accuracy of initial estimates for the current account is judged to be better than that of initial estimates for the financial account.






    Previous PageNext Page