|Page tools: Print Page Print All|
ABS COMPILATION METHODS
CHART 4.3: INTERNATIONAL MERCHANDISE TRADE PROCESSING SYSTEM - PROCESS FLOWCHART
OVERVIEW OF THE INTERNATIONAL MERCHANDISE TRADE PROCESSING SYSTEM
4.12 A high level depiction of the compilation of international merchandise trade statistics is shown in Chart 4.3. The chart shows the movement and processing of exports and imports information from the DIBP system to the release of ABS statistical outputs. In most cases the data moves in one direction but the two-way movement of data between the ICS and the ABS environment reflects the normal transfer of the DIBP information to the ABS and the occasional transfer of information from the ABS to the DIBP, e.g. delivery of new export and import statistical codes. The movement of data in both directions between the data load, IDW, edit/amendment system and aggregate and output editing reflects the interaction, amendment and re-checking of the data between these parts of the International Merchandise Trade System (IMT system).
4.13 The IMT system uses standard ABS statistical infrastructure and tools with adaptations to cover the type of data (export and import declarations, not survey questions and answers) and the volume of data received on a daily basis.
4.14 The IMT system has a separate test environment for testing changes or new functionality prior to implementation in the production environment. The test environment is also used as a training environment.
4.15 In reality the system used to receive, process and disseminate the statistics is much more complex than shown in Chart 4.3, which is only a representation of the key processes. The following sections describe the key processes and stages culminating in the release of monthly international merchandise trade statistics on the ABS website.
START THE PROCESSING MONTH
4.16 The timing of the processing cycle differs between exports and imports due to exports being compiled on a departure date basis and additional time allowed for the finalisation of export manifests. Consequently, for exports the commencement of the new processing month normally occurs around the middle of the reference month. For imports the commencement of the new processing month happens earlier, generally on the sixth working day.
4.17 Threshold edit values are set to manage processing workloads. These are used to ensure that all records with values above the thresholds and records with fatal or warning error messages are checked by ABS editors.
4.18 Before processing commences it is necessary to ensure that metadata changes taking effect from the current reference month have been implemented correctly. For example, if a new 10 digit commodity code is active in the ICS from 1 January 2014 this new code must be loaded to the IDW metadata before processing of January 2014 data commences.
INPUT DATA WAREHOUSE (IDW)
4.19 The IDW is a key database in the processing of international merchandise trade data. It is the repository for all export/import data and metadata, including:
EXPORTS AND IMPORTS DATA LOAD (TO THE IDW)
4.20 The data load processes initially reformat and map some customs data items to ABS data items. The most important re-mapping is for any fields which use the UN/Locode (i.e. the country, port and state fields). This is because when the DIBP introduced the UN/Locode in their system (with the introduction of the ICS) the ABS decided to minimise changes to the IMT system and statistical output by continuing to use the previous country, port and state codes (see Diagram 6.1 in Country and Overseas Port Classifications and Processing).
4.21 Processing stage codes are added to records in the data load to IDW. These codes are used throughout the IMT system to identify particular record types and their status at particular processing points.
OUT OF SCOPE RECORDS
4.22 The load process identifies out of scope transactions which are not edited but are still loaded to the IDW for analytical purposes. The following record types are out of scope of international merchandise trade statistics.
4.23 Replacement or amendment records which alter an out of scope record to an in-scope record are edited and included in international merchandise trade statistics. Chart 4.4 illustrates the scope of international merchandise trade statistics, that is, international merchandise trade statistics equals customs declarations and some bunkers, and excludes international non-merchandise trade and out of scope records.
CHART 4.4: SCOPE OF INTERNATIONAL MERCHANDISE TRADE
EXPORTS AND IMPORTS EDIT
4.24 Once the export and import data are loaded to the IDW, all in-scope records are passed through their respective edit process (exports or imports edit). The edit process checks the details of each transaction against a set of edit conditions which are categorised as fatal, warning (notifiable) or informative. Transactions which fail fatal or warning edits then have messages sent to the edit/amendment system for investigation by an ABS editor. The investigation may require contact with the exporter, importer or agent to resolve. Informative edits usually indicate an automated system adjustment has been applied. Transactions which fail informative edits are not viewed by an ABS editor unless they also contain a fatal or warning edit.
4.25 The following list describes the most common edit checks applied to both export and import data.
4.26 To ensure that ABS editors focus on the most significant transactions, and to minimise contact with exporters (or their agents), significance editing is used in conjunction with many of these edit checks. Significance editing gives a score to all transactions to indicate how the reported value (exports FOB value, imports customs value) differs from a predicted value. The size of the score is an indication of the potential error. Transactions with scores beyond the significance score thresholds are flagged for review by an ABS editor. The score is allocated based on data calculated by the input editing benchmarks. The input editing benchmarks use the previous 12 months of data to determine if quantity or gross weight is the best predictor of value.
4.27 The re-edit process is essentially the same as the edit process but occurs following editor amendment whereas the edit process occurs when a new transaction (or revised version of a transaction) is received from the DIBP.
4.28 Chart 4.5 shows the imports data load where the IDN information from the DIBP is re-formatted and loaded to the IDW. Earlier versions of an IDN which are already on the IDW are identified (to ensure that the new version is not loaded as a new transaction) and up-dated in the Load to the IDW process. In Chart 4.5 this is shown as data moving from the IDW back to the Load to IDW stage and then through to the IDW again. The metadata which is used in the Reformat and Imports edit processes includes the edit checks and country/port metadata. After the edit check a new version of the transaction with any relevant alerts is created in the IDW (shown as a two way arrow between the IDW and Imports edit stages).
CHART 4.5: IMPORTS DATA LOAD - PROCESS FLOWCHART
4.29 There are many similarities in the exports and imports data load processes. However, they are separate processes involving different data items and metadata and there are some aspects which are unique to the exports load. The differences in the processes can be seen by comparing Charts 4.3 and 4.4. The unique export processes are described in the following paragraphs.
EXPORTS MATCHING PROCESS
4.30 Exporters must complete an export declaration with the DIBP and obtain an Export Declaration Number (EDN) in advance of the goods arriving at the wharf, or airport, for export. The EDN is an intention to export and includes preliminary information about when the goods will leave Australia and on what carrier the goods will be transported. The actual departure details are listed on the ship or aircraft manifest. When the ABS receives both the EDN and its manifest, the transaction is said to be acquitted 'matched' and the goods exported.
4.31 The exports matching process matches EDNs and manifests and adds the departure details from the manifest to the EDN information. In most cases the EDN is received by the ABS before the manifest. The EDN will be loaded to the IDW and identified as an EDN awaiting acquittal. Once the corresponding manifest is received the EDN is acquitted and, if the transaction has not been reported in Australian dollars, the RBA exchange rate for the day of departure will be used to convert values to Australian dollars. If the manifest is received first it is also loaded to the IDW but identified as a manifest record. When the EDN is received, the matching process retrieves the manifest from the IDW and the EDN is acquitted. This process of retrieving EDNs/manifests from the IDW and matching them with their corresponding EDNs/manifests is depicted with a two way arrow between the exports matching process and the IDW.
4.32 A small number of records are 'self acquitting'. This means the EDN is marked as acquitted (and therefore exported) without being shown on a manifest. Self acquitting EDNs are those export declarations which do not get recorded on a manifest. These are postal records, ships and aircraft exported under their own power, ship and aircraft stores and spares and unaccompanied baggage, identified by the relevant export goods type.
4.33 Because EDNs are lodged and passed to the ABS prior to export, the exchange rate can be for a future date and is therefore unknown. When these EDNs are loaded to the IDW the FOB value is estimated using the latest available exchange rate. To update these estimated records the load process checks for new exchange rates and retrieves the estimated records from the IDW. These records are passed through the load process again to update the information. In Chart 4.6 this is shown as data moving from the IDW back to the Match and Load to IDW stages (with input of exchange rates) and then through to the IDW again.
CHART 4.6: EXPORTS DATA LOAD - PROCESS FLOWCHART
FINALISING MONTHLY EXPORTS DATA
4.34 Towards the end of the processing month, a report is produced showing all unacquitted (no manifest received) large value EDNs with expected departure dates within the processing month. If the report shows an abnormally high value of unacquitted EDNs sometimes a decision to load more daily files to the IDW is made in the hope that manifests to acquit the EDNs are received. However, there comes a point where meeting the publication deadline means that no more data is loaded. At this point the monthly exports telephone follow-up occurs.
4.35 The follow-up involves direct contact with exporters (or their agents) about their large value unacquitted exports to determine if the goods have actually been exported. Goods that have been exported are acquitted using the manifest information supplied by the exporter and the transaction is included in monthly estimates. If the goods are not exported the expected departure date is changed to the following month and the goods are not included in the monthly estimates. Because ABS resources are limited, and experience shows that most EDNs with expected departure dates in the current processing month have been exported, all records below a given value (normally around $1 million) are included in the monthly estimates without checking their actual departure.
4.36 The edit/amendment system is used to view and, if necessary, amend transactions that have been flagged by the edit process as requiring examination by an ABS editor. The system is also used to view and, if appropriate to do so, amend unusual transactions identified as requiring ABS editor attention (e.g. by output edit processes). Amendments are applied as a result of contact with the exporter or importer, identification of a repeated mis-reporting error or other information provided in the transaction or similar transactions. Any amendment of a transaction requires the ABS editor to include comments (which are appended to the transaction) to explain the change(s) made. All amended transactions go through the re-edit process to ensure data integrity and transactions failing re-edit checks are returned to the edit/amendment system for re-examination. The system also provides the means to view any transaction using its unique identifier. The system is also used by ABS editors and managers to monitor work flow, e.g. the number of transactions awaiting editor attention (because they have failed fatal or warning edits).
AGGREGATE AND OUTPUT EDITING
4.37 Aggregate editing is an important component of statistical compilation and finalisation of output for release. Aggregate editing involves creating monthly totals by particular levels of commodity (HS 6 digit or SITC 2 digit), country, state, mode of transport and unit of quantity and comparing these to previously released aggregates. Significance scoring (scoring the current month aggregates against a benchmark derived from the previous 12 months of data) is used to highlight the most significant aggregate movements and to set workloads. Where necessary further follow-up with exporters and importers (or their agents) is undertaken and transactions amended. Records of any investigations and findings are kept and reviewed by more senior staff during the finalisation of outputs.
4.38 Output editing involves examining the data for particular commodities or at the 2 digit level of the HS. Examination of data at these levels may reveal a significant change. When this occurs the transactions contributing to the change are identified and scrutinised.
4.39 The trade query tool is an integral component of aggregate and output editing. The query tool allows staff to retrieve data from the IDW using a defined set of criteria. Simple reports can be produced, data aggregated and sorted to help ABS editors to understand movements or to resolve mis-reporting errors in the data. The trade query tool is also used in editing and verifying individual transactions. Where mis-reporting errors are found the relevant transactions are amended using the edit/amendment system.
4.40 International merchandise trade statistics are based on a large number of individual transactions and are released at a very detailed aggregated level, e.g. 8 digit exports and 10 digit imports by Australian port, country and overseas port. When ensuring the quality of international merchandise trade statistics, most resources are focussed on the data required for macroeconomic statistics.
OUTPUT FOR BALANCE OF PAYMENTS, NATIONAL ACCOUNTS AND PRICES
4.41 Once editing is finalised and prior to the public release of international merchandise trade statistics, data are prepared for inclusion in the balance of payments.
4.42 The preparation of data for the balance of payments signals the first approval stage in the release of international merchandise trade statistics. The export and import data supplied for the balance of payments includes totals on a SITC and, for imports only, a Balance of Payments Broad Economic Categories (BoPBEC) basis, information about significant month to month movements and other information used to compile goods data on a balance of payments basis. For more information regarding classifications such as SITC and BoPBEC and how data is prepared for the balance of payments see the Classifications and Balance of Payments Statistics sections.
4.43 Import data are delivered to the Balance of Payments Section on the 6th working day after the reference period, and export statistics are delivered on the 13th working day after the reference period. The later delivery of export data is due to the compilation of export statistics on a departure date basis.
4.44 International merchandise trade statistics are first released with international trade in services on a balance of payments basis in International Trade in Goods and Services, Australia (cat. no. 5368.0). Goods credits and debits on a balance of payments basis are closely related to international merchandise exports and imports but have been adjusted to take account of the principles of economic ownership and residence (see paragraphs 12.4-12.7 in Balance of Payments Statistics).
4.45 Australia's balance of payments is published quarterly in Balance of Payments and International Investment Position, Australia (cat. no. 5302.0). In this publication, the quarterly estimates of goods and services credits and debits are derived from the months published in the corresponding release of International Trade in Goods and Services, Australia (cat. no. 5368.0) with any revisions kept to a minimum. The quarterly exports and imports statistics which are included in the Expenditure on Gross Domestic Product estimates in the publication Australian National Accounts: National Income, Expenditure and Product (cat. no. 5206.0) are the same as those published in the quarterly Balance of Payments and International Investment Position, Australia (cat. no. 5302.0), meaning they include both goods and services on a balance of payments basis. Detailed exports and imports data on an international merchandise trade basis are supplied to the National Accounts Branch for use in compiling the annual supply-use tables, for input-output analysis and for other analytical purposes.
4.46 On a quarterly basis, data are supplied to the Prices Section of the ABS. The data includes detailed HS commodity level data (8 digit exports and 10 digit imports) by country, value, quantity and gross weight. The data are used in the compilation of the International Trade Price Indexes, Australia (cat. no. 6457.0).
4.47 International merchandise trade statistics provided to areas of the ABS which compile the balance of payments, national accounts and prices statistics include aggregates and some transaction level data so that the compilers are able to fully understand and interpret the data for their statistics. This is reflected in Chart 4.3 where the data provided for these areas are shown as coming from the IDW (transaction level data) and from the ABSIW (aggregate level data).
OTHER SPECIAL REPORTS
4.48 Any special reports produced for users from the IDW are aggregated and confidentialised in a separate process to the confidentialisation process described in the following paragraphs. Separate aggregation and confidentialisation is necessary because the reports contain non-standard aggregations of commodities and/or time periods, or are released (with consent) after a period of time has elapsed (see paragraphs 9.47-9.52 in Special Data Releases). These reports are produced on an ad hoc basis in response to user needs and are prepared on as a 'user pays' service.
4.49 In Chart 4.3 the process of confidentialising international merchandise trade statistics is depicted as a single box which represents a database and set of utilities. The utilities enable:
4.50 Once the monthly exports and imports data are finalised and provided to the Balance of Payments Section, the output files for release to the public are prepared. It is in the creation of these output files that the confidentiality restrictions are applied to the data. Separate output files for exports, imports and import clearances are created.
4.51 The Data Confidentiality section explains the ABS confidentiality legislation and how confidentiality is applied to international merchandise trade statistics.
ABS INFORMATION WAREHOUSE (ABSIW)
4.52 The ABSIW is the repository for data available for dissemination, including international merchandise trade statistics. Also stored on the ABSIW are all the metadata which make the statistics meaningful, e.g. international merchandise trade commodity, country codes, descriptions and associated information. The ABSIW was created to enable all ABS data to be captured, defined and delivered in a consistent way.
4.53 At the end of the processing month after the data have been finalised and the output files created, international merchandise trade statistics are loaded to the ABSIW User database (an intermediate database where data are checked prior to release). The publication tables, time series spreadsheets and data cubes found on the ABS website are all extracted from the ABSIW User database. At 11.30am (Canberra time) on the day of release all storage tables involving international merchandise trade statistics and metadata for the particular collection, time series spreadsheets and data cubes are copied from the ABSIW User database to the ABSIW Output database (output environment) and ABS website.
4.54 Consultancy and subscription data are created from the ABSIW Output database from 11.30am (Canberra time) on the day of release of the publication.
These documents will be presented in a new window.