|Page tools: Print Page Print All RSS Search this Product|
Data sources and compilation methods
Table 4.1: Exports data received from the Department of Home Affairs
(b) Where amendments or changes to earlier declarations are received these are identified using the combination of EDN and Line number and status fields.
Imports and clearances information
4.8 After midnight each day the ICS produces a file of import and clearance records which were finalised by the Department of Home Affairs in the previous 24 hours. The file is transmitted to the ABS overnight. Table 4.2 below lists the import data received daily from the Department of Home Affairs. While some data items have been grouped in the table, for example, duty calculation fields, each item included in the grouping is listed in the description. Appendix 5 (in the Downloads tab) lists these data items and explains whether they are available in ABS aggregate output.
Table 4.2 Daily imports and clearances information received from the Department of Home Affairs
(b) Where amendments or changes to earlier declarations are received these are identified using the combination of Line Action Code, IDN and Line number.
4.9 The relationship between the ABS and the Department of Home Affairs is defined in a signed agreement between the two agencies. Meetings between the deputies of both agencies are held annually and regular discussions occur at the work level. Together these arrangements ensure that both agencies understand the requirements of the other agency and ensures that the relationship remains strong.
4.10 The ABS receives a range of information about the customs procedures applied to transactions. This includes the line nature type, treatment code, duty fields and GST fields. The information supplied by Department of Home Affairs can be used to ensure the correct treatment of transactions in ABS statistics.
ABS compilation methods
4.11 In 2001 when the concepts, sources and methods (CSM) for international merchandise trade statistics was first published, ABS statistics were processed on a mainframe computer. Between 2003 and 2006 the international merchandise trade system was re-designed, tested and transferred to a new platform. Over a similar period the Department of Home Affairs were re-engineering their systems for collecting exports and imports information. While these system changes were significant in terms of the way data are collected, delivered and processed there was only minimal impact on statistical output (see Appendix 2 in the Downloads tab). If users of the statistics are interested in the previous systems and methods they should read the 2001 version of the CSM.
4.12 The IMT system uses standard ABS statistical infrastructure and tools with adaptations to cover the type of data (export and import declarations, not survey questions and answers) and the volume of data received on a daily basis.
4.13 The IMT system has a separate test environment for testing changes or new functionality prior to implementation in the production environment. The test environment is also used as a training environment.
Start the processing month
4.14 The timing of the processing cycle differs between exports and imports due to exports being compiled on a departure date basis and additional time allowed for the finalisation of export manifests. Consequently, for exports the commencement of the new processing month normally occurs around the middle of the reference month. For imports the commencement of the new processing month happens earlier, generally on the sixth working day.
4.15 Threshold edit values are set to manage processing workloads. These are used to ensure that all records with values above the thresholds and records with fatal or warning error messages are checked by ABS editors.
4.16 Before processing commences it is necessary to ensure that metadata changes taking effect from the current reference month have been implemented correctly. For example, if a new 10 digit commodity code is active in the ICS from 1 January 2014 this new code must be loaded to the IDW metadata before processing of January 2014 data commences.
Input Data Warehouse (IDW)
4.17 The IDW is a key database in the processing of international merchandise trade data. It is the repository for all export/import data and metadata, including:
Exports and imports data load (to the IDW)
4.18 The data load processes initially re-format and map some customs data items to ABS data items. The most important re-mapping is for any fields which use the UN/Locode (i.e. the country, port and state fields). This is because when the Department of Home Affairs introduced the UN/Locode in their system (with the introduction of the ICS) the ABS decided to minimise changes to the IMT system and statistical output by continuing to use the previous country, port and State codes, (see Diagram 6.1 in the Country, Overseas Port, State and Australian Port chapter).
4.19 Processing stage codes are added to records in the data load to IDW. These codes are used throughout the IMT system to identify particular record types and their status at particular processing points.
Out of scope records
4.20 The load process identifies out of scope transactions which are not edited but are still loaded to the IDW for analytical purposes. The following record types are out of scope of international merchandise trade statistics.
4.21 Replacement or amendment records which alter an out of scope record to in-scope record are edited and included in international merchandise trade statistics.
Exports and imports edit
4.22 Once the export and import data are loaded to the IDW, all in-scope records are passed through their respective edit process (exports or imports edit). The edit process checks the details of each transaction against a set of edit conditions which are categorised as fatal, warning (notifiable) or informative. Transactions which fail fatal or warning edits then have messages sent to the edit/amendment system for investigation by an ABS editor. The investigation may require contact with the exporter, importer or agent to resolve. Informative edits usually indicate an automated system adjustment has been applied. Transactions which fail informative edits are not viewed by an ABS editor unless they also contain a fatal or warning edit.
4.23 The following list describes the most common edit checks applied to both export and import data:
4.24 To ensure that ABS editors focus on the most significant transactions, and to minimise contact with exporters, significance editing is used in conjunction with many of these edit checks. Significance editing gives a score to all transactions to indicate how the reported value (exports FOB value, imports customs value) differs from a predicted value. The size of the score is an indication of the potential error. Transactions with scores beyond the significance score thresholds are flagged for review by an ABS editor. The score is allocated based on data calculated by the input editing benchmarks. The input editing benchmarks use the previous 12 months of data to determine if quantity or gross weight is the best predictor of value.
4.25 The re-edit process is essentially the same as the edit process but occurs following editor amendment whereas the edit process occurs when a new transaction (or revised version of a transaction) is received from the Department of Home Affairs.
4.26 Chart 4.3 below shows the imports data load where the IDN information from the Department of Home Affairs is re-formatted and loaded to the IDW. Earlier versions of an IDN which are already on the IDW are identified (to ensure that the new version is not loaded as a new transaction) and up-dated in the Load to the IDW process. In Chart 4.3 below, this is shown as data moving from the IDW back to the 'Load to IDW' stage and then through to the IDW again. The metadata which is used in the Reformat and Imports edit processes includes the edit checks and country/port metadata. After the edit check a new version of the transaction with any relevant alerts is created in the IDW (shown as a two way arrow between the IDW and Imports edit).
Chart 4.3 Imports data load - process flowchart
4.27 There are many similarities in the exports and imports data load processes. However, they are separate processes involving different data items and metadata and there are some aspects which are unique to the exports load. The unique export processes are described in the following sections.
Exports matching process
4.28 Exporters must complete an export declaration (EDN) with the Department of Home Affairs in advance of the goods arriving at the wharf for export. The EDN is an intention to export and includes preliminary information about when the goods will leave Australia and on what carrier the goods will be transported. The actual departure details are listed on the ship or aircraft manifest. When the ABS receives both the EDN and its manifest, the transaction is said to be acquitted 'matched' and the goods exported.
4.29 The exports matching process matches EDNs and manifests and adds the departure details from the manifest to the EDN information. In most cases the EDN is received by the ABS before the manifest. The EDN will be loaded to the IDW and identified as an EDN awaiting acquittal. Once the corresponding manifest is received the EDN is acquitted and, if the transaction has not been reported in Australian dollars, the Reserve Bank of Australia exchange rate for the day of departure will be used to convert values to Australian dollars. If the manifest is received first it is also loaded to the IDW but identified as a manifest record. When the EDN is received, the matching process retrieves the manifest from the IDW and the EDN is acquitted. This process of retrieving EDNs/manifests from the IDW and matching them with their corresponding manifests/EDNs is depicted with a two way arrow between the exports matching process and the IDW.
4.30 A small number of records are 'self acquitting'. This means the EDN is marked as acquitted (and therefore exported) without being shown on a manifest. Self acquitting EDNs are those export declarations which do not get recorded on a manifest. These are postal records, ships and aircraft exported under their own power, ship and aircraft stores and spares and unaccompanied baggage, identified by the relevant export goods type.
4.31 Because EDNs are lodged and passed to the ABS in advance of export, the exchange rate can be for a future date (and is therefore unknown). When these EDNs are loaded to the IDW the FOB value is estimated using the latest available exchange rate. To update these estimated records the load process checks for new exchange rates and retrieves the estimated records from the IDW. These records are passed through the load process again to update the information. In Chart 4.4 below, this is shown as data moving from the IDW back to the Match and 'Load to IDW' stages (with input of exchange rates) and then through to the IDW again.
Chart 4.4 Exports data load - process flowchart
Finalising monthly exports data
4.32 Towards the end of the processing month, a report is produced showing all unacquitted (no manifest received) large value EDNs with expected departure dates within the processing month. If the report shows an abnormally high value of unacquitted EDNs sometimes a decision to load more daily files to the IDW is made in the hope that manifests to acquit the EDNs are received. However, there comes a point where meeting the publication deadline means that no more data is loaded.
4.33 The edit/amendment system is used to view and, if necessary, amend transactions that have been flagged by the edit process as requiring examination by an ABS editor. The system is also used to view and, if appropriate to do so, amend unusual transactions identified as requiring ABS editor attention (e.g. by output edit processes). Amendments are applied as a result of contact with the exporter or importer, identification of a repeated mis-reporting error or other information provided in the transaction or similar transactions. Any amendment of a transaction requires the ABS editor to include comments (which are appended to the transaction) to explain the change(s) made. All amended transactions go through the re-edit process to ensure data integrity and transactions failing re-edit checks are returned to the edit/amendment system for re-examination. The system also provides the means to view any transaction using its unique identifier. The system is also used by ABS editors and managers to monitor work flow e.g. the number of transactions awaiting editor attention (because they have failed fatal or warning edits).
Aggregate and output editing
4.34 Aggregate editing is an important component of statistical compilation and finalisation of output for release. Aggregate editing involves creating monthly totals by particular levels of commodity (HS 6-digit or SITC 2-digit), country, state, mode of transport and unit of quantity and comparing these to previously released aggregates. Significance scoring (scoring the current month aggregates against a benchmark derived from the previous 12 months of data) is used to highlight the most significant aggregate movements and to set workloads. Where necessary further follow-up with exporters and importers is undertaken and transactions amended. Records of any investigations and findings are kept and reviewed by more senior staff during the finalisation of output.
4.35 Output editing involves examining the data for particular commodities or 2-digit chapters of the HS. Examination of data at these levels may reveal a significant change. When this occurs the transactions contributing to the change are identified and scrutinised.
4.36 The trade query tool is an integral component of aggregate and output editing. The query tool allows staff to retrieve data from the IDW using a defined set of criteria. Simple reports can be produced, data aggregated and sorted to help ABS editors to understand movements or to resolve mis-reporting errors in the data. The trade query tool is also used in editing and verifying individual transactions. Where mis-reporting errors are found the relevant transactions are amended using the edit/amendment system.
4.37 International merchandise trade statistics are based on a large number of individual transactions and are released at a very detailed aggregated level e.g. 8-digit exports and 10-digit imports by Australian port, country and overseas port. As the ABS has limited resources available for ensuring the quality of international merchandise trade statistics, resources are focussed on the data required for macroeconomic statistics.
Output for Balance of Payments, National Accounts and Prices
4.38 Once editing is finalised and prior to the public release of international merchandise trade statistics, data are prepared for inclusion in the balance of payments.
4.39 The preparation of data for the balance of payments signals the first approval stage in the release of international merchandise trade statistics. The export and import data supplied for the balance of payments includes totals on a SITC and, for imports only, a BoPBEC basis, information about significant month to month movements and other information used to compile goods data on a balance of payments basis. More information regarding classifications such as SITC and BoPBEC and how data is prepared for the balance of payments is presented in the Classifications and Relationship with Other ABS Macroeconomic Statistics chapters respectively.
4.40 International merchandise exports statistics are first released with goods and services on a balance of payments basis in International Trade in Goods and Services, Australia (cat. no. 5368.0). Goods credits and debits on a balance of payments basis are closely related to international merchandise exports and imports but have been adjusted to take account of the principles of economic ownership and residence. More information can be found in the Relationship with Other ABS Macroeconomic Statistics chapter (paragraphs 12.4 to 12.7).
4.41 Australia's balance of payments is published quarterly in Balance of Payments and International Investment Position, Australia (cat. no. 5302.0). In this publication, the quarterly estimates of goods and services credits and debits are the sum of the months published in the corresponding release of International Trade in Goods and Services, Australia (cat. no. 5368.0) without any changes. The quarterly exports and imports statistics which are included in the Expenditure on GDP estimates in the publication Australian National Accounts: National Income, Expenditure and Product (cat. no. 5206.0) are the same as those published in the quarterly balance of payments (so they include both goods and services and are on a balance of payments basis). Detailed exports and imports data on an international merchandise trade basis are supplied to the National Accounts Branch for use in compiling the annual supply-use tables, for input-output analysis and for other analytical purposes.
4.42 On a quarterly basis, data for the Prices Branch of the ABS are also prepared. The data include detailed Harmonized System commodity level data (8 digit export and 10 digit import) by country, value, quantity and gross weight. The data are used in the compilation of the International Trade Price Indexes, Australia (cat. no. 6457.0).
4.43 International merchandise trade data provided to the areas of the ABS which compile the balance of payments, national accounts and prices statistics include aggregates and some transactions level data so that the compilers of those statistics are able to fully understand and interpret the data for their statistics.
ABS Information Warehouse (ABSIW)
4.44 The ABS Information Warehouse (ABSIW) is the repository for data available for dissemination, including international merchandise trade statistics. Also stored on the ABSIW are all the metadata which make the statistics meaningful e.g. international merchandise trade commodity and country codes and descriptions and associated information. The ABSIW was created to enable all ABS data to be captured, defined and delivered in a consistent way.
4.45 At the end of the processing month after the data have been finalised and the output files created, international merchandise trade statistics are loaded to the ABSIW User database (an intermediate database where data are checked prior to release). The publication tables, time series spreadsheets and data cubes found on the ABS website are all extracted from the ABSIW User database. At 11.30am (Canberra time) on the day of release, the publication, all storage tables involving international merchandise trade data and metadata for the particular collection, time series spreadsheets and data cubes are copied from the ABSIW User database to the ABSIW Output database (output environment) and ABS Website.
4.46 Consultancy and subscription data are created from the ABSIW Output database from 11.30am on the day of release of the publication.
Other (non-Department of Home Affairs) data sources
4.47 There are only a small number of regular non-Department of Home Affairs data sources used in the compilation of international merchandise trade statistics. These data sources are:
4.48 Export declarations are not required for fuel provided to foreign airlines and ships in Australia. The value of these transactions is estimated and included in Australia's merchandise export statistics using price and historical quantity data while alternative data sources are explored. In addition, because exports are recorded on a date of departure basis, occasionally research shows that a significant export has occurred for which no details have yet been received from the Department of Home Affairs. These transactions are added to the IDW towards the end of the processing month and processed in the same way as customs records i.e. once added the transaction(s) are passed through the edit process, subject to output editing and included in the final results.
These documents will be presented in a new window.