1.1 The Australian Bureau of Statistics (ABS) Consumer Price Index (CPI) is a robust measure of household inflation, providing trusted official statistics to the Australian community for nearly 70 years. The CPI uses internationally endorsed methods, and aims to build on this by exploiting opportunities to use big data for compiling official statistics. In the environment of delivering the best possible statistical programs in more efficient and innovative ways, the ABS has undertaken a research program investigating priorities to enhance the CPI. This publication focuses on one component of that program, that being to maximise the use of transactions data to compile the Australian CPI.
1.2 Transactions (scanner) data refers to point-of-sale purchases from retailers and contains detailed information about transactions, dates, quantities, product descriptions, and values of products sold. From March quarter 2014 the ABS significantly increased its use of transactions data to compile the Australian CPI, now accounting for approximately 25 per cent of the weight of the Australian CPI. The approach adopted was a 'direct replacement' of observed point-in-time prices with a unit value calculated from the transactions data.(footnote 1)
1.3 While the implementation of this method represented an improvement over traditional CPI practices (e.g. average unit value replaced a point-in-time price), further enhancements are possible. This includes using all the products available in the datasets (rather than a sample of products) and weighting products by their economic importance (rather than using unweighted price indexes). Ideally, National Statistical Offices (NSOs) can achieve both these improvements using automated processes with minimal manual intervention.
1.4 One option for using timely expenditure information available in transactions datasets is the calculation of weighted bilateral indexes (e.g. Fisher, Törnqvist). Weighted bilateral indexes compare prices and expenditure across two points in time. They treat expenditure patterns symmetrically and can be compiled either directly or indirectly (chained). Unfortunately, both these bilateral approaches have shown weakness when applied to transactions data.
- Direct bilateral indexes compare prices and quantities from the current period relative to an earlier base period (e.g. period 0 to 1, period 0 to 2). They have the problem of item attrition (i.e. product entries and exits) decreasing the amount of matched products overtime. Additionally, the period chosen as the base period is given special importance and will exclude some items (e.g. seasonal items) that are not available in the base period (Diewert 2013).
- Indirect (chained) bilateral price indexes compare prices and quantities from consecutive time periods (e.g. period 0 to 1, period 1 to 2) which can be chained together to form a continuous series. While indirect bilateral methods address the item attrition issue observed with direct comparisons, they suffer from a 'chain drift' problem where the index fails to return to parity after prices and quantities revert back to their original values. 'Chain drift' is caused by quantities spiking when consumers stock up goods that are on sale, and not returning to their normal level immediately after the sales period (Ivancic, Fox and Diewert 2011; van der Grient and de Haan 2011).
1.5 The limitations of traditional bilateral index formulae have motivated research by NSOs and academics into new methods for compiling price indexes from transactions data. Typically, multilateral index methods have been used in the spatial context to compare price levels across different regions, however academics and NSOs are proposing they be used to make price comparisons across multiple (three or more) time periods. Multilateral methods have a number of advantages for temporal aggregation including:
- Using a census of products available in datasets;
- Weighting products by their economic importance; and
- Producing price indexes that are free of chain drift.
1.6 Statistics Netherlands (Chessa 2016) and Statistics New Zealand (SNZ 2014) are currently the only NSOs compiling components of their respective CPIs using multilateral methods. The Consumer Price Index Manual (ILO 2004) is in the process of being revised to include material recommending the use of multilateral methods for temporal aggregation when using transactions data (Dippelsman and Diewert 2017). Where appropriate, the ABS will contribute information to the update of the revised Consumer Price Index Manual.
1.7 When a multilateral method is used to produce a temporal index, each bilateral price comparison depends on prices observed in other periods of the multilateral comparison window. As a result, incorporating a new period into the multilateral comparison window may alter the price comparisons of earlier periods. To overcome the issues of revisions and produce a CPI in 'real time', NSOs must choose an extension method to use in a production setting.
1.8 As part of the ABS research program into Enhancing the CPI, a number of well-known multilateral and extension methods have been considered for use in the CPI. This research assessed different multilateral methods against a framework associated with the ABS Data Quality Framework (DQF) and was supported by empirical evidence to observe how methods behaved in practice. While this work demonstrated support for using a multilateral method in the Australian CPI, it did not recommend a specific method for the ABS. In light of further research and external consultation with CPI stakeholders and international experts, the ABS is now in a position to provide details on the preferred methods for using transactions data in the CPI.
1.9 The remainder of this publication provides on an implementation plan for maximising the use of transactions data in the CPI. Section two of this paper describes the specific methods the ABS will use in the production of the CPI. The justification for choice of methods borrows heavily from the framework established for assessing multilateral methods in the Information paper: Making Greater Use of Transactions Data to compile the Consumer Price Index
(cat. no. 6401.0.60.003) as well as supporting empirical evidence. Section three provides empirical results comparing the published CPI to price indexes produced using a multilateral method. The remaining sections detail consultation conducted by the ABS and a planned timeline for implementation into the Australian CPI.
1 Introduction of Information paper: Making Greater Use of Transactions Data to compile the Consumer Price Index
(cat. no. 6401.0.60.003) provides a more detailed explanation of current ABS methods used with transactions data. <back