APPENDIX 2 REVIEW BY PROFESSOR KEVIN FOX


COMMENT ON "MAKING GREATER USE OF TRANSACTIONS DATA TO COMPILE THE CONSUMER PRICE INDEX, AUSTRALIA", AUSTRALIAN BUREAU OF STATISTICS 6401.0.60.003, RELEASED: 29 NOVEMBER 2016

By Kevin J. Fox (footnote 1) 20 April 2017


SUMMARY

This comment assesses the plans of the ABS for expanding the use of transactions data in CPI construction. The focus is on two issues that were noted in ABS (2016) as being unresolved: (i) the best multilateral index number method to use, and (ii) the best way of extending the resulting series when new observations become available. Drawing on research conducted since the publication appeared, recommendations are made regarding both issues. Specifically:

1. It is recommended that the CCDI (or "GEKS-Törnqvist") multilateral index be used.
2. It is recommended that the "mean splice" be used as the extension method.

Using transactions data with multilateral methods to construct the CPI is an improvement over current practice. Internationally, there is much interest in moving to such methods. Since the 1990s, the ABS has been a leader in research on this topic, engaging with academics and researchers from other NSOs. Now, as other NSOs begin to implement new methods for using transactions data in their CPIs, Australia should not be left behind, as deviating from international best practice would bring the credibility of the Australian CPI into question.

Besides the benefits from using more information in a more efficient way, there are additional benefits that can arise from the proposed changes. These include the potential for improving the timeliness of CPI releases and the potential to move resources from data collection and processing to analytics, which can better inform policy formulation.


ABOUT THE AUTHOR

Kevin Fox is a Professor of Economics in the UNSW Business School, and Director of the Centre for Applied Economic Research. He works primarily in the field of economic measurement, with a focus on productivity and prices. He is a Fellow of the Academy of the Social Sciences in Australia, a Fellow of the Society for Economic Measurement, and an elected member of the NBER-affiliated U.S. Conference on Research in Income and Wealth. He was appointed as an Advisor to the Australian Treasury in 2016.

He has had a long association with the ABS, being a member of the ABS Methodology Advisory Committee since 1999. He chaired the ABS Advisory Group for the 16th Series CPI Review, 2009-2010, and is a member of the Productivity Measurement Reference Group and the Input-Output User Group.

He has conducted research on the use of transactions data in the CPI, much of it stemming from collaborations with the ABS, including the following:

2003-2005 Australian Research Council Linkage Grant, for "Can Electronic Point-of-Sale (POS) Data Improve the Australian Consumer Price Index?" with R.J. Hill. Industry Partner: Australian Bureau of Statistics.

2006-2008 Australian Research Council Linkage Grant, for "Scanner Data in the Consumer Price Index: How to expand and improve their use," with J. de Haan, P-H. van Mulligan and M. Silver. Industry Partners: Australian Bureau of Statistics and the Central Bureau of Statistics (Netherlands).


1. THE USE OF TRANSACTIONS DATA IN THE CPI

The availability of transactions level data in large quantities provides huge advantages for National Statistical Offices (NSOs). The recent ABS publication (ABS 2016), henceforth referred to as "the report" convincingly sets out many of the advantages of using such Big Data, so they will not be repeated here. A point worth emphasising, however, is that there is nothing sacred about current CPI construction methodology.

Existing methods, while differing somewhat across countries, are largely based on price information sourced from sampled prices (e.g. from sending price collectors to record prices listed on supermarket shelves). These prices are combined with importance weights for the different product categories, where the weights are sourced from periodic expenditure surveys. Clearly there are potential problems, for example if the sampled prices are not representative of actual consumer purchases (e.g. no one buys the goods at those listed prices) or the importance weights are out of date due to infrequent surveys. In addition, these weights are usually only available at expenditure class levels rather than at the level of individual goods, requiring the use of average prices (across goods, outlets and time). With information available on all goods prices and the quantities purchased at these prices, this is not how one would design a price index. Rather, current CPI construction has developed as a response to historically limited data availability.

The advent of readily available transaction level data then allows for an overhaul of traditional methodology, as the data constraint has been enormously relaxed. However, this opportunity for improved price index construction has been somewhat offset by the complexities involved in the use of high-frequency data. One of these has been the observation that traditional bilateral index number theory can break down when using such data.

The response by NSOs to date has been to make limited use of transactions data, such as to directly replace some price sampling, but not making direct use of the available quantity information. The ABS report (p. 1, 1.3) indicates that from March 2014 there was a significant increase in the use of transactions data, "now accounting for approximately 25% of the weight of the Australian CPI". This can be thought of as using a more comprehensive data source in combination with traditional CPI compilation methodology. This is an improvement (more price information means a more representative price for each product), but is still far from making optimal use of the available information.

Drawing on recent developments in the academic literature, starting with Ivancic, Diewert and Fox (2009, 2011), some NSOs have started to use multilateral index numbers in CPI construction when using transaction level data, but in a very limited fashion; see Krsinich (2015) and Chessa (2016). Multilateral methods are drawn from the international comparisons literature, and provide a solution to the chain drift problem when applied in the time series context. This problem can be severe when constructing price indices with transactions data, even when using superlative index formulae as recommended in the ILO (2004) CPI Manual; see Ivancic, Diewert and Fox (2011) and de Haan and van der Grient (2011).

Some complexities also arise in the use of such methods. First, a choice of multilateral index number formula needs to be made. In traditional CPI construction, the choice of formula is typically constrained by data availability, especially the availability of expenditure weights, leading to the use of the (non-superlative) Lowe index; see ABS (2011, p. 23, 4.38).

Second, as the CPI is non-revisable, a way of extending the price index as new information becomes available is required. The issue is the following. In international comparisons, multilateral methods are applied over a set of countries to make comparisons of relative prices or output. The addition of another country to this set of comparisons can change some or all relativities, as its addition contributes further information. In the time series context, multilateral methods are applied over an initial time period, or "window", and as data for a new period becomes available, simply extending the window by the additional period would mean that earlier estimates may be changed. This is unacceptable for a series which should not be revised, such as the CPI. Hence, a method that respects this constraint is needed for extending the original window as data for more periods become available.

The report assesses, theoretically and empirically, the use of alternative multilateral methods and extension methods. It also addresses practical issues of implementation, specifically around the level of aggregation at which these methods are applied.

To put the report in context, it is worth noting that it is the latest in a long series of research on transactions data by, and supported by, the ABS over almost twenty years. Much of this has been in collaboration with academics and experts from other NSOs(footnote 2) . The approach of the ABS has thus been to demonstrate appropriate caution when considering new data sources and methods, while collaborating broadly to explore opportunities that can provide significant benefits. In taking this approach, in my view the ABS has provided an exemplar for productive government engagement with external experts.

In what follows, this comment will provide an overall assessment of the report, make some specific comments, and provide recommendations. The focus is on two issues that were noted in the report as being unresolved: (i) the best multilateral index number method to use, and (ii) the best way of extending the resulting series when new observations become available.


2. OVERALL ASSESSMENT AND SPECIFIC COMMENTS

1. The ABS Data Quality Framework (DQF) is central to considerations, along with the relevant academic literature and research conducted by other NSOs. This is all very satisfactory.

2. The reported empirical results using actual transactions data enhances our understanding of the relative performance of the methods. The amount of work and care that has gone into these empirical assessments is impressive.

3. The descriptions of the background issues in Section 1, the multilateral methods in Section 2, and the extension methods in Section 3, are accurate and informative.

4. In Section 4, the use of the "test approach" to assess the performance of the alternative multilateral methods is appropriate and presented in a thoughtful and informative manner. In particular, the report does not simply present a mechanical application of received results, but assesses the relevance of tests for the context faced by NSOs. It is impressive that new results are derived and presented in Appendix 2.

5. 4.23, page 18: "To the best of our knowledge, the TPD and QAUV_TPD methods have not yet been assessed rigorously from this economic approach." Diewert and Fox (2017) have now assessed the TPD from this perspective, and show that it is an approximately additive method that is consistent with linear (perfect, or infinite, substitutability) and Cobb-Douglas (elasticity of substitution equal to one) utility functions. Clearly the first case is unrealistic in general, while the second case may approximate actual preferences in some cases. The QAUV_TPD method will share the same substitution possibilities as the GK method (discussed in the following point), as the only difference is in how the quality adjustment factors are estimated.

6. 4.24, page 18: "In contrast, the GK and other additive methods are consistent only with linear utility functions." The GK method is also consistent with Leontief (perfect nonsubstitutability) as well as linear (perfect substitutability) utility functions; see Diewert (1999, p. 28). Clearly neither case is realistic, and hence unlikely to appropriately represent actual consumer preferences.

7. 4.24, page 18: The test used for assessing substitution bias in the multilateral indexes is limited by not knowing the actual substitution elasticities. The approach of Diewert and Fox (2017), partly in response to this work, is to generate artificial data under specific assumptions on the elasticity of substitution. This then allows an assessment of the performance of methods when the true elasticity that generated the data is known. For elasticities in a range regarded as being reasonable (based on empirical evidence), the GK and TPD methods were found to perform quite poorly. In any empirical application, it could be found that these methods approximate each other and the results from methods with better substitution modelling properties. However, this is not an argument for choosing these methods as the default ex ante. Fortuitous empirical correspondence of theoretically poor methods with theoretically better methods is not a basis on which to justify the use of the theoretically poor methods. Diewert and Fox (2017) find in favour of the CCDI multilateral method, call the GEKS method in the report and in much of the literature(footnote 3) .

8. 4.31, page 19, Interpretability: It is noted that the CCDI/GEKS approach is perhaps the easiest to grasp, as price movements are derived by combining superlative bilateral index numbers. There is an additional property of the CCDI index which facilitates interpretability. It is based on the use of the Törnqvist bilateral index, which is a share weighted geometric mean of price ratios. Following the results of Caves, Christensen and Diewert (1982) and Inklaar and Diewert (2016), Diewert and Fox (2017) note that the CCDI method can be interpreted as comparing the level of prices in each period with the corresponding level of prices in an average observation. This provides a significant interpretability advantage over other methods, including the Fisher-index-based GEKS method.

9. 4.36 and 4.37, page 20: The simple multiplicative decomposition property of the bilateral Törnqvist index (having a weighted geometric mean form) is inherited by the CCDI multilateral index. This is appropriately noted as an attractive property, and provides an additional advantage of using the CCDI method.

10. Summary of Section 4: In terms of the Test Approach to assessing multilateral methods, there is no clear winner. In terms of the Economic Approach, following the recent results of Diewert and Fox (2017), the arguments in favour of the CCDI method are somewhat stronger than in the report. In terms of Flexibility, the CCDI method is again found to be amongst the preferred methods. In terms of Interpretability, the CCDI method has advantages in addition to what is noted in the report (see point 8 above). Hence, based on the considered criteria and complemented by the results of Diewert and Fox (2017), it appears that there are strong arguments in favour of using the CCDI method.

11. 5.3, page 22: The data constraints that led to the traditional CPI compilation methods are very nicely explained. The emergence of current practice as a response to data constraints is a point worth emphasising in proposing to change practice now that the data constraint has been significantly relaxed.

12. 5.5, page 22: The proposed structure, illustrated in Figure 5.1 for aggregation seems sensible. Moving away from traditional unweighted elementary aggregates is an appropriate response, and the reasons provided (both practical and theoretical) are very sensible. Practical reasons (motivated by the nature of the data received by the ABS from providers) aside, being able to better account for substitution across a wider range of items is attractive.

13. 5.10, page 25: The empirical deviations of the CCDI ("GEKS") method from the other multilateral methods is interesting. As noted above, the relative properties of the CCDI method and the evidence of Diewert and Fox (2017) suggest that it should do well in approximating the truth. The "truth" is hard to assess in any uncontrolled empirical application, so it is hard to determine how these deviations should be interpreted. This suggests that, while the CCDI method is ex ante to be preferred, it would be worthwhile running another multilateral method in parallel to identify any method specific differences. The use of the regression based (weighted) TDP method seems sensible, as it allows more reasonable substitution possibilities compared to the GK and QAUV_TPD methods, and being regression based it can provide measures of statistical significance for the estimated coefficients.

14. 5.14, page 28: The results presented in figures 5.8-5.11, for the example of Snacks and Confectionery, are very interesting. Multilateral methods, having the property of being "transitive", will not be subject to chain drift bias, hence the proposal for their use. However, extending the series may introduce chain drift bias, and the extent of this bias may depend on the extension method chosen. These figures provide evidence on this, and show that there can be significant downward bias introduced by the extension methods. In the presented examples, the "half splice" performs relatively well (closest to the corresponding multilateral method applied to the full data period). The "direct extension" method of Chessa (2016) seems hard to justify (especially the varying window length), and unsurprisingly it is shown that it can perform very badly, i.e. it can introduce significant chain drift. It seems that for both theoretical and now empirical reasons, this can be excluded from further consideration. The relatively good performance of the "half splice" may be a result of the particular data examined. The difference between the "window splice", "half splice" and the "movement splice" methods is the choice of linking period between the old and new windows (first period of the new window, a middle period, and last period of the old window, respectively).

A problem with the approach used to assess these methods is the following. The benchmark for each multilateral method is the method applied over the full 58 time periods. As noted in 1.19 (page 4), with the use of multilateral methods there is a tension between transitivity and "characteristicity"; as multilateral methods use data from periods other than those being directly compared, there can be a loss of relevance of the index, i.e. if the comparison between two periods is affected by data from periods with limited relevance. As the window length becomes longer, data from more chronologically remote periods influence all bilateral comparisons to some extent. Hence, the benchmark indexes used in figures 5.8-5.11 may be suffering from a serious loss of characteristicity due to the long window length. The problem with using this approach to find a definitive answer on the best extension method can be thought of as one of not knowing the truth.

Diewert and Fox (2017) therefore used a different approach to assess the methods(footnote 4) . By using their artificial data generated assuming various elasticities of substitution, they know the truth. They concluded that rather than choosing one period for linking the old window with the new window, where the resulting index could depend significantly on that choice of period, a geometric mean approach could be used. That is, using an average of the resulting price level for the new period from using all overlapping periods for the linking in turn. This has the advantage of treating all overlapping periods symmetrically, which as an ex ante default principal can be argued to be reasonable(footnote 5) , and the approach can moderate the influence on the resulting index of a linking period or periods with unrepresentative data patterns.


3. SUMMARY AND CONCLUDING COMMENTS

This is an excellent and timely piece of work by the ABS. It continues their research dating back almost twenty years on the potential for transactions data for improving the CPI. It is thoughtful in its structure, systematic in its approach and it provides insightful new evidence on the behaviour of methods of interest.

A convincing case is made for the desirability of expanding the use of transactions data in the CPI, something which is now very possible due to the work over many years done by the ABS, both internally and through collaborations. Other NSOs have recently started to make changes to CPI construction in light of this work, and more are likely to follow. To maintain the credibility of the Australian CPI, it would be timely for the ABS to also move to implementation.

A goal of no field collections seems feasible and desirable. This will allow a reallocation of resources from collection and processing to automation and analytics. It can also facilitate the production of a monthly CPI, a key recommendation of the last major review of the CPI; see ABS (2010). Potential improved accuracy issues aside, additional analytics and improved timeliness can better inform policy formulation.

The report identifies two unresolved issues for the broader use of transactions data in the CPI: (i) the best multilateral index number method to use, and (ii) the best extension method to use.

The paper which originally proposed multilateral methods in this context - Ivancic, Diewert and Fox (2009, 2011) - proposed either the GEKS method or the TPD method (both gave similar results in their empirical work). It was combined with what has now become known as the "movement splice" for extending the CPI series when a new period's data became available. Different multilateral methods and extension methods have subsequently been suggested and applied in this context. The report reviews and assesses these alternatives, and provides empirical evidence on their use. This is illustrative of broader ABS experimentation on these methods using data on a diverse range of product classes at different levels of aggregation.

This is all very informative. It also represents ample caution in relation to understanding the impact of changes in CPI data sources and construction methodology, as well as significant in-house expertise in analysing these issues.

Recent work by Diewert and Fox (2017), stimulated in part by the ABS report, recommended the following in regards to the two unresolved issues noted above:

1. Recommended multilateral method: CCDI (or "GEKS-Törnqvist") multilateral index.
2. Recommended extension method: The "mean splice".

As with current CPI methodology, there is always the possibility for future research to reveal means for enhancing the methods used(footnote 6) . At this time, however, implementation of the Diewert-Fox recommendations in points 1 and 2 above seem not only feasible but desirable, especially relative to the limitations imposed by data constraints on current CPI construction methodology.


REFERENCES

Australian Bureau of Statistics (2010), Outcome of the 16th Series Australian Consumer Price Index Review, Information Paper, Catalogue No. 6469.0. Commonwealth of Australia, Canberra.

Australian Bureau of Statistics (2011), Consumer Price Index: Concepts, Sources and Methods, Information Paper, Catalogue No. 6461.0. Commonwealth of Australia, Canberra.

Australian Bureau of Statistics (2016), Making Greater Use of Transactions Data to Compile the Consumer Price Index, Australia, Information Paper, Catalogue No. 6401.0.60.003. Commonwealth of Australia, Canberra.

Caves D.W., Christensen, L.R. and Diewert, W.E. (1982), 'Multilateral comparisons of output, input, and productivity using superlative index numbers', Economic Journal 92, 73-86.

Chessa, A.G. (2016), "A New Methodology for Processing Scanner Data in the Dutch CPI," Eurona 1/2016, 49-69.

Diewert, W.E. (1999), "Axiomatic and Economic Approaches to International Comparisons", pp. 13-87 in International and Interarea Comparisons of Income, Output and Prices, A. Heston and R.E. Lipsey (eds.), Studies in Income and Wealth, Volume 61, Chicago: The University of Chicago Press.

Diewert, W.E. and K.J. Fox (2017), "Substitution Bias in Multilateral Methods for CPI Construction using Scanner Data," Discussion Paper 17-02, Vancouver School of Economics, University of British Columbia, Canada.

Fox, K.J. and I. Syed (2016), "Price Discounts and he Measurement of Inflation", Journal of Econometrics 191, 398-406.

Feenstra, R.C., H. Ma and D.S. Prasada Rao (2009), “Consistent Comparisons of Real Incomes across Time and Space”, Macroeconomic Dynamics 13(S2), 169-193.

de Haan, J. and H.A. van der Grient (2011), "Eliminating Chain drift in Price Indexes Based on Scanner Data," Journal of Econometrics 161, 36-46.

Inklaar, R. and W.E. Diewert (2016), "Measuring Industry Productivity and Cross-Country Convergence", Journal of Econometrics 191, 426-433.

Ivancic, L., W.E. Diewert and K.J. Fox (2009), "Scanner Data, Time Aggregation and the Construction of Price Indexes," Discussion Paper 09-09, Department of Economics, University of British Columbia, Vancouver, Canada.

Ivancic, L., W.E. Diewert and K.J. Fox (2011), "Scanner Data, Time Aggregation and the Construction of Price Indexes," Journal of Econometrics 161, 24-35.

Jain, M. and R. Abello (1999), "Measurement of Price Indexes and Biases Using Scanner Data," Analytical Services, Methodology Division, Australian Bureau of Statistics.

Jain, M. and R. Abello (2001), "Construction of Price Indexes and Exploration of Biases Using Scanner Data" Analytical Services, Methodology Division, Australian Bureau of Statistics. Room document at the Sixth Meeting of the International Working Group on Price Indices, 2-6 April 2001, Canberra.

Jain, M. and J. Caddy (2001), "Using Scanner Data to Explore Unit Value Indexes," Analytical Services, Methodology Division, Australian Bureau of Statistics.

Krsinich, F. (2015), “Implementation of Consumer Electronics Scanner Data in the New Zealand CPI,” Statistics New Zealand. Paper presented at the New Zealand Association of Economists conference, Wellington, New Zealand, 3 July.

Moore, K. (2014), "Transactions Data: From Theory to Practice," Australian Bureau of Statistics, presented to the Group of Experts on Consumer Price Indices, UNECE, 26 - 28 May 2014.

1 School of Economics and Centre for Applied Economic Research, UNSW Sydney, NSW 2052. Tel: 0434 147 405. Email: K.Fox@unsw.edu.au <back
2 See e.g. Jain and Abello (1999, 2001), Jain and Caddy (2001) and Moore (2014). The idea of using transactions data with multilateral methods came out of research supported through two Australian Research Council Linkage Grants with UNSW researchers, the second of which also involved Statistics Netherlands. Research experience and results shared at the annual UNSW Economic Measurement Group Workshop, supported by the Linkage grants, also significantly advanced understanding and informed practice. <back
3 The bilateral index formula used in the standard GEKS multilateral method is the Fisher Ideal index. Sometimes it is replaced by the Törnqvist bilateral price index formula. As Fisher and Törnqvist indexes often approximate each other closely in empirical applications, it is not uncommon for the resulting multilateral indexes to be both called "GEKS", as in the report. De Haan and van der Grient (2011; 41) called the Törnqvist based multilateral indexes GEKS-Törnqvist indexes. Feenstra, Ma and Rao (2009; 171-172) also noted that Törnqvist bilateral price indexes could be used in place of Fisher price indexes in the GEKS methodology. Fox and Syed (2016; 401) call the Törnqvist-based indexes CCD indexes. Caves, Christensen and Diewert (1982) (CCD) used the GEKS methodology in the quantity context, using bilateral Törnqvist quantity indexes as their basic bilateral index formula. Inklaar and Diewert (2016; 429) extended the CCD methodology to making price comparisons across production units. Hence, to distinguish the Fisher- and Törnqvist-based indexes, and to reflect the contributions of CCD and Inklaar and Diewert, Diewert and Fox (2017) called the Törnqvist-based multilateral indexes as CCDI indexes, and reserved GEKS for the traditional Fisher-based indexes. <back
4 This method was initially suggested by Ivancic, Diewert and Fox (2011, footnote 19, page 33). <back
5 In any particular set of data, there may be circumstances where one or more periods are excluded from consideration as linking points, depending on the characteristics of the data or exogenous expert knowledge. This implies a zero or one weighting scheme in forming a geometric average in obtaining the index value for the new period. Using a more general weighted mean approach is also possible, but introduces an extra level of complexity in determining the appropriate weights. There does not appear to be sufficient evidence at this stage to support a weighted mean approach, but it is a topic worthy of research. <back
6 Diewert and Fox (2017) also note some promising research directions. <back