Enhancing measures of non-market output in economic statistics: Progress paper
Jason Annabel - Economic Research Hub
As indicated in the 'roadmap' paper (footnote1), the ABS defines the industries of Health Care and Social Assistance, Education and Training, and Public Administration and Safety, as being predominantly non-market in nature. Non-market economic activity occurs when goods and services are provided to consumers free of charge, or at highly subsidised prices.
Non-market measurement is important given the sizeable contribution it makes to the Australian economy. These three industries contribute 17% of Australia’s gross value added (footnote 2) and provide 3,875,200 jobs (footnote 3). This research is also a step along the road to being able to measure productivity for non-market industries, which will help governments assess the ‘value for money’ received through the provision of public services.
ABS is currently enhancing estimates of six non-market service areas:
Market prices allow the output of different products to be weighted appropriately in aggregation. For instance, imagine that comparing the market prices of cars and bicycles tells us that a car is worth 100 times more than a bicycle. When we aggregate across all products to estimate GDP, each car produced will carry 100 times the weight of each bicycle made. The prices of individual products are needed to derive indicators of relative value – or, weights – which are essential in aggregating the spectrum of all goods and services into a single measure of total output.
When there are no prices, it is difficult to weight the output of products. For instance, we might intuitively assume that a brain surgery procedure provided by public hospitals is more valuable to society than a hernia repair and should ‘count’ for more in aggregation, but we can’t tell by how much, because both services are provided to the consumer free of charge. It is difficult to aggregate brain surgeries with hernia repairs, because no price information exists to weight them.
There are a range of measurement options outlined in the System of National Accounts that are available in the absence of market prices (footnote 4). One option is to assume that the quantity of output produced is equal to the quantity of inputs consumed in producing them. For example, the inputs to a public school include salaries paid to teachers, administrators and ancillary staff, as well as materials and supplies that are used up in production. Under this scenario, if public school teachers earn a real pay rise, meaning input costs increase, it would imply that there has been an increase in output, even if the service being provided is unchanged (for instance, if there is no change to the number of teachers employed or the number of students enrolled). Since outputs and inputs are highly correlated under this approach, productivity growth cannot meaningfully be estimated.
A second measurement option is to model a set of ‘pseudo’ price indexes for various types of non-market output. These ‘pseudo’ price indexes would then serve as the aggregation weights, similar to the cars and bicycles example outlined earlier. The indexes would theoretically reflect the prices that would be paid for non-market output, if prices existed. An example might be to model the market value of a tonsillectomy provided in a public hospital, and then estimate how this value changes over time.
A third measurement option, which is conceptually equivalent to the second option described above, is to estimate the growth in the quantity of different types of non-market output, and then explicitly adjust them for perceived changes in quality over time. Examples of explicit quality adjustment might include accounting for improvements in school test scores over time, or adjusting for changes in hospital waiting periods over time.
A final fourth possibility is to use the costs of production to drive the aggregation weights. It might be difficult to determine by how much more society values public hospital brain surgeries compared to hernia repairs in terms of the prices paid by consumers, but expenditure data permits an objective way to determine how much more expensive one procedure is to perform compared to the other.
When data is appropriately stratified, this approach allows quality change over time to be implicitly reflected as the cost weights change over time. For example, if tonsillectomies previously required an overnight hospital stay, but technological progress now means they can be performed as ‘day’ surgery, the procedure now costs less to perform than it did before. Implicitly, this is an improvement in quality. Differentiating the overall service into homogenous pieces via stratification allows the concept of utility to be reflected in the statistics, the only drawback being that it is observed from the perspective of the producer and not the consumer. It also means that observed data can be used to compile the aggregation weights as opposed to estimating or modelling them.
ABS will be pursuing the fourth option. This is considered to be international best practice in terms of compiling national accounts statistics.
Outputs and outcomes
By way of example, the ABS aims to construct a volume index of the output of school education by aggregating lower-level index series. These lower-level series will be compiled from source data stratified as follows:
Update on progress
The section below describes progress in measuring each of the six non-market service areas listed earlier. It outlines the data landscape, stakeholder engagement, opportunities and challenges presented by the available data, and next steps.
Inclusion of emergency department and non-admitted patient care in the output measure is important in the context of capturing shifts across different types of care over time. For example, a shift from admitted care to non-admitted care can be considered an efficiency gain if the latter costs less but can achieve the same or a better outcome. These shifts can be partially captured through annual updating relative cost weights across different types of care.
Emergency department care and non-admitted patient care are provided mainly in public hospitals. Private hospital activity in these areas is not being pursued as it is estimated to represent less than 4% of non-admitted events in 2016-17 (footnote 8).
ABS is working closely with the Australian Institute of Health and Welfare (AIHW) who collect data for emergency department activity and non-admitted care. AIHW publishes the number of emergency department presentations and non-admitted care occasions or events back to 1994-95. The Independent Hospital Pricing Authority (IHPA) publishes cost weights for emergency department and non-admitted care by relevant classifications back to 2009-10. These cost weights can be used for constructing indexes.
There are challenges in relation to tracking patients through the hospital system. For instance, if a patient is treated as an admitted patient and is then subsequently treated for the same condition as a non-admitted patient, this conceptually represents one treatment but it would be 'counted' twice in the source data. Constructing an integrated dataset that links all three types of care from patient level records is not statistically practical at this time.
It is therefore more realistic to construct separate quantity indexes for emergency department care and non-admitted patient care, with these indexes aggregated with the admitted care index to produce a quantity index for total hospital activity.
Additional challenges arise from variations in the scope and coverage of the available cost data for emergency department and non-admitted care over time, as well as from changes in classifications and coding standards in reporting of emergency department and non-admitted care activity.
Two options exist for measuring growth in emergency department activity. One involves stratification using the Urgency Related Group (URG) classification, whereas an alternative is to regroup the data by triage and ICD chapter. These options are currently being worked through. One benefit of mapping from URG to ICD is that it is stratified similarly to admitted care, but a drawback is that the mapping is approximate, and refinements will be required.
Data for non-admitted care provided by hospitals is classified by the IHPA's Tier 2 classification, which reflects the nature of the service provided and the type of clinician providing the service. Again, there are two stratification alternatives - to use the Tier 2 classification as is, or to map to ICD. The pros, cons and limitations of each option are similar to those applying to emergency care.
There is a lack of expenditure data to match the level of details of activity prior to 2009-10, which means that historical quantity indexes for emergency department activity and non-admitted care will likely have to be constructed from aggregates prior to this year. This will make it difficult to capture shifts across different types of emergency and non-admitted treatment in earlier years.
In addition, ABS aspires to extend the time series on admitted care back to 1994-95, though there are challenges in doing so, not least of which is sourcing relevant datasets.
The benefit of using a finer disaggregation of diagnosis-based data for admitted patient care has been considered. In theory, lower level detail is better for controlling for compositional changes. However, in practice, the benefit is limited by variability in source data quality over time. It is considered that the significant investment in data acquisition, mapping and repair needed for a lower level of disaggregation would not justify the analytical benefit.
The ABS is in the process of compiling a set of experimental indexes covering emergency department activity and non-admitted patient care. Variation in source data scope and coverage are challenges currently being dealt with.
By June 2020, ABS intends to have refined and published the indexes covering emergency department and non-admitted patient services, and aggregated them with indexes for admitted patient care. This aggregated index will reflect a holistic view of hospital services, and is intended to support improvements to the annual national accounts in 2021.
The ABS already compiles a directly observed quantity indicator of ambulance services. This is primarily compiled from data sourced from the ROGS, where the quantity metric is the number of patients treated (whether subsequently transported to hospital or not).
The primary data source is likely to be the Productivity Commission's Report on Government Services (ROGS) (footnote 9). The ROGS data covers public ambulance services and is available on a State/Territory basis. It covers both urban and rural services, as well as services provided by salaried staff as well as volunteers. Amongst a range of other metrics, it captures the expenditures incurred at the State level to provide ambulance services, as well as the number of incidents, which are further categorised into emergency, urgent and non-emergency.
Measurement alternatives include the number of patients treated, the number of incidents requiring ambulatory services (an incident is defined as an event that results in a demand for ambulance services to respond), or the number of responses (defined as the number of vehicles sent to incidents). Using the number of responses would include output produced when a vehicle is dispatched to an incident, irrespective of whether patients are subsequently treated or not.
The ROGS notes that "The role of paramedics is expanding to include the assessment and management of patients with minor illnesses and injuries to avoid transport to hospital." While this objective tends to increase the expenditure per person on ambulances, it has the tendency to deliver downstream savings by reducing hospital admissions as well as reducing the burden on hospitals.
The percentage of patients not transported to hospital has increased over the past decade, reflecting an increase in paramedics treating patients at the scene. This reflects increased capability of paramedics (underpinned by improvements in training and equipment) as well as actions to reduce hospital emergency congestion. Therefore, the volume index needs to be interpreted in the context that the role of the paramedic has changed over time in pursuit of a broader goal, rather than becoming less efficient over time.
There is currently 10 years of data available and analysis is indicating that aggregate indicators were not sensitive to cost weighted aggregation across states instead of total unweighted numbers, so the totals will be used directly. Further analysis is expected to be published in June 2020, with implementation into national accounts and productivity statistics to follow in 2021.
While compositional change at the State level of activity can be captured, the ABS will investigate whether cost weights are available by severity of incident (i.e. costs incurred on emergency incidents versus urgent incidents and non-emergency incidents) within each State. We will explore whether further disaggregation can improve the quality of the aggregate output volume index for ambulances.
The ROGS data on ambulances also includes statistics on performance metrics, including service response times (both in terms of time taken to respond to triple-zero calls for assistance and in time taken to physically respond to emergency incidents). It also includes data on the extent to which patients reported a clinically meaningful reduction in pain from paramedic services, as well as survey-based patient experience metrics. These data could potentially be used to underpin a first experimental foray into explicit quality adjustment in the future.
ABS has also obtained detailed data from the NDIS, but exploration of this data in terms of measuring output volumes is still in its infancy.
The analysis will focus on the specialist disability services provided under the National Disability Agreement (NDA) and the National Disability Insurance Scheme (NDIS). These services encompass disability supported accommodation, respite and community support services such as therapy, early childhood interventions, life skills and case management.
The NDA, formerly the Commonwealth State and Territory Disability Agreement (CSTDA), is a high-level agreement between the Commonwealth and State governments around the provision of disability services to people with disability. It is a framework that sets out the roles and responsibilities of each level of government in supporting people with disability. NDA data extends back to 2003-04.
The NDIS began rolling out in 2013-14, with clients of the NDA being progressively transitioned across to the NDIS. This transitioning process is expected to be complete by the end of 2019-20.
Clearly, disability care is an important measurement objective, especially given the expected ongoing increase in demand for assistance following the rollout of the NDIS. However, there are a number of significant measurement challenges.
Disability care is not identified as an industry in the Australian industry classification, nor are disability products identified in the product classifications that underpin Australia's national accounts. It follows that to implement any improvements in measuring disability care in the national accounts and productivity statistics, output will need to be mapped to existing product and industry classifications. This will make it difficult to 'see' disability care in the accounts. However, policymakers, businesses and employees that provide disability services view it as if it was an industry. For research purposes, the ABS will estimate disability care output as if it was an industry in its own right.
As pointed out in the ROGS, there are difficulties in drawing the boundaries between disability care and aged care, especially given they often overlap. Both disability care and aged care have similar goals, but with different programs. The ABS is assuming that the datasets used to measure disability care and aged care accurately reflect the scopes of the two activities, with no additional adjustment required.
Further, there are difficulties in constructing a seamless time series as the rollout of the NDIS is taking place before the NDA closes. Preliminary analysis shows a marked increase in the aggregate provision of disability services early in the life of the NDIS, but care is required to make sure the NDA and NDIS datasets are combined correctly.
The ABS will continue working through the issues raised above with a view to publishing a comprehensive volume index time series representing growth in output of disability services. Owing to the complexity involved, this work is not expected to be completed by June 2020.
Aged care services
While aged care services are supported by government funding, they are generally delivered by non-government service providers such as religious organisations, charitable organisations, and private-for-profit organisations. In terms of calculating cost weights, it is likely that the weights will reflect government funding only, and not necessarily the full economic cost of service provision.
The National Aged Care Data Clearinghouse (NACDC) holds data on the number of clients, classified by variables such as State, age bracket and type of care, and expenditure data is likely to be sourced from ROGS. Conversations are underway with AIHW and the Department of Health with a view to bringing together an integrated, holistic dataset matching the specifications required.
The ABS aims to construct a volume index of total output provided by the aged care services industry. However, given research for this topic is still in its infancy, the extent to which this index will implicitly capture quality change is still to be determined.
The two main building blocks are student enrolment data (on a full-time equivalent basis) and school expenditure data. The enrolment data represents the quantity variable whereas the expenditure data underpins the cost weights.
As outlined earlier in the paper, the output indexes for schools will be built up by school type (pre-school, primary, secondary, and special needs), by jurisdiction (i.e. State/ Territory), and by affiliation (i.e. government / non-government). As the main focus of this work is to enhance measures of non-market activity, no attempt will be made to disaggregate non-government school activity into sub-categorisations such as religious affiliation.
Most of the student enrolments data at the above level of granularity is collected by the ABS (footnote 10), except for special needs school education. In terms of compiling comparable indexes for primary and secondary schools, expenditure data presents a challenge, especially for non-government schools. There are a range of data possibilities for expenditure data, including ABS’ Government Finance Statistics data (footnote 11), statistics from the ROGS, and Australian Curriculum, Assessment and Reporting Authority. The ABS is also examining whether it can repurpose some of its own survey data for this exercise for non-government schools, with a prime candidate being the dataset underpinning the annual Australian Industry publication (footnote 12).
Enrolments data by individual school year of enrolment is available for primary and secondary schools, but the ABS is currently unable to stratify by year of enrolment. The lowest level of expenditure data currently available is total primary (which, for most jurisdictions, groups kindergarten to year 6) and total secondary (years 7 to 12). No published information has been found which shows how much each individual year of school education costs relative to other years, and in some cases (especially for schools which provide continuous education from kindergarten to year 12), obtaining a primary/secondary split is not possible.
Without an appropriate data source, the ABS intends to explore possibilities around modelling expenditure by individual school year, as well as enhancing the accuracy of the enrolments data by adjusting it for student absentee rates.
Additional challenges currently being worked through involve accessing expenditure and enrolments data for special needs schools, as well as historical enrolments data for pre-schools.
The ABS plans to extend this analysis to cover all school types by constructing a volume index of total school output. It is expected that indexes for pre-school, primary and secondary education will be completed by June 2020, though refinements around stratification by individual school year and accounting for absenteeism will not be incorporated by this time. Designing an output volume index for special needs schools is likely to extend beyond June 2020.
The volume indicator currently used by ABS is a headcount of the number of students enrolled at university. Opportunities exist for us to refine this metric by:
At this stage, the most promising option for measuring tuition output is by utilising student load statistics (Equivalent Full-Time Student Load (EFTSL)) from the Department of Education, disaggregated by university.
The most likely path for measuring research output would be the count of published research output, disaggregated by university, also available from the Department of Education.
Both of these outputs will be cost adjusted using university-level expenditure data from the Department of Education. By utilising data on staff hired for teaching versus research purposes, the total costs for each university can be decomposed into a teaching and research component, and these costs will serve as the cost weights for the output index for tuition and research respectively. This will be premised on the assumption that the ratio of teaching to research staff reflects the university’s ratio of teaching to research costs.
This disaggregation of costs will also allow the two volume indexes to be weighted together to derive a volume index covering total university output.
In addition to the above, the ABS is investigating possibilities to obtain expenditure differences across postgraduate and undergraduate programs, which can then serve as an additional dimension for disaggregation and would capture the notion that postgraduate qualifications reflect a higher quality in education output relative to an undergraduate qualification.
The Department of Education has recently commenced publishing expenditure and activity statistics disaggregated by broad academic discipline, but it is limited by the lack of a lengthy time series, and a lack of full coverage. Stratification by discipline could become possible in the future as this dataset matures.
Over the period to June 2020, the ABS will explore stratification options with a view to designing the most appropriate indicator possible. As mentioned in the 'roadmap' paper, the challenges of measuring output volume growth in the university sector are complex, and it is not anticipated that this work will be completed by June 2020.
For further queries please email <email@example.com>.
Clemmensen, A. J., 2017, “Quality Adjustment of Hospital Services”, Statistics Denmark
Diewert, W. E., 2011, “Measuring productivity in the public sector: Some conceptual problems”, Journal of Productivity Analysis, vol 36 no 2 pp 177-191
Gemmell, G., Nolan, P. and Scobie, G., 2017, “Estimating Quality-Adjusted Productivity in Tertiary Education: Methods and Evidence for New Zealand”, Victoria University
Gibbs, T., 2017, “Sources and Methods: Public service productivity estimates: Education”, Office for National Statistics, United Kingdom
Gu, W. and Morin, S., 2014, “Experimental Measures of Output and Productivity in the Canadian Hospital Sector, 2002 to 2010”, University of Chicago
Gu, W. and Wong, A., 2012, “Measuring the Economic Output of the Education Sector in the National Accounts”, Statistics Canada
Harris, L. and Martin, J., 2019, “A guide to quality adjustment in public service productivity measures”, Office for National Statistics, United Kingdom
Lewis, J., 2019, “Public service productivity: adult social care, sources and methods, 2019 update”, Office for National Statistics, United Kingdom
Lewis, J. 2019, “Public service productivity estimates: Healthcare QMI”, Office for National Statistics, United Kingdom
Luo, Q., 2018, “Disease Based Output Measures for Hospitals”, ABS Chief Economist series
New Zealand Productivity Commission, 2018, “Improving State Sector Productivity”
New Zealand Productivity Commission, 2018, “Measuring State Sector Productivity”
Productivity Commission, 2019, “Report on Government Services”
Schreyer, P., 2010, “Towards Measuring the Volume Output of Education and Health Services: A Handbook”, OECD Statistics Working Papers 2010/02, OECD
Trewin, D., 2004, “Measurement of Services: Recent ABS experience”, Statistical Journal of the United National Economic Commission for Europe, vol 21, pp 277-290
United Nations, 2008, System of National Accounts
York, J. 2010, “The Devil Is In The Detail: Demonstrating the Impact of Measurement Choices on Inputs to Government Sector Productivity”, Statistics New Zealand
FOOTNOTE 1 https://www.abs.gov.au/websitedbs/D3310114.nsf//home/ABS+Chief+Economist+-+Enhancing+measures+of+non-market+output+in+economic+statistics:+A+roadmap
Figure 1 shows the majority of Education output (in current prices) comes from Tertiary education (27% of total Education output). Primary education (24%) and then Secondary education (23%) comprise the second and third largest shares. The 'Remaining education services' category includes adult education, vocational education, sports and physical recreation instruction.
For Health services, Hospitals (25%) and Aged care (21%) comprise the largest shares of total output (in current prices). Ambulance services have a relatively small share of total health output at 2%. However this component of the research has been very beneficial in terms of data availability. The 'Remaining health services' category covers activity that is predominantly market in nature, including dental, physiotherapy, general practice and specialist medical and pathology services.
Figure 1: Shares of Education and Health output in the Australian National Accounts