Think Outcomes conference
Presentation: Using Data to Measure Outcomes and Inform, Melbourne, 12 April 2016
David W. Kalisch, Australian Statistician
First, I would like to acknowledge the Wurundjeri people of the Kulin Nation, the Traditional Custodians of the land we are meeting on today and pay my respects to their Elders both past and present, and to acknowledge members of the Aboriginal and Torres Strait Islander community who may be attending this conference today.
What a refreshing title for the conference, with the focus on outcomes and the measurement of outcomes. From my background in public policy, I have taken considerable interest in the way new programs and policies are portrayed to the public.
When I reflect on the many public policy statements over recent years programs are largely judged on two key dimensions; how many people are affected and how much it costs. The focus is on the immediate impact – how many people are likely to be winners and how many are losers if the policy is introduced and generally on the basis that no-one changes their behaviour.
This last assumption generally made is the most heroic one, as most public policy is introduced with the explicit purpose of changing behaviour and improving outcomes.
This is where the measurement of outcomes is key, as is the importance of having rigorous, reliable approaches to monitor and evaluate outcomes.
From an international perspective I would suggest we are very good at measuring outcomes.
Every year the ABS delivers 500 to 600 statistical releases – providing information to governments, business and the community on economic, social and environmental outcomes.
Through these statistics, the community has reliable, relevant and timely information on:
In total, once you also take account of the further information available from other government agencies, universities and business, there is considerable information available on outcomes.
Even with the impact of efficiency dividends over many years – reducing the resources available to the ABS and other information providers, there is still a lot of outcome information that is regularly produced and available for use.
Increasingly, we are confronting choices about how to best use the scarce public/taxpayer resources available to us.
In recent years, the ABS has made decisions to stop a number of lower priority statistical series because they could not be produced within the limited resources provided by government.
I expect that these difficult choices may need to continue to be made in coming years with our prevailing funding arrangements.
The internal focus on efficiency and productivity in the ABS is real and ongoing, but ultimately I suspect the future will also be marked by a more limited statistical program, and we need to consider other cost-effective way of producing the information needed to measure outcomes.
So, you will not be surprised, with my role as the Australian Statistician, that I would emphasise the value of official statistics to inform key decisions and measure key outcomes. These purposes, in addition to holding governments to account are reasons for having an independent national statistics office.
What the ABS, and other national statistical offices, want to measure across the economy, society and the environment are, generally, the outcomes that matter to the community. However, myself and the ABS want to take this even further and unleash the power of statistics.
But first I want to reiterate that measuring outcomes, especially from public policy, is not simple.
The challenge with measurement especially when measuring the value of government policies and programs is how sure are we that a program or policy made a difference to people’s lives. That it was not the result of other changes in their life, in the broader economy etc?
In particular, we need to recognise that:
There have been collaborations across the Australian government to produce new data insights using snapshots of data over time to start to present a longitudinal view of the lives of Australians, and in turn an increasing evidence-base in which to measure the impact of social policy on outcomes.
We have already demonstrated by bringing together existing data, how new information can be produced in timely cost-effective way, without any additional burden on individuals and businesses – and is a real data source that can be used to understand and measure outcomes.
For a longitudinal view of the outcomes of individuals, the Australian Census Longitudinal Dataset integrates a sample from 2006 Census with the 2011 Census. The ABS will also link the 2016 Census with the 2011 Census. This dataset provides insights into the changes between each five year period of individuals and families. Examples of analysis using this dataset include understanding housing transitions for older people, transitions for recent migrants, changes in family and individual situations, and employment outcomes for people in different industries such as the automotive industry.
The ABS has also built an expanded analytical business longitudinal database combining administrative tax data with business survey data. One of the purposes of the dataset is to understand the productivity of medium and small business over time – and it has already provided insights to Government into the innovation and growth potential of different types of businesses.
A brand new data source is a linked employer and employee dataset which opens up source of information on employment. This brings together personal income tax data with business data and has already become a new data source for understanding multiple job holding.
Other data integration projects the ABS has been involved in include:
The ABS is continuing to work with other Commonwealth departments with existing large administrative datasets to demonstrate how an integrated data resource can be used to measure the outcomes and impacts of policy and services, in a timely manner.
The Multi-agency Data Integration Project will bring together, for the first time, Census data with administrative data on health, income, and social security payments, to establish a foundational, linked dataset which could be extended both cross-sectionally (such as through to businesses or other data sources such as education) and longitudinally to provide a publically accessible linked data resource.
However I recognise that Australia is well behind the frontier in terms of measuring and reporting on outcomes. We are just scratching the surface on measuring outcomes, and in many cases not even being clear about the outcomes that matter and the ones we prioritise as a community.
However, we have an opportunity to catch up to what is already available in New Zealand, in terms of their Integrated Data Initiative. This integrated data resource utilises data linkage of key government programs with their Census and survey outcome information in a sophisticated and broad fashion – and when combined with smart analytics – can measure outcomes better and also provide more informed assessments of likely outcomes – both positive and negative.
The New Zealand Resource is widely used for answering policy questions, allowing for efficient evaluation and research analysis, and the production of person-centred statistical outputs on the pathways, transitions, and outcomes.
More specifically, it has been central to Government decision-making and is used to provide a statistical link between programs and liabilities, allowing for cost-benefit analysis of spending programs to target early interventions and for the evaluation of government program delivery.
However to get the usefulness of the prevailing New Zealand resource in the Australian context we must also include key state government services, such as hospital, other community services and schools.
And in the longer term, also drawing on private administrative data – such as big data – which I will talk more about in the big data panel discussion later this morning.
I would hope that, in a number of years’ time, we would have an Integrated Data Resource similar in usefulness to what the New Zealand government achieved in August 2013.
This is a realistic goal.
Case study of a recent Census debate
However, it is not smooth sailing as we seek to provide improved public policy insights for government and the community.
You may be aware that the ABS has announced, following a Privacy Impact Assessment, that it intends to retain name and address information from the 2016 Census later this year.
There were a number of key dimensions to this decision by the ABS:
The other key development is that the ABS, and many other National Statistical Offices in other countries, are using Census data increasingly to measure outcomes.
This is the prime reason we want to keep names and addresses, and use them safely.
If we are to get reliable results on outcomes, especially for small population groups, we need to use names and addresses in the linkage process rather than the probabilistic linking that works well with much larger populations.
Some examples where name and address linkage is necessary to produce reliable insights includes:
One feature of the current debate around more extensive use of Census data for public purposes, even with a very safe environment at the ABS is that:
My one challenge to you today is that if you really think outcomes are important, and if you think the ABS can help deliver valuable outcome information, then you do need to be more active and more vocal in making this view known.
I would hope we can see more balance in the media reporting, where the value of the outcome information is better appreciated, alongside the privacy dimensions we are managing well.