Learnings from the 2016 Census, 2017 Marriage Law Survey, and Beyond
David W Kalisch, Australian Statistician
IPAA NSW State Conference, 21 May 2019
In December 2016 I presented to IPAA ACT on our learnings from the 2016 Census, as they were at that time. ABS had finished the Census data collection, and the MacGibbon Review and Senate Inquiry had both reported a few months earlier.
Our intensive data quality checks were still underway and final Census results would only start emerging from mid-2017.
Today, I want to take up the opportunity to provide an update on the Census 2016, and how it ended up. I want to also take the opportunity to highlight further leadership learnings from the ABS journey, including the unexpected and ambitious task of conducting the Australian Marriage Law Postal Survey in less than 100 days in the second half of 2017, and how this has shaped our ongoing organisational transformation.
How did the 2016 Census end up?
Each five yearly population Census is one of the ABS’ most important statistical collections.
Each Census is used to re-benchmark our population estimates that drive GST distributions and electoral distributions. Updated population projections feed into future infrastructure and local service planning. The Census provides a vast array of information on how Australia, and importantly how small areas and small population groups have changed over time, not available from other data sources.
Now that we are in 2019, what was the outcome with the 2016 Census?
The 2016 Census has provided us with an accurate and updated snapshot of how the nation has changed over recent years, as was the case with the 2011 and 2006 Census collections.
Whether the Census succeeds or fails is ultimately determined by the community response to the Census over the entire eight week collection period and the commitment and expertise of the ABS to collect quality information from the community and subsequent actions to process and release quality data.
From research tracking community sentiment towards the 2016 Census, 97-98% of the community were committed to completing the Census fully and accurately, even while the Census privacy controversy raged in late July and early August. The proposed disruption to the 2016 Census from some groups across the community (privacy advocates, some journalists and a small number of politicians) did not translate into non-completion by the broader community.
The ABS sets an expectation for the anticipated Census response, over the eight week collection period, through the respective phases of Approach (end July to 9 August), Remind (10-25 August) and Visit (26 August to end September). The Census cumulative response temporarily dipped below our expectations from 9 August for a number of days as the online form was unavailable. However, the total Census response was back on track by 13 August, and for many subsequent weeks was more than a million household responses ahead of schedule.
Each Census includes an extensive quality assurance process before the final numbers are released.
Some of the key metrics on Census data quality from 2016 were that:
- The overall Census household response rate of 95% in 2016 was comparable to that achieved in recent Australian and overseas Census collections.
- The estimated total net undercount of 1.0% in 2016 was the lowest level recorded across the previous five Australian Censuses.
- The 2016 Census still achieved around an 80% increase in online Census responses compared to the 2011 Census, delivering improvements to overall Census data quality.
- The 2016 Census had fewer hard refusals from households to complete the Census than was the case in 2011.
Given the process difficulties with the 2016 Census collection, one of ABS’ most important decisions, in August 2016, was to establish an Independent Assurance Panel to report on the quality of 2016 Census data.
The Panel comprised domestic and international experts. They were given full access to ABS internal information and to ABS staff. They produced a comprehensive report, transparently and fully published in June 2017, coinciding with the first major release of 2016 Census data.
The Census Independent Assurance Panel found that
- “… the 2016 Census is fit-for-purpose and can be used with confidence."
- “… the 2016 Census data is of comparable quality to previous Censuses, is useful and usable, and will support the variety of uses of Census data as was the case for previous Censuses."
- “… the 2016 Census continued to collect a comprehensive, detailed picture of Australia’s population... "
- “The Panel examined a number of key topics including population counts, sex, age, income, Indigenous status, country of birth, language, ancestry and family structure. The Panel’s analysis revealed that the levels and distribution of characteristics matched expectations well and were comparable to other independent data sources...”
(Census Independent Assurance Panel 2017)
Other National Statistical Offices around the world are considering whether to establish their own Census Independent Assurance Panel, following the ABS’ lead; Statistics New Zealand did this with its recent 2018 Census. The ABS has already decided it will again establish an Independent Assurance Panel to report on data quality with the 2021 Census.
The first major Census data release took place in June 2017, and traditional media was very enthusiastic in its exploration and use of this new and interesting range of insights about Australia, and how our nation had changed. Further data released in October 2017 focussed more on the employment-related information and journey to work data. There were more data releases through 2018, including homelessness estimates and updated socio-economic indexes for Australia (SEIFA), and enhanced linked data resources, in order to draw new statistical insights for policy or evaluation purposes.
While we have seen considerable use of the quality 2016 Census data to date, there is plenty more information for expert data analysts to still exploit.
From 2016 Census to 2017 Australian Marriage Law Postal Survey (AMLPS)
2016 Census was a “teachable moment” for the ABS. The 2016 Census process and surrounding experience provided many insights to us, and other organisations, around cyber security, risk management, community service expectations and stakeholder and media engagement.
The ABS leadership group recognised that the ABS should quickly learn the lessons from the 2016 Census. These valuable insights accelerated and reshaped our ongoing ABS Transformation program.
In hindsight, this was the best approach we could take. Do not waste a good crisis, and do not obfuscate the key lessons, as this will slow down progress building organisational capability.
This can be challenging in the current political and media environment where there appears to be no tolerance for any errors.
Then on 9 August 2017, exactly a year to the day from the 2016 Census reference night, the Treasurer directed the ABS to conduct the Australian Marriage Law Postal Survey (AMLPS). This Direction was subject to two High Court challenges, both rejected by the Full Bench 7-0 within a month.
Following High Court agreement that the AMLPS could continue, stakeholder and media perspective towards the AMLPS was generally very positive. Active engagement of ABS with key stakeholders and media was crucial during this time to build and maintain support.
The Australian Government had set a very ambitious time frame for completion of a survey, with reporting of results required in less than 100 days, without strong appreciation of the complexity of running such an exercise. The ABS process also was starting from scratch with little forewarning of the task that we were about to be given on 9 August 2017.
Following the 2016 Census experience, the challenge was not just to deliver quick survey results, but to deliver results that would be trusted by the public and their elected Federal Parliamentarians. To do this, the ABS needed to ensure the process was as smooth as possible for the public, taxpayer funds were used efficiently, and to deliver quality results with integrity.
This was a project with considerable risks and a very high public profile, but also where innovation was the key to delivering a quality survey outcome in the available time. The 2017 AMLPS provided a great opportunity for the ABS to work differently, and to embed new ways of working. Use of past ABS operating approaches would have increased risks, and reduced the likelihood of success. Our learnings from the 2016 Census experience were key to a successful AMLPS just a year later.
More broadly, the AMLPS demonstrated that the public service can deliver an extraordinary service.
With the AMLPS, we were able to
- Use Agile work methods, critically drawing more on the expertise of Executive Level and junior staff, driving accountability and collaboration among teams across the ABS and other partner organisations, with stand-ups each morning and afternoon increasing the transparency and cadence of decision making, key dependencies and outcomes;
- Have necessary community awareness material, encouraging electoral roll registration in the first instance, operating in the media within days of project initiation;
- Focus on delivering smooth, timely services for eligible Australians, including through user centred design and service automation;
- Complete a number of high value, high risk but compliant procurement processes within days that led to the engagement of Fuji Xerox and Amazon Web Services;
- Leverage effective working relationships with the Australian Electoral Commission, Department of Human Services, Finance, Australian Signals Directorate/Australian Cyber Security Centre, Australia Post and others to deliver AMLPS outcomes;
- Introduce new and improved practices with stakeholder engagement, media engagement, risk management, issues management and privacy by design.
ABS has prepared a Conduct Report on the AMLPS. The Nous Group were engaged to draw further insights through an independent evaluation of the 2017 AMLPS, available for the ABS and others.
Preparing for the 2021 Census
Preparations for the 2021 Census, already well progressed, are drawing upon learnings from the 2016 Census, the 2017 AMLPS and our organisation-wide transformation.
We have set three key objectives for the 2021 Census:
- Smooth running: The census experience is easy, simple and secure
- Strong Support: Governments, businesses and the community have confidence in the Census and there is a high level of community participation
- High quality data: Census data is high quality and widely used to inform on areas of importance to Australia
Many changes to the Census process have been introduced for 2021, including:
- More extensive governance oversight compared to past Censuses, commensurate with the scale and importance of the Census. The Australian Statistician now chairs the Census Executive Board, which also includes external members. Independent program assurers have been engaged, and we have a more rigorous approach to program and risk management;
- Adopting user centred design so the Census process better meets citizen expectations, already influencing proposed self-service and contact centre approaches;
- Conducting full open tender procurement processes for a secure, simple and smart digital service and recruitment of a 30,000-strong temporary workforce;
- Cyber security expertise from across government has been actively drawn into preparations for the 2021 Census, in contrast to the more limited engagement from key agencies prior to the 2016 Census DDOS attack;
- Using agile work practices where appropriate, building on the AMLPS experience, to increase the cadence of program activity and decision making;
- Building preparedness for key issues that may emerge through the Census implementation process;
- Additional funding was provided in the 2019-20 Budget to secure the Census on-line system, but ABS still has concerns over the adequacy of funding to deliver a good Census response from across the community and a better response from key disadvantaged groups such as Aboriginal and Torres Strait Islanders in 2021.
Overall, ABS is doing what it can to deliver a quality 2021 Census process, mindful of the critical uses to which this data is required and used.
Key learnings I have taken
There have been many learnings for myself, for ABS, and others from these experiences.
Over time, I have also reflected on these experiences and learnings, so the perspective you are getting today has moved on from what it was at the end of 2016 (Kalisch 2016), or even the end of 2017 (Kalisch 2017). As leaders, we should never stop considering how to improve our organisations.
I want to highlight four main learnings that I have taken from the experience of the past four years:
1. Eyes on the prize
In many ways, every population Census is too important to fail, given the many critical uses of Census data across our nation, from fair electoral distributions and financial grants, through to effective planning of infrastructure and services, through to informed population estimates and projections, and much more.
Each Census is a major delivery on the long-standing purpose of the ABS to inform Australia’s important decisions.
The information derived from each Australian Census is now greater than was the case with the Census conducted just twenty years ago. Data integration techniques, and the information that can then be derived from joining the Census data with other information, has enhanced the potential statistical insights to the community from each Census collection.
This emphasises the importance of conducting a Census every five years in Australia, rather than following the international standard of a 10-yearly Census collection, to more regularly update some key outcome measurements across our nation.
Against the backdrop of the privacy debate in the media and social media and then the very regrettable unavailability of the on-line form for nearly two days, the ABS was focussed on strategies that would produce the best data from the 2016 Census process.
This included more closely managing key Census processes, shifting our Census field force to areas with lower response, taking necessary steps to prioritise Census related activity in the ABS, and effectively supporting our Census staff working long hours in a challenging public and social media environment.
Some of the leadership behaviours we found particularly important were honesty and authenticity, trust in our expert middle managers and their staff to deliver, increased accessibility to staff who are experiencing a range of emotions and pressures, leaders removing blockages to progress, and helping inform key strategy calls across an organisation that was facing increased public scrutiny.
2. Learn fast, learn more
With the Census, we had a number of reviews (MacGibbon, Senate Inquiry) undertaken within weeks and finalised within a few months of the collection process issues in August 2016.
Overall, these reviews documented some of the challenges of delivering a modern Census, and delivered some recommendations for ABS and across Government.
However, the timing established for these reviews required them to draw their conclusions while the Census process was still only part way through. Census data quality assessments were still to be undertaken, with Census data due to be released in the following year and beyond.
While the Government and ABS responded to the respective recommendations, ABS also took the time over coming months to draw on other perspectives, reflect on our own performance and make further changes. This had the advantage of considering the entire Census process.
Overall, this did provide the advantages that:
- ABS captured a number of the key learnings within a matter of a few months of the Census process difficulties, helping us deliver quality data from the 2016 Census and positioning us well for the unexpected task of delivering the AMLPS;
- ABS has drawn more learnings from these experiences over time and built them into our ways of working, with benefits to our production of over 500 statistical releases each year and preparations for the 2021 Census; and
- ABS has sought out and received greater support from partners across Government and necessary external suppliers across our work program.
Ultimately, everything we do should be a learning experience, and one of the biggest challenges for all organisations is making sure we better capture, share and enshrine these learnings within our organisational DNA over time.
3. Organisational transformation is a dynamic, full on contact activity!
The ABS had initiated a comprehensive transformation program in 2015, following a number of reviews in 2013 and 2014 which concluded that the ABS needed to markedly improve its organisational performance.
This transformation was established to be wide ranging – covering our environment, strategy, governance, workforce, culture and infrastructure.
This Census experience did highlight that, in mid-2016, there was still a considerable performance gap in our governance, culture, workforce and partnerships if the ABS was going to achieve the outcomes expected by our key stakeholders and ourselves.
As a result, we recalibrated our transformation efforts in many ways:
- Our partnerships with policy agencies were taken to a new level while we also progressed opportunities for improved collaborations with service delivery agencies inside and outside of government;
- We have broadened access to more independent voices on key program boards and advisory groups, to act as critical friends, and boosted our use of independent assurance to test claims of external suppliers and internal managers;
- We improved the maturity of our risk management practices and developed more comprehensive contingency plans and responses in the event that our best laid plans did not eventuate;
- We have more actively encouraged innovation to be generated across our workforce;
- Our key leaders have learnt from the 2016 Census and 2017 AMLPS experiences, and we have better equipped our middle managers to play their necessary leadership roles;
- Our workforce has been further broadened and refreshed in some key areas, to include more of the range of skills and experiences valuable for a modern information agency;
- Our infrastructure transformation has placed greater attention to ongoing management of statistical risk, reflecting the potential risks to ABS reputation and community trust in ABS statistics if we had a major statistical incident.
You need regular communication with your staff to listen, adopt, respond and persuade – they have valuable insights and are also the main conduits through which organisational transformation takes place.
You also need to engage with key stakeholders to understand and shape expectations and opportunities. There are also opportunities to engage with the community, including through media and social media, to help build trust, and respond to any misunderstandings.
4. Capitalising on a new information age
This is an exciting time to be working with data, while also recognising we are confronting new risks.
There are many opportunities and challenges for national statistical agencies, such as the ABS.
More data is being produced in this digital age. New information sources are emerging, colloquially captured under the title of big data, including government administrative data, sensing data such as satellite imagery, and commercial information from retail and housing sectors as a few examples. Many of these new data sources can be useful, while some “data” should clearly come with consumer warnings about its accuracy and utility.
Technology is enabling us to use large data sets more easily and draw new information insights from linked data. Data can be a powerful tool enabling evidence based policy and rigorous evaluations, and data is no longer the main constraint to evidence based decisions.
On the challenges front, we are seeing increased community concern around data security following a number of (largely private sector) data breaches and poor data practices, by organisations including Sony, Verizon, Red Cross, Facebook, Cambridge Analytica, etc. Cyber incidents and low levels of trust in governments, businesses and NGOs provide further challenges.
Providing some contrast, the Productivity Commission report into Data Availability and Use highlighted the value for the community from more effective use of data (Productivity Commission 2017). More effective use of data, and especially its evaluation and policy insights, can contribute to more effective public services and better use of taxpayer funds.
The ABS continues to enhance what we do and how we work. We have improved our measurement of the labour market, economy and population, have enabled broader but still safe access to our national data resource through application of the Five Safes Framework and our Virtual Data Lab, and our data linkage resources are contributing significant new policy insights.
Following substantial changes to what ABS does and how we work over the past few years, the ABS will need to keep transforming over coming years to ensure we are still well placed to meet our nation’s continuing and emerging data needs.
Concluding Remarks
A major contemporary development has been reduced trust in governments, business, NGOs and the media, as measured by the Edelman Trust Barometer across a large number of countries.
Trust is critical for many organisations to operate, including across the public sector.
ABS relies on the trust of households, businesses and other data suppliers to provide us with accurate and sometimes very sensitive data so we can produce Australia’s official statistics. ABS carefully and transparently uses this information to produce quality statistics on the economy, our population, society and environment. ABS information is made available to expert analysts and researchers, to enhance Australia’s understanding of our collective situation, while also ensuring the secrecy of individual and business information.
Public trust in the integrity, accuracy and security of data by the ABS ensures that this information is available and used to inform important decisions by governments, business and the community. Public trust in ABS data supports our democratic processes and institutions.
ABS is not immune from the questions of trust. As noted earlier, data breaches and cavalier data practices of some companies or organisations can reduce public trust that their data will be kept safe, even by professional data organisations such as the ABS.
As the ABS found, problems with our 2016 Census collection process led to reductions in community trust in the ABS, at least temporarily. The quality of the service experience we deliver to the community does matter, not just the quality of the final data.
The good news from the ABS experience is that public trust can be rebuilt, as we saw when it became more widely understood that the Census collection was still of high quality and when quality data from the 2016 Census was released in the following year and beyond. Response rates to our other business and household surveys have remained at internationally high levels. Conduct of the Marriage Law Survey in late 2017, with its seamless process and delivery of trusted data that enabled swift Parliamentary resolution of this issue, also helped rebuild public trust in the ABS, now back to pre-2016 levels.
Building and maintaining trust will be a major ongoing challenge for public agencies, including the ABS. As the environment in which we operate and as community and government expectations change over time, I expect that the dimension of public trust, and how it affects the delivery of public services, will receive greater attention over coming years.