Survey of Income and Housing 2023-24 Review Report

On 17 July 2025 the Australian Bureau of Statistics (ABS) announced that it will not release statistics from the 2023-24 Survey of Income and Housing (SIH), noting that the data did not meet the ABS’ high standards for official statistics. This paper provides additional background on this decision, summarises ABS investigations into sources of statistical error, and outlines ABS learnings and how they will be applied to future collections.

The ABS has conducted surveys of household income in various forms since 1974, and the SIH has been conducted in its current form since 2003-04. It gathers data on household income sources and amounts, net worth, housing situations, and both household and personal characteristics.

While SIH has been conducted on a 2 yearly basis since 2003-04, the last annual results were released from the 2019-20 cycle. The scheduled 2021-22 cycle was cancelled in December 2021 due to disruption and statistical impacts associated with the COVID-19 pandemic.

The SIH provides high quality and trusted statistics to inform important social and economic policy analysis on issues such as:

• the distribution of income and wealth in Australia
• household experiences of economic hardship
• analysis and modelling of the effects of taxation and income support systems on the wellbeing of people and households
• home ownership rates, housing affordability and the circumstances of owners and renters.

Across its survey program the ABS evolves its collection approach, wherever possible, to improve efficiency and reduce cost, reduce demands on the Australian public to provide their data, improve respondent experiences and take advantage of methodological and technological innovations. While these innovations are not always visible to data users, they are critical to maintaining the quality of official statistics and have been a feature of the SIH program since its inception.

While the high-level design of the SIH has remained consistent since 2003-04, ABS has repeatedly updated the survey to ensure the content remains relevant, that respondent imposition is reduced as much as possible, and that the best methods are applied within budget constraints. In most cases design changes have delivered the intended benefits and improved the relevance, quality and/or timeliness of the information collected.

The ABS introduced a computer assisted web interview (online form) for the SIH 2019-20, which was a major technological and methodological innovation. While some households have continued to prefer a face-to-face or telephone interview, the introduction of an online option is consistent with community expectations for digital engagement and has been welcomed by many respondents.

The introduction of an online option has also allowed the ABS to reduce costs relative to an exclusively face-to-face survey approach. Difficulties in contacting busy households are reflective of a gradual decline in the ability and willingness of households to participate in face-to-face surveys, increasing costs for the ABS. Results from the 2019-20 SIH met the ABS’ high standards for official statistics – including those from the new online form – and were released in April 2022.

The SIH 2021-22 was impacted by the COVID-19 pandemic and subsequently cancelled. Following the COVID-19 pandemic disruption, the ABS began exploring the availability of alternate data sources to meet users’ needs and replace parts of, or all, the SIH. The use of alternative sources by national statistics offices around the world has increased dramatically over the last decade to supplement and replace official statistics. The ABS has successfully applied new data sources across a wide range of statistical products and has championed the benefits to the Australian public of reusing rather than recollecting required data. A successful recent example is the replacement of the ABS Retail Trade Survey with aggregated, de-identified bank card transactions data provided to the ABS from participating banks, along with supermarket transactions and new vehicle sales data to produce the Monthly Household Spending Indicator.

The ABS explored alternative data sources for SIH from November 2021 to November 2022. Unfortunately, this work did not deliver the expected administrative data sources to replace SIH and was closed. Development work then commenced for SIH 2023-24 with compressed timeframes, which limited the amount of design, build and testing before the survey went into the field in July 2023.

The design changes outlined below span the collection approach and sample design, content design, and use of alternative data sources. Unfortunately, some design changes implemented for the SIH 2023-24 reduced data quality significantly.

Given an increasing public preference for digital engagement, ABS made changes to the SIH 2023-24 sample design with the intention of delivering high quality data while making it easier for households to complete the survey. A new approach was implemented whereby the sample was divided into two groups to reduce any potential statistical impact arising from households not responding to a digital survey.

One group was randomly selected for ‘self-enumeration only’ using a digital (web or phone) approach, with no interview follow-up of non-responding households. In the second group, those households that did not complete a web or phone interview were assigned for field follow-up by an interviewer. This sample design allows the ABS to apply statistical adjustments to address potential under-representation of groups that were less likely to provide a digital survey response.

Unfortunately, when implemented, system and business process limitations resulted in some households in the second group not receiving the field follow-up required to support this design. This left the survey results open to bias arising from systematic differences between those who responded to the survey and those who did not. 

Graph 1 presents the SIH 2023-24 responding sample by household tenure type, compared against previous Census of Population and Housing data. This demonstrates that renter households were underrepresented in SIH 2023-24 data, while outright owners were over-represented. This pattern would usually be corrected through statistical weighting techniques. However in this case, attempts to do so adversely impacted the quality and coherence of a range of other statistics, such as income and housing costs.

a. Data from 1996 to 2021 is from Census of Population and Housing

b. SIH 2023-24 unpublished data

c. SIH 2023-24 results are provided to demonstrate data quality limitations and should not be used for statistical purposes

Survey content is reviewed on a regular basis to maintain relevance and inform the highest priority information needs. Further, with the introduction of online survey forms, other changes have been required with the aim of improving respondent experience. For SIH 2023-24, changes were made to question ordering, wording and response options.

While some changes were expected to improve respondent experiences, a number had unintended negative impacts on data quality. For example, the ABS increased the number of questions which were ‘skippable’ for the 2023-24 collection. This change was made to reduce respondent frustration in cases where they had difficulty providing accurate estimates.

Within a traditional face-to-face interview, an ABS interviewer would have the opportunity to support respondents to make a best estimate to some questions (e.g., the value of the family home). This is more challenging using an online survey because the respondent may not know, become frustrated, and will attempt to move to the next question. If they cannot skip the question, then there is a risk of survey drop-out, leading to higher survey non-response.

In this example, the decision to increase the number of ‘skippable’ questions resulted in a larger than anticipated increase in questions not being answered, which had an adverse impact on several critical data items. This outcome shows that the ABS undertook inadequate testing of these design changes within the constrained testing period available for SIH 2023-24.

While Proof of Concept testing during survey development suggested that this was a viable approach, large scale implementation during data processing revealed several challenges. These were largely associated with the scope, coverage, conceptual alignment and timeliness of the available alternative datasets. For example, administrative sources of superannuation unit record data suffer from lags associated with financial year reporting, limited coverage for groups that are new to the Australian labour force (e.g., school leavers, recent migrants) and different reporting requirements for Self-Managed Superannuation Funds. As a result, superannuation account balances and government payment questions were reinstated in the SIH 25-26.

The ABS had planned to process SIH 2023-24 data progressively from the commencement of data collection, allowing an early assessment of data quality. Unfortunately, initial data extraction was significantly delayed by system problems, which in turn delayed the commencement of data processing. SIH 2023-24 data collection ceased on 30 June 2024. The inability to transfer the collected data from the data collection system to the statistical compilation system was not resolved until December 2024.

Once raw data were extracted, various phases of cleaning, coding, editing and amendment, imputation, derivation, data integration and estimation were undertaken consistent with established practice. Preliminary aggregate statistics were then produced from weighted survey data and compared against various other statistical sources to assess coherence and data quality. Detailed, aggregated data comparisons were made against previous iterations of the SIH, the Australian System of National Accounts (ASNA), the Person Level Integrated Data Asset (PLIDA), and non-ABS sources such as the Household, Income and Labour Dynamics (HILDA) survey.

These assessments identified inconsistencies, including:

• Estimates of personal and household income were much lower than expected when compared to previous iterations of SIH or information in the National Accounts and other sources.
• Aggregate liabilities and loan values were lower than expected, and exhibited lower coverage compared to the National Accounts than previous SIH cycles.
• An observed decrease in superannuation income since SIH 2019-20 was not consistent with other sources and population trends.
• Falls in outstanding mortgage values since SIH 2019-20 were also inconsistent with trends observed in other sources.

A comparison of select SIH 2019-20 and SIH 2023-24 estimates can be found in Table 1.

Taken together, these results suggested the SIH 2023-24 results were not fit-for-purpose and led ABS to initiate an internal Quality Incident Response Plan (QIRP) process, led by its Methodology and Data Science Division (MDSD). This formal process includes assembling a multidisciplinary expert ABS team to conduct an intensive review and data assessment. This team was also tasked with identifying potential treatments to address identified data quality issues and support the production of fit-for-purpose statistics.

Following review, the QIRP process confirmed that the observed data quality issues stemmed from two key statistical limitations discussed above, namely:

• A pattern of survey non-response that reduced the representativeness of the results.
• Pervasive question level non-response and response error, largely caused by ‘skippable’ questions in the survey form.

The QIRP panel found that the combined effect of these errors rendered the data unsuitable for its intended purpose.

The QIRP panel considered several potential treatments to address or reduce the impact of the observed quality issues. 

These included:

• Using alternative data sources to calibrate estimates to address the under-representation of key groups of interest. For example, Department of Social Services data sources regarding the number of government recipients and other tenure related benchmarks were investigated but considered not feasible.
• Varying the level of relative statistical influence (weighting) given to the followed-up sample (group two) compared to those not followed-up to address under representativeness.
• Imputation, removal or reweighting of records with an increased occurrence of zero reporting believed to be due to the instrument design.
• Further investigation into the potential use of alternative sources of superannuation data.

While most of these treatments potentially improved the quality of some selected data items, they were not considered sufficient to address the widespread quality concerns observed across the SIH datasets. On this basis, ABS management considered the outcomes of the QIRP and agreed that even with significant additional effort and cost, the SIH 2023-24 dataset would not meet ABS quality standards and could mislead those using SIH results to inform important social and economic programs and policies.

The data quality issues noted above were progressively identified during the development of the SIH 2025-26 collection and informed ABS’ approach to testing, implementing, monitoring and assuring the new collection. These learnings are also informing modernisation of the ABS’ survey program more broadly.

Evaluation reports produced during processing of 2023-24 results were used by the 2025-26 survey development team to pinpoint areas of concern. Solutions to individual form design deficiencies were tested through multiple rounds of user testing, comprising cognitive testing and useability testing with both the public and ABS interviewers. This testing resulted in several improvements to question ordering (identified as a source of misreporting through the QIRP), removal of the ‘skip’ functionality and additional instructions encouraging respondents to provide their best estimates. For critical data items, in-field edits were developed to query unexpected responses. 

Given clear public expectations and preferences, the SIH 2025-26 will continue to offer an online form. However, as noted above, improvements to the new sampling approach are required to maintain statistical robustness in the context of a digital collection. 

Improvements made for SIH 2025-26 include:

• Increasing the proportion of the sample allocated for field follow-up.
• Implementing additional field monitoring and reporting, including clear indicators, thresholds and escalation pathways where issues are identified.
• Additional and more granular management information for weekly reports to improve allocation of work to interviewers to enable prioritisation and maintain representativeness.

Finally, more timely assessment, validation and assurance of survey data has been instituted to support earlier detection of quality concerns. Improvements to data extraction processes are supporting more frequent interim datasets for SIH 2025-26 and allowing rigorous assessment of key issues such as the representativeness of incoming data. Finally, new governance arrangements and additional statistical quality gates have been established through the production process to ensure early detection and escalation of quality concerns.