Raising Survey Response Rates by Using Machine Learning to Predict Gold Providers

Maintaining high response rates for surveys is becoming more difficult for the Australian Bureau of Statistics (ABS) and other National Statistical Organisations. With limited budgets for data collection, this has led to a search for more effective strategies for following up respondents. This paper focuses on predicting survey respondents who will complete their survey without requiring any follow-up calls – hereinafter referred to as Gold Providers (GPs). Accurately predicting GPs enables follow up efforts to focus on the rest of the providers – those for whom follow up is likely to improve the likelihood they will respond. This responsive data collection protocol of allocating follow up resources is called the GP strategy.

This paper examines a live trial of this GP strategy for the 2018-19 cycle of the Rural Environment and Agricultural Commodities Survey (REACS), one of the ABS surveys struggling to achieve its target response rate. Two approaches were used to predict GPs: a rule based descriptive approach and a model based response propensity approach. The model based response propensity approach used a machine learning method called the random forests with regression trees method.

In the live trial, the machine learning approach outperformed the rule based approach by more accurately predicting GPs and non-GPs, and enabling more flexibility to set the required proportion of GPs in the full sample.

Key words: Gold Provider strategy; Intensive Follow-up; response propensity; machine learning; random forests method

Maintaining consistently high response rates for surveys is becoming more difficult for the ABS and other National Statistical Organisations. This fact, coupled with the increasing cost and constrained budgets for data collection, has led to these agencies searching for more effective follow-up strategies that aim to increase the response rates of those that will help to decrease non-response bias.

Where the target survey variables of interest are statistically independent of response propensity, this can be achieved through a responsive data collection protocol called Gold Provider (GP) strategy that strategically delays and redirects what is known in the ABS as Intensive Follow Up (IFU) efforts (i.e. telephone calls) otherwise spent on GPs to non-GPs during the GP strategy period. Here GPs are the survey respondents who self-respond, i.e. complete their survey without requiring any follow-up calls. During the GP strategy period, the non-GPs will have normal follow up calls being undertaken towards them, while the GPs will on purposely not be followed up, and the follow up calls saved from them will be re-allocated to the non-GPs. Once the GP strategy period has ended, the GPs that haven’t self-responded will be followed up as are the non-GPs. Given the key element of the GP strategy is to delay rather than stop or cancel IFU resources towards the GPs, conceptually it won’t cause any additional non-response bias to the estimates, and therefore won’t pose any significant statistical risk, if any, on data quality. On the other hand, the saved calls from delayed GP follow up can be used to follow up non-GPs, thus helping improve response rates and reduce non-response bias.  Therefore, this GP strategy ultimately aims to direct follow up efforts most efficiently to improve overall response rates with no reduction, or even improvement, on data quality.

Conducted annually, the Rural Environment and Agricultural Commodities Survey (REACS) is one of the ABS business surveys that have been faced with the difficulty in achieving their target response rates. For the REACS, the IFU period is around 3 months with 3 important milestones, namely the second and third Reminder Letter and the end of the IFU. We should note that the first Reminder Letter is not considered as a critical milestone due to its early occurrence. Throughout the IFU period, the key IFU strategy is to prioritise the IFU resources (i.e. calls) towards non-respondents in the Completely Enumerated (CEd) sector, followed by the sampled sector. Here the CEd sector refers to the one that contains the respondents that have a selection probability of 1 because they have a significant impact on the quality of the estimates. Nevertheless, within the CEd sector and likewise the sampled sector, there is an implicit assumption that all respondents have an identical propensity to respond and require equal resource to respond. Therefore, resource allocation intensity within the sectors is not differentiated for different respondents.

Clearly this is a less cost-effective IFU strategy than the GP strategy. To illustrate the efficacy of the GP strategy, we conducted a live trial on half of the sample for the 2018-19 cycle of REACS.

This paper provides an overview of the methodology used to predict and select the GPs from the 2018-19 sample, the setting up of the live trial, and assessment of the efficacy of the GP strategy in terms of prediction accuracy, cost saving as well as impact on data quality.

The key to the success of a GP strategy is to accurately identify GPs. There are different ways to achieve that. A straightforward approach is to use a rule based descriptive (RBD) method to define and predict GPs based on their response behaviour in previous survey cycles. This has historically been used for a limited number of business surveys in the ABS.

Alternatively, since model-predicted survey response propensities have been used to develop data collection protocols (McCarthy et al., 2009; Peytchev et al., 2010; Earp et al., 2013; Buskirk et al., 2013; Phipps and Toth, 2012; Wilson et al., 2015; Plewis and Shlomo, 2017), a model based response propensity (MBRP) method could also be tailored to define and predict GPs. For the live trial, both the RBD and MBRP methods have been assessed to compare their performance in accurately predicting GPs.

To develop the rules for RDB or train the algorithm for MBRP to predict GPs for the REACS 2018-19 cycle, we used a dataset that pulled together survey information from 4 historical survey cycles ranging from 2014-15 to 2017-18. The consolidated data set comprised such survey information as the response status during each IFU stage (i.e. the second and third Reminder Letter and the end of the IFU), the final response status, the final response date, total number of Reminder Letters sent to a respondent before response, number of calls made to a respondent during each IFU milestone period, etc. After data consolidation and confrontation, a total of 157,000 observations from around 100,000 respondents (many of them occurred in more than 1 cycle) were included in the data set.

It is worth noting that only the survey respondents of 2018-19 cycle that have historical information in the consolidated data set will be predicted with their GP status, and the ones that were newly selected into the 2018-19 cycle or did not have complete historical information were automatically classified as non-GPs.

With the previously used GP strategy in the ABS, GPs were defined as survey respondents who completed their survey without requiring any follow up in the previous survey cycle (referred to as "Definition 1").

For this live trial, a couple of alternative definitions with a specific relaxation on the rule relating to the number of calls at different milestones were also explored. These were:

• Survey respondents that responded before Reminder Letter 3 with 2 calls or less in the previous survey cycle (referred to as "Definition 2")
• Survey respondents that responded by IFU end with 2 calls or less in the previous survey cycle (referred to as "Definition 3")

A retrospective analysis was conducted on all three definitions from the perspectives of prediction accuracy on GPs, and potential savings on the number of IFU calls.

From the perspective of prediction accuracy, the accuracy level of each definition in correctly predicting GPs was assessed. That is, to decide whether they are true GPs or not for the REACS survey cycle of 2015-16, the set of respondents that were predicted as GPs based on the chosen definition using information from survey cycle of 2014-15, were assessed against their actual response status and attempted IFU efforts. The results showed that Definition 3 provided the most accurate prediction of GPs with an accuracy rate over 80% against 75% and 70% of Definition 1 and 2 respectively. The assessment for the REACS survey cycles of 2016-17 and 2017-18 showed consistent results.

From the perspective of cost saving, potential call savings from not following up GPs that were predicted by each definition were assessed. The results showed that the number of calls that would have been saved by using Definition 3 are larger than those from using Definitions 1 and 2. More specifically, compared to Definition 1, the percentage of calls that would have been saved from Definition 3 is 6.3%, 10.5% and 8.5% more for cycle 2015-16, 2016-17 and 2017-18 respectively. And compared to Definition 2, the percentage of calls that would have been saved from Definition 3 is even bigger, namely, 7.8%, 15.4% and 13.2% more for cycle 2015-16, 2016-17 and 2017-18 respectively.

Based on results from the retrospective analysis, Definition 3 was chosen as the final RBD definition. That is, the GPs are survey respondents that responded by the end of IFU with 2 calls or less in the previous survey cycle.

The random forests with regression trees method

A number of methods to model response propensities have been advocated in the literature. Traditionally, response propensities are estimated by fitting a logit or probit regression model (Black et al., 2010; Peytchev et al., 2010; Chen et al., 2012; Whiting and McNaughtan, 2013; Plewis and Shlomo, 2017). With a logit or probit regression model, there are multiple validating measures to test the robustness and fitness of the model results. However, Earp et al. (2013) pointed out that logit regression requires the analyst to hypothesise variables thought to be associated with non-response and then use the observed data to fit the model parameters. Therefore, there is a risk that these “explanatory” variables may be mis-specified or under-identified, and the logistic regression models are often difficult to interpret because of interactions between the characteristic variables.

Newer machine learning (ML) techniques for classification and predicting including classification and regression trees (CART) (Phipps and Toth, 2012; Valiiant et al., 2013; Earp et al., 2013; Toth and Phipps, 2012; Buskirk and Kolenikov, 2015; Lohr et al., 2015; Wilson et al., 2015) and random forests (Brieman, 2001; Buskirk et al., 2013; Buskirk and Kolenikov, 2015) have been proven to be powerful tools to predict survey response propensities. Whilst these ML techniques also suffer from model misspecification if the explanatory variables are not correctly and comprehensively identified, they are superior to logistic regression models because they do not require the assumption of linearity in the modelling. More importantly, the automatic interaction detection inherent in trees provides a straightforward method to account for and easily interpret interactions between auxiliary data and paradata and the propensity to respond (Earp et al., 2013; Toth and Phipps, 2014; Buskirk and Kolenikov, 2015).

Of these different ML techniques, the random forests method is an example of a nonparametric “ensemble” tree-based method because they generate estimates by combining the results of several classification or regression trees rather than using the results of a single tree. By aggregating estimates across many trees, random forests tend to generate more stable estimates and with less variance as compared to those generated from a single tree as it overcomes the associated problems of overfitting by using bootstrapped datasets and limiting the number of features selected by the algorithm at each node (Brieman, 2001).

A forest of classification trees and a forest of regression trees are the two main propensity estimation methods that have been developed in random forests. For this live trial, the random forests with regression trees method was chosen as the MBRP approach as it can generate continuous response propensities.

Selection of predictors

The predictors selected to be included in the random forests play a vital role in model's fitness and prediction accuracy. Therefore, efforts are required to select relevant predictors to further improve the prediction accuracy.

Standard response models which only include survey variables as predictive variables have been shown by the literature (Durrant et al., 2017) to perform poorly in terms of prediction. Instead, an ABS study (Black et al., 2010) recommends using a framework that covers 6 main categories, including area characteristics, business characteristics, survey design features, respondent characteristics, interviewer characteristics, and interviewer observations. The first three categories are survey variables, while the last three categories are referred to as paradata. Using this framework, descriptive analysis was conducted to determine which survey variables and paradata items to be adopted as predictors for the live trial. These included state, industry, size, significance level and weighting contribution towards estimates (as the survey variables) and number of calls being made and reminder letters being sent (as the paradata). Due to lack of data, characteristics from the categories of interviewer characteristics and interviewer observations were not able to be included as predictors.

Based on the final definition for each approach, the total sample of 27,159 respondents for REACS 2018-19 cycle were predicted as GPs or non-GPs. It is worth noting that there were respondents whose GP status couldn't be predicted by either approach due to lack of historical information. These respondents were automatically classified as non-GPs.

The GPs predicted from both approaches for the 2018-19 cycle are listed as follows:

• 10,722 GPs were predicted under the RBD definition
• 12,452 GPs were predicted under the MBRP definition (with the averaged response propensity score of 0.85 as the cut-off threshold); and
• 7,955 GPs were commonly predicted across both definitions.

In order to simplify workflows and have a large enough treatment group, it was decided to classify 40% of the total sample size as GPs using a combination of RBD and MBRP approaches. GPs for the trial were thus comprise of:

• All of the 7,955 respondents that were commonly predicted as GPs under both definitions
• A random sample of 1,454 respondents uniquely predicted as GP by the RBD definition; and
• A random sample of 1,454 respondents uniquely predicted as GP by the MBRP definition.

This collective way of recognising GPs for the live trial is called the GP_final approach. The other 60% of the total sample, including 1,718 CEd respondents and 14,155 sampled respondents were treated as non-GPs for this live trial.

Both the predicted GPs and non-GPs were evenly split into two homogenous sub-groups – a control group and a treatment group – making sure to balance the number of total respondents, taking into consideration the similarity of specifications including the number of total new respondents, the number of total CEd respondents, the number of total GPs, and the number of CEd respondents that are GPs. It is worth noting that the even split was conducted at stratum level and the REACS sample is stratified by characteristics including size and geographic location. Therefore, the distribution by these characteristics between control and treatment group should also be similar.

The detailed composition of the allocated control and treatment groups is provided in Tables 1 and 2 below.

For the live trial, while the control group would have "normal" IFU follow up being undertaken towards them throughout the entire data collection period, the treatment group was set up to be implemented with the GP strategy, with the aim to increase the overall response rates using the same IFU resources. The components of this were as follows:

• The GP strategy period was set to last for about two months starting from the beginning of the IFU until the sending out of the third Reminder Letter
• During the GP strategy period, while the non-GPs will have "normal" IFU action being undertaken towards them similar to the ones in the control group, i.e. being followed up with calls, the GP units will not have any IFU action being undertaken. Additionally, the IFU resources saved from these GP units will be re-allocated to the non-GPs; and
• After the GP strategy period, all non-respondents, whether GPs or not, will be resumed with "normal" IFU action until the end of IFU period, the same as for the control group.

The live trial of the GP strategy was successfully conducted under the guideline of a rigorous implementation and monitoring framework. Upon its completion, an assessment was made by comparing the performance of treatment and control groups from three main perspectives, namely the prediction accuracy, cost-effectiveness and the impact on data quality. The assessment aimed to evaluate whether the GP strategy was effective in increasing the overall response rate by delaying the IFU actions towards the GPs and re-allocating these additional IFU resources to non-GPs during the GP strategy period. Additionally, the assessment was critical in informing the decisions of whether and on what scale to adopt the GP strategy on an ongoing basis for REACS, and identifying any future improvements that could be implemented to enhance its efficacy.

To evaluate the success of the GP strategy, the first and foremost aspect was to assess how accurately the GPs were predicted from a retrospective perspective.

Table 3 below presents an overview of GP status defined by different approaches, as compared to the actual GP status for the 2018-19 cycle (numbers in the final row). Here the column "Unclassified" refers to the situation where a respondent's GP status couldn't be predicted by either approach due to lack of historical information. From Table 3, we can see that the population of the actual GP was about 60% of the total sample, 20% higher than the 40% that we set up. And the GP_MBRP only approach was able to identify closer total GP numbers. Additionally, it had much smaller number of unclassified respondents compared to the RBD only approach. 

Table 4 below presents the prediction accuracy of different approaches in rate terms. From the results we can see clearly that all approaches have achieved high accuracy in predicting GPs, with an accuracy rate ranging from 76% to 80% respectively. We can further observe that the GP_MBRP only approach outperformed the others in achieving high accuracy rates for predicting both GPs and non-GPs.

To further compare the prediction accuracy between the RBD and MBRP approaches, the 10,863 respondents that were predicted as GPs using the GP_final approach were broken down by the approach from which they were predicted, and their prediction accuracies were assessed against their actual GP status. This breakdown is presented in Table 5 below, from which we can observe that of the 7,955 GPs that were commonly predicted by both approaches, 6,495 were actual GPs, resulting in an accuracy rate around 82%. While of the two sets of 1,454 GPs that were uniquely predicted by either approach, the one predicted by the MBRP approach achieved a much higher accuracy rate than the one of the RBD approach, which again proved the higher prediction accuracy of the MBRP only approach.

From the retrospective analysis conducted above, we concluded that the GP identification approach that was implemented for the live trial was a successful one.  In addition, it could potentially be further improved by adopting the MBRP only approach as it outperformed the others in predicting both the GPs and non-GPs. Moreover, the MBRP only approach was more flexible in allowing the user to specify the overall GP proportions by adjusting the cut-off threshold of the predicted response propensity scores, and thus was more adaptable to changes and improvements based on historical information. As mentioned above, the actual GP respondents were about 60% of the full sample, rather than the 40% that we set up for the live trial. Therefore, for next cycle, to reflect this actual GP status, we should lower the cut-off threshold from 0.85 to for example 0.75 to allow more GPs to be predicted by the MBRP approach.

Given that the IFU resources remained the same and the main objective was to increase the overall response rate, it is critical to analyse the success of the GP strategy from the cost-effectiveness perspective, more specifically, the response rate achieved (i.e. effectiveness) and the IFU resources allocated as indicated by calls during the IFU period (i.e. costs) for the GP and non-GP respondents between the control and treatment group. From Table 6 below, we can see that the treatment group achieved a similar response rate compared to the control group with 708 less calls made to GPs. It was also an obvious observation that the GPs in the treatment group achieved a much smaller rate of average calls per response, compared to the GPs in the control group. These were strong indications of the effectiveness of the GP strategy in generating savings in IFU resources.  However, the live trial also demonstrated that diverting saved resources to non-GPs was not cost effective.  As seen in Table 6, there was hardly any improvement on the non-GP response rate even with an increase of over 900 calls to them.

Table 7 looks further at the split of non-GPs between the control and treatment groups in terms of the response status and the IFU resources allocated.  We can see that for the treatment group, an additional 875 calls (228 plus 647) unexpectedly resulted in an overall smaller number of responders of 24 (38 minus 14), as compared to the control group. We can also see average number of calls allocated to the respondents within the treatment group was bigger than those of the control group, without an overall higher response rate being achieved.

Results from both Table 6 and 7 suggested that, for the treatment group, the calls that were saved from the GPs and redirected to non-GPs did not result in a better response outcome. This may have been due to the ineffective re-allocation of these resources to non-GPs. Therefore, to conduct the re-allocation more efficiently, the additional resources should be targeted on specific non-GPs that are more likely to respond with more intense follow up calls, i.e. the ones with relatively higher predicted response propensities. To validate this proposed approach, the response rate achieved and the IFU resources allocated of the non-GPs between control and treatment groups were examined further by their response propensity score ranges as presented in Table 8 below.

From Table 8 we can see, as expected for both the treatment and control groups, there was strong correlation between  response propensities and response rates. This again was a strong indication of the high predictive power of the MBRP approach in predicting GPs. Similarly, for each response propensity score range, we can also observe a positive relationship between the percentage of respondents and calls for both the treatment and control groups. We note, however, with the treatment group, the calls that were allocated to the respondents within the response propensity score ranges that are annotated (a) and (b) respectively in Table 8 were disproportionally bigger or smaller as compared to those of the control group, without converting to a commensurate higher or lower response rate. This suggest that if these additional calls had been re-allocated towards the non-GPs based on their predicted response propensity score ranges from highest to lowest, the overall cost-effectiveness could have been improved as indicated by the results of the control group. It is worth noting that the result in Table 8 should be interpreted with the focus on the RP score ranges that have substantive number of units, i.e. the ones starting from (0.2, 0.3].

To test this assumption, potential new response rates for each score ranges of the treatment group were also simulated, assuming that the treatment group's response conversion per calls remain the same, and call distribution towards non-GPs was aligned to those of the control group. Simulation results as presented in Table 9 suggest that with the same total number of 3,906 calls being made towards the non-GPs within the treatment group, the overall responders could potentially be increased from 5,025 to 5,189, resulting in an overall response rate increase of 2%. It is worth noting that these simulation results were conservative as theoretically the new approach of targeted call re-allocation should be more effective than the one of the control group.

To summarise, the assessment of cost-effectiveness suggested that the GP strategy was effective in saving IFU resources from the GPs by delaying the commencement of IFU resources towards them, without causing a reduction in response rate for these respondents. This result was a strong indication of the success of the GP strategy in diverting resources away from respondents that didn't need IFU. However, the assessment also indicated that the IFU resources that were saved from the GPs and redirected to non-GPs did not result in a better response outcome due to ineffective re-allocation of these resources. A further simulation study showed that if these saved resources were directed to non-GPs in the same manner as the control group, a higher response rate by 2% could have be achieved. This suggests that, if the saved resources were directed to only non-GPs with the highest response propensity, even more gains in response may be achieved. Further research should be conducted to develop an effective deployment of the saved resources to non-GPs with a view to reducing non-response bias.

As already stated in the introduction section, given the key part of the GP strategy is to delay rather than stop IFU resources towards the GPs, conceptually it shouldn’t cause any non-response bias to the estimates, and therefore won’t pose any significant statistical risk if any on the data quality of the survey. To validate this statement quantitatively, we also analysed the estimates between the treatment and control group.

Based on the comparison between the control and treatment group on their estimates produced by states and variables of interest as presented in the scatter plot below, it was found that no systematic bias (i.e. over or under estimation issue) occurred one way or the other. Therefore, we can conclude that the implementation of the GP strategy in the way the live trial was conducted hasn't posed any additional non-response bias to overall estimates.