6227.0.55.002 - Experimental estimates of education and training performance measures based on data pooling, Survey of Education and Work, 2007 to 2010, Sep 2011
ARCHIVED ISSUE Released at 11:30 AM (CANBERRA TIME) 09/09/2011 First Issue
Page tools:
 Print Page
 Print All
| ||
DIFFERENT OPTIONS FOR DATA POOLING The potential advantage of pooling is to increase the sample size and reduce the sampling error of the measures or variables of interest without having to undertake additional and costly sampling. Pooling generally assumes that each survey is an independent non-overlapping sample of the same population and that the characteristics of the population and the variables of interest do not change substantially from one survey to another. The ABS Analytical Services Section conducted an initial assessment on five different options for survey data to be used in data pooling:
Option A - Combining current SEW with one or more previous SEWs Advantages The advantage of Option A is its simplicity. It combines successive surveys that have a similar survey design, scope, coverage, sampling unit, reporting method, mode of survey, reference period and weighting method. All the performance measures of education are included in these surveys, and since 2001, similar wording has been used for the education questions. Given that the education variables are conceptually the same and similarly measured there is little need to harmonise or adjust the measures before pooling. Deriving combined or pooled estimates is simpler as composite or adjusted survey weights for pooled data can more easily be computed. Disadvantages By pooling successive cycles of the same survey we lose the independence of the annual time series for the data unlike the situation where several concurrent surveys are pooled together. It becomes more difficult to interpret the estimate obtained by combining or pooling several consecutive SEWs as estimates obtained in this way are no longer a point estimate but are a smoothed measure of the combined period. Option B - Combining SEW with another MPS Supplementary Survey Advantages Option B allows independent pooled estimates to be provided over a shorter period, for example, one year. Given that the surveys involved are supplements to the MPS they are all based on similar methodology and survey design. Disadvantages The increase in sample size is modest as given the nature of the MPS there is considerable overlap in the sample. The MPS sample is comprised of eight staggered groups which each spend eight successive months in the survey. Each month, one of the groups moves out of the sample and a new group moves in. Each monthly sample, therefore, retains seven out of eight of the respondents interviewed in the previous month. Furthermore, in Option B there are differences in opportunities for engagement in the reference period that must be taken into account with surveys conducted in different months in the year. The SEW for example is conducted in May, when most students will be enrolled or in work, while a survey conducted early in the year may find students between study and work. Moreover, the gains in sample size may differ by age group due to differences in the scope of the surveys with some of the surveys having different sampling distributions. For example, the Job Search Experience Survey (JSE) only interviews persons who are unemployed or employed and started their current job in the previous 12 months. All these issues mean that this approach requires complex re-weighting. Option C - Combining SEW with a Special Social Survey (SSS) Advantages Option C, combining SEW with an SSS, such as the National Health Survey or the Survey of Disability, Ageing and Carers, would increase the effective sample and reduce RSEs. Disadvantages The reductions in RSEs are small while variation in scope, reporting method, response rates, frequency and timing of these surveys require substantial adjustment or harmonisation of data prior to pooling. There is also substantial risk of introducing additional non-sampling error in the combined estimates under this option: e.g. differences in scope/coverage between surveys and the way response variables are defined/measured etc. Furthermore, with these surveys conducted every 3-6 years, pooled estimates would only be provided several years apart. Option D - Combining SEW with the Multi-Purpose Household Survey (MPHS) Advantages Option D involves combining SEW with the MPHS, itself a supplement of the monthly MPS. As the MPHS and SEW are both conducted as supplements of the MPS, they have the same survey design, sampling unit, coverage, mode and weighting method. This option allows the annual time series to be maintained for pooled data. Disadvantages The additional sample made possible by the inclusion of the MPHS is small as it is only conducted on one-third of the out-going MPS rotation group each month and there is overlap between the MPHS sample enumerated from June to December and the May SEW. Option E - Combining SEW with another SEW, MPS Supplementary Survey and MPHS Advantages Estimates from the May SEW would be pooled with an additional SEW conducted in October, and with all MPS Supplementary Surveys and the MPHS conducted in the current year. All these surveys are supplements of the monthly MPS and hence consistent in terms of survey design, sampling unit, reporting method, mode of survey, questionnaire wording and weighting method. Combining the surveys doubles the sample size and has a similar impact in reducing RSEs to Option A. Option E’s advantage over Option A is that it maintains the annual time series in the education data. Disadvantages Option E does no more than double the sample because the October SEW would be carried out only five months subsequent to the May SEW with a consequent overlap of the sample reducing the effective gain in sample size. Option E's main disadvantage is in terms of increased complexity of combining data from different sources with varying scope and enumeration periods. An additional SEW would involve substantial costs, which would have to be user-funded. ABS RECOMMENDED APPROACH FOR SEW DATA POOLING After assessing these options, the ABS concluded that only Options A and E provided a sufficient increase to the effective sample and consequent reduction in RSEs to be worth pursuing for data pooling. Option A, combining the most recent SEW with one or more previous SEWs, was preferred for its overall advantages in increasing the effective sample size and relative simplicity. These were seen as outweighing the disadvantage that annual estimates would not be produced. Moreover there was only limited support from the stakeholders for a user-funded SEW. ALTERNATIVE METHODS FOR COMBINING/POOLING CONSECUTIVE SURVEYS Estimates for a variable of interest can be prepared either by using an aggregate ‘composite’ or a unit level ‘pooled’ approach. Composite estimates are produced as a weighted average of the individual year aggregate estimates of interest, while pooled estimates are derived by merging unit level data from the relevant surveys into a single dataset, adjusting the individual unit weights in the merged dataset and producing estimates using these new weights. The composite approach is used where there are significant differences in population characteristics or the variable of interest. The pooled approach is generally used where the population does not change significantly from one year to the next, where the survey samples are independent of each other and where they measure more or less the same variables. Composite methods are simpler requiring, in theory, less computation and time to produce estimates, while methods based on the pooled approach are slightly more computationally intensive, requiring larger amounts of information and time to produce the estimates. The methods were compared in terms of information and time required and most importantly in terms of accuracy. For reasons of efficiency and theory the primary methods considered were:
Testing the alternative methods A key assumption in data pooling is that the surveys to be pooled are from the same population and that the characteristics of the population and the variables of interest have not changed substantially from one survey to another. This assumption was tested by examining distributions across the surveys for selected variables, testing for the significance of difference in key education measures across the surveys and tests for survey effect for the variables being measured. Over the period 2006 to 2009, education-related outcomes for the major socioeconomic and demographic groups were broadly stable. The major concern was that a MPS sample size decline in 2009 led to a slightly larger error for pooled estimates compared to the case if no reductions had occurred. This should be a passing problem as the full MPS sample was reinstated for the 2010 SEW. A test for survey effect was also conducted to see if the estimates in the pooled data set were affected by the period or the survey in which they were collected. The model results showed no consistent pattern and the survey effects that were found, although significant, were small. Overall the surveys were found comparable and pooling feasible. In preliminary testing with three trial measures both methods produced moderately better estimates (in terms of reduced RSEs) than single year estimates. The simpler composite method generally produced lower RSEs for two measures but higher ones for the remaining one than the pooled method. Nevertheless, the RSEs for the two methods were strongly correlated and the differences between them were on average small. In choosing the optimal method it is important to choose one based not just on the most efficient results but on wider theoretical and practical considerations. Ideally the method chosen should produce reliable and improved estimates without requiring too much information, computation, time and effort. Each method has its own advantages and disadvantages. The composite method appears simple to use and in testing of three trial measures produced on average marginally lower RSEs than the unit level GREG estimator method. The composite method just requires data on the estimates and their associated RSEs. A problem with this method is that the weighting factor has to be calculated separately for each variable included in the analysis and it can be time consuming to implement if many time periods are involved and if more than two surveys are being combined. Furthermore the estimates of the weighting factor under this particular method vary over time, between measures and between the levels at which the measures are produced. This means that estimates are not being produced on a consistent basis and as such comparisons between the values across national, jurisdictional and Socio-Economic Indexes for Areas (SEIFA) quintile level may not be valid or appropriate. This is of particular concern given this study's objective of providing more accurate measures, especially of trends over time, of key indicators of participation and attainment. The GREG method is a more efficient method of combining population samples across surveys. It automatically handles the multiple benchmark classes used by the ABS and pooled data can be weighted consistently with the post-stratified survey weighting procedure used by the ABS for all the independent survey samples. Unlike the aggregate composite estimator method, the GREG estimator method requires only one set of weights for the different pooling periods, so comparisons across measures are seamlessly coherent. Since the weights under this method are calibrated to the benchmark totals of the end-period survey, the reference point for the estimates is the most recent survey. This is in contrast to the composite method where the reference period is difficult to determine or interpret. The way the sample weights are benchmarked under the GREG method also helps to capture the variation in data better, particularly at the jurisdictional level and below, since the calibration of the weight is done at a final level through the use of population benchmarks at jurisdictional and sub-jurisdictional level. The main drawback of the GREG method is that more information, data and computing are required to derive the weights for the pooled dataset with implications for time, effort and resources. At present initial weights have to be derived outside of the final SEW data file and two sets of benchmark data (demographic and labour force) are required before this information can be fed through the GREG SAS macro to derive the pooled weights. However, in reality the task of obtaining the benchmarked data is relatively simple. Since the final survey weights in each SEW file are already calibrated to the relevant demographic/labour force data for that year the required benchmark data for the pooled dataset can simply be obtained from the relevant end-period SEW itself. If the initial weights and their corresponding replicate weights are included along with the final weights and their corresponding replicate weights, which are already in the final SEW data files, no outside data will be needed for analysis. While the GREG method may appear more computationally intensive, requiring more data and information, since the data required are already included in the SEW datasets, then the procedure for weighting and producing the estimates under this method becomes routine. Given its overall analytical and practical advantages, the GREG estimator method has been adopted as the preferred approach to data pooling. The lagged GREG estimator method The lagged GREG estimator method obtains the final survey weights through the calibration of the initial survey weights (which are based on the initial probability of selection) to a set of external population/labour force benchmarks using the ABS GREG SAS macro. With SEW data this involves merging the surveys to be pooled into one dataset and then calibrating in the initial weights in the combined data set to the set of external population/labour force benchmarks for the end reference date. For example if SEW 2009 and SEW 2010 are to be pooled then the two datasets are merged together into one dataset and their initial weights then calibrated to the population/labour force benchmark totals of SEW 2010. For the SEW the following two benchmarks are used to produce the weights using the GREG SAS macro:
In brief the steps involved in calibrating weights in the pooled SEW dataset to demographic/labour force benchmarks involves the following steps:
The weights in the pooled data are benchmarked to produce estimates of selected population characteristics of the last surveyed year included in each dataset because this is the year for which estimates are being made. The method is equivalent to the current SEW weighting and estimation method for a single year. However, since the pooled data are two combined years of SEW sample, as such the estimates produced are ‘lagging’ with respect to estimates that may in principle have been produced from a comparable independent survey, of the same sample size of the pooled dataset, conducted in the reference year of the benchmarks. An advantage of the individual year estimate is that it applies to the particular year concerned; whereas a pooled sample reflects the combined behaviour of the population (subject to socioeconomic, migration factors etc) of the years included and hence may be best understood as a lagging estimator for the reference year representing the situation somewhere during the time period between the two surveys depending on the weighting given to each. The denominator for the key measures of participation and attainment, which are estimates of proportion, The ABS benchmarking method for population surveys, as used here, is a generalised regression technique which is a model-assisted method of estimation. The choice of benchmark variables for weighting and the model assumption about the independence of educational attainment variables on the labour force, age and gender has a small impact on the estimates produced. The SEW weights for an individual year are calibrated to align with independent estimates of the population by sex, 5-year age group, state or territory of usual residence, section of state or territory and labour force status. For pooled data across several years, extra model assumptions that the characteristics of interest and benchmark variables (eg. labour force proportions) do not depend on the year of collection mean that the pooled data is not a simple average of the single year estimates. On occasion this can lead to the final pooled result being outside the proportion estimates derived from the single year data that make up the pooled data set. For example, the estimate derived from the two-year pooled result including 2009 data could be fractionally above or below the estimates from the data of the two single years that go into it because of the different overall SEW sample size in 2009 compared to adjacent years. The data pooling method could also be varied by using a different set of initial weights. For example, using the final weights for the individual SEW years as initial weights for the pooled data may reduce the non-response differences in sample take between the two years. However, it would reduce the benefits of lower RSEs for fine output categories under the model assumption that educational attainment changes slowly over two years. It is also possible that education questions will be added in the future to other LFS supplements where the initial LFS weights are readily available and hence including such initial weights in this report may give a better indication of the data performance that may occur for future pooled estimates. The extra implicit assumptions with pooling data means that more care should be taken with computed levels of significance. The estimate of significance used here is based on the generally accepted p-value of 0.05, which is an estimate related solely to sampling error. There are many other factors in a pooling project, particularly in the weighting, that could cause a marginal increase or decrease in the value of an estimator. Most of the factors are model assumptions related to non sampling error, e.g. assumptions on change of characteristics and non-response across two years, and hence will not directly effect the variance estimate. Hence care should be taken when the calculated p-value is near 0.05. In summary, the presented pooling results arise from one choice of initial weights and population benchmarks. As such that choice and the validity of model assumptions about population behaviour and survey response across multiple years will have some effect on the estimates produced. The method, however, is identical to the method for SEW weighting of individual years. In analysing the results for 2007-2010, some very small impacts on the comparability of pooled data and individual year time series data have been observed. The general behaviour of the two time series however was consistent with the pooled data exhibiting lower RSEs for the time point estimates. APPROACH TO PRODUCING POOLED SERIES In pooling data, two alternative procedures need to be considered for pooled series: non-overlapping pooled estimates; and rolling pooled estimates. Non-overlapping estimates compare pooled data from an initial set of surveys with data from a subsequent set of surveys. Rolling estimates combine adjacent data that overlaps in the middle. Non-overlapping estimates are computationally simpler and the trends produced are easier to interpret. However, regular estimates cannot be produced as frequently, for example trends using two-year pooled data would require four years of data for two data points, and four-year pooled data would require eight years of data. The advantage of rolling estimates is that estimates can be produced each year maintaining the time series in the data. The disadvantage is that it is difficult to interpret what changes in the data are measuring. An estimate of the change between two sets of rolling pooled data will include some cases that are in both sets of data. In effect this will reduce the difference but more importantly the standard error of the change between the two consecutive rolling estimates will require evaluation of the resulting covariance between the two estimates. Consequently, while rolling estimates may be appropriate for producing level estimates on an annual basis rolling estimates are not appropriate for measuring change. For these reasons non-overlapping data have been preferred. For further information, see Research Paper: Weighting and Standard Error Estimation for ABS Household Surveys (Methodology Advisory Committee), Jul 1999 (cat. no. 1352.0.55.029) Document Selection These documents will be presented in a new window.
|
||