SAMPLE DESIGN
The following discussion will give a brief introduction to some basic terms and ideas in sampling and an outline of sample designs commonly used. The main focus of the discussion will be on determining an appropriate sampling method. PROBABILITY AND NON-PROBABILITY Non-Probability Samples If the probability of selection for each unit is unknown, or cannot be calculated, the sample is called a non-probability sample. Non-probability samples are often less expensive, easier to run and don't require a frame. However, it is not possible to accurately evaluate the precision (ie. closeness of estimates under repeated sampling of the same size) of estimates from non-probability samples since there is no control over the representativeness of the sample. If a non-probability sample is carried out carefully, then the bias in the results can be reduced. As it is dangerous to make inferences about the target population on the basis of a non-probability sample, non-probability methodology is often used to test aspects of a survey such as questionnaire design, processing systems etc. rather than make inferences about the target population. Different types of non-probability samples are discussed below. Quota Sampling To select a quota sample, the interviewers select respondents until a pre-determined number of respondents in certain categories are surveyed (eg. the interviewers might select the sample to achieve a certain age/sex breakdown reflective of the target population). This is the method of sampling commonly used by market researchers and political pollsters as it can produce fairly good estimates if it is properly conducted. When top up units are selected randomly to fill a quota, and no element of judgment is used by the researcher for unit selection, it is very similar to a probability sample. However, when non-response is significant (which is almost always the case for voluntary surveys), quota sampling can under-represent those portions of the population that are unwilling to respond or hard to contact. This is of particular concern when the data items collected influence the likelihood of response. See also the section on Non-Response in Errors in Statistical Data for further details. Convenience and Haphazard Sampling Street corner interviews, magazine and newspaper questionnaires and phone-in polls are all examples of convenience or haphazard samples. These types of surveys are subject to biased or unrepresentative samples as only persons who feel strongly about the topic will respond. These surveys also have a tendency to ask questions that are loaded or have a biased wording. Street corner interviews can be biased depending on the timing and the placement of the interviewer. There is no control over selecting the sample of respondents in any of these methods, however they are very cheap and easy to administer. Judgement or Purposive Sampling Judgement sampling is where a 'representative' sample is chosen by an expert in the field of study. Judgement sampling is subject to unknown biases but may be justified for very small samples. This form of sampling can be used to choose a sample for a pilot test of a probability survey but inferences about the population should not be made from judgement samples. Judgement sampling is also known as purposive sampling. Probability Samples A probability sample is one in which every unit of the population has a known non-zero probability of selection and is randomly selected. A probability sample allows inferences about the target population to be made. By knowing the selection probability for each unit, objective selections can then be made which should produce a more representative sample. Known probabilities also allow the measurement of the precision of the survey estimates in terms of standard errors and confidence intervals. Probability samples require a frame for selection purposes and thus are relatively expensive in terms of operational costs and frame maintenance. The most common sampling techniques, such as simple random, systematic, stratified, multi-stage and cluster sampling, are all examples of probability samples. These will be looked at later in this chapter. Choosing Between Probability and Non-Probability Samples The choice between using a probability or a non-probability approach to sampling depends on a variety of factors:
Probability sampling is normally preferred when conducting major surveys, especially when a population frame is available ensuring that we are able to select and contact each unit in the (frame) population. However, where time and financial constraints make probability sampling infeasible, or where knowing the level of accuracy in the results is not an important consideration, non-probability samples do have a role to play since they are inexpensive, easy to run and no frame is required. For this reason, when conducting qualitative (investigative), rather than quantitative research, non-probability samples & techniques such as case studies are generally superior to probability samples & quantitative estimation. Non-probability sampling can also be useful when pilot testing surveys. If a non-probability sample is carried out carefully, then the bias in the results can be reduced. Note that with non-probability methods it is dangerous to make inferences about the whole population. Quota sampling may be appropriate when response rates are expected to be low. True probability sampling would be more expensive and may require top up units to be selected. If quota sampling is used, selection of units should be as random as possible and care should be taken to avoid introducing a bias. Unlike certain non-probability samples, probability sampling involves a random selection of units. This allows us to quantify the standard error of estimates and hence allow confidence intervals to be formed and hypotheses to be formally tested. The main disadvantages with probability sampling involve cost, such as the costs involved with frame maintenance and surveying units which are difficult to contact. SIMPLE RANDOM SAMPLING Simple random sampling (SRS) is a probability selection scheme where each unit in the population is given an equal probability of selection, and thus every possible sample of a given size has the same probability of being selected. One possible method of selecting a simple random sample is to number each unit on the sampling frame sequentially, and make the selections by generating "selection numbers" from a random number table or, from some form of random number generator. With (SRSWR) and Without (SRSWOR) Replacement Simple random sampling can involve the units being selected either with or without replacement. With replacement sampling allows the units to be selected multiple times whilst without replacement only allows a unit to be selected once. Without replacement sampling is by far the more commonly used method. Advantages The advantage of simple random sampling lies in its simplicity and ease of use, especially when only a small sample is taken. Disadvantages Simple random sampling does, however, require a complete list of all population units as each unit needs to have a unique number associated with it to enable random selection. This sampling scheme also becomes unwieldy for large sample sizes and can be expensive if the sample is spread over a wide geographic area. In practice, simple random sampling is rarely used because there is almost always a more efficient method of designing the sample (in terms of producing accurate results for a given cost). Nevertheless, simple random sampling forms the basis of a number of the more complex methods of sample design, and is used as a benchmark to which other designs are compared. USE OF AUXILIARY INFORMATION We have discussed methods of drawing simple random samples. There is no point in using any other kind of sample selection method if one knows no more about the population to be sampled than the existence of each of the units in the population. However, some further information is often available about each of the population units from simple observation or data from a previous study (for example, a census). This information can be in the form of demographic variables such as age, sex and income or geographical/business types such as industry, employment, state, region and sector (private or public). This further information about the population units is called auxiliary information. Such information can be used in the selection and the estimation process to obtain more accurate estimates or reduce costs. Sampling techniques using this information include systematic sampling, stratified sampling (including post-stratification) and cluster sampling. Auxiliary information is also used in estimation techniques such as ratio and regression estimations. One of the major aspects of sample design is the efficient use of auxiliary or supplementary information. As there is often a cost involved in obtaining auxiliary information, it is necessary to quantify the gains that are obtained through using auxiliary information and balance it against the cost of acquisition. SYSTEMATIC SAMPLING Systematic sampling provides a simple method of selecting the sample when the sampling frame exists in the form of an explicit list. Where the frame contains auxiliary information then the units in the frame are ordered with respect to that auxiliary data (eg employment size of a business). A fixed interval (referred to as the skip) is then used to select units from the sampling frame. Systematic sampling is best explained by describing how the sample selections are made. Method Assume that the population has N units (eg. 37) and that we wish to select a sample of size n (eg. 5). The list of N population units is ordered in some way. The steps taken in selecting a systematic random sample are:
The value of k is usually not an integer. In this case we either
Example: Calculating the Skip Interval Say that we wanted to take a systematic sample of size 5 from a population of 37 units. The sample size does not divide evenly into the population. The two options for coping with this are discussed below. Order the population units in some way and number them from 1 to 37.
n = 5 k = 37/5 = 7.4 k = 7 r = 4 Then the sample units are : 1st unit=4, 2nd unit=4+7=11, 3rd unit=4+14=18, 4th unit=4+21=25 and 5th unit=4+28=32. Therefore, sample = ( 4, 11, 18, 25, 32) k = 7.4 r = 4.2 sample = ( 4.2, 11.6, 19, 26.4, 33.8) = ( 4, 12, 19, 26, 34) Features of Systematic Random Sampling The usefulness of systematic random sampling depends upon the strength of the relationship between the variable of interest and the benchmark variable/s. The more highly correlated they are, the greater the gains in accuracy achieved over simple random sampling. This is because we are ensuring a more representative sample of population units are selected. If there is a strong relationship between the variable of interest and the benchmark variable/s then ordering the list by the variable of interest will yield more accurate results using systematic sampling than simple random sampling. Systematic random sampling using ordered lists ensures a range of units will be selected in the sample. Advantages
Disadvantages
Stratified Sampling Stratified sampling is a technique which uses auxiliary information which is referred to as stratification variables to increase the efficiency of a sample design. Stratification variables may be geographical (eg. state, rural/urban) or non-geographical (eg. age, sex, number of employees). Stratified sampling involves
Stratification almost always improves the accuracy of estimates. This is because the population variability can be thought of as having components within strata and between strata. By independently sampling within each stratum we ensure each stratum is appropriately reflected in the sample, so between stratum variability is eliminated and we are left only with the within stratum component. With this factor in mind we see that the most efficient way to stratify is to have strata which are as different from each other as possible (to maximise the variance which is being eliminated) while being internally as homogeneous as possible (to minimise the variance remaining). Practical Considerations When planning a stratified sample, a number of practical considerations should be kept in mind:
Example of Stratification As an example of stratification, if we were interested in the educational background of members of a Science faculty at a University, we could select a sample from the faculty as a whole or select samples independently from each of the departments within the faculty, such as mathematics, physics, chemistry etc. This latter method would ensure that each department was adequately represented (which would not necessarily happen otherwise), and should increase the precision of the overall estimate. If on the other hand, we were interested in the level of education (PhD, Masters, Bachelor) rather than the background we should stratify the faculty by level (Professor, Senior Lecturer, Lecturer) rather than by the department. Using this stratification we are more likely to find uniformity of educational standards within a level rather than an area of work, and we are also more likely to separate the better qualified from the less qualified. Advantages The four main benefits of stratified sampling are:
Disadvantages
Number of Strata There is no rule as to how many strata the population should be divided into. This depends on the population size and homogeneity and the format in which the output is required. If output is required for some sub-groups of the population these subgroups must be considered as separate strata. ABS Surveys All surveys conducted by the Australian Bureau of Statistics employ stratification. Household surveys (such as the Monthly Population Survey and the Household Expenditure Survey) use geographic strata. Business surveys use variables such as state and industry strata and use some measure of size (eg employment) to form size strata. Allocation of Sample An important consideration after deciding on the appropriate stratification is the way in which the total sample is to be allocated to each stratum. There are three common methods of calculating the number of units required from each stratum.
Estimation As samples are selected independently from each stratum, estimates are also usually made separately for each stratum, then added to give the overall estimate (eg estimated unemployment for Australia will be the sum of the state unemployment estimates). Similarly, standard errors or variances (measures of sample variability) are calculated for each stratum and then all strata specific variances are added up to obtain the overall variance. The addition of variances is possible because the sample is selected independently from each strata. This overall variance can then be used to calculate an overall standard error. Post Stratification There will be occasions when we may like to stratify by a certain variable, say age or sex, but we cannot because we do not know the age and sex of our population units until we select them. Post-stratification is a method used when stratification is not possible before the survey. The stratification variable can then be used after the survey is conducted, to improve the efficiency of estimates or, to obtain estimates corresponding to different categories of that variable (eg. sex) by stratifying the sample as if the benchmark information had been available previously. CLUSTER AND MULTI-STAGE SAMPLING So far we have considered a number of ways which a sample of population units can be selected and population characteristics estimated on the basis of this sample. In this section consideration is given to a sampling scheme where the selection of population units is made by selecting particular groups (or clusters) of such units and then selecting all or some of the population units within selected groups for inclusion in the sample. Cluster Sampling Cluster sampling involves selecting a sample in a number of stages (usually two). The units in the population are grouped into convenient, usually naturally occurring clusters. These clusters are non-overlapping, well-defined groups which usually represent geographic areas. At the first stage of selection, a number of clusters are selected. At the second stage, all the units in the chosen clusters are selected to form the sample. Practical Considerations
Advantages Cluster sampling involves selecting population units that are "close" together and does not require all the population units to be listed. Cluster sampling has two advantages:
Disadvantages In general, cluster sampling is less accurate than SRS (for samples of the same size) because the sample obtained does not cover the population as evenly as in the case of SRS. However it is often preferred because it is more economical. For example, if we take a simple random sample of 10,000 households across the whole of Australia then we are more likely to cover the population more evenly, but it is more expensive than sampling 50 clusters of 200 households. Multi-stage Sampling Multi-stage sampling involves selecting a sample in at least two stages. At the first stage, large groups or clusters of population units are selected. These clusters are designed to contain more units than are required for a final sample. At the second stage, units are sampled from the selected clusters to derive the final sample. If more than two stages are used, the process of selecting "sub-clusters" within clusters continues until the final sample is achieved. The same practical considerations apply to multi-stage sampling as to the cluster sampling. Example: A Three-Stage Sample The following is an example of the stages of selection that may be used in a three-stage household survey.
Uses of Multi-stage Sampling Multi-stage sampling is generally used when it is costly or impossible to form a list of all the units in the target population. Typically, a multi-stage sample gives less precise estimates than a simple random sample of the same size. However, a multi-stage sample is often more precise than a simple random sample of the same cost, and it is for this reason that the method is employed. Advantages and Disadvantages The advantages and disadvantages of multi-stage sampling are similar to those for cluster sampling. However, to compensate for the lower accuracy, either the number of clusters selected in the first stage should be relatively large (but this increases the cost of the survey) or the sampling fraction for later stages should be high (i.e. a large percentage of each cluster should be selected). SAMPLE SIZE ISSUES AND DETERMINATION An important aspect of sample design is deciding upon the sample size given the objectives and constraints that exist. Since every survey is different there are no fixed rules for determining sample size. However, factors to be considered include
Once these issues have been addressed, you are in a better position to decide on the size of the sample. Variability The more variable the population is, the larger the sample required to achieve specific levels of accuracy. However, actual population variability is generally not known in advance; information from a previous survey or a pilot test may be used to give an indication of the variability of the population. When the characteristic being measured is comparatively rare, a larger sample size will be required to ensure that sufficient units having that characteristic are included in the sample. Population Size An aspect that affects the sample size required is the population size. When the population size is small, it needs to be considered carefully in determining the sample size, but when the population size is large it has little effect on the sample size. Gains in precision from increasing the sample size are by no means proportional to population size. Resources and Accuracy As discussed earlier, the estimates are obtained from a sample rather than a census, therefore the estimates are different to the true population value. A measure of the accuracy of the estimate is the standard error. A large sample is more likely to have a smaller standard error or greater accuracy than a small sample. When planning a survey, you might wish to minimise the size of the standard error to maximise the accuracy of the estimates. This can be done by choosing as large a sample as resources permit. Alternatively, you might specify the size of the standard error to be achieved and choose a sample size designed to achieve that. In some cases it will cost too much to take the sample size required to achieve a certain level of accuracy. Decisions then need to be made on whether to relax the accuracy levels, reduce data requirements, increase the budget or reduce the cost of other areas in the survey process. Level of Detail Required If we divide the population into subgroups (strata) and we are choosing a sample from each of these strata then a sufficient sample size is required in each of the subgroups to ensure reliable estimates at this level. The overall sample size would be equal to the sum of the sample sizes for the subgroups. A good approach is to draw a blank table that shows all characteristics to be cross-classified. The more cells there are in the table, the larger the sample size needed to ensure reliable estimates. Likely level of Non-response Non-response can cause problems for the researcher in two ways. The higher the non-response the larger the standard errors will be for a fixed initial sample size. This can be compensated for by assigning a larger sample size based on an expected response rate, or by using quota sampling. The second problem with non-respondents is that the characteristics of non-respondents may differ markedly from those of respondents. The survey results will still be biased even with an increase in sample size (ie. increasing the sample size will have no effect on the non-response bias). The lower the response rate, the less representative the final sample will be of the total population, and the bigger the bias of sample estimates. Non-response bias can sometimes be reduced by post-stratification as well as through intensive follow up of non-respondents, particularly in strata with poor response rates. Sampling Method Many surveys involve complex sampling and estimation procedures. An example of this is a multi-stage design. A multi-stage design can often lead to higher variance in resulting estimates than might be achieved by a simple random sample design. If, then, the same degree of precision is desired, it is necessary to inflate the sample size to take into account the fact that simple random sampling is not being used. Relative importance of the variables of interest Generally, surveys are used to collect a range of data on a number of variables of interest. A sample size that will result in sufficiently precise information for one variable may not result in sufficiently precise information for another variable. It is not normally feasible to select a sample that is large enough to cover all variables to the desired level of precision. In practice therefore, the relative importance of the variables of interest are considered, priorities are set and the appropriate sample size determined accordingly. Calculation of sample size When determining an appropriate sample size, we take as a general rule, the more variable a population is, the larger the sample required in order to achieve specific levels of accuracy in survey estimates. However, actual population variability is not known and must be estimated using information from a previous survey or a pilot test. It is worthwhile keeping in mind that the gains in precision of estimates are not directly proportional to increases in sample size (i.e doubling the sample size will not halve the standard error, generally the sample has to be increased by a factor of 4 to halve the SE). In practice, cost is a major consideration. Many surveys opt to maximise the accuracy of population estimates by choosing as large a sample as resources permit. In complex surveys, where estimates are required for population subgroups, enough units must be sampled from each subgroup to ensure reliable estimates at these levels. To select a sample in this case, you might specify the size of the standard error to be achieved within each subgroup and choose a sample size to produce that level of accuracy. The total sample is then formed by aggregating this sample over the subgroups. Sample size should also take into account the expected level of non-response from surveyed units. When the characteristic being measured is comparatively rare, a larger sample size will be required to ensure that sufficient units having that characteristic are included in the sample. Sample Size Formulae If a survey is designed to estimate simple proportions without any cross-classifications in a large population (approximately over 10,000 units), the following formulae can be used to determine the size of the sample: where n = sample size, p = sample proportion, SE(p) = required standard error of the sample proportion However, to be able to use this formula, the proportion being estimated needs to be roughly known from supplementary information or a similar study conducted elsewhere. For example, suppose a survey seeks to estimate the proportion of Richmond residents in favour of Sunday night football at the MCG. The standard error (SE) desired is 0.04, while the proportion (p) in favour of the proposal is thought to be about 0.40. The size of the sample would need to be n=150. If this survey was then completed with a sample size of n=150 and it was a found that the sample proportion (p) in favour of the proposal was 0.8 (not 0.4 as guessed), then the standard error of this sample proportion of 0.8 would be 0.033 not 0.04 as originally planned for. A proportion of 0.5 gives the highest standard error for a fixed sample size or, requires the highest sample size for a fixed standard error, hence p=0.5 is the worst case scenario. It is for this reason that an estimate of p=0.5 is often used when calculating sample sizes when there is no information on the proportion to be estimated. Example: Gains From Sampling Suppose we wish to take a sample from a population. We have a preliminary estimate of the proportion of the population having the characteristic we are interested in measuring (50%). The level of accuracy we require from our survey is an RSE of 5%. Using the formulae for the sample size in a finite population. For various population sizes, the sample size that we would need is:
The gains from employing sampling are greatest when working with large populations. |
