 |
||
Australian Bureau of Statistics |  |
|
|
|
4522.0.55.001 - Information Paper: Measuring Crime Victimisation, Australia: The Impact of Different Collection Methodologies, 2002
Latest ISSUE Released at 11:30 AM (CANBERRA TIME) 05/02/2004 |
Page tools:
 Print Page
 Print All
 RSS
 Search this Product
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Introduction
Through sections 3, 4 and 5 the scope and definitions of the data from various collections is progressively adjusted to assist in comparisons. The paper concludes with a summary of the main points raised throughout. CRIME VICTIMISATION DATA SOURCES There are many national collections which present statistics on crime victimisation. Broadly speaking, these collections can be divided into:
This section outlines a number of the key collections and presents the objectives, scope and characteristics for each. Although the focus is on Australian sources, the International Crime Victims Survey (ICVS) is discussed below as it produces data at the national level for Australia and has been referred to by some users in the context of providing national crime victimisation data. ADMINISTRATIVE DATA Administrative data on crime victimisation can be derived from a number of sources including police, hospitals and community service agencies. The ABS has two relevant national collections based on administrative data: Recorded Crime - Victims and Causes of Death. Recorded Crime The annual Recorded Crime statistics collection (RCS) provides indicators of the level and nature of recorded crime in Australia and a basis for measuring change over time in criminal victimisation. The collection is a census of all victims of selected crimes reported to, or detected by, police, whose details are subsequently recorded on police administrative systems. Data have been provided by state and territory police agencies on a calendar year basis since 1993. National standards and classifications are applied to all aspects of the collection. Causes of Death The annual Causes of Death collection brings together statistics and indicators for all deaths, including perinatal deaths, registered in Australia. The statistics are compiled from data made available to the ABS by the Registrar of Births, Deaths and Marriages in each state or territory. Deaths from external causes are attributed to the event leading to the fatal injury as described in Chapter 20 of International Classification of Diseases Revision 10 (ICD-10). The Causes of Death collection has been collected and published on a calendar year basis since 1964. National standards and classifications are applied to all aspects of the collection. The ICD-10 category of death through external causes can be further disaggregated to the category of death through assault which can be compared to murder/manslaughter in the RCS collection. SURVEY DATA One of the limitations with administrative data on crime victimisation is that incidents may never come to the attention of authorities such as police, or the victim may never speak of the incident to anyone else. Surveys of individuals in the community provide a way of asking people directly about their experiences of crime, and therefore the victimisation rates from surveys are generally considerably greater than rates from administrative systems. General Social Survey The ABS General Social Survey (GSS) is a new multi-topic survey conducted for the first time in 2002 and expected to be repeated four-yearly. The survey was conducted to give social researchers the ability to cross-classify a broad range of social characteristics relating to the same person at a single point in time. The GSS is an interviewer based survey in which one respondent, randomly selected per household and aged 18 years or over, is asked a series of questions by an ABS interviewer for each area of social concern. For the crime and safety topic the GSS questions asked about assault, break and enter and feelings of safety. National Crime and Safety Survey The ABS National Crime and Safety Survey (NCSS) is a regular survey (run in 1983, 1993, 1998 and 2002 to date) and expected to be conducted three-yearly in future. It is specifically designed to collect crime and safety information from individuals in selected households and enable investigation of movements across crime topics, populations, geography and time. In 1983 the NCSS was completed by personal interviews. Since 1993 the NCSS has been a mail-back collection conducted as a supplementary survey to the ABS monthly Labour Force Survey. All persons aged 15 years and over within selected households are in the scope of the survey, with the exception of a separate module on sexual assault which is only enumerated for persons aged 18 years or over (1993 and 1998 was female only). Women's Safety Survey The first Women's Safety Survey (WSS) was conducted by the ABS in 1996 and a repeat is planned for 2005, but with the coverage anticipated to be extended to include men as well as women. This survey was conducted to provide information on the safety of women at home and in the community. The WSS was an interviewer based survey in which one respondent, a female randomly selected per household and aged 18 years or over, was asked a series of questions by a female ABS interviewer regarding her safety in the home and the community. The majority of interviews were conducted face-to-face, however telephone interviews were available for respondents who did not wish to proceed with a face-to-face interview. International Crime Victims Survey The International Crime Victims Survey (ICVS) is an international survey. It was conducted in 1989, 1992, 1996 and 2000 across a selection of industrialised and developing countries spanning all continents. The survey was conducted to monitor crime and perceptions of crime and criminal justice internationally in a standardised way, and to provide comparative information on the patterns, contours and effects of victimisation in both developed countries and the rest of the world. The ICVS is a telephone based survey (in Australia and most other countries) in which one respondent, randomly selected per household and aged 16 years and over, is asked a series of questions by an interviewer. The Australian component of the ICVS was coordinated by the Australian Institute of Criminology (AIC) with fieldwork conducted by Roy Morgan Research. COMPARISON OF CRIME VICTIMISATION RATES This section compares statistics on crime victimisation from various sources, including survey and administrative data. COMPARISONS USING ADMINISTRATIVE DATA Differences between various sources of official statistics may include any number of the following:
Differences may also occur within the same source of administrative data (e.g. state/territory police agencies), due to factors such as:
To provide an indication of the numeric differences, the following table illustrates homicide rates sourced from the two administrative datasets considered in this paper. 1. NUMBER AND RATES PER 100,000 OF HOMICIDE IN AUSTRALIA, 1997-2001
The figures show that homicide rates according to RCS are consistently higher than those from the Causes of Death collection, and in 1999 the rates differed by 0.4 deaths per 100,000 persons. There are two main differences between these two sets of figures that make direct comparisons difficult:
MEASUREMENT OF A SINGLE CRIME CATEGORY USING ADMINISTRATIVE AND SURVEY DATA To illustrate the differing statistics between administrative data and surveys, and surveys themselves, the following section focuses on assault victimisation statistics. Assault was chosen because:
2. UNADJUSTED ASSAULT PREVALENCE RATES AS PUBLISHED BY COLLECTION
The above table indicates that prevalence of assault victimisation ranges from 0.8% (2002 RCS) to 9.0% of the population (2002 GSS and 2000 ICVS). These rates are significantly different between the surveys, with the exception of the ICVS and GSS not being significantly different. The smaller victimisation rates for the RCS collection are to be expected as it is well documented that only a small proportion of victimisation is subsequently reported to and recorded by police (e.g. ABS (a), 2002; Statistics Canada, 1997). A crime must progress through a series of victim, and then police, decisions before it may be recorded by police in official administrative systems. These decisions include:
Looking just between surveys, there are some basic scope differences between the survey collections relating to age and sex of victims which will affect the victimisation rates. These include:
It is possible to adjust for these age and sex scope differences. The table below presents prevalence rates for a comparable scope of females aged 18 years and over. 3. ASSAULT PREVALENCE RATES, Adjusted only for comparable respondent group (females aged 18 years and over)
However, even with the output amended for comparability on age and sex of victims the assault rates are still significantly different. The assault rates for the GSS, WSS and ICVS are significantly higher than the NCSS rate. The ICVS assault rate is significantly higher than the WSS rate. Therefore, consideration must be given to other factors within each of the collections where differences may arise and ultimately impact on an individual's decision to report being a victim of assault. These are outlined and discussed in detail in section 4 and the impact these differences may have is discussed in section 5. COMPARING SURVEY METHODOLOGIES This section describes methodological differences between survey sources. SAMPLE DESIGN AND SELECTION Decisions on the appropriate sample size, distribution and method of selection are dependent on a number of considerations. These include the aims and content of the survey, the level of disaggregation and accuracy at which the survey estimates are required, and the costs and operational constraints of conducting the survey. The General Social Survey (GSS), National Crime and Safety Survey (NCSS), Women's Safety Survey (WSS) and the International Crime Victim's Survey (ICVS) aim to provide national estimates of the nature and extent of victimisation for selected crime, with assault being the selected crime type collected in all four surveys. General Social Survey - 2002 The GSS sampled from private dwellings in urban and rural areas and excluded sparsely populated areas. A random sample of households was selected using a multistage, area based, sample design. Respondents were selected within households using an ARA (Any Responsible Adult) selection methodology. National Crime and Safety Survey - 2002 The NCSS was a supplement to the ABS Labour Force Survey. The NCSS included private dwellings and excluded sparsely settled areas, similar to the GSS above. A random sample of households was selected using a multistage, area based, sample design. All persons aged 15 years and over in the selected household were selected as respondents. Women's Safety Survey - 1996 The WSS sampled from private dwellings in urban and rural areas and excluded sparsely populated areas. Specific training and support was provided for interviewers, and therefore the size of the sample selected in rural and remote areas was reduced due to interviewer costs. A random sample of households was selected using a multistage, area based, sample design. Respondents were selected in households using an ARA selection methodology applied to women who were usual residents of the household. International Crime Victims Survey - 2000 The ICVS sampled from private dwellings with at least one phone connection, in urban and rural areas of the mainland states and the urban areas of Tasmania. The Northern Territory was included as part of rural South Australia, and the Australian Capital Territory as part of rural New South Wales. An adaptation of random digit dialling was used to select private dwellings. Within each household one respondent aged 16 years or over was randomly selected. SCOPE AND COVERAGE Inclusions The scope and coverage for each of the above surveys was based on usual residents of the selected household and included:
Exclusions The following persons were specifically excluded from the GSS, NCSS and WSS:
In addition, the NCSS also excluded members of the permanent defence forces. For the ICVS, where persons selected as respondents were absent for the duration of the survey period, were incapable of responding, deaf, suffered from an illness or disability or were too old, another person in the household was selected. QUESTIONNAIRE Each of the surveys used a different introductory statement and different questionnaire format. General Social Survey - 2002 The GSS was introduced to respondents as being a general social survey using the following introductory statement:
The GSS collected information in three parts and included information on selected topics such as crime and safety. The survey averaged 45 minutes per household to complete and had the following structure:
National Crime and Safety Survey - 2002 The NCSS was introduced as a crime and safety survey using the following statement:
The survey collected data on safety followed by data on crime (the same order as the GSS). Feelings of safety were measured at the beginning of the survey straight after a question relating to perceived neighbourhood problems. The second part of the survey relates to experiences of crime and the reporting of incidents. The NCSS was conducted using a supplementary questionnaire to the Labour Force Survey. Three paper forms were used in the NCSS to collect crime victimisation information and the completion of the relevant form/s averaged 8 minutes per person (the average for victims only was 13 minutes).
Women's Safety Survey - 1996 The WSS was introduced to respondents as being about public safety using the following statement:
The WSS interviews averaged 50 minutes per household and collected information in two parts:
International Crime Victims Survey - 2000 The ICVS is introduced by a Roy Morgan interviewer as follows:
The ICVS interviews averaged 19 minutes per household and the structure of the questionnaire was as follows:
SURVEY PROCEDURES Survey procedures relate to the mode of the data collection and any special procedures required (e.g. communicating with respondents who did not speak English). The procedure used for each of the surveys is outlined below. General Social Survey - 2002 The GSS involved a face-to-face personal interview using Computer Assisted Interviewing. Interviewers were required to make a number of attempts to contact a household and leave call-back cards before recording a non-response or non-contact. Interviewers were predominately female (around 90%). Follow-up was conducted by telephone (for interviews with less than three missing responses) or in person (for more than three missing responses). National Crime and Safety Survey - 2002 The NCSS involved a self-administered questionnaire. Questionnaires were either delivered to the selected households by interviewers or mailed to those respondents who completed the Labour Force Survey by telephone. Respondents were asked to complete the relevant questionnaires and return them by mail. Each respondent was provided with an individual mailback envelope, to minimise the likelihood of other members of the household seeing another person's form. Completion of the third form relating to sexual assault was voluntary. A series of mail and telephone follow-ups were undertaken to increase response. The first follow-up was by mail, and the second by mail or telephone. Women's Safety Survey - 1996 The WSS involved a face-to-face personal interview using pen and paper techniques. However, the option of a telephone interview was offered to respondents who did not wish to proceed with a face-to-face interview. Only trained female interviewers were used. Interviews were only conducted with women in private. Women selected in the survey were advised that they were under no obligation to respond. Special procedures were used to conduct interviews with selected respondents who could not speak English (110 respondents in total). These respondents provided a telephone number and an interviewer contacted them. There was no follow-up conducted. International Crime Victims Survey - 2000 The ICVS is a telephone survey using Computer Assisted Telephone Interviewing (CATI). There was no follow-up in this survey, and phone calls resulting in no replies or engaged numbers were considered ineligible contacts. Special procedures were used to conduct interviews with selected respondents who could not speak English. These interviews were completed with a bilingual interviewer. RESPONSE RATE Response rates varied between the surveys. The details for each are outlined below. Fully completed refers to forms where significant amounts of information were not missing. General Social Survey - 2002 The GSS was fully completed by 15,510 persons aged 18 years or over, achieving a response rate of 91%. National Crime and Safety Survey - 2002 The NCSS was fully completed by 41,200 persons aged 15 years or over, achieving a response rate of 76%. The 24% of non-responses represented cases where potential respondents refused to participate in the survey, failed to mailback their survey form, or mailed back the form with a significant amount of missing data. Women's Safety Survey - 1996 The WSS was fully completed by 6,333 women aged 18 years or over, achieving a response rate of 78%. Full refusals accounted for 13% of the sample (1,038). A further 7% of the sample (579) were either unable to be contacted or unable to be interviewed due to language difficulties, death or illness. The remaining 2% of non-respondents were made up of those women who discontinued with the survey part way through. International Crime Victims Survey - 2000 The ICVS was fully completed by 3,031 persons aged 16 years or over, achieving a response rate of 57% (as a proportion of eligible telephone numbers). However, it should be noted that the sampling was conducted in two stages. The first stage collected data from 2,005 persons aged 16 years or over, whilst the second stage consisted of an additional sample of 1,026 persons aged 65 years or over. The purpose of the older sample was to provide a large enough sample of older Australians to examine crime victimisation among this group. However, available data (as used in section 3) relates to the full sample of 3,031 respondents. Non-responses represented refusals, terminations, appointments not met and calls terminated due to language problems. IMPACT OF DIFFERING SURVEY METHODOLOGIES Steps are taken in all aspects of development and conduct of surveys to ensure that data collected are as accurate as possible. However, as with all surveys, information collected can be affected by various types of errors. Sampling error can usually be quantified; however, non-sampling error is difficult to quantify. The impacts of various methodological differences are discussed below. Where possible, the assault victimisation data (or break-in data for NCSS/GSS comparisons only) from each survey is used to illustrate the extent of the differences. SAMPLING ERROR Sampling errors occur in sample surveys because a sample, rather than the entire population, is surveyed. Various factors that can influence the size of the sampling error include sample design, sample size and population variability. The extent of this variability can be determined by calculating the relative standard error (RSE) of the estimates, which expresses the standard error of the estimate as a percentage of the estimate's value. The standard error is a measure of the variation among the estimates from all possible samples, and thus a measure of the precision with which an estimate from a particular sample approximates the average results of all possible samples. At the national level, RSEs for the assault victimisation rates from the GSS, NCSS, ICVS and WSS were less than 25%. However, when assault victimisation rates were disaggregated by other data items (e.g. age, sex, state/territory) the RSEs increased (in some cases to over 50%). Where RSEs are greater than 50% the estimates are considered too unreliable for general use. NON-SAMPLING ERROR Errors made in collecting, recording and processing data can occur regardless of whether a sample or complete enumeration of a population is undertaken. These errors are referred to as non-sampling errors. Specific types of non-sampling errors and their possible impact on assault victimisation data are discussed in detail below. Respondent Errors Errors made may arise through ambiguous or misleading questions, inadequate or inconsistent definitions of terms used, by the complexity of questionnaire sequence guides causing some questions to be missed, by the use of long recall periods, or by respondent behaviours to particular types of questions or situations. Respondent error may also occur due to respondents answering incorrectly to protect their personal integrity, their personal safety or to protect another person. For example, respondents may not have reported incidents they experienced, particularly if the perpetrator was somebody close to them, such as a partner or family-member. The WSS attempted to minimise this effect by conducting interviews alone with respondents. The GSS (which reported a higher assault victimisation rate for females than the WSS, and referred to 2002 rather than 1996) did conduct personal interviews with respondents, although there was no requirement for respondents to be interviewed alone and it is unknown what proportion of respondents were interviewed in the presence of a third party. The NCSS involved leaving questionnaires to be mailed back, providing the possibility of respondents completing forms in private or in the presence of other persons, or even for a single member of the household to complete the NCSS on behalf of other in-scope respondents. The ICVS is a telephone based survey with no requirement for the respondents to be alone so, similar to the GSS and NCSS, it is unknown whether another person was present when the respondent completed the interview. Recent research on surveys conducted in the USA, Great Britain and Germany found that between 37% and 57% of respondents were interviewed face-to-face in the presence of a third party (cited in Zipp & Toth, 2002), and that when the respondent is married, his or her spouse is present in approximately one of every three interviews. Comparable findings in the Australian context are not available. One could infer that for the GSS, NCSS and ICVS between 40% and 60% of respondents reported in the presence of a third party, and may have introduced some level of respondent error to protect personal integrity, safety or safety of another person. However, the true nature and extent of this issue cannot be quantified at this time. All four surveys asked respondents to recall their experiences of victimisation. An important element of this recall is the time period. Each of the ABS surveys asked about experiences within the previous 12 months from interview date (the WSS also asked about experiences in the last five years). The 2000 ICVS asked about experiences in 2000, 1999 and the last five years respectively, and for publication purposes used the 1999 recall period. Therefore the recall period used for output of results is inconsistent between the ABS surveys and the ICVS, with the ABS surveys using a 0-12 month recall period and the ICVS using a 3-15 month recall period. In addition, it is not possible to identify for the ICVS whether persons reporting as victims did so where victimisation occurred in the three months between the end of the 1999 recall period and the interview dates in March 2000, and therefore whether there was any over-reporting of victimisation. One study using data from the US National Crime Survey found that victimisation rates for personal crimes based on a three month recall period was almost 40% higher than victimisation rates based on a 12 month recall period (Bushery, 1981). This research excluded series crime (i.e. two or more similar or related incidents which occur during the same reference period). Therefore the results reflect the recall rates of persons experiencing a single type of incident once in the reference period, either within 3 months or 12 months prior to interview. In addition, further research found that recall rates for crimes by relatives and marital partners were worse than recall rates for crimes by other non-strangers (Skogan, 1990). The GSS questions on crime victimisation are preceded by questions on family contact, whilst the NCSS questions on crime victimisation are preceded by questions on neighbourhood problems. The immediacy of reference to family contacts in the GSS may have aided recall of crimes by family contacts leading to a higher rate for older incidents in the reference period that are more difficult to recall (i.e. crime by family members). Non-Response Bias Non-response can introduce errors into results when non-respondents have different characteristics and experiences of victimisation from those individuals who did respond. Non-response bias is the difference between the estimate based on the responding sample and the estimate if there were no non-respondents. Non-response bias arises if the respondents in the sample are not representative of the non-respondents with respect to the characteristic under measurement (in this case crime victimisation). Measuring non-response bias is difficult without information about non-respondents, though the likely size of its effect can be considered. The proportion of non-respondents for the surveys under discussion ranged from 9% for the GSS, 22% and 24% for the WSS and NCSS respectively, through to 43% for the ICVS. Therefore, greater impact due to non-response bias is expected from the ICVS than the NCSS or WSS, and GSS is expected to have the least impact due to non-response bias. There has been conflicting research regarding the impact of non-response bias on crime victimisation reporting rates. Van Dijk (1990) reviewed the 1989 International Crime Victims Survey and found that there was no clear evidence on the effects of non-response. Strangeland (1995) concluded the same in his survey in regional Spain and went on to state that there appeared to be two counter-balancing effects operating. Firstly, the lifestyle of non-respondents which makes them difficult to locate may also make them more vulnerable to crime. Secondly and alternatively, victims may be more motivated to respond because they have something to tell. The magnitude of difference between prevalence of crime victimisation among non-respondents and respondents would have to be large for non-response bias to be the primary cause of the discrepancy between survey estimates. This point is made in the following simple example using the NCSS and GSS which reported the greatest differences in assault victimisation. In the NCSS, which had a sample size of 54,418, the non-response rate for victims of assault was approximately 24% or 13,248 individuals. An estimate of the number of respondents who reported to be victims of assault was 1,935, calculated as 4.7% of the number of respondents. Non-response is greatest in the younger age groups and reduces as age increases, whilst prevalence of assault decreases with age. If we assume the overall prevalence of assault for non-respondents in the NCSS was in fact 9.4% (double the rate for respondents), the estimated number of assault victims among the non-respondents would be 1,245. The estimate of the total number of victims from the sample would then be 3,180 persons (1,245 + 1,935), producing an overall estimate of prevalence of 5.8% which is still well below the 9.0% prevalence rate in the GSS. One of the benefits of the NCSS is that because it is a supplement to the ABS LFS, selected characteristics of most non-respondents to the NCSS are known (i.e. those characteristics that are collected for the LFS). An analysis was undertaken on a range of socio-demographic characteristics for NCSS respondents and non-respondents to determine if there were any significant differences. There were no significant differences. The weighting methodology applied in the NCSS and GSS may have also reduced the non-response bias, though the size of this reduction is not possible to measure. Such a reduction would have occurred if, within age by sex by state categories, the non-respondents would have provided similar answers to the respondents. However, comparing the two surveys with the greatest difference in assault victimisation prevalence rates (NCSS and GSS), and with a large difference in non-response rates, non-response bias may explain a small amount of the difference in assault victimisation. Sample representation To gain some insight into the representativeness of the sample, the socio-demographic characteristics of survey respondents can be compared to independent estimates of the corresponding population at the time of the survey. Where there are differences in sample representation and these differences are not adjusted for in the final results, the victimisation rates may not be representative of the entire population. The distribution of GSS, NCSS and WSS survey respondents (classified by geographic location, age and country of birth) conformed reasonably well with the geographic location, age and birthplace distribution of the relevant population at the time of each survey. Each of the surveys under discussion did not sample from sparsely populated areas. As noted in section 4, the ICVS over-sampled persons aged 65 years or over. However these respondents were weighted down in the ICVS to produce estimates that conformed with the age distribution of the population aged 16 years or over at the time of the survey. As stated above, the NCSS is a supplementary questionnaire to the ABS LFS. Underenumeration in the NCSS results primarily from non-response (in the NCSS), but also, to a lesser extent, from undercoverage in the LFS. Undercoverage, which is present in all household surveys, occurs when certain individuals in the population do not have a chance of being included in a survey (e.g., because they have no place of usual residence, or they are overseas, or because of imperfections in survey procedures). The NCSS weighting process ensures that estimates conform to the distribution of the population by age, sex and geographical area, thereby compensating for any underenumeration in the survey. Bias will only occur if persons not enumerated in the NCSS have significantly different characteristics to the survey respondents in the same age by sex by geography category. Mode Effects Mode effect refers to the impact of the survey delivery method on the response to the survey. Different survey modes include self-administered questionnaires (e.g. mailback questionnaires), telephone interviews and face-to-face interviews. Modes can also be different within surveys (e.g. a mailback questionnaire that involves follow-up via telephone interview), and these are often referred to as mixed-mode surveys. Below is a summary of some of the advantages and disadvantages of using a self administered questionnaire, telephone interview and face to face interview. 4. ADVANTAGES AND DISADVANTAGES OF DIFFERENT SURVEY MODES
The mode of the survey is likely to have some impact on the reporting of crime although it would not solely explain the differences between the survey results. Mode effect is costly to investigate, though it can be quantified. An investigation would require the conduct of parallel surveys using different delivery methods. In the case of crime victimisation this has not been done. What follows is a qualitative assessment of the possible impacts of different modes that should be considered when developing surveys on crime victimisation. Self administered questionnaires (SAQ), the method used for the NCSS, allow respondents to look ahead and go back to earlier items and may therefore reduce the impact of question order. The likely occurrence of this cannot be quantified at this stage. Personal face-to-face and telephone interviews (GSS, WSS and ICVS) do not allow respondents this opportunity. There are two factors which should be considered when completing crime questions using different modes: sequencing and the notion of the word 'threat':
The GSS used Computer Assisted Interviewing (CAI) as the means of data collection and this may also have subtler effects on data quality. Notebook computers may still have a novelty value for many respondents, and the use of notebook computers in the respondent's home may effect the respondent's perception of the interview - a survey may be seen as more important or more objective when computers are used to collect data (Tourangeau & Smith, 1996). The involvement of an interviewer helps maintain respondent motivation (Fowler, 1993). Therefore, for all interviewer based surveys (i.e. GSS, ICVS and WSS) this may assist in keeping respondents focused on providing satisfactory answers rather than trying to find a way to complete the questionnaire as quickly as possible. In the mail back NCSS, there is more information that is given to the respondent, such as the inclusion and exclusion boxes that provide information for the respondent on what types of events to include or exclude for the questions. Generally for the four surveys being considered the definition of what to include or exclude as 'use force or violence against you' is unclear for the assault-related questions. Furthermore, respondents may attempt to accumulate a common ground based on previous information thereby using this information to determine the context of subsequent questions (Clark & Schober 1992). As the previous topics in sequencing differ in the different surveys it is important that this information on what to include or exclude be readily available to all respondents. SAQs (such as the NCSS) allow respondents to change their answers. For example, in order to make their responses consistent with the title, some respondents may decide to go back and change their responses after being asked question 61 (form A) 'Do you consider this to be a crime?'. The quantitative impact of this on the NCSS victimisation rates is not known, though it is expected to be minimal. The NCSS questionnaire included a number of prompts for inclusions and exclusions for main questions. This is necessary in a SAQ as the respondent does not have the presence of an interviewer to assist them. These prompts were developed over a number of iterations to cover the key issues related to the question. The GSS, WSS and ICVS questions are asked of the respondent by the interviewer without prompting which is normal practice for an interviewer based survey. The interviewer will only assist the respondent if asked about inclusions or exclusions and interviewer instructions and training cater for these circumstances. Within the NCSS there is a specific exclusion of incidents involving name calling, swearing, etc. which did not involve a physical threat. In the GSS this was only stated by the interviewer if the respondent specifically asked. Therefore for the GSS, WSS and ICVS respondents may adopt a wider definition of assault than in the NCSS. The table below shows that if the assault victimisation responses are disaggregated into actual versus attempted assault within each survey, there is a similar difference in the proportions of actual versus attempted assault victimisation for the GSS and NCSS (i.e. 54% actual in GSS and 60% actual in NCSS) whereas the WSS shows a different pattern with 85% of identified assaults being actual assaults. When comparing prevalence rates for actual assault for the GSS, NCSS and WSS, the GSS and WSS are not significantly different from each other though both are still significantly different from the NCSS. This difference in the reporting of actual versus attempted assault in the WSS may be due in part to the difference in the question responses available between the GSS/NCSS and the WSS. In the GSS and NCSS respondents can answer yes or no to a question regarding attempted/threatened assault, whereas for the WSS the respondent is asked to answer yes or no to specific examples. 5. COMPARISON OF PREVALENCE RATES FOR ASSAULT FROM ABS SURVEYS, By actual versus attempted assault
The possible impact of prompts on break-ins was also compared for the NCSS and GSS. The table below indicates that any differences in prompts for attempted break-in does not explain differences in reporting between the two surveys as there is still a significant difference in the prevalence rates for actual break-in. The possible impact of different population bases (i.e. GSS used person population whilst NCSS used household population) has been dealt with by the re-calculation using households as the base population for the GSS. This resulted in a slight increase (approximately 3%) in the rate for the GSS. 6. COMPARISON OF PREVALENCE RATES FOR BREAK-IN FROM ABS SURVEYS, By actual versus attempted break-in
A final difference between self-administered questionnaires and personal interviews involves whether the questions are read to the respondent (auditory) or by the respondent (visual). The personal interview questions are presented aloud for the GSS, ICVS and WSS and this eliminates the requirement that the respondent be able to read. This may be important within sub-populations where literacy problems are common. On the other hand, when the questions are only read aloud, the respondent has less control over the pace of the interview and may be prone to 'primacy' effects, favouring options presented early in the list of permissible answer categories over those presented toward the end (Tourangeau & Smith, 1996). The latter would be unlikely to have had an impact on the GSS and ICVS with regard to victimisation questions as they involve 'yes' or 'no' answers. However, this could be a consideration for the WSS which used a series of possible responses for the crime victimisation questions. Context Effects Context effects occur when the preceding questions influence responses to subsequent questions (directional) or when the order in which the questions are administered affects the correlation between the target and the context questions. Context effects are difficult to predict because of their subjective nature, with the impact varying for different people. It is possible that the context effects may contribute to the differences in reporting of assault victimisation in the surveys listed above. However, as with mode effects, without developing an experiment to compare the differences between the survey methodologies there is no way of quantifying how much each factor contributes to the overall difference in results. Therefore, the following is a qualitative assessment of the possible context effects in the different crime victimisation surveys. There are two ways that context effects can influence the cognitive process of respondents: Assimilation (respondents might interpret new information by associating and including it with existing knowledge) and Contrast (respondents focus on the differences between new information and existing knowledge and exclude the new information). A respondent's comprehension and interpretation of question meaning is affected by the context in which the question is presented. Hence, the more ambiguous the wording, the more the respondent relies on the context of the question to understand the question. One of the mechanisms producing context effects is the context and retrieval processes. Respondents tap only a small portion of the potentially relevant information when formulating their responses to survey questions. Information that is highly accessible in memory is likely to dominate the respondents response i.e. respondents take what comes to mind easily (Schuman & Presser, 1981). The NCSS may appear to respondents to be focused on a narrower concept of crime than the other surveys, in particular the GSS. The NCSS is introduced as a crime and safety survey and respondents are able to refer to the purpose of the survey at any time to give them a context or framework to answer the questions (i.e. they may assimilate or contrast new information with the purpose of the survey at any time because the NCSS is a SAQ). Furthermore, the NCSS states that its purpose is about crime prevention and forming a basis for community programs. Respondents may only report assaults which they believe meet this purpose. Interestingly, when the scope of the ABS survey being considered is confined to females aged 18 years and over, the assault prevalence rate for the NCSS increases and decreases for the GSS, bringing the prevalence rates for assault victimisation closer together (see table 7 below). Looking at this in another way, under the GSS there is a greater proportion of males reporting assault victimisation than females compared to the NCSS. This may be related to the context of the surveys, whereby males do not see themselves as being victims of crime and/or are not interested in crime prevention (the purpose of the NCSS), although they do acknowledge that they have experienced an actual or attempted act of force or violence. The WSS and GSS were both interviewer based surveys and introduced a more general concept of assault. The WSS referred to 'experiences of aggressive or threatening behaviour' and the GSS referred to 'crimes that may have happened to you'. WSS used specialised procedures to ensure that respondents felt safe and to encourage respondents to report all incidents of aggressive behaviour rather than narrowing this to only crime reported to police. The general survey context along with specific question order context encourages GSS respondents to have a broader definition of crime. The GSS does not mention the topic of crime until this module is reached approximately half way through the interview, and there are no specialised procedures involved in eliciting information on assault victimisation. The questions asked prior to the questions under analysis (i.e. questions asked prior to those on assault victimisation) can impact on the response. The NCSS and ICVS ask questions regarding robbery prior to questions regarding assault. Any element of force or violence (assault) that is associated with theft is designed to be counted as part of robbery in these two surveys, and therefore should be excluded from the assault counts. The NCSS asks about sexual assault after assault. Checks on the NCSS questionnaire allow any sexual assault data for females aged 18 years or over to be excluded from the assault counts. The WSS asks about sexual assault before assault, and therefore any assault that was in a sexual context is designed to be excluded from the assault counts. The GSS asks about assault victimisation before any other type of victimisation involving the use of force or violence. Therefore, the assault victimisation figures from the NCSS should not include sexual assault or assault associated with robbery, and the WSS assault victimisation figures should not include any sexually related assault experiences, whilst the GSS may include both sexual assault and assault associated with robbery in the assault victimisation data. For comparability, all assault related offences are aggregated and presented in the table below for the same age and sex respondent group (note that ICVS data for the scope requested is not available at this stage). The assault-related prevalence rate for NCSS is significantly lower than those for the GSS and WSS, even after adjusting for age, sex and offence scope differences. 7. COMPARISON OF PREVALENCE RATES FOR VICTIMS OF ASSAULT RELATED OFFENCES FROM ABS SURVEYS, Adjusted for age and sex and offence scope differences
Question Wording What may appear to be slight differences in the wording of a question can have a great impact on the responses to that question. For the assault questions in the four surveys being discussed, the wording is the same for the two surveys with the greatest differences in results (GSS and NCSS). For break-in victimisation there is one key difference between the GSS and NCSS. In the NCSS the question stated break-in to your home, with garage and shed in an include/exclude box below the question, whereas for the GSS, home, garage or shed were all read out in the actual question. An analysis of the Recorded Crime statistics indicates that approximately 8% of residential unlawful entries recorded by police are for outbuildings (i.e. garages and sheds). However, these are less serious matters and are often not reported to police, therefore the 8% figure should be seen as a guide only. Using Recorded Crime data as an indication, it seems likely that at least 8% of break-ins will occur in garages or sheds, and therefore differences in question wording (i.e. the exclusion of the mention of garages or sheds in the question) may have some impact on results for break-in between the GSS and NCSS. Coding Differences When survey responses are returned to the initiating agency there may be a number of forms where further detail or comment indicates that the response provided may be incorrect. Depending on the proportion of recoding that is completed (or able to be completed) this may result in changes to the crime victimisation rates that are actually published. For the ABS surveys mentioned above (GSS, NCSS and WSS), only the NCSS contained detailed comments (due in part to being a SAQ). An analysis of the recoding indicated that the change in prevalence and incidence rates based on recoding was not significant (6% of all break-ins were taken out and 3% of all assaults). Information available from the NCSS has indicated that recoding alone would not make a significant change to victimisation rates for assault or break and enter. CONCLUSION Sources of crime data can vary on many different levels and for a number of valid reasons. Ultimately the choice of which data source is used should be an informed decision made by the user based on an understanding of the purpose of the data source and the methodology behind it. The ABS assesses the fitness-for-purpose of data sources using the ABS Quality Framework (Allen, 2002). The framework contains 6 dimensions of quality that should be considered when choosing a data source: relevance, accuracy, timeliness, accessibility, interpretability and coherence. Users should also refer to the Explanatory Notes in publications in order to assess the fitness of data for use in their situation. Both victimisation survey data and police recorded crime data contribute to informing users about the nature and extent of crime victimisation. Data from victimisation surveys can be used to contextualise information from the police recorded crime data. Alternatively, the two types of data sources can be used to test alternative hypotheses related to criminal activity (Statistics Canada, 1997). Neither administrative statistics nor victimisation surveys alone can provide comprehensive information about crime. Each is useful for addressing specific issues. However, even within a particular method of data collection (i.e. survey or administrative data) there are differences between collections. Administrative collections may have limitations due to different practices between agencies supplying the relevant data. For example, information recorded by police agencies may vary between states and territories due to legislation, recording systems and recording practices. Surveys may have limitations due to differences in methodology and changes between successive cycles. Following a brief comparison of two ABS administrative collections, Causes of Death and Recorded Crime statistics, this paper has presented comparisons of four surveys to illustrate the nature and extent of differences in survey methodologies:
Each collection has its own purpose, such as the comparison and analysis of different social phenomena (e.g. GSS) or the measurement of crime and safety across states and territories over time (e.g. NCSS), and each provide different ways of measuring similar phenomena. As illustrated in section 4, there are numerous elements that combine to make up a single survey methodology. These include:
Differences in any one of these elements may impact on statistics from the collection and on response to an individual question within it. This paper's review of the survey methodologies, using assault victimisation results from the four surveys, is summarised in the following table. 8. SUMMARY OF REVIEW OF SURVEY METHODOLOGY USING ASSAULT VICTIMISATION RATES
As the table shows, there are a number of methodological factors expected to contribute to the differences between the survey results. However, a number of these factors have not been quantifiable, including the impact of the mode of the survey and the context of the survey and its questions. The ABS is currently investigating the crime victimisation survey methodology, with the aim of ensuring continued collection of quality crime victimisation statistics. This paper aims to increase community understanding of the nature of crime measurement in Australia, and why findings from data sources may differ. Ultimately, users must decide which measure of crime is fit for their purpose. The information in this paper can help inform that decision. FURTHER INFORMATION For further information about this product, please contact Catherine Andersson on Melbourne 03 9615 7375, email catherine.andersson@abs.gov.au or email crime.justice@abs.gov.au LIST OF REFERENCES Allen, B 2002, Qualifying Quality - Issues of Presentation and Education. Presented at the International Symposium on Data Quality, Quebec, Canada, 16-19 October 2001. Australian Bureau of Statistics (a), Causes of Death, Australia (ABS cat. no. 3303.0) Australian Bureau of Statistics (b), General Social Survey (ABS cat. no. 4159.0) Australian Bureau of Statistics (c), Labour Force, Australia (ABS cat. no. 6202.0) Australian Bureau of Statistics (d), National Crime and Safety Survey, (ABS cat. no. 4509.0) Australian Bureau of Statistics (e), Recorded Crime, Australia (ABS cat. no. 4510.0) Australian Bureau of Statistics (f), Women's Safety, Australia (ABS cat. no. 4128.0) Australian Bureau of Statistics (g), Women's Safety - User Guide, Australia (ABS cat. no. 4129.0) Bushery, J 1981, Recall Biases for Different Reference Periods in the National Crime Survey. Proceedings of the American Statistical Association Survey Research Methods Section: pp. 238-243. Carcach, C & Makkai, T 2003, The Australian Component of the 2000 International Crime Victims Survey (ICVS), Technical and Background Paper Series, no.3, Australian Institute of Criminology. Clark, C H & Schober, M F, 'Asking questions and influencing answers', In, Tanur, J.M. (ed) 1992, Questions about Questions: Inquiries into the Cognitive Bases of Surveys. Russell Sage Foundation: New York. Coleman, C & Moynihan, J, 'Haunted by the dark figure: criminologists as ghostbusters?', In, Maguire. M (ed) 1996, Understanding Crime Data: Haunted by the dark figure. Open University Press: Buckingham UK. Fowler, F J 1993, Survey Research Methods. Sage Publications: Newbury. Holbrook, A, Green, M & Krosnick, J 2003, 'Telephone versus face-to-face interviewing of national probability samples with long questionnaires: Comparisons of respondent satisficing and social desirability bias', Public Opinion Quarterly, Vol 67: pp. 79-126. Schuman, H & Presser, S 1981, Questions and answers in attitude surveys: experiments on question form, wording and context. Academic Press: New York. Skogan, W G 1990, 'The Polls - A Review: The National Crime Survey Redesign', Public Opinion Quarterly, Vol 54: pp. 256-272. Statistics Canada 1997, An Overview of the Differences between Police-Reported and Victim-Reported Crime. Strangeland, P 1995, The Crime Puzzle: Crime Patterns and Crime Displacement in Southern Spain, cited in The effect of interview method and response rate on victim survey crime rates, presented at the AIC conference in Adelaide. Tourangeau & Smith 1996, Asking Sensitive Questions. The Impact of Data Collection Mode, Question Format, and Question Context. Public Opinion Quarterly, Vol 60: pp. 275-304. Van Dijk, J M 1990, Experiences of Crime around the World: Key Findings from the 1989 International Crime Survey, Kluwer Law and Taxation Publishers, Deventer, Netherlands. Zip, J F & Toth, J 2002, 'She said, he said, they said: The impact of spousal presence in survey research', Public Opinion Quarterly, Vol 66: pp. 177-208. Document Selection These documents will be presented in a new window.
This page last updated 8 December 2006
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
