ABS QUALITY FRAMEWORK
The ABS uses a quality framework as the conceptual basis for the data quality assurance procedures undertaken across all ABS data collections. Its dimensions include accuracy, timeliness, integrity, relevance, accessibility, interpretability and coherence. Each of these dimensions is related and the ways that they interact can also impact on data quality. The various elements of quality have a complex relationship and any action taken to address or modify one aspect of quality may affect one or more of the other elements. For example, all the other dimensions of quality impact on relevance and information provided to ensure statistics are interpretable will also serve to define coherence. Inevitably trade-offs must also be made between accuracy and timeliness, between continuity over time and revisions, between depth and completeness and response burden on data suppliers. Achieving an acceptable level of quality is the result of addressing, managing, and balancing the various factors or elements that constitute better quality.
A brief discussion of these aspects of quality, as they relate to causes of death statistics, is included below
The relevance of statistical information reflects the degree to which it meets the real needs of clients. Relevance is concerned with whether the available information sheds light on the issues most important to users. Relevance of data must be tested against the requirements of any particular user of the statistics. Users need to make a determination of the relevance of the dataset, taking into account all of the aspects of quality, for the purpose to which the statistics are being used.
Issues for causes of death data:
- The primary objective of the owner of the source data can differ from the information needs of the statistical users. Registrars of Births, Deaths and Marriages and Coroners have legislative and administrative obligations to meet, as well as being the source of statistics. As a result, the population covered by the source data, the time reference period for some data, and the data items available in the registration system, may not align exactly with the requirements of users of the statistics.
- There can be differences between the defined scope of the population (i.e. every death occurring in Australia) and the actual coverage achieved by the registration system. Levels of registration can be influenced by external factors and coverage achieved will be influenced by the steps taken by the owners of death registration systems to ensure all deaths are registered. For example, a death certificate may need to be produced in order to finalise certain other legal requirements e.g. finalisation of a person's estate.
- There are eight different registration systems within Australia. Each jurisdiction's registration system, whilst similar in many ways, also has a number of differences. These can include the types of data items collected and the definition of those data items, and business processes undertaken within Registries of Births, Deaths and Marriages including coding and quality assurance practices.
The accuracy of statistical information is the degree to which the information correctly describes what it was designed to measure. It may be described in terms of major sources of error that potentially cause inaccuracy. Quality standards used to achieve administrative goals can be different to quality standards needed to produce statistics, although the application of sound quality assurance principles by the administrative program will usually result in quality suitable for statistical use.
Non-sample errors are the main influence on accuracy in datasets such as this which are a complete census of the population rather than a sample. Non-sample error arises from inaccuracies in collecting, recording and processing the data. The most significant of these errors are: misreporting of data items; deficiencies in coverage; non-response to particular questions; and processing errors. Every effort is made to minimise error by working closely with data providers, the careful design of forms, training of processing staff, and efficient data processing procedures.
Issues for causes of death data:
The main sources of non-sample error are:
- completeness of an individual record at a given point in time ( e.g. incomplete causes of death information due to non-finalisation of coronial proceedings)
- completeness of the dataset e.g. impact of registration lags, processing lags and duplicate records
- extent of coverage of the population (whilst all deaths are legally required to be registered some cases may not be registered for an extended time, if at all)
- lack of consistency in the application of questions or forms used by data providers. For example, sometimes old forms are used before being replaced with new forms and so there could be a period of overlap when a mixture of questions is used, or different questions are asked on death registration forms on the same subject. For occupation, the questions include main occupation during working life, usual occupation, and current occupation
- particular data items which would be useful for statistical purposes may not be collected by jurisdictions where that item is not essential for administration purposes
- question and ‘interviewer’ biases given that information for death registrations are supplied about the person by someone else. For example, Indigenous origin as reported by a third party can be different from self reported responses on a form. Forms are often not subject to the same best practice design principles as statistical questionnaires, and respondent and/or interviewer understanding is rarely tested
- level of specificity and completeness in coronial reports or doctor's findings on the Medical Certificate of Cause of Death will impact on the accuracy of coding
- errors can occur in coding of the causes of a death to ICD-10. Consistency between mortality coders is a contentious issue with literature suggesting only a 50% concordance between coders at the very detailed (four digit) level (McKenzie et. al., 2001).
The timeliness of statistical information refers to the delay between the end of the reference period to which the information pertains, and the date on which the information becomes available.
Issues for causes of death data:
- A balance needs to be maintained between accuracy (completeness) and timeliness, taking account of the different needs of users.
- Causes of death statistics are released with a view to ensuring that they are fit for purpose when released. Supporting documentation for causes of death statistics are published and should be considered when interpreting the data to enable the user to make informed decisions on the relevance and accuracy of the data for the purpose the user is going to use those statistics.
- The availability of information from the administrative system to support processing will depend on the reasons for the system, timeliness requirements imposed by legislation or the system, resourcing of the administrative system.
- To meet user requirements for timely data it is often necessary to obtain information from the administrative source before all information for the reference period is available (e.g. finalisation of coronial proceedings).
The coherence of statistical information reflects the degree to which it can be successfully brought together with other statistical information within a broad analytical framework and over time. Coherence encompasses the internal consistency of a collection as well as its comparability both over time and with other data sources. The use of standard concepts and classifications promotes coherence.
Issues for causes of death data:
- Use of the supporting documentation released with the statistics is important for assessing coherence within the dataset and when comparing the statistics with data from other sources. For example, differences in the scope of collections, classifications used, point in time at which the analysis or coding occurred, availability of information for coding purposes, purpose for which the data/information is being produced, and confidentiality protocols may be different for ABS data and other sources of causes of death information.
- Changes to questions, scope etc. over time can affect the consistency of data collected over the period, even when the source of the data is the same. These changes can be the result of legislative or program objective changes.
- The completeness or quality of older versus newer data can also impact on comparisons across time or domains.
- Statistical concepts for questions are not always suited to the administrative purpose or the means of collection.
The accessibility of statistical information refers to the ease with which it can be referenced by users. This includes the ease with which the existence of information can be determined, as well as the suitability of the form or medium through which the information is being accessed.
Issues for causes of death data:
- Often an administrative source can provide the basis for statistical information which has a different nature and focus to the source's principal administrative purpose. There may be a reduced focus or availability of funding within the program to ensure the accessibility of information for non-administrative uses.
- Each jurisdiction has its own legislation governing death registration as well as that governing the coronial process. Jurisdictions also have privacy legislation which governs the accessibility of the statistics.
- The ABS observes strict confidentiality protocols as required by the Census and Statistics Act (1905). This may restrict access to data at a very detailed level which is sought by some users.
The interpretability of statistical information reflects the availability of the supplementary information (metadata) necessary to interpret and utilise it appropriately, including concepts, classifications and measures of accuracy. In addition, interpretability includes the appropriate presentation of data to aid in the correct interpretation of the data.
Issues for causes of death data:
MANAGEMENT OF CAUSES OF DEATH STATISTICS QUALITY
- Information on some aspects of statistical quality may be hard to obtain as information on the source data has not been kept over time. This is related to the issue of the administrative rather than statistical purpose of the collection of the source data.
- Changing business rules over time and/or across data sources can affect consistency and hence interpretability of statistical output.
The following table defines each of the six dimensions of the ABS quality framework and describes their relationship to causes of death statistics. The third column of the table provides a brief summary of the mechanisms used by the ABS to assure quality in relation to each of the quality dimensions
Management of Causes of Death Statistics Quality
SPECIFIC ISSUES REGARDING CAUSES OF DEATH DATA QUALITY
|Dimension||Quality Elements||Mechanisms used to assure quality|
- Key user needs addressed
- Basis for recording
- Client and stakeholder feedback mechanisms in place
- Regular program reviews, priority setting and planning
- Conformity with international and national standards
- Coverage of data sources
- Completeness and specificity of collected data
- Methodological errors associated with data collection or compilation procedures
- Non-sampling error including coding error/uniformity
- Statistical discrepancy
- Revisions to data
- Regular source data reviews and assessments
- Quality control mechanisms in place for identifying errors
- Systems and methods are reviewed regularly
- Regular training of staff
- When data becomes available
- When data is released
- Setting preannounced release dates
- Monitoring of release dates
- Regular reviews of timeliness in conjunction with stakeholders
- Reliability of key meta data
- Over time
- With other data sources
- Internally consistent
- Use of standard international and national frameworks, concepts and classifications
- Data comparison
- Ease with which data can be obtained
- Methods of dissemination
- Cost of data
- Delivery systems in line with user needs
- Access to data for analytical purposes
- Provision of search tools and advisory services
- Promotion of statistics
- Clarity and availability of metadata
- Provision of information on concepts, classifications, methodology and quality
- Interpreting data on release
Availability of high quality death statistics, and mortality data, is of concern to all levels of government (Commonwealth, State and Local). High quality mortality statistics are used as a fundamental measure of the health of the population. Whilst there are numerous issues surrounding production of death statistics, key issues include: recognising the statistical role of Registrars; standardisation of data items between states and territories; registration lags; indigenous status and occupation data.
Recognising the statistical role of the Civil Registration Process
In 2005, the ABS commenced a program to develop better relationships with RBDMs, with the aim of increasing the quality of birth and death statistics. This program has included a range of activities:
- developing a national standard set of data items for collection of birth and death statistics
- having more frequent and structured communication, at both a strategic and operational level, between the ABS and RBDMs
- providing ABS assistance to RBDMs in areas such as form content and design, technical assistance in developing appropriate formats for provision of data to the ABS
- Developing Memorandums of Understanding (MOU) in order to clarify the type of data required by the ABS, security of the information and how the data will be used.
The program has had a number of successes including delivery of data in standard file formats, regular and productive meetings and increased information sharing at an operational level. While MOUs have not been signed, negotiations are continuing positively and the ABS anticipates a number of MOUs during 2008.
Standardisation of Data Items
The ABS recognises that the process required for standardisation across the eight states and territories is an evolutionary one. Since 1993 the ABS has endeavoured to work with RBDMs on forms, systems and quality issues. The range of information currently collected varies significantly across states and territories; therefore standardisation of forms across all states and territories remains an important goal. These projects have had varying degrees of success. During 2005 and 2006 the ABS undertook negotiations with all RBDMs to develop a standard set of data items to be supplied from the administrative system. These changes were implemented with respect of 2007 registration year data.
The second stage of the standardisation process, in which consistency of definitions, questions and procedures is achieved, is in its initial phase. There are still a considerable number of data items for which the information collected is inconsistent. From a national point of view it would be preferable to collect identical information using identical wording and definitions for all birth and death registrations. Although significant progress towards achieving consistency of the information collected has been made, there remain a number of items for which collected data is not comparable across states and territories.
Over the last 6 years, on average 96% of deaths registered in a year are registered in the year in which the death occurred. The majority of the remaining deaths are registered in the following year. However, lags by coroners have occurred and these have led to delays in the release of the causes of death statistics, and in some cases, increased coding to non-specific codes. This affects interpretation of statistics for some external causes, with apparent declines occurring because the detailed causes of death is not available at the time that the statistics are finalised
There are several data collection forms on which people are asked to state whether they are of Indigenous origin. Due to a number of factors, the results are not always consistent. The likelihood that a person will identify, or be identified, as Indigenous on a specific form is known as their propensity to identify as Indigenous. Propensity to identify as Indigenous can be thought of as the proportion of the total, unknown, number of Indigenous people who identify as such on a specific form.
Propensity to identify as Indigenous is determined by a range of factors, including how the information is collected; who completes the form; the perception of how the information will be used; education programs about identifying as Indigenous; and cultural issues associated with identifying as Indigenous.
While it is considered likely that most deaths of Indigenous Australians are registered, a proportion of these deaths are not identified as Indigenous by the family, health worker or funeral director. That is, whilst data is provided to the ABS for the Indigenous status question for 99% of all deaths, there are concerns regarding the accuracy of the data. The Indigenous status question is not always being directly asked of relatives and friends of the deceased by the funeral director
The ABS publishes two statistical series that provide counts of annual number of Indigenous deaths.
The first is a count of the number and characteristics of registered deaths which have been identified as Indigenous. The second, Experimental population Estimates and Projections, is derived from the previous Census, adjusted for undercount, and also uses registered deaths information. The most recent published data in this series is published in Experimental Estimates and Projections of Aboriginal and Torres Strait Islander Australians, 1991 to 2009 (cat. no. 3238.0). In this publication the level of mortality is presented in the 1996-2001 experimental life tables.
The ratio of the number of Indigenous deaths registered to the number of expected deaths compiled from population projections is referred to as the 'implied coverage rate' and is used to assess the extent to which identification of Indigenous people occurs in the deaths collection. The implied coverage of Indigenous deaths is a comparison of the number of deaths registered as Indigenous with the census-based estimates and projections of Indigenous deaths. Given this volatility, and the experimental nature of the base populations, any estimates of coverage are only indicative. The assessment of the completeness of coverage of Indigenous deaths should be interpreted with caution. Over-precise analysis based on Indigenous death registrations, Indigenous deaths coverage or projected Indigenous deaths should be avoided. The table below provides current estimates of implied coverage rates for each of the states and territories.
INDIGENOUS DEATHS (a), Implied coverage - 2002–2006
Deaths registered as Indigenous
Projected Indigenous deaths
Implied coverage of Indigenous deaths (b)
|State / Territory|
|New South Wales|
|Australian Capital Territory|
|n.p. not available for publication but included in totals where applicable, unless otherwise indicated.|
(a) See Deaths Australia, 2006, cat. no. 3302.0 Explanatory Notes for further information.
(b) Calculated as the ratio of deaths registered as Indigenous to projected Indigenous deaths.
(c) Not calculated due to small numbers of Indigenous deaths.
(d) Includes Other Territories.
Occupation data has not been coded by the ABS from 2003 and is only available for 2002 and before.
All state and territory deaths registration forms ask for occupation at death; however, the way in which the question is asked still differs across the states and territories. Main tasks performed, which assists in more accurate coding of occupation, is asked by only half of the states and territories. Even with access to additional information, occupation of deceased is questionable given the reporting protocol (i.e. funeral directors usually ask the questions of next of kin or friend). The table below shows the differences in the occupation question and other information that is collected on the registration forms. Given the disparate nature of these questions across the states and territories, reasonable analysis of occupation at death has been limited. Investigation by the ABS needs to be undertaken to fully understand the links between occupation and mortality, with consequential discussion with RBDMs to implement appropriate methods of collecting data of sufficient quality for analysis.
Status of Occupation Question on Death Registration Form
|State/Territory||Occupation question||Main Task Performed||Other|
|NSW||Usual occupation during working life |
|Retired at date of death|
Was deceased a pensioner at date of death? (Y/N)
(If yes, state pension type)
|Victoria||Usual occupation during working life|
|No||Was the deceased retired and/or a pensioner (Y/N)|
|Queensland||Usual occupation during working life |
|No||Retired at date of death (Y/N)|
|Yes||Was the deceased retired (Y/N)|
|Western Australia||Main occupation (including working life)||Yes||Was deceased retired (Y/N)|
|Tasmania||Main occupation during working life |
|Was deceased retired at the time of death (Y/N)|
|Was the deceased retired (Y/N/U)|
Was the deceased a pensioner (Y/N/U)
|ACT||Usual occupation of deceased||No|