|Page tools: Print Page Print All|
CHAPTER 4 A FRAMEWORK FOR STATISTICAL MEASUREMENT
While the Productivity Commission Report provides a valuable context for measuring the contribution of the sector, the application of such a framework to the NPI sector is in its infancy. Defining appropriate measures for outcomes and impacts is problematic, and improvements in data available are likely to be over the longer term.
The Productivity Commission Report presents the Framework in more detail. It is reproduced below as Figure 2.
FIGURE 2. MEASUREMENT FRAMEWORK AND TYPES OF INDICATORS
Source: Productivity Commission 2010, Contribution of the Non–for–Profit Sector, Research Report, Canberra
The framework provides for measurement (collectively or individually) at 4 levels:
The statistical measurement framework for NPIs should also incorporate a variety of other statistical classifications and standards developed for the NPI sector and for social and economic statistics more generally. These include the International Classification of Non–Profit Organisations (ICNPO) (see appendix 5), The Australian and New Zealand Standard Industrial Classification (ANZSIC), the Australian Culture and Leisure Classifications, frameworks for the measurement of social capital and wellbeing, and a number of other social and economic statistics frameworks.
In assessing the data needs for improved evaluation of the contribution of the sector, the Productivity Commission (p.88) notes that it is important to distinguish two broad purposes of data development, as they will typically require different information:
4.3 APPLICATION OF THE FRAMEWORK FOR MACRO–LEVEL MEASUREMENT
At the macro level, populating the framework with inputs (mainly financial) data is relatively straightforward, provided data collections are in place. Much of the macro data currently available for NPIs relates to inputs.
The measurement of outputs in physical terms is also relatively straightforward for groups of organisations providing the same or similar service, such as school education. Non–financial indicator data are generally not additive across the range of activities undertaken by NPIs, so it is not possible to present a wider view of the performance of the sector. One approach that might have value for aggregating outputs is to use the costs of providing services to weight the different non–financial indicators into a composite index to measure change over time (the approach used in the Australian national accounts to measure change in a range of government services outputs). Such an approach can be envisaged for some subsets of the NPI sector such as health, education and employment services.
Outcomes and impacts measurement becomes increasingly more contentious because of issues around the choice of appropriate indicators. There is a large body of work already available or underway on outcomes indicators for areas of government policy interest. The COAG Reform Council (CRC) has prepared or is preparing framework papers on various aspects of government service measurement under National COAG Agreements. The Productivity Commission’s annual Report on Government Services (ROGS) assembles and publishes data on broad level indicators of public sector performance. Although these indicators mainly relate to outcomes against government policy targets regardless of who is actually delivering the services on the ground, they are the types of broad level indicators that could be considered for the NPI sector. The challenge is to distinguish or attribute outcomes to the services delivered by the NPI sector.
The ABS, AIHW and other Australian Government, and state and territory organisations have also been active in developing indicators of social, economic and environmental conditions. Again, the challenge is to be able to attribute outcomes to the various service providers involved, and to take account of the external factors impinging on outcomes. Care has to be taken in the interpretation of indicators. If particular indicators of wellbeing are deteriorating, it does not necessarily mean that the outcomes from NPI interventions and services have declined. The opposite may be true, as there may be significant other factors at play affecting the demand for services (such as deteriorating economic conditions). Ultimately, outcome indicators may still be regarded as useful as long as NPIs are dominant or substantial contributors, but not necessarily the only contributors. They are also useful to NPIs as indicators of need.
In terms of what is feasible, the potentially high cost of data development and collection (including the costs to individual organisations providing data) has to be weighed against the benefits of having new or improved information. Consultation with stakeholders is needed to set priorities for new data development. The task of developing an agreed set of macro level statistical indicators of outcomes and impacts should not be underestimated. Progress is likely to be slow.
4.4 APPLICATION OF THE FRAMEWORK FOR MICRO–LEVEL MEASUREMENT
Given the diversity of the NPI sector, and the difficulty of aggregation of indicators not denominated in units of currency, the Productivity Commission considers outcomes measurement and analysis to be most suited to the assessment of micro level service delivery programs rather than to macro level measurement across program types or segments of the sector.
Contract agreements between service funders and NPI service providers include reporting requirements to account for funds spent. They are also likely to include a measure of physical units of output and, increasingly, a measure of performance against expected outcomes. Funders will also require information on the financial health and suitability of service providers before a contract is let. These data are required for administrative purposes, but if the data can be collected and stored in a suitable way they have the potential to be captured for statistical purposes and for analysis and use in evidence based policy advice. The ABS has a role in providing advice on data frameworks and systems in order to increase the coherence of information across agencies, and to maximise the statistical potential of data.
The Productivity Commission highlights a need for better and more standardised financial reporting and evaluation for organisations involved in service contracting. The recent adoption of the Standard Chart of Accounts for financial reporting by NPIs to governments should have significant benefits for streamlining administrative processes, and is potentially a significant boost to data that might become available for statistical use. The need for improved measurement and reporting approaches has been echoed in the stakeholder consultation for this IDP.
These documents will be presented in a new window.