Module 19: Monitoring and Evaluation

19.3. A framework for performance monitoring

Outputs...
... Clearly defined performance criteria/targets
    ... Selected and benchmarked performance indicators

If they are to improve the process of service delivery, action plans need to be based on an evaluation of actual performance. Evaluation needs to be based on the use of indicators and targets.

Any improvement in the performance of PPP contracts depends on the existence of a management system within which performance improvements can take place; this leads to a “virtuous circle” of improvement, as shown in Tool 19-1.

The key point underlying all of the above is that action will be taken as a result of measuring performance in any particular area of activity. It is important to understand that in some cases the basic management structures at the institutional level (both within the municipality and the private sector) are underdeveloped; in such cases it is difficult for the municipality both to gather relevant information on performance and from that to develop action plans to improve that performance.

The more detailed framework shown in Tool 19-2 outlines a number of stages in the process for measuring and evaluating the effectiveness of the partnership.

Each of the stages from Tool 19-2 is described in more detail below:

Stage 1: Performance evaluation

If they are to improve the process of service delivery, action plans need to be based on an evaluation of actual performance. Evaluation needs to be based on the use of indicators and targets.

Action plans to improve service delivery need to be based on a sound evaluation of the performance of different methods of service delivery. Performance evaluation needs to answer specific questions; these questions are relevant to those actors who are in a position to take action relating to service delivery, whether it is using conventional tender contract systems or by involving local communities.

Performance evaluation needs to take place against a number of clearly defined criteria or targets, which have been set for a particular reporting period and within the local context. The approach adopted is to define performance indicators having quantitative or qualitative values, which cover the field of service delivery. Associated with each performance indicator is a performance target; the status, or “performance”, of the PPP in meeting its objectives is then assessed by comparing each performance indicator with its respective target. This enables performance comparisons to be made:

◊ between different time periods for a given partnership arrangement; and

◊ between different methods of service delivery.

This Tool does not attempt to prescribe arbitrary performance targets; these must be developed within the local context. However, benchmark values are presented for a number of selected key indicators.

Stage 2: Performance reporting

Investigate the existing performance reporting systems to see whether they are sufficiently well developed to permit a thorough evaluation of the service delivery process.

Performance reporting provides the essential input to performance evaluation. It not only reveals whether planned actions have achieved their objectives, but also identifies common problems and allows improvements to be built into the system for the future. The development of a sound performance reporting system along with the choice of appropriate performance indicators are important elements in the evaluation of the service delivery process.

Stage 3: Characterising performance indicators

Make sure that performance indicators display the appropriate characteristic.
Use the groupings as a starting point to focus attention on the key areas.

Performance indicators can be defined as “variables whose purpose is to measure change in a process or function”. They provide the information from which performance reports are compiled, in order to assist in answering the questions posed by performance evaluation of the service delivery process. Performance indicators should be selected based on managerial requirements.

Characteristics of a good performance indicator are:

◊ there is a valid link between the indicator and the question being addressed;

◊ problems will be detected by use of the indicator;

◊ the indicator gives an idea of the magnitude of the problem; and

◊ the information required to define the indicator is readily available.

Information relevant to the partnership, which performance indicators can measure, can be grouped usefully as follows: general, time, cost, quality, inter-organisational cooperation and partnership and socio-economic issues (guidelines on each indicator group can be found in Sohail and Cotton, 2000).

General

It is useful to have an overview of some of the fundamental aspects of a particular contract, for example who initiated the works; the degree of community involvement; and who has responsibility for operation and maintenance. A number of indicators are presented covering issues of ownership, roles and responsibilities and power relations.

Time

Delays may occur at different stages of the procurement process depending on the method of procurement, the management systems in place and/or the personnel involved. Performance indicators can measure the time performance of the procurement process at different stages of the contract. Avoidable delays may be identified and measures put in place to reduce these in the future.

Cost

The cost of the work is generally estimated at several stages of the procurement process:

– a preliminary estimate is prepared using “rule of thumb” methods;
– a cost estimate is prepared by the engineer, based on an existing schedule of rates;
– a more detailed cost estimate is prepared by the engineer at the tender stage; and
– the initial contract cost (the contractors’ estimate for carrying out the work).

Performance indicators can be selected to monitor the relationship between the engineers’ and contractors’ estimates and the final cost of the contract.

Quality

If the work carried out by the contractor does not meet the required standard, it will be rejected. A single indicator based on the number of times work has been rejected is proposed as a measure of quality performance. However, in many situations truly independent valuations of quality may be lacking, and it is valuable to investigate this as part of an in-depth participatory assessment.

Inter-organisational cooperation and partnership

The ways in which officials and the community interact, and the degree of co-operation between them, can affect the efficiency and success of a project significantly.

Socio-economic issues

In addition to the infrastructure itself, a community may receive a number of other benefits as a result of infrastructure procurement. These may relate to skills development through training, increased employment and empowerment.

Stage 4: Selecting performance indicators for the service delivery process

There is no fixed set of questions that can be applied to all situations. The indicators selected by the municipality will be different for conventional tender contract systems and community-partnered procurement processes. These in turn will then vary from place to place according to the local context. Once the indicators have been selected, the local government must check that they will assist in answering important questions about the service delivery process.
It is advisable for the municipality to avoid collecting large amounts of data (either through objective means or using participatory techniques), which cannot subsequently be put to the intended purpose.

Stage 5: Defining and selecting information

Information about the service delivery process should be available through a management information system.
For community-partnered procurement, the most important knowledge lies with the community of users and may not be recorded in a formal sense.

The nature and form of the information systems is crucially important for determining performance indicators and developing performance reports. The local government must know:
– what information needs to be collected in relation to each indicator; and
– where that information can be found.
This knowledge requires a careful review of the different indicators in order to see whether or not information will be readily available, and if necessary to plan for the collection of the information required. Some information may require processing before it can be used as an indicator. It is also possible that information about the service delivery process is available through a management information system; however, often such information may be poorly developed or non-existent within the municipality.
(Likely sources of information for each indicator can be found in Sohail and Cotton, 2000).

Stage 6: Collecting the information

Distinguish clearly between indicators which require different data collection methodologies.

Collecting information is more complex than appears at first sight, as there are two distinct methods that can be employed.

– Some performance indicators can be assessed in an objective manner by collection of performance data; collection of this information could be done internally using the staff of the institution or by using external consultants.

– Other performance indicators are based on qualitative information; collection of this information often requires either processing of available data or collection of new data using participatory assessments of performance.

© 2004 UNDP,  Manufactured by Margraf Publishers GmbH, Germany