An evaluation of evaluation: problems with performance measurement in small business loan and grant schemes

This project researched the problems of evaluating public sector small busi ness loan and grant schemes. The methodology was to use case studies of fou r loan and grant schemes run by enterprise agencies on contract from task f orces or city challenges. These case studies were selected following a mapp ing exercise of 27 finance schemes in north east London. Research showed that evaluation relied on performance indicators. The main measures used were deployment of funds, job creation, leverage of additiona l funds, lender of last resort, ethnicity of recipients, default rate, and enquiry levels. Four technical problems with these performance indicators w ere found. First, the interpretation of indicators varied between organisat ions. Second, performance indicators judged agencies on areas which were ou tside their control. Third, indicators failed to take account of the full r ange of work involved in managing a loan and grant scheme. Fourth, lender o f last resort and leverage were incompatible in their preferred risk positi on and no attempt was made to reconcile the two. That these technical weaknesses were not taken into account in the weight g iven to performance indicators led to four organisational effects. First, a gencies seemed to have made their own attempt to render the indicators mean ingful, reflecting the conditions under which the schemes had been first es tablished. The existence of strong political, auditing, and time pressures led to three management styles designated as client oriented, bureaucratic, and entrepreneurial. Second, the system encouraged managing agencies to di stort figures to produce more favourable results. For instance, reschedulin g of debts allowed the agency to avoid recording high levels of default. Th ird, the focus of information gathering on external needs (those of the fun ders) seemed to have discouraged agencies from contemplating acid developin g their own data systems. This might reflect either a lack of resources or a fear that any information could be used against them. Several possible ex planations for these problems are explored in this report. Objectives were ambiguous because of the under-development of programme theory. Data was no t always available because its collection was not of value to the enterpris e agency. Communication between funders and agencies was poor because funde rs had come to see their main power as residing in the re-tendering of cont racts. These points present part of the explanation, but a further level of analys is is possible. That theory is under-developed, data unavailable, and commu nication weak are symptoms of a deeper problem. This is the problem that pe rformance management is operating under a positivist theory of knowledge. S everal possible improvements to the evaluation of small business loan and g rant schemes are explored in this paper. Key among these is an acknowledgem ent of the limitations of evaluation through adopting a developmental appro ach within a culture of organisational learning. This paper illustrates the value of adopting a developmental approach, but also the fundamental trans formation which it implies. "With each new evaluation, the evaluator sets out, like an ancient explorer , on a quest for useful knowledge. not sure whether seas will be gentle, te mpestuous, or becalmed. Along the way the evaluator will often encounter an y number of challenges: political intrigues wrapped in mantles of virtue: d evious and flattering antagonists trying to co-opt the evaluation in servic e of their own narrow interests and agendas: unrealistic deadlines and absu rdly limited resources; gross misconceptions about what can actually be mea sured with precision and definitiveness; deep-seated fears about the evils- incarnate of evaluation, and therefore, evaluators: incredible exaggeration s of evaluators' power: and insinuations about defects in the evaluator's g enetic heritage (Patton, 1997: 38; Utilisation Focused Evaluation)." (C) 20 01 Published by Elsevier Science Ltd.