How do you manage evaluation properly?
Posted on August 30, 2017
Don Macdonald, a trainer, trustee and former charity CEO, is writing a series of New to Management blogs for us in anticipation of his new book, Twenty First Century Skills for Non Profit Managers, being published by BEP in November.
Impact evaluation is now essential in our sector, with increasing numbers of funders requiring evaluation results and systems as part of their bidding process. If you manage a small charity where you are responsible for organising evaluation yourself or commissioning a consultant, you must put effective systems into place. Even if you have not studied social policy, it is still possible to organise something worthwhile.
To evaluate properly you need to know your targets and objectives. Of course, with some contracts from statutory agencies, targets are defined for you. With trust funders, you may be asked to define your own targets, therefore any that you propose must be both realistic, so that you can achieve them, and challenging, so that the trust will approve them (more about target-setting in an upcoming blog).
You must then make sure you have the right systems in place for collecting data on your participants, activities and results (See below). Unless your organisation is tiny, you will almost certainly need a computer programme to record and analyse any data collected, either a spread sheet or a Customer Relationship Manager (CRM) database and there are even some free ones available.
If you have collected it properly in the first place, this data can always be evaluated in more depth at a later time. Make sure you get participants’ agreement to collect data and follow any other legal requirements, as new ones arrive with GDPR. Don’t collect irrelevant data, and any data collected must be measurable and objective, using phrases such as ‘distance travelled’ is too woolly. You need to measure:
- Numbers and profile of users (ethnic origin, gender, age, residence, background regarding service provided e.g. employment status for job-finding projects)
- Inputs, against targets, namely actual services provided to different clients (e.g. reaching the right client group, courses and support provided)
- Outputs against targets (e.g. starting work, qualifications)
- Outcomes against targets (e.g. staying in work, changes in behaviour)
- Feedback and self-assessment from clients and stakeholders and partner agencies
Another way to ensure that monitoring and evaluation are soundly based is to use a framework known as the Theory of Change (TOC). Developed by the Aspen Institute, it defines long-term goals such as the actual changes that are desired, and works backwards to identify critical success factors that are necessary to achieve the long-term goal. The table below shows a TOC mapping exercise, outlining the various outcomes, outputs and activities necessary to achieve long-term outputs.
Theory of Change showing the mapping of a reduction in offending project
||Outcomes at end of programme
||Long-term wider impact
|Action planning, information & advice, one-to-one counselling, positive change course & skills training for 100 offenders & ex-offenders from one region
||Programme attended by 90% of participants for 75% of sessions
||75% of participants complete positive change course & also achieve a skills qualification
||75% of completers progress into work, further education or self-employment, with a reduction in reported offending
||Lower levels of offending, greater levels of employment, better health and well-being in the community
It is important that staff, volunteers and beneficiaries are consulted in development of evaluation systems to bring a wide range of ideas crucial for the success of the project. If you can organise a thorough evaluation, it is also useful for publicising your organisation’s work and effectiveness.
Please share your comments about this blog below, or join the conversation on Twitter.
Have your say!
(Max approx. 500 words)