Impact evaluation: overview, benefits, types and planning tips
May 4, 2021
After a thorough consultation with our in-house M&E experts, we have created a 2-part series on impact evaluation and this article is the first part of the two. It provides an in-depth overview of impact evaluation and answers questions such as – what is impact evaluation, when is a good time to conduct an impact evaluation, whom to engage in the evaluation, what are the benefits, types and challenges of impact evaluation, plus some top tips and easy to follow steps to help you and your organisation plan and manage your own impact evaluation.
In the second part, we will walk you through the process of designing an actual impact evaluation work plan and help you grasp its key elements, including the objectives, purpose and scope of the evaluation, key questions, methodologies and more.
Governments, donors, multilateral institutions and development organisations around the world are investing billions of dollars every year in developing and implementing projects, programs and policies to help reduce poverty, improve lives, encourage learning, and protect the environment. As much as it’s important to have these projects in place, good intentions alone are not enough – It is equally crucial to understand the impacts these interventions are making.
Since the Paris Declaration on Aid Effectiveness in 2005, many development actors are now required to report against their core strategic objectives and demonstrate their effectiveness. Meaning, NGOs and civil society organisations have to go beyond just tracking and reporting on what they have achieved to focusing on identifying the real difference their interventions are making in the lives of the poorest and the most vulnerable populations and justify that these efforts are effective in manifesting change.
In order to meet this growing demand for aid effectiveness, development actors are increasingly incorporating impact evaluation into their monitoring and evaluation plans as a tool for learning and accountability and to design projects and policies that are evidence-based.
This article is the first part of our 2-part series on impact evaluation. It provides an overview of impact evaluation, its benefits, types, and some tips on how to plan and manage it. In the second part, we walk you through the process of designing an actual impact evaluation work plan to help you grasp its key elements, including the objectives, purpose and scope of the evaluation, key questions, methodologies and more.
What is impact evaluation?
OECC-DAC defines impact as, “Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended.”
According to the DAC Evaluation Network report, impact evaluation serves both objectives of evaluation: lesson-learning and accountability. When done correctly, impact evaluations should measure both positive and negative changes in development outcomes that can be attributed to a specific intervention, whether they are short or long-term, intended or unintended, direct or indirect. The intervention could be a small project, a large program, a collection of activities, or a policy.
But how do we understand and report on the changes our interventions are making? Change is not an easy concept to capture and explain as it does not happen in a linear path. In impact evaluation, we cannot understand change by simply asking what we have achieved but rather we have to ask ourselves how our efforts were connected to this change and who or what was involved in the change, what strategies were used to bring about the change, what were the contexts that affected how change happened and what was the process or pathway of change?
What are the benefits of impact evaluation?
Impact evaluation helps to demonstrate project success or failure, and provide accountability to all stakeholders, including donors and beneficiaries. It helps to determine if, and how well, an intervention worked to create a change in a particular community of interest or in the lives of our target populations, while demonstrating the extent of the impact and how it came about.
Impact evaluations are also useful in navigating the real needs on the ground and providing answers to project or program design questions to determine which, among several alternatives, is the most effective approach, represents the greatest benefits to the target communities, offers the best value for money and is the most suitable for scale-up and replication. This provides organisations with evidence to make informed decisions for redesigning the current project or for planning of future interventions. Impact evaluation also helps organisations to use the findings from the evaluation to advocate for changes in behaviour, attitudes, policy and legislation at all levels.
Impact evaluations have already proven to be valuable for development interventions.
In 2015, a World Bank report found that, “Projects with impact evaluations are more likely to implement their activities as planned and, in so doing, are more likely to achieve their objectives.”
However, it is also important to note that impact evaluation might not be applicable in all contexts for a range of reasons such as budget, timing, questions of interest etc. It is therefore important to consider impact evaluation as one tool in a wider spectrum of evidence-generating activities.
Outcomes vs. Impact - what's the difference?
Oftentimes, people confuse project outcomes with impact. It’s important to understand that intermediate outcomes are evident during the life of the evaluation as opposed to the long-term impacts of the intervention. Outcomes are the benefits an intervention is designed to deliver, whereas, impacts are higher level strategic goals or long term effects of an intervention.
Achieving the intermediate outcomes may contribute to the intended final impact. For example when a project achieves an increased number of women’s participation in community decision-making (intermediate outcome), this might contribute to the improved economic, social and physical well-being of women, which would be the long term effect of the intervention (impact). So, in other words, outcomes precede, and are usually a precondition for impact to occur.
Types of impact evaluation
An impact evaluation can be undertaken during as well as towards the end of an intervention but the planning must begin early on. Based on the timing and the purpose of evaluation, impact evaluation is categorized into two types.
How to plan and manage impact evaluation?
Before planning and implementing impact evaluation, organisation staff and relevant stakeholders must clarify a few points and should proceed with the evaluation only if it’s appropriate and necessary. Here are some point to consider and steps to follow:
First of all,
- It’s important to determine how relevant the evaluation will be for your organisation’s development strategy.
- An impact evaluation should only be implemented when there is clearly a need to understand the impacts of an intervention and when impact evaluation is the best way to answer the questions about the intervention.
Once the organisation has clarity on the above points, they can proceed with:
- Identifying what needs to be evaluated and generating evaluation questions according.
- Identifying the availability of resources and determining how to mobilize them. To estimate the amount of resources for impact evaluation, organisations can refer to the budgets of previous similar evaluations or use a budget analysis template made available by other organisations. Check USAID’s budget template for reference.
- Given the availability of resources and time, they have to determine whether the findings will be credible and relevant.
- Determining whom to engage in the evaluation, decision making and management, and outlining the required skills of the evaluation team – It is also important to garner commitments from all individuals who will be invested in this process.
- A clear understanding of the appropriate timing for impact evaluation is also crucial.
- Developing an evaluation design, methods and implementation work plan.
- Development and distribution of evaluation reports.
- Encouraging utilization of evaluation results. Impact evaluation is useful when there is a scope to use the findings from the current intervention to inform decisions about future projects. Therefore, it is important to have a clear understanding of how the findings from impact evaluation will be used and by whom?
- Maintaining the quality of evaluation throughout the project cycle.
It is recommended that organisations first take an evaluability assessment before proceeding with the planning. (Check out this article by BetterEvaluation to learn more about evaluability assessment.)
When is a good time to conduct an impact evaluation?
Impact evaluation planning should begin in the early stages of a project. The impact evaluation methodologies need significant investment in preparation and enough time for the collection of baseline data, and where appropriate, the creation of a randomized control trial or comparison group or the use of other strategies to investigate causal attribution.
Instead of leveraging it as a stand-alone component, It is important to address impact evaluation as part of an integrated monitoring and evaluation (M&E) plan. Meaningful impact evaluation cannot be carried out without drawing on data from other ongoing M&E activities and components. M&E enables impact evaluation by providing information on the nature of the intervention, context of the information and additional evidence on how the intervention has been progressing and whether impact evaluation is necessary and when it’s a good time to undertake it.
Although it is a good practice to undertake impact evaluation sooner as it provides useful information to make modifications to the project and improves its efficiency and benefits, one must also be wary that when impact evaluation is undertaken too early, there is a probability that impacts may be underestimated or unnoticed because in some cases, the impacts may not have had sufficient time to develop. When implemented too late, it might miss the timing window to inform decisions.
Whom to engage in impact evaluation - taking a participatory approach
Thinking about whom to involve, why and how in the evaluation process is a crucial step in M&E, therefore, evaluation management arrangements should be clearly described from the beginning of the evaluation process. This helps to develop an inclusive and context-specific participatory approach, which can bring a lot of value to the evaluation. The underlying rationale for choosing a participatory approach to impact evaluation could be pragmatic or ethical, or a combination of both and being clear and intentional about participatory approaches in impact evaluation is an essential step for managing expectations and guiding implementation.
The nature of the project, the purpose of the evaluation, the expectations of the donors, the goal of the intervention, plus the skills and competencies available within the team, all determine how different team members and stakeholders could be engaged in different stages of the evaluation process to maximize the benefits.
Additionally, asking these questions laid out by BetterEvaluation can also help in designing an impact evaluation that is participatory in nature:
- What purpose will stakeholder participation serve in this impact evaluation?
- Whose participation matters, when and why?
- When is participation feasible?
Tip: a common practice includes, creating an ‘Evaluation Management Team,’ which is a steering committee responsible for creating and supervising the ‘Evaluation Team.’ Evaluation management team also provides technical guidance, oversees quality assurance and manages the budgets and field visits and other operational aspects of the evaluation. Additionally, the management team also creates and manages an ‘Evaluation Reference Group’ which is responsible for providing technical and cultural advice. The members for this group are selected from a range of relevant stakeholders.
Challenges of impact evaluation
Impact evaluations are known to be time consuming and expensive, require special skills to conduct and pose many managerial and technical challenges – this is often a challenge for organisations that do not have large budgets and technical expertise to carry out evaluation work. It is also very difficult to determine the appropriate time to execute impact evaluation – depending on the timing of the evaluation, different purposes and results may be reached.
Another key challenge is that many impact evaluation methodologies need to be agreed upon from the start of an intervention, especially if they rely on baseline surveys or randomisation.
This can be difficult in more complex interventions where goals and objectives evolve over time. In such cases it may be more appropriate to use methodologies that do not require extensive baselines.
Counterfactual is another big challenge in impact evaluation – to identify what would have happened in the community in the absence of the intervention. To figure that out, a comparison group must be selected to represent the counterfactual and that requires careful thought, if it’s not done properly then an inappropriate comparison group may invalidate the evaluation results.
Another major challenge of undertaking an impact evaluation is measuring the impact itself. Assessing impact is not an easy task as it is often not visible during the life of a short-term intervention and is more likely to be affected by other interventions and other factors. In practice, a particular intervention is rarely sufficient to produce the intended impacts alone, oftentimes, a combination of similar interventions and projects are required to achieve an impact.
There are several reasons for choosing impact evaluation for an intervention. It plays a crucial role, not just in identifying project impacts but also in assessing them and understanding their dimensions.
We hope this article was helpful in explaining impact evaluation, it’s benefits, types and challenges and in providing easy to follow steps to help you plan and manage your own impact evaluation. Check out the second part of our 2-part series on impact evaluation – “Designing an impact evaluation workplan: a step-by-step guide” and learn how to design and conduct appropriate and effective impact evaluation and make it a part of your overall M&E plan to improve your learning and accountability.
If you have any comments on this article or if you’d like to simply suggest an M&E topic that you would like us to cover on our next blog post then please write to us in the comment section below.
Key Resources:
- Overview of Impact Evaluation, by Patricia Rogers at UNICEF, 2014.
- Impact Evaluation Methods for Youth Employment Interventions – ILO
- Choosing Appropriate Designs and Methods for Impact Evaluation – Australian Government
- Impact Evaluation: Resources – IDB
- Guidance for the Terms of Reference for Impact Evaluations – European Commission
By Chandani Lopez Peralta, Content Marketing Manager at TolaData.
