Qualitative indicators and their
relevance in M&E
August 11, 2021
Indicators are integral to the design, implementation and monitoring and evaluation of every development project. Different types of indicators are used in monitoring and evaluation, such as ‘quantitative’, ‘qualitative’ and ‘hybrid or mixed’. In this blog post, we will deep dive into qualitative indicators and explain what they are, how they differ from other indicators, how to formulate them, how to use progress markers to set them and last but not least, how to measure qualitative indicators and the tools you can use to measure them.
But before we jump into qualitative indicators, let’s understand what indicators are and why they are important in M&E.
What are indicators and why are they important?
Indicators are important tools to benchmark, target and monitor performance. Simply put, indicators are variables that provide reliable means to measure achievements, changes and transformations brought about by an intervention which helps the project team to gain a holistic understanding of their intervention, assess their actual progress against their targets, set priorities and allocate resources based on real needs, learn from their experiences and provide a clearer picture of their results to their stakeholders for making evidence-based decisions.
What are qualitative indicators?
Development projects are complex, therefore, in order to fully understand their processes and outcomes, we need approaches that are not only based exclusively on the measurement of tangible and material outcomes but also approaches that are able to explain the intangible characteristics, properties, abstract concepts and qualitative processes of a project. This is where qualitative evaluation comes into play – qualitative evaluation sees development projects as dynamic and evolving and not necessarily following a predetermined direction.
Qualitative indicators are naturalistic which means it does not attempt to manipulate the project or its participants for evaluation, rather, it studies the processes naturally as they unfold. The naturalistic approach helps to go beyond pre-determined and expected outcomes to capture the unexpected, differential impact and the actual changes that occur as a result of a project.
Unlike quantitative indicators, qualitative indicators do not strictly involve enumeration, which allows them to capture a much broader picture and nuances of a project and describe the nature of the changes more thoroughly. These indicators provide detailed information on any changes taking place, the nature, character, extent and scope of these changes or the process leading to those changes. The qualitative approach also emphasises the importance of getting close to project participants in order to understand more authentically their realities and the details of their everyday lives.
Qualitative evaluation is holistic, meaning it sees a project as a working whole which allows the team to understand and analyse the project from many different perspectives. This holistic approach is also useful in capturing different dimensions of a development project, including specific information about the context, participants, interrelationships with other projects, activities, patterns of behaviour or the nature of the relationships among groups, individuals, organisations, etc. Therefore, qualitative measurements are best suited to help track the progress of innovative, complex, multifaceted or multi-dimensional projects, contexts and objectives that are not easily measurable by just quantitative means.
How are qualitative indicators different from quantitative and hybrid indicators?
How to formulate good qualitative indicators?
There are many criteria to consider when developing an appropriate set of indicators for a project. The nature of indicators depends on your project’s specific objectives and its social and cultural aspects. Identification of indicators must be done in the project planning or design phase and it must be a participatory activity with inputs from different team members and stakeholders. Indicators must be reviewed and redefined on a regular basis to match with the evolving context of the project. When developing indicators, make sure they are realistic, intelligible, unambiguous, replicable, comparable, easy to monitor, record and report on, yield high-quality data and are context-specific. More on qualitative and quantitative data collection methods in M&E.
Once indicators are developed, it is equally crucial to understand how to use them. One single indicator can be measured with different methods and at different levels of quantification. Therefore it is best to begin by asking these questions: what do I want to measure and what do I want to use it for? Being explicit about your assessment objectives is crucial for setting sound and reliable indicators
To establish a qualitative indicator, it is necessary to set strict criteria to measure change over time and to reflect on the outcome, so that you can track your actual progress against your set objectives. A qualitative indicator is usually expressed in descriptive terms, which can include statements and narratives. Examples of qualitative indicators include – NGO functional capacity; level of participation of women in local governance; involvement in decision making about service delivery; level of employee satisfaction; changes in knowledge and attitudes, etc.
Top tips:
- Structure your indicator to allow annual assessments against predetermined and well-defined criteria.
- Your statement needs to describe the gradual or milestone changes so they can be assessed over time.
- Your indicators must meet the standards for indicator validity and reliability and they must be SMART (specific, measurable, achievable, relevant and time-bound.)
- Consider the system for data collection, including who will do it and how often. Do you have the required capacity and resources for data collection?
- Establish controls to maintain the data quality.
- Leave room for flexibility, unexpected developments and changes emerge all the time, so make sure you are able to adjust, remove or add new indicators as needed.
Progress Markers as an innovative methodology to set qualitative indicators
Outcome mapping (OM) is a methodology for planning, monitoring and evaluating development initiatives in order to bring about sustainable social change. This methodology is useful when you need to identify individuals, groups, organisations or stakeholders with whom you will work directly to influence behavioural change, plus when you need to track and monitor behavioural change and the strategies to support those changes.
The methodology of Outcome Mapping introduces the concept of boundary partners that can be useful in identifying intended users of an evaluation by focusing on those who are directly associated with the initiative itself. It’s a participatory approach to develop outcome challenges and progress markers together with the boundary partners.
Progress markers are a set of statements describing a gradual progression or milestone changes in a boundary partner leading to the outcome challenge. Progress markers are very useful to measure progression through a qualitative approach towards the outcome, therefore progress markers are central in the monitoring and evaluation process. Progress markers can be seen as qualitative indicators as they can be used to monitor the progress of the project towards achieving its overall goal in a systematic manner.
Examples of milestones that you can set in Progress Markers are:
✔ Love to see: expanding influence, sharing expertise, behaviour change.
✔ Like to see: active engagement, learning, commitment.
✔ Expect to see: early encouraging response to the project, initial engagement.
Each Progress Marker describes a changed behaviour by the target group and can be monitored and observed. Progress Markers could be established as a set and could be graduated by stages from easier to a more difficult stage to achieve the change process of a target group.
How to measure qualitative indicators?
When you set up qualitative indicators, it is important to think of your data collection system. Some tools are more appropriate for process-oriented indicators, some for indicators that measure certain criteria. Other formats are good tools to assess particular attributes or for capacity building. It is important to pick the one that provides the best kind of information you need to measure your qualitative indicator.
Some tools you could use to collect data for qualitative indicators:
✔ Focus groups: these allow participants from a particular sample or sub-set of the population to answer questions and offer opinions or insights on a topic, event or perceived change.
✔ Direct observations: are consistent and organized site visits by observers. These are ideal for collecting data about what goes on in practice, rather than what goes on in theory.
✔ Assessment of attributes: this is useful to assess the development of specific outcomes, components or elements of a project. It takes a set of attributes and asks questions about those attributes to gain a better understanding of the progress of the report.
✔ Rubrics: you could develop a rubric that consists of statements that describe criteria for assessing different levels of performance within your outcomes.
✔ Milestone scales: this tool outlines sequential stages or milestones of your project and its processes and measures movement along this scale. These work best when each stage is clearly defined and when the defined stages realistically represent the local context and processes involved.
✔ Key informant interviews (KII): this is a tool that could be structured or semi-structured relying on a list of issues or topics to be discussed according to the qualitative indicator.
✔ Questionnaires: the questions are organised around key aspects of a particular subject or key aspects of a particular process, phenomenon, function, or organisation.
✔ Case study: this method is an important tool for gathering, integrating and accessing information across a manageable set of well-selected cases.
Utilizing qualitative indicators in development projects is still in its infancy stage. However, with new changes and progress within the development sector, key development players and professionals have realized that numerical indicators alone are rarely adequate to evaluate the complex dynamics of development projects and capture their holistic picture. Therefore, many are adopting a combination of qualitative, quantitative and hybrid indicators into their projects to strike a balance between practicality and comprehensiveness, and demonstrate a richer understanding of the dynamics at play.
We hope you found our article helpful. If you have any suggestions or comments on this article or ideas for improvement then please mention them in the comment section below.
Key References
- Indicators, INTRAC
- Wageningen University & Research (2018). Outcome Mapping.
- Outcome Mapping Practitioner Guide, Outcome Mapping Learning Community
- Handbook on Qualitative Indicators, USAID
- Combining Quantitative and Qualitative Aspects of Indicators for Assessing Community Resilience, Daniel Becker, Stefan Schneiderbauer, John Forrester and Lydia Pedoth.
By Chandani Lopez Peralta and María José de León at TolaData.
