Building a Monitoring Framework for Animal Advocacy Projects: A Step-by-Step Guide
After working on a Theory of Change (aka your roadmap to create change for animals), the next step is to plan what you will do with it: planning a framework to gather data and use it to improve your ToC further. This is key to helping your advocacy work become more impactful over time. This cycle is part of Monitoring, Evaluation and Learning (MEL).
“This was a new but useful tool for me. As I’m going into my second year of utilizing MEL and ToC, I’m getting more familiar with it and it actually improves a lot of the work especially while strategizing.”
Monitoring = Collecting routine data on outputs and outcomes to answer learning questions
Evaluation = More in depth assessment of the your work using additional data collection beyond monitoring data to answer learning questions
If you haven’t already, check our previous post to learn more about Theory of Change.
What to expect from this post:
Monitoring
Learning Questions
Indicators
Monitoring framework
Q&A
Monitoring
Let’s have a look at the example ToC we had from the previous post. It was created for a school engagement program aiming to invite them to provide vegan lunch to their students once a week.
Now that we have a ToC drafted, we can use it to test whether our program is working as expected. To do this we use learning questions. What questions immediately surface when you look at the example ToC above? Where might it fail? To help think through possible questions to ask we can use the following six categories:
Activities: What activities are we doing?
Targeting: Are we targeting the right stakeholders?
Engagement: Are those stakeholders participating and engaging?
Feedback: Are those stakeholders satisfied?
Outcomes: Is stakeholder knowledge/attitude/behavior changing?
Financials: Are we running the program cost-efficiently?
These questions build on one another - before knowing whether you are achieving your intended outcomes, it is useful to know whether you are conducting the number of activities you think you are, and whether you’re reaching the right people and whether they are engaged.
Assumptions: remember our assumptions from the last blog post? These can also be a good source of learning questions. Take a look at your most important assumptions and think about whether any of them need tracking.
It is important to tailor these questions to be specific to your program. We can see how this works by applying it to our example. Let’s look at some examples for each category you could focus on if you were piloting this intervention:
When starting out with building a monitoring framework, it is helpful to start small! Selecting 1-3 learning questions for the first 3 months is a great start, and you can add and swap out questions as you progress with your program.
Indicators
“Working on MEL really made me realize that measuring our impact in communicable ways has been a weakness of ours. I really appreciated all the guidance in identifying risks that would undermine the impact of our program and different ways (not just quantitative) of presenting the results of our work”
To answer our learning questions, we will need to collect regular monitoring data. For each question we develop indicators that will help us define what data we need to collect to get to an answer.
Indicators can be quantitative (e.g. number of attendees at a workshop, % of people from a particular demographic) or qualitative (e.g. reasons for attending the workshop, barriers preventing behavior change). It is good practice to select 1-3 indicators for each learning question.
A good indicator follows the CAR principles:
Credible
Valid: Do our indicators capture what we are trying to measure?
Reliable: Can the data be collected in the same way consistently?
Actionable
Specific action: Do we have a specific action that we’ll take based on the results of measuring that indicator?
Ability to act: Do we have the commitment and resources to take that action?
Responsible
Costs to us: Are the time and financial costs of collecting and analyzing this data worth it compared to the value it delivers?
Burdens on stakeholders: Does collecting this data require too much time from stakeholders?
Let’s look at one of our example learning questions and define some strong CAR indicators for it:
What is the best type of school to work with?
To answer this question we need to know what types of school we are working with and whether they are committing to the pledge. So we first need to define some different variables for school type, e.g. size, primary vs secondary, state vs private, religious affiliation etc.
Then for each type of school we can develop some indicators and assess them against the CAR Principles:
Monitoring Framework
Finally we can pull together our learning questions and indicators into a monitoring framework to guide data collection. Here is a simple example for one of the indicators from our previous section:
Learning Question: What is the best type of school to work with?
The final step is to decide on a data review cycle. For this learning question, we could collect data for 3 months, and then have a team meeting to review that data and see if we can answer our learning questions.
Let’s say we find that primary schools tend to not serve very many meals because they are smaller, and they are also less likely to commit to the pledge. If we have worked with a large number of schools over that period, we might be able to conclude that this is likely to be true for most primary schools, and we can decide to refine our target group to focus on secondary schools only.
We can then select a new question to focus on for the next quarter. Or if we haven’t worked with enough schools to be sure, we might decide to continue collecting data on this question and review again at the end of the next cycle. With each cycle, we update and refine our ToC, and over time our program should become more effective, and we will have stronger evidence to understand its effectiveness.
Q&A
-
Your ToC should be seen as a living document. As you learn more about your intervention you reflect that in the ToC by making updates.
“Do not overstress your ToC because it will change during the process of working and it's very normal.” – Poppy, Indonesia Director
-
When building your ToC, it is helpful to include a variety of different stakeholders. Talking through it with others can be a useful way of surfacing assumptions or risks that you missed.
Sometimes you won’t be able to think of all of the critical assumptions and risks ahead of time, they will only become apparent when you start working on the intervention. But keeping alert to unsurfaced assumptions and risks, and updating the ToC when they appear can help you to adapt quickly to the new circumstances.
-
Working on MEL does take time away from delivering the actual program, and this can be a real source of frustration especially for small teams with limited resources. The risks of doing no MEL however are greater: imagine spending years working on a program and not knowing whether it is actually having an impact on the animals you are trying to help. The solution is to do MEL that is appropriate to your organization’s capacity. Start with at least building a ToC, and gradually work on tracking indicators for a few learning questions, and build over time. Don’t try to build out a large MEL system if you don’t have the resources to dedicate to it. And don’t be afraid to ask your funders for dedicated MEL resources as part of a project.
“I was relatively resistant to the whole process at first, as I thought we already had a workable ToC underlining our program. But looking back, I can see that all that extra time spent thinking about the program, both close-up and from far away, really helped hone our approaches to be as in line with our goals as they could be.” – An AAA Country Director
-
A good monitoring framework will specify when data needs to be collected, and who is responsible. That person should set reminders that trigger when data needs to be collected, so that they don’t have to remember all the time what needs reporting. Then, set a periodic review meeting to look over the data and see if the learning questions are answered and the ToC can be updated.
-
THANK YOU Jamie and The Mission Motor for supporting AAA team and sharing this practical guide on today’s blog post!
