In pursuit of Impact
By Maung Maung Nyein Chan
Monitoring, Evaluation and Learning Manager, MyJustice
I consider myself a seasoned monitoring and evaluation (M&E) specialist. However, when I joined MyJustice in 2016, my confidence was tested. I wondered: would I be up for the challenge of measuring impact for an access to justice programme?
My M&E challenge
MyJustice was built on a Theory of Change which was rather ambitious to achieve during the first five years of the programme. It aims to empower communities with rights awareness and legal knowledge, while creating a better sense of fairness and accountability among service providers.
The programme also set out to avoid importing external ideas, intending rather to “go with the grain” by promoting existing local practices. In accordance with this concept, local organisations are invited to work with MyJustice.
In my past jobs, I have already experienced challenges in applying M&E disciplines to constantly evolving programme contexts. The challenge was greater with MyJustice because of the scale of the programme and sheer number of implementing partners: approximately 50 organisations, working in their own different contexts.
I soon learned that it was impossible to develop one overall M&E plan which could capture the progress of each partner. I needed to treat each partner individually; explore impact at their level and then collate them to demonstrate impact collectively.
Limits to the surveys
Initially I tried to apply scientific approaches. I wanted to demonstrate impact attributable to the programme as ‘facts’. As much as I wanted to use empirical methods such as surveys, I have come to realise that most of the impact we have produced were at the individual level, in the form of journeys that our partners and beneficiaries have travelled in search of justice. A survey was not the right tool to capture all these important details.
Besides, impact at community level and beyond were yet to be seen. Significant changes in the justice sector are still not yet taking place and it is beyond the capacity of MyJustice as an international NGO to push an agenda of holistic justice sector reform. This also counted against surveys as our measurement tools.
Capturing the journey
Accordingly, the strategy switched to in-depth individual interviews to document powerful stories on how MyJustice has helped overcome the barriers in seeking justice. It took almost one year and over 60 interviews for me and my colleague to collect a valid amount of information. Instead of numbers, we used strong quotes and diverse views, reflecting voices of service providers and community members, extracted from thorough analysis of information collected. Beneficiaries like the mother of a girl victim of sexual abuse, who told us:
They abused my daughter and made her look like the wicked one. We felt insulted and embarrassed. I did not know what to do. Before I came across MyJustice, I thought there was no one who would care about us.
These individual stories, combine with basic quantitative data already collected, told us a meaningful story about the journey in promoting access to justice for Myanmar, and the road ahead.
Impacts regarding access to justice are likely to be intangible and cannot be measured through traditional and quantitative methods. The challenge is knowing where the impact lies. MyJustice has taught me to look for impact at all levels: individual, community, and system. This has been a whole new, exhausting yet productive experience. Maybe that is what justice reform is about.