Related Project:
Coordinating Implementation Research to Communicate Learning and Evidence (CIRCLE)
Author(s):
Ana Claudia Franca-Koh, Deputy Director/Research Advisor, CIRCLE Project
It is great to be here in Liverpool at the Fifth Global Symposium on Health Systems Research. We kicked off the week by hosting the Developmental Evaluation (DE) skills building satellite session based on our early experience implementing a DE in Tanzania. Jointly led by Dr. Beati Mboya, the Principal Investigator of CIRCLE’s* DE team, and Dr. Ashwin Budden, a DE expert, the session aimed to apprise the audience about the fundamentals of the DE approach. The diverse audience representing NGOs, government, research institutions, and funding organizations mirrored the Tanzanian panelists – Rose Kyando (USAID/Tanzania), Dr. Bakari Salum (Government of Tanzania), and Albert Ikonje (Deloitte, Inc.), who ably led the audience through the nuts and bolts of Tanzania’s DE.
Participants reflected on the need to improve on traditional evaluations which produce results at the end of a program cycle, which is often too late to inform subsequent interventions.
“I was looking for new ways of doing things. The traditional ways are not working well, due to rapidly changing environments. DE tries to capture the needs of our times. We aren’t really getting what we need from traditional evaluations, in terms of the need to adapt our projects.”
– Ryan Guinaran, DITENG Inc. and Benguet State University, Philippines.
What DE is and is not
- DE is not a tool or a method, it is a broad approach to evaluation that supports different kind of methods.
- DE focuses on learning and program improvement, not on judging whether a program succeeded or failed.
- DE is ideal for complex programs that are developing and or operating in dynamic contexts.
Photo credit: Ant Clausen - DE is flexible in terms of how it is designed and implemented. But this also means it needs a flexible enabling environment. This may be challenging for organizations that have defined reporting periods.
- DE is not the same as process evaluations. Although DE does focus on process, it can also target outcomes and especially unintended and emergent outcomes that are not part of a predefined results framework.
- DE is not the same as performance reviews. Performance reviews focus on examining indicators and to what extent targets have been met. DE goes further; it looks at broader contextual issues by asking more questions: For example, why are we seeing these changes? Are there factors that are affecting these results? Where are the opportunities for course correction or innovation?
Tanzania panelists highlighted some key lessons learned that resonated with session participants:
- The beginning was not easy. It is always challenging starting something new. It took time and persistence to bring all stakeholders on board, both within and across organizations. Some feared embedded evaluators were like auditors, but later understood their purpose.
- Partners need time to adapt, to understand the work of embedded evaluators and to include them in their work plans. Embedded evaluators are now part of quarterly plans, and are included in field visits and meetings.
- DE is already making a difference. The evaluators have only been embedded for a few months, but program improvements have already been made with some preliminary results. For example, Albert Ikonje (Deloitte) said that recent field visits showed that referrals within larger health facilities were not happening effectively, patients did not know where to go. Partners are now working with the facilities to address this gap. Embedded evaluators have been able to immediately share any issues uncovered.
- The DE process is integrated into existing systems and structures: For example, the DE team uses routine meetings to share results with stakeholders, incorporate themselves in quarterly planning, share offi
Photo credit: Ant Clausen ces and travel with partners to conduct field visits.
- Key country stakeholders have become strong advocates for DE: Dr. Bakari Salum (from the Government of Tanzania) said the government may decide to adopt the DE approach for other programs because it provides the type of information they need. Rose Kyando from USAID/Tanzania emphasized the importance of being able to see change in real time – she has already witnessed emergent issues being addressed in implementation and partners learning from one another.
“This approach of continuously informing stakeholders, in particular those implementing the programs, about areas of improvement is something that we need.”
— Habib Hussain, Contech International, Pakistan
Stakeholder engagement
- DE employs a continuous stakeholder engagement process, implemented from the beginning. This takes time, effort and commitment.
- All stakeholders need to be included. Mapping them out from the beginning to decide who to engage, when and how is important.
- Getting stakeholder buy-in is only the beginning. DE keeps stakeholders involved throughout the process, from developing evaluation questions and frameworks together, to embedding evaluators that work alongside implementers.
- Embedded evaluators are key to ongoing engagement and cultivating trust relationships with stakeholders.
To learn about our activities at HSR click here, and for more information about DE, read other recent blogs at www.harpnet.org.
Stay tuned for more about implementation research in MNCH at HSR 2018.
*USAID’s Coordinating Implementation Research to Communicate Learning and Evidence (CIRCLE) Project
