Navigating ethical considerations in evaluations: 3ie's TREE…

EEN members share a commitment to generating high quality evidence on approaches that can support positive outcomes for young people. This blog presents an example of how collaborations as part of the Global Trials Fund can support collaborative conversations on conducting impact evaluations.

Earlier this year, the TREE team at 3ie collaborated with the Education Endowment Foundation (EEF) and Effective Basic Services Africa (eBASE Africa) to implement the TREE Review Framework in two of their upcoming evaluations in Cameroon. Tangang Andrew, a Research Fellow at eBASE Africa, provides us with an overview of how the TREE Review Framework has impacted their evaluations and the lessons learned from its implementation.

3ie’s Transparent, Reproducible and Ethical Evidence Framework (TREE) is a public good designed to complement the Institutional Review Board (IRB) process by helping research teams evaluate and document the ethical considerations inherent in social science research. The main objectives of the framework are to:

(i) establish ethical standards for evaluations;

(ii) better integrate best practices for transparency, reproducibility, and ethics into evaluation workflow; and

(iii) establish timely, independent review of risks facing the evaluation team to produce credible, meaningful, ethical evidence and the efforts to mitigate such risks.

Q: What motivated eBASE to use the TREE Review framework in their research process?

A: At eBASE, we want to make sure our research meets the highest standards of ethical integrity. There’ is also increasing appetite for evidence-informed decision-making in developing countries. While this is hugely welcome, evidence generation brings with it ethical challenges such as risks to credibility and usability of evidence, and risks to research participants and staff. To mitigate some of these risks, we decided to adopt the TREE Review framework and embed it within our ethics review processes.

Q: Can you describe how eBASE has used the TREE Review framework in practice?

A: Our first experience with the TREE Review framework was in 2022 during a pilot evaluation of our Menstrual Hygiene Management project, which aimed to support girls’ attendance at school by providing menstruation-related education and materials. We had identified security concerns as the research was being carried out in conflict-affected regions of Cameroon.

The 3ie team, which included regional peers with a deep understanding of the local context, undertook a comprehensive and iterative review of the protocol. One key takeaway from the process was that the TREE Review could support capacity building for stronger evidence generation. So, we decided to adopt the TREE framework as a complementary tool for our ethical reviews and offered training to our staff on the utility of the framework.

Since then, we’ve reviewed over 10 projects using the TREE Review framework, including the menstrual hygiene management impact evaluation and our social and emotional learning pilot.

Q: What have been the key takeaways from using the TREE Review framework?

A: The key takeaways have been quite positive. eBASE staff have reported a substantial improvement in their understanding of ethics reviews since adopting the TREE framework. They appreciate that the framework offered clear guidelines for identifying both ethical and security issues. Overall, the staff feel that the TREE framework has enhanced their approach to conducting ethical reviews.

The evaluation teams have also benefitted from the framework, noting that it has served as a secondary review panel of their evaluation protocols. Initially, the review was commissioned mainly to identify potential security risks to project staff and propose mitigation strategies. Over time, however, it has also served to identify salient points in the evaluation design and the relationship with key stakeholders. Some team members have observed a shift in their perspectives on their roles in the evaluation, recognizing the need to be more aware of how the results of the evaluation will be useful to participants and policymakers.

I believe these experiences highlight how the TREE Review framework has enhanced the ethical integrity of our research, as well as the practice of evaluation teams. However, it is worth noting that a TREE review process conducted with little or no understanding of the evaluation contexts, could result in the misrepresentation of the realities of an evaluation.

Q: How does eBASE plan to use the TREE framework in their future research activities?

A: With the increased demand for policy research activities in Sub-Saharan Africa over the last 14 years, there is a need for researchers to establish ethical standards that directly address issues within the context of the research or evaluation activities. It is also important that these ethical issues and standards are disseminated for peer-learning and experience sharing. The TREE framework offers a template which is replicable and adaptable to research in various contexts.

At eBASE Africa, we are going to build on our experiences to spearhead the adaptation of the framework in Cameroon and Sub-Saharan Africa. One of our plans is to translate the TREE Review Questionnaire into French as it is currently only available in English. Translating the questionnaire would make the tool available to researchers in over 21 Francophone African countries, including Cameroon.

We also plan to organize capacity-building workshops to help researchers in ethics better understand how the framework can be used to complement and improve already existing ethics review processes. Currently, we’re collaborating with the Education Endowment Foundation (EEF) to hold a workshop in Cameroon. Additionally, we aim to encourage more research and publications in peer-reviewed journals on the adaptation and contextualization of the TREE Review Framework. This could help stimulate reflections and perspectives on ethical issues and ethical standards in policy research in various contexts.

At eBASE Africa, we’re going to build on our experiences to spearhead the adaptation of the framework in Cameroon and Sub-Saharan Africa. One of our plans is to translate the TREE Review Questionnaire into French as it is currently only available in English. Translating the questionnaire would make the tool available to researchers in over 21 Francophone African countries, including Cameroon.

Q: Lastly, how do you see the TREE framework influencing the broader research community?

A: The global evidence community has a set of agreed standards which researchers are required to adhere to. Variations in context means that researchers are often unaware of emerging ethical issues that could bring into question the credibility and usability of the evidence. Comprehensive tools like the TREE Review Framework can not only effectively address this issue, but mobilize the evidence community towards robust ethical standards in policy, research, and practice. Researchers using this tool should continue sharing their experiences so that we can all work together to shape and enhance ethical standards.