Terms of Reference (ToR)
Impact Assessment of the Baytna Syria Program 2019-2021
Baytna is a Syrian-led organization, based in Brussels in Belgium and registered as an association under the name “Association Baytna Pour le Soutien de la Société Civile “. Baytna’s three-year strategic program for 2019-2021 is funded by the Swedish MoFA (SIDA) with a total of 17,250,000 SEK, the Danish MoFA (DANIDA) with a total of 13,000,000 DKK, the Dutch MoFA with a total of 250,000 Euro, the French MoFA with a total of 500,000 Euro, and the Swiss MoFA with a total of 173,000 Euro.
Baytna is established as a response to address the challenges that the Syrian civil society is facing. The organization attracts Syrian civil society organizations (CSOs) and contributes to their institutionalization and project delivery. Baytna is 100% Syrian-led, which means that the organization enjoys broad knowledge about the local context in Syria. Baytna’s mandate covers grant awarding, capacity development, networking, and advocacy.
Baytna’s overall goal for the ongoing program is to empower civil society as a force for democratic change and inclusive peace and stability; to support transitional justice; and to promote respect for human rights and civil liberties using different tools and approaches including advocacy, lobbying, and policymaking.
Baytna’s expected outcomes include that Baytna’s supported civil society groups and activists effectively monitor the governing structures and the democratic processes, and document violations and promote accountability; Baytna’s supported civil society groups and activists foster public participation in policy making and decision-making; Baytna’s supported civil society groups and activists influence national and international policies to contribute to democratic change; and Baytna contributed to a more mature, diverse, and inclusive civil society narrative.
Baytna aims to provide a credible, independent, and sustainable entity capable of networking amongst Syrian CSOs and promoting public participation, accountability, and empowerment. Baytna provides three core services: (1) a hub for Syrian civil society organizations, Syrian refugees, diaspora, and others to meet and coordinate; (2) knowledge capacity building workshops to Syrian CSOs; and (3) small-scale grants to which Syrian CSOs can apply, a mechanism facilitated through its Syria-based field officers. Baytna is using these three core intervention competencies in its three proposed programs for the period 2019-2021: Syrian Online, Syria Inline and Syria Outline.
Baytna’s board of directors is comprised of four prominent Syrian figures, all independent and not
remunerated. The board has a written bylaw and meets at least three times a year. They support the general orientation of Baytna and provides leadership in setting policy, vision, and strategic objectives to the organization.
Baytna consists of four main units: programs, development and quality, admin and finance, and communication all led by the Executive Director of Baytna.
PURPOSE OF THE ASSESSMENT AND INTENDED USE
The main purpose of this impact assessment is to measure through an independent evaluation the changes that can be attributed to the current Baytna’s program.
The purpose of this study can be summarized in terms of accountability and learning; (i) to gather data about the effectiveness and impacts of the program to make sure that the intervention is still on track and is likely to reach its objectives, and (ii) to provide Baytna, the donors and its partners with an input to upcoming discussions concerning the preparation of Baytna’s new strategy for 2022-2024.
For the purposes of this assessment, impact will be understood as the wider effects of the program – social, economic, technical and environment – on individuals (disaggregated by gender and age groups), communities and institutions. The impact can be direct or indirect, intended, or unintended, positive, or negative, macro (within the sector) and micro (household).
The assessment results will generate evidence of key achievements and challenges to inform decision making process for donors,Baytna’s senior management, and key stakeholders as well as providing an opportunity for understanding how the program delivery mechanism works best, under which circumstances, and how this can be improved in the future. In addition, these assessment results are expected to inform the next phase of Baytna and its programming work inside and outside Syria. This impact study will examine Baytna’s program from 2019 through 2021.
This study will mainly focus on the geographical locations, themes and intervention tools covered by the program. Findings and conclusions from this study will be shared with Baytna’s Board of Directors, SIDA, Danish MoFA, French MoFA, Swiss MoFA, Dutch MoFA, and any other potential donor or key stakeholder.
SCOPE OF WORK AND METHODOLOGY
The impact assessment should review all aspects of the current Baytna program. Assessment methodology will include:
- Desk review: The assessment team will examine project documents, project data, donor grant agreements, subgrant agreements (SGA) with partners inside and outside Syria, projects monitoring and evaluation documents, tracer studies, need assessments, donor proposals, donor progress reports (narrative and financial), external audit reports, internal policies, Baytna’s approved strategy document, guides and training documents and any other relevant documents. The main purpose of this desk review is to build a good understanding of the available information on Baytna within the wider context of civil society support mechanisms before conducting an in-depth study of this intervention. More specifically, this review will aid in understanding the long-term trends in terms of program implementation and outputs since its inception, design of the sampling frame, defining the original scope of this program, geographical coverage, and target populations; determining sampling methodology; and analysing the risks posed by various contextual realities.
- Case studies: The assessment team may pick 10 to 15 case studies of beneficiaries who successfully completed their projects to study in further depth and to assess the impact on local communities. The team may also want to select a few partner employees who are direct beneficiaries of Baytna capacity development efforts both inside and outside Syria to study in further details the impact on individual and organizational capacity improvement as a result of this program. In addition, case studies of networking events may be chosen to further study their effectiveness and evaluate their impact. Use of case study is an appropriate evaluation tool in this context because it provides the opportunity to probe further the in-depth experiences of specific target beneficiaries or a subset of beneficiaries of Baytna as a distinct part of whole. More specifically, this will allow for greater latitude in seeking out and assessing the program impact through exposing a lot more about program processes and outcome and way they interact with each other.
- Field visits: The assessment team will have the opportunity to visit some of the awarded projects should they have the ability to. The project areas are currently limited to the provinces of Aleppo, Idleb, Raqqah, Sweida, Deir Ezzor, Hassakah, Turkey, Lebanon, and Belgium. They will be able to study Baytna’s activities in all areas, and will interview beneficiaries who agree to share information, survey target population, conduct focus groups randomly with selected beneficiaries who benefited from Baytna’s grants or capacity development programs. Field visits to the program targets will be critical for both collecting data and to building good understanding of Baytna’s target beneficiaries, geographical coverage disparities and risks related to project implementation in an authentic setting.
- Interviews with stakeholders: The assessment team should pay particular attention to the interviewing of stakeholders, namely subgrant recipients, beneficiaries of capacity development program, Baytna staff (in Belgium, Turkey, and inside Syria), targeted communities, Baytna’s Board of Directors, Baytna’s donors and others as relevant. The main purpose for employing this methodology is to understand the impact of Baytna’s program both from the perspective of those involved and how they have benefited or affected by the interventions of this program.
ISSUES TO BE COVERED
This assessment will mainly aim at identifying any changes resulting from Baytna’s program interventions, establish causal connections between the changes and the program inputs and measure the magnitude of the change. It will particularly focus on and try to assess a wide range of broad, key and strategic performance indicators, including project effectiveness, efficiency, relevance, performance, sustainability and coverage. The impact assessment will adapt OECD/DAC evaluation criteria and concepts which defined as following:
RELEVANCE: IS THE INTERVENTION DOING THE RIGHT THINGS?
The extent to which the intervention objectives and design respond to beneficiaries (Beneficiaries is defined as, “the individuals, groups, or organizations, whether targeted or not, that benefit directly or indirectly, from the development intervention.”), global, country, and partner/institution needs, policies, and priorities, and continue to do so if circumstances change.
Note: “Respond to” means that the objectives and design of the intervention are sensitive to the economic, environmental, equity, social, political economy, and capacity conditions in which it takes place. “Partner/institution” includes government (national, regional, local), civil society organizations, private entities and international bodies involved in funding, implementing and/or overseeing the intervention. Relevance assessment involves looking at differences and trade-offs between different priorities or needs. It requires analyzing any changes in the context to assess the extent to which the intervention can be (or has been) adapted to remain relevant.
COHERENCE: HOW WELL DOES THE INTERVENTION FIT?
The compatibility of the intervention with other interventions in a country, sector or institution.
Note: The extent to which other interventions (particularly policies) support or undermine the intervention, and vice versa. Includes internal coherence and external coherence: Internal coherence addresses the synergies and interlinkages between the intervention and other interventions carried out by the same institution/government, as well as the consistency of the intervention with the relevant international norms and standards to which that institution/government adheres. External coherence considers the consistency of the intervention with other actors’ interventions in the same context. This includes complementarity, harmonization and co-ordination with others, and the extent to which the intervention is adding value while avoiding duplication of effort.
EFFECTIVENESS: IS THE INTERVENTION ACHIEVING ITS OBJECTIVES?
The extent to which the intervention achieved, or is expected to achieve, its objectives, and its results, including any differential results across groups.
Note: Analysis of effectiveness involves taking account of the relative importance of the objectives or results.
EFFICIENCY: HOW WELL ARE RESOURCES BEING USED?
The extent to which the intervention delivers, or is likely to deliver, results in an economic and timely way.
Note: “Economic” is the conversion of inputs (funds, expertise, natural resources, time, etc.) into outputs, outcomes, and impacts, in the most cost-effective way possible, as compared to feasible alternatives in the context. “Timely” delivery is within the intended timeframe, or a timeframe reasonably adjusted to the demands of the evolving context. This may include assessing operational efficiency (how well the intervention was managed).
IMPACT: WHAT DIFFERENCE DOES THE INTERVENTION MAKE?
The extent to which the intervention has generated or is expected to generate significant positive or negative, intended or unintended, higher-level effects.
Note: Impact addresses the ultimate significance and potentially transformative effects of the intervention. It seeks to identify social, environmental, and economic effects of the intervention that are longer term or broader in scope than those already captured under the effectiveness criterion. Beyond the immediate results, this criterion seeks to capture the indirect, secondary, and potential consequences of the intervention. It does so by examining the holistic and enduring changes in systems or norms, and potential effects on people’s well-being, human rights, gender equality, and the environment.
SUSTAINABILITY: WILL THE BENEFITS LAST?
The extent to which the net benefits of the intervention continue or are likely to continue.
Note: Includes an examination of the financial, economic, social, environmental, and institutional capacities of the systems needed to sustain net benefits over time. Involves analyses of resilience, risks, and potential trade-offs. Depending on the timing of the evaluation, this may involve analyzing the actual flow of net benefits or estimating the likelihood of net benefits continuing over the medium and long-term.
The following serve as guiding questions that can be integrated within the adapted OECD/DAC evaluation criteria and should be addressed based on the theory of change and results framework attached in Annex I:
- Did the support provided by Baytna (including grants, capacity development, networking, amongst others) reach the target group as intended, or did it reach a large portion of unintended population? Were benefits distributed fairly between gender and age groups and across social and cultural barriers?
- To what extend has the intervention achieved its objectives (outputs and outcomes)?
- To what extent are the resulted changes in the target geographical areas of Baytna consistent with the planned outputs, purpose, and goal of the evaluated intervention?
- To what extent can the identified changes be attributable to the intervention rather than extraneous factors? What could have been done better to make the intervention more effective?
- What are the intended and unintended, positive, and negative, direct or indirect, counterfactual effects of Baytna’s intervention on people and institutions?
- How has the intervention affected the overall situation of the target beneficiaries and stakeholders?
- How is the intervention consistent with the needs and priorities of the targeted communities, especially women and rural communities?
- In which ways the overall interventions consistent with capacity development needs and priorities of the targeted communities? Has there been discernible change in the institutional, organizational, and individual capacity in management and provision of civil society services and empowerment? What are the key capacity development changes resulted from this intervention? To what extent has the intervention contributed to capacity development and system strengthening of community-based institutions? To what extent can changes that have occurred during the period covered by the evaluation be identified and measured?
- What do beneficiaries and other stakeholders affected by the intervention perceive to be the effects of Baytna on themselves?
- To what extent can the identified changes in the target geographical areas be attributed to the intervention of Baytna? What would have occurred without the intervention of Baytna?
- How is Baytna’s intervention consistent and complimentary with activities supported by other LNGOs and INGOs?
- Is the intervention consistent and complementary with activities supported by other donor organizations? If so, to what extend can this be justified by the activities?
- Do the targeted beneficiaries have the (technical and financial) capacity to maintain the benefits from Baytna after cessation of external support from donors?
- Will the benefits produced by this intervention be maintained after cessation of external support? To what extent can it be maintained?
- To what extent and in what ways is the intervention supported by local institutions and well-integrated with local social and cultural conditions?
- To what extent is the interventions (especially small grant initiatives) identified, needs-based and initiated by the beneficiaries?
- Is the intervention compatible with a sustainable use of natural resources? Or is it harmful to the natural environment? How can this be justified by the results?
- Has the Baytna program been managed with reasonable regard for efficiency? What measures have been taken during planning and implementation to ensure that resources are efficiently used?
- In what ways could the intervention have been implemented with fewer resources without reducing the quality and quantity of the results?
- How could more of the same result have been produced with the same resources?
- How could an altogether different type of intervention have solved the same development problem but at a lower cost?
- Was the intervention economically worthwhile for the targeted communities, given possible alternative uses of the available resources? If not, how should the resources allocated to the intervention have been used for another, more worthwhile, purpose?
- Is the intervention producing the most appropriate impact, given the context and resources available? To what extent can this be justified by results?
- How efficient and effective was Baytna’s adaptation to the pandemic of COVID-19? How affected was the implementation of programs? What should have been done? Can the results be sustained? What are the missed gaps?
EVALUATION TEAM COMPOSITION AND REQUIRED COMPETENCIES
The Evaluation Team will be comprised of a team leader and technical assistants.
- Team Leader: The assessment should be led by an individual with competence in project/program evaluation, and who has worked with civil society support mechanisms and development particularly with more focus on gender and local community empowerment in both humanitarian and development context. A proven track record in operation in a conflict environment is a must. In particular, the Team Leader should demonstrate the following:
- Must hold a postgraduate degree in conflict analysis, research, Project Management, or any other relevant field;
- Must have at least 10 years professional work experience in the areas of program evaluation, especially in civil society development and capacity building in conflict settings;
- Knowledge and/or experience of civil society’s role, including both men and women, and boy and girls in democratization processes;
- Extensive conceptual and methodological skills and experience in applying qualitative and quantitative research evaluation methods;
- Prior impact assessment experience is required;
- Experience in organizational management, structures and systems, operations, capacity development, reporting, and monitoring is essential;
- Excellent understanding of the Syrian context and conflict dynamics;
- In country or regional similar work experience will be an added advantage;
- Excellent communication skills necessary for building rapport with stakeholders, facilitating participation and effective presentation of result to diverse audience;
- Perfect command of English language and working level Arabic.
- Detailed knowledge of the role and development of Syrian civil society is desirable;
Prior experience in the region is an added advantage.
- Technical Assistance Team:
- Must hold a graduate degree in conflict analysis, research or any other relevant field. A bachelor’s degree with 7 years of relevant experience will be considered;
- A minimum of 5 years of professional work experience in the areas of program evaluation, especially in civil society development and capacity building in conflict settings;
- Knowledge and/or experience of civil society’s role, including both men and women, and boy and girls in democratization processes;
- Extensive knowledge of and experience in applying qualitative and quantitative evaluation/research methods.
- Prior research or evaluation research is required;
- Excellent communicative, both spoken and written skills in Arabic and English.
- Able and willing to be deployed to the provincial and district areas covered by Baytna Syria programs;
Detailed knowledge of the role and development of Syrian civil society is desirable;