Indicators

A strong design and monitoring framework, with better defined outcomes and outputs and measurable indicators, and the setting up of a project performance and monitoring system during project preparation or at project startup could help track implementation challenges. These may have helped mitigate the risk of scope changes, loan cancellations, and early implementation delays due mainly to delays in startup activities, such as signing of subsidiary loan agreements, detailed engineering design, bidding document preparation, and delays in land acquisition and resettlement.

Strong linkages between indicators across the different levels of the project result chain need to underpin project design and monitoring frameworks, The linkage between the project impact performance indicators of increased marine passengers and increased domestic and international freight volume was not a strong measure of success in this project; it should have been revised during project implementation.

Executing and implementing agencies need to find ways to sustain data collection and analysis throughout the project implementation phase to ensure high-quality M&E systems. Monitoring and evaluation was not completed before project closing because, at the critical late stage of implementation, the pandemic brought consultant services to a stop before they can be completed. Some baseline data were collected in 2015, but the function was not completed because of early implementation delays.

Guidance for completing CPRO DMFs underscored the dynamic nature of a crisis operation and the need to be prepared to strengthen, amend or add DMF indicators during implementation. The fact that this was not followed reflects a built-in lack of flexibility with DMFs (and perhaps the difficulty of making changes in accordance with existing Project Administration Instructions).

The design and monitoring frameworks (DMFs) of the COVID-19 Pandemic Response Option (CPRO) extracted aspects of the government’s response plan that were important to ADB and restricted the number of indicators and targets in order not to burden the government. However, this made demonstrating outcome-level achievement difficult because it relies too much on one or two indicators and targets (weak horizontal logic) and is narrowly and inadequately supported by a minimal number of outputs (weak vertical logic).

The framework for a program’s data management system should be put in place before its start to measure the achievement of targets and beneficiaries and (the ambitiousness of goals should be balanced against the availability of data and capacity to quickly augment granular data collection. Although the government has increased its focus on gender equality and has made efforts to strengthen gender data tracking, data disaggregated by sex and age of end recipients were often not available at the time of program preparation and monitoring.

When designing future programs, ADB needs to encourage the government to implement long-term reforms that expand sex-disaggregated data collection and strengthen the capacity of line ministries and statistical agencies. With ADB dedicated technical assistance (TA) support, the government should promote the establishment of a monitoring and evaluation system, interagency exchange of data, and systemic links between planning, budgeting, and execution.

Ownership of the reform process is critical and, given the experience of this, it will remain so. The reform agenda was fully in line with government and ADB strategies and priorities. This allowed for timely and quick support that resulted in comprehensive products. Detailed discussion on outcome indicators at the design stage is critical to finding the best possible indicators for measuring outcome.

The DMF indicators and results chain, to effectively serve as a monitoring and evaluation tool, need to reflect the changes in TA scope and focus during implementation. Output performance indicators and targets were changed to align with the government’s requests for support to emerging priorities. The outcome statement and targets should also have been updated to reflect the changed focus of the TA which has evolved and become broader.

The design and monitoring framework (DMF) and monitoring and reporting mechanism need to be carefully developed at the planning stage. The outcome statement of the TA was not appropriate as it is too narrow and should be more focused on the TA benefits to Maldives. Outcome targets were too ambitious since there were several contributing factors outside the scope of the TA that affected their achievements. With a small country portfolio, unexpected delays in one or two projects will make the target unachievable. Having an outcome indicator at the broader level could help avoid that.