Where is the sector at on VFM analysis?
This blog summarises the discussions from the 12th May workshop organised by the VFM-WASH team on measuring VFM in the WASH sector.
On Tuesday 12th May the VFM-WASH team held a workshop to share the guidance we developed to do VFM analysis with sector organizations and discuss how this type of analysis can be useful for the sector. Our goal was to help clarify how VFM metrics can be computed and used and to support the development of a VFM culture within the sector. The event attracted a very diverse group of practitioners, who all shared a deep interest in better understanding the approach and its practical applications. A dozen organizations were represented including UNICEF, Plan UK, Practical Action, Water Aid and the Global Sanitation Fund. Researchers and consultants from WEDC, Oxford University, OPM, Trémolet Consulting, University of Leeds LSTHM and from the Independent Monitoring and Evaluation Project (IPME) were also present.
The summary presentation on how to do VFM analysis can be found here. Presentations from other partners are available here. In addition a more complete summary of the discussions can be found here. To continue sharing experiences and questions with the community of practice of VFM WASH practitioners, join our LinkedIn group!
Overall, there is appetite for better, more in-depth and accurate VFM analysis in the sector. However organisations are still unsure how to go about it. There is also a concern that isolated unit costs indicators will be used out-of-context to make funding decisions, without consideration for events out of the implementers’ control or for the added costs of providing services for marginalised groups. As a sector, we need to change this perception of VFM analysis and show that it can be more than a unit cost calculation and actually provide meaningful findings on the cost efficiency and cost-effectiveness of programmes. We need to demonstrate how it can actually be useful for funders as well as for implementers and improve monitoring tools to do better VFM analysis.
So why do VFM analysis?
Partners shared different reasons for doing VFM analysis, including responding to donors requests, providing evidence of cost efficiency to attract donor funding, and improving procurement and programmes’ design. Presentations are available here. The summary note also presents examples shared by partners doing the workshop.
How can we do VFM Analysis better?
At present doing ex-post VFM analysis is akin to detective work. Linking together data from different sources, with funding on the one hand and outputs and outcomes on the other does take time. However the workload depends on whether the organisation has activity-based budgeting and financial reporting. Rapidly calculating VFM indicators on a routine basis requires adapting M&E and financial reporting systems. Collecting good outcome data also requires a baseline and end-line survey. Obviously this increases M&E costs for implementers and therefore reduces available budget for implementation. Jeremy Toubkiss from UNICEF mentioned that the first priority should be to create a monitoring, evaluation and learning culture, around quality of program design and implementation. Hence VFM analysis could become more routine once these basics are in place. However, having an idea of the type of VFM analysis envisaged will help define the M&E framework required to collect the data.
Moreover, more emphasis needs to be put on the qualitative aspects of the analysis related to outputs, sustainability and equity. Outcome data is an essential piece of VFM analysis to better understand the sustainability of the programme. Programmes deliver good VFM if they deliver sustainable outcomes, i.e. if people who gained access through the programme continue to benefit from good service levels over time. Barbara Evans Ian Ross explained that the concern of sustainability lies at the heart of our VFM action-research programme, as we initially considered the possibility of defining a WASH effectiveness metric (with the related cost-effectiveness indicator), based on the concept of DALYs. This indicator would reflect the decrease in exposure to specific risks experienced by people who are able to use improved WASH services as a result of the service level they receive. This requires detailed outcome data on service levels achieved in terms of functionality, quality, quantity, reliability and accessibility of services. The idea received positive feedback from the participants. It could not be done as part of our analysis because of lack of outcome data, but the concept will be further developed through a journal article.
How can we create a VFM culture in the sector?
The sector needs a common VFM framework. As argued by Sophie in her blog” VFM analysis in the WASH sector: why is it needed?” the more VFM analyses are done using the same methodology, the easier it will be to make evidence-based funding decisions. Currently guidance on VFM analysis for the WASH sector is limited. We collected many good ideas on practical tools needed to facilitate VFM analysis going forward. Common sector-specific definitions of outputs, outcomes, efficiency and effectiveness indicators are required to avoid misinterpretation. Organizations could then apply these to their activities: each organization would still have their own indicators, based on their M&E reporting, but they will analyse outputs and outcomes of main WASH activities in the same way. Our proposed framework is presented in this Guidance Note: we welcome comments on it so that it can become a joint sector-effort to set a standard. Participants also suggested having a checklist, to verify that reporting systems (M&E and financial ones) can provide the data required to do VFM analysis, would be helpful. This will be developed based on the existing Guidance Document. However, it will be difficult to develop a common VFM software, as VFM analysis is based on each organizations specific activities and M&E reporting.
Creating a VFM culture in the sector will mostly require thought leadership from sector stakeholders. Participants called for creating a community of practice for VFM WASH practitioners of donors, VFM specialist and program specialist to discuss discussion around a common framework from VFM. You are invited to join our linked in group here!