Mid-Term Evaluation of the USAID/Pakistan Maternal, Newborn and Child Health Program

Author

Pinar Senlet

Susan Rae Ross

Jennifer Peters

Global Health Technical Assistance Project

Publication Date

September 1, 2008

Summary

The purpose of this evaluation is to provide the United States Agency for International Development's Mission to Pakistan (USAID/Pakistan) with an independent mid-term evaluation of its maternal, newborn, and child health (MNCH) programmes. These programmes, which are managed by the Office of Health, are implemented through two projects: PAIMAN is a maternal and newborn project led by John Snow Inc. (JSI); Save the Children (SC) implements the Improved Child Health Project (ICHP) in the Federally Administered Tribal Areas (FATA). This 74-page document evaluates both of the projects; summaries of each section of the report follow.

Conducted from June-July 2008 by a team of 3 consultants through the Global Health Technical Assistance Project, the evaluation assessed whether the project is achieving intended results, reviewed its organisational structures and technical approaches, and formulated recommendations for future directions. During field visits to selected project sites, the team interviewed a broad range of stakeholders, including health managers, service providers, trainees, and community members.

I. The goal of the Pakistan Initiative for Mothers and Newborns (PAIMAN) project is to improve maternal and neonatal health in selected districts in Pakistan. Its framework outlines 5 strategic objectives: to increase awareness of and access to maternal and newborn health services, to improve the quality of services, and to increase the capacity of health care managers and providers. Evaluators found that the strategy and approaches of PAIMAN are appropriate and sound. Implementation has been challenging due to the high number of activities, multiple consortium partners, and the geographic spread and diversity of project districts.

Key Findings:

The project has a clear and evidence-based communication strategy that reaches out to 9 groups, with multiple messages for each group. "Of the numerous interventions underway, advocacy and mass media interventions seem strong. However, most interventions have not been implemented at a frequency or scale that would reach the majority of the target populations. For example, theater shows, a potentially effective means of outreach, have reached only a small percentage of the total population. Similarly, all events, local and larger-scale, have reached only about two percent of the population. This is not sufficient to effect population-level change."
PAIMAN has successfully created support groups and trained lady health workers (LHWs) to be behaviour change agents. The creation of support groups is likely to be taken to scale in coming years. This appears to be a sustainable and high-impact intervention.
PAIMAN recently created 3 mass media interventions: a music video about male responsibility, a TV drama tackling issues of family planning and breastfeeding, and television advertisements addressing maternal and newborn issues. "If these interventions prove successful, they seem likely to have impact."
Through its non-governmental organisation (NGO) subgrantees, PAIMAN has helped establish 9 birthing centres, which have been successful in meeting community needs. In one, PAIMAN successfully assisted the community to receive civil community board (CCB) funding to sustain it.
PAIMAN has been successful in supporting the development of pre-service training for community midwives (CMWs). There were no consistent responses about how the CMWs will be established and supervised in the community. PAIMAN has had discussions about accessing microfinance schemes to support the birthing rooms, but plans are not finalised. PAIMAN is also preparing to test different supervision models once CMWs are in the field, but these plans need to be finalised and better communicated to all levels.
PAIMAN has increased the number of Ministry of Health (MOH) facilities that can provide basic maternal and newborn care (MNC). Overall, the number of births in upgraded facilities has increased, though it varies greatly by district and facility.
PAIMAN upgraded 13 rural health centres (RHCs) to provide basic emergency maternal and newborn care (BEmONC) and 18 district headquarters hospitals and tehsil headquarters hospitals (DHQ/THQs) to provide comprehensive emergency maternal and newborn care (CEmONC). While some facilities are still waiting for instruments and staff, most are now equipped and staffed. Since most have only recently become functional, it is too early to expect major changes or assess the quality of services.
PAIMAN trained many providers who had never had any refresher training, "which is a major accomplishment". The strategy of working through the provincial and district Health Development Centers has been important for developing a sustainable system.
The project trained 1,600 providers in essential maternal and newborn care (EMNC). A major weakness of the training is that it does not include a clinical practicum or use of the partograph. Initially, the curriculum included both, but PAIMAN removed these two subjects so that male providers and paediatricians who do not conduct deliveries did not have to learn them.
The team found that there is confusion about the difference between minor pregnancy and newborn ailments, such as urinary tract infections, aenemia, and vomiting, and major complications that lead to maternal and newborn deaths.
The behaviour change communication (BCC) messages and training curriculum are not specific enough to make these distinctions.
While the referral system is evolving as providers and health facilities become functional, the team did not find a clear referral system that can be communicated to women to ensure that they have access to facilities that meet their needs.
PAIMAN helped establish and reinforce district health management teams (DHMTs), provided management training to a broad range of health managers, and worked to improve the quality and utilisation of health information systems.
As of June 2008 the project had achieved 5 of the 6 indicator targets set for 2007. Between 2005 and 2008, in 10 project districts:
- Births assisted by traditional birth attendants (TBAs) increased from 36% to 38%. Women who received 3 or more antenatal visits during pregnancy increased from 27% to 35%.
- Pregnant women who received at least 2 doses of tetanus toxoid (TT) during the most recent pregnancy increased from 40% to 43%.
- Women who had a postpartum visit within 24 hours of giving birth increased from 34% to 39%.
- Facilities upgraded and meeting safe birth and newborn care quality standards increased from 0 to 26.
- On average, district health budgets increased by 52%.

"These results are impressive. However, there are vast differences in performance between districts, requiring district-level analysis for future directions. Project targets set for 2009 are significantly higher than the 2007 targets. At the current implementation rate, it is unlikely that the project will achieve its final targets for most of the indicators. It should be understood that it is too early to expect population-level impact in terms of significant behavior change and increased service utilization."

A series of conclusions, short-term recommendations for the PAIMAN project, and long-term recommendations for USAID follows. On the whole, evaluators feel that "PAIMAN has been successful in putting key building blocks in place through training, communication and outreach, and facility renovations to enhance maternal and newborn care. The foundation is in place for increased utilization of services....Due to the political climate, with additional funding the project might be urged to expand into other districts..." The evaluation team suggests phased graduation of PAIMAN-supported districts over the next 5 years.

Evaluators extend a series of recommendations to be integrated into any follow-up project; to cite only one: Behaviour change messages need to be refined before any expansion, in part by working with the national MNH and LHW programmes to simplify and consolidate the number of BCC messages and intended audiences. It should also clarify messages on major maternal and neonatal danger signs and the definition of skilled birth attendant (SBA). The evaluation team suggests that PAIMAN consider segmenting messages by theme (e.g., antenatal care) and conduct intensive campaigns for 3 months on one subject, so that the same messages are reaching intended audiences via numerous channels, such as interpersonal outreach, theatre, and television. "LHWs and CHWs reach the largest numbers of beneficiaries and their work should continue. PAIMAN should conduct a mini-evaluation of the other BCC activities (events and theater) to determine the two or three most effective strategies. These interventions should be scaled up and the rest dropped. Where possible, events should only be conducted if they provide service in inaccessible areas."

II. The strategic objective of the ICHP is to increase use of key child health services and behaviours in Federally Administered Tribal Areas (FATA), a belt of 7 semi-autonomous tribal agencies with a total population of 3.6 million people stretching north to south along the border between Pakistan and Afghanistan. ICHP aims to increase access to and availability of quality child health services and to increase knowledge and acceptance of child health services and behaviours at the community level.

Key Findings:

Effective relationships have been established with FATA officials, health care providers, and community leaders, which has greatly facilitated implementation in the 6 months prior to the evaluation.
A major obstacle to project implementation has been shortages of Ministry of Health (MOH) health staff, especially female staff, worsened by staff absenteeism due to the fluid security situation.
To support programme implementation and facilitate reinforcement of health system management, ICHP helped to form an agency health management team (AHMT) for each agency in FATA.
To increase access to child health services, ICHP has started major and minor renovations of health facilities, upgraded medical stores, and facilitated Child Health Days (CHDs). CHDs aim to ensure that health care providers, medicines, and vaccines are available on the same day to increase community access to needed services. The CHD approach seems to be successful, although inadequate supplies of essential medicines continue to be a major issue. The project has asked USAID to provide a waiver so it can purchase medicines.
To improve the knowledge and skills of health care providers, the project conducts trainings in essential newborn care (ENC) and in integrated management of newborn and childhood illnesses (IMNCI). It is too early to assess the results of these trainings.
The project formulated a community mobilisation strategy to identify areas for intervention and to help bridge the gap between service providers and communities. Key activities include: increasing community awareness through lady health workers (LHWs) and community health workers (CHW) managed through subgrants with local NGOs; community sensitisation events, such as formation of local committees; and advocacy at agency, tehsil, village, and facility levels.

"The evaluation team concludes that the project can make a substantial contribution to increasing access to and the availability of quality child health services and increasing knowledge, awareness, and healthy behaviors related to child health issues, if the security situation allows them to implement the interventions."

The evaluators argue that, since project momentum is just building, the project should be extended for two years, through 2011, so it can achieve the targets initially set. Since FATA is culturally, politically, and structurally different from the rest of Pakistan, USAID should design an integrated maternal, neonatal, and child health (MNCH) project with a water and sanitation component specifically for FATA. Depending on the funding available, the project could be expanded to cover other areas of primary health care. USAID needs to continue to advocate with FATA officials to enhance resources for MNCH. USAID and the project also need to engage traditional tribal leaders (Jirga elders) in areas where the project is working in order to obtain greater support and buy-in from community leaders. It would be useful to provide pre-service training for women from FATA to become lady health visitors (LHVs) to increase the number of female health care providers who reside and work in these areas.

Click here to access the 74-page report in PDF format.

Click here to access a 124-page document of Annexes.