HealthLearn’s First Clinical Practice Evaluation: Results

Executive Summary

• Working with local partners, we evaluated whether our online Newborn Care Foundations course could improve birth attendants' practices in Nigeria by directly observing care practices before and after they took the course.

• We found high course completion rates and significant knowledge gains among course participants. Overall clinical practice scores also improved significantly among observed birth attendants.

• Most notably, we saw substantial improvements in early breastfeeding rates after the training, along with related supportive practices like coaching mothers and maintaining skin-to-skin contact. 

• We trained over 5,000 health workers in this course in 2024, and this evaluation suggests that the course improves early initiation of breastfeeding, an outcome associated with substantial reductions in neonatal mortality. 

• The results of this evaluation feed into updated impact estimates that suggest the program has had a large impact to date and could be even more impactful as we scale up.

Introduction

We are excited to share the results of the first clinical evaluation of our Newborn Care Foundations course. This study was designed to test key uncertainties in our Theory of Change – in particular, assessing whether our online course can improve birth attendants’ clinical skills. In this post, we summarize our preliminary findings, which suggest that the course significantly improves health workers’ clinical practice and key indicators associated with lower newborn mortality.

In a follow-up post, we provide updated impact modeling and our plans for next steps.

What is HealthLearn?

We develop and provide engaging, accredited, case-based, mobile-optimized online courses for health workers (HWs) in Nigeria and Uganda. This includes one HealthLearn course (Newborn Care Foundations) and two courses (focused on epidemic preparedness and hypertension diagnosis and management) from another NGO. HealthLearn is distinct from many other digital health solutions (for example, data collection tools, chatbots, or clinical decision support apps) in the sense that our intervention is specifically designed to deliver in-service professional development for facility-based health workers who are using their own devices. 

Methods

Summary

Our primary goal with this evaluation was to assess whether HWs’ clinical practices improve after they take our Newborn Care Foundations course. To test this, we worked closely with partners in Nigeria to directly observe HWs’ birth care practices before and after taking the course. These data were collected by trained observers using a standardized checklist. The primary outcomes we assessed were overall checklist scores and the scores of key indicators associated with lower mortality.

Details

The HealthLearn Newborn Care Foundations course teaches the basic care that every mother and newborn should receive. It is targeted to facility-based birth attendants (a role filled in Nigeria by nurses, midwives, community health workers, and other cadres of HWs). The course was developed using Nigeria’s national guidelines for newborn and maternal care. The full contents of the course have been validated and approved by the Newborn Care Subcommittee of Nigeria’s Federal Ministry of Health and Social Welfare (FMOHSW) and have also been reviewed and accredited for continuing professional development by the Nursing and Midwifery Council of Nigeria and the Community Health Practitioners Registration Board of Nigeria. The course places special emphasis on care in the first 90 minutes after birth and practices that are proven to reduce neonatal mortality. Some of the specific practices highlighted in the course are described in more detail below.

We conducted a pre-/post-course observational study of birth attendants working in primary health centers (PHCs) in the Federal Capital Territory (FCT), Nigeria. PHCs in the Bwari and Gwagwalada area councils (both outside of the Abuja city center) were selected with a convenience sampling approach based on delivery volume, accessibility to observers, and security considerations. Pre-course clinical observations (described below) were collected before the course was disseminated to participating HWs. When pre-course data collection was complete, 100 HWs in the selected facilities were encouraged to take the online course using their own devices. All HWs in participating facilities were eligible to take the course. Our implementing partner in FCT, Thriving Up Initiative, led the dissemination of the course which involved organizing the HWs, sharing the course, and motivating them to take the course. They encouraged HWs to take the course in a shared WhatsApp group and provided each HW with a small stipend (2,000 Naira in total, just over one US dollar) to support the costs of mobile data. Thriving Up Initiative also conducted a post-course asynchronous WhatsApp discussion of key practices, which was led by a certified essential newborn care trainer. Post-course clinical observations began after this discussion was complete. On average, post-course observations were collected approximately one month after participants took the course.

Our evaluation partner, Brooks Insights, led data collection, which involved observing HWs attending deliveries before and after taking the HealthLearn Newborn Care Foundations course. Each delivery was scored with a standardized checklist based on Nigeria’s national guidelines and international best practices. The checklist was reviewed and revised by two experts: a neonatologist who sits on the FMOHSW Newborn Care Subcommittee, and a nurse/midwife who is a certified and experienced trainer in essential newborn care. Observers were experienced HWs who had been certified in essential newborn care and went through a required classroom training and pilot observation. Observers used KoboCollect to gather checklist data. Eligible participants in observed deliveries included any HW who attended deliveries in the selected PHCs. Eligible observations had to be uncomplicated vaginal deliveries; multiple deliveries (e.g. twins) and those with major complications (e.g. birth asphyxia) were excluded during observation. Mothers and birth attendants gave informed consent for each observation. Our objective was to collect two observations before the course and two observations after the course for at least 30 birth attendants. 

Brooks Insights conducted spot checks of some observations, and all observations included in this analysis passed through routine data quality checks for geolocation, duration, and internal consistency. Checklist responses were converted to a binary scale (1 - correct practice; 0 - incorrect practice) for analysis. We analyzed results at the HW level (comparing pre- and post-course checklist scores) with the Wilcoxon Rank-Sum test, and at the individual indicator level using Fisher’s Exact Test. 

We also collect some data within the Newborn Care Foundations course itself. Participants take a short pre-test when they enroll in the course, and repeat the same test at the end of the course. These in-course pre- and post-test scores were compared with paired t tests. The Newborn Care Foundations course remains open for enrollment, but the results presented in this writeup (e.g. course completion) are for data through November 25, 2024. The full study design passed through review and approval by the Federal Capital Territory Ethics Review Board.

Key Results

In-course learning gains

Of the 100 HWs in the group who were asked to take the course, 91 completed it. The average in-course pre-test score was 67% in this group, which increased to 80% on the post-test (P < 0.0001). Average pre-test scores in this group were higher than those of the wider population of >5,000 people who completed the Newborn Care Foundations course in Nigeria and the group of course completers in FCT, but all groups had roughly equivalent post-test scores (Figure 1). This suggests that the training brings groups of HWs with different knowledge levels up to roughly the same level. When considering all course completers, average pre-test scores were only slightly lower in the group that reported attending no births (58%) vs. those that reported attending ≥1 birth/month (62%), or ≥5 births/month (63%). This suggests that the HWs working in facilities that participated in the clinical evaluation had higher baseline knowledge of newborn care than the broader population of HWs in Nigeria, but experience as a birth attendant had only modest impacts on prior knowledge of the best clinical practices assessed on the test.

Figure 1: Pre/post-test scores in three populations: all learners who completed the course, those who self-reported working in FCT, and those who worked in the facilities participating in this evaluation. The difference in mean scores (Δ) is shown for each group. There was a statistically significant improvement in scores from pre- to post-test within each of these groups (P<0.0001).

Direct observation of HWs

In total, we collected 56 observations before the course and 52 observations after the course (note that, in alignment with our original protocol, some HWs were observed twice). 33 birth attendants were observed in total. Among the 28 HWs who were observed both before and after the course, pre-course checklist scores were 44% and post-course checklist scores were 57% (an improvement of 13 percentage points, P < 0.001). The group of HWs observed had an average pre-test score of 73% in the course; scores increased to 80% on the post-test, but this difference did not reach statistical significance (P = 0.1). 

We assessed whether there were changes in key indicators that we emphasized in the course because they have a demonstrated benefit on newborn and/or maternal mortality. This analysis focused on the following indicators:

1. Giving mothers a uterotonic (medication that reduces the risk of bleeding, such as misoprostol or oxytocin) within 5 minutes of delivery.

2. Waiting 1-3 minutes to clamp and cut the umbilical cord (this substantially reduces preterm neonatal mortality).

3. Promoting early breastfeeding by keeping the baby and mother in skin-to-skin contact for the first hour after birth and coaching the mother on early initiation of breastfeeding (delayed breastfeeding is associated with a 33% increase in newborn mortality).

For the third indicator, we also tested whether there were changes in two key practices that support early initiation of breastfeeding: coaching the mother to encourage her to begin breastfeeding early (3a), and keeping the baby in continuous skin-to-skin contact with the mother for the first hour after birth (3b). Note that early skin-to-skin care is distinct from kangaroo care, which has much longer duration (days or weeks) and is specifically targeted towards low birthweight babies. 

There were high rates of delayed cord clamping in the pre-course observations, which did not change significantly in post-course observations. However, there were significant increases of 23 and 26 percentage points, respectively, in early administration of a uterotonic and initiation of breastfeeding within the first hour after birth. The two practices we tracked that support early breastfeeding (indicators 3a and 3b) also increased significantly in the post-course condition.

Discussion

Interpretation

These results suggest that Newborn Care Foundations can improve clinical practice and key outcomes (such as early initiation of breastfeeding) associated with reductions in newborn mortality. These results are in alignment with prior research. Other studies have demonstrated that digital training can improve HWs’ clinical practice, even in resource limited settings, which is consistent with the modest clinical practice improvements we observed in this evaluation. 

We found that certain practices still had low adherence in the post-course condition. Notably, continuous skin-to-skin care in the first hour after birth increased, but it was still only practiced in roughly one-fifth of observed births. Nigeria’s most recent Demographic Health Survey reports just 36% of neonates begin breastfeeding within an hour of birth, which suggests that there is massive room for improvement in this indicator. Given the large gaps in basic newborn care practices identified here, policy and advocacy work, or even mass media campaigns, may be particularly impactful ways to increase early breastfeeding and thereby improve neonatal outcomes at scale in Nigeria.

There were significant improvements in knowledge (as measured by pre-/post-test scores) among all course completers, course completers in FCT, and HWs in the targeted facilities. However, the subset of HWs who were directly observed for this study had test score improvements that were not statistically significant. It may be that the test has limited sensitivity, or that there is only a modest correlation between improvements in knowledge and in clinical practice, or that the most salient behaviors emphasized in the course aren’t reflected perfectly by the test questions. Consequently, test scores alone may not fully capture the training's impact on health outcomes.

Limitations

This evaluation did not assess a non-intervention group, which could help to control for changes unrelated to the HealthLearn course. For example, after facility selection, we learned the facilities were required to participate in a five-day government training program that happened during pre-course observations. This training was principally done through lectures and comprised 17 modules, including one module on perinatal health. We reviewed this module’s contents and found that many of the specific practices covered in the Newborn Care Foundations course were not in the four slides that covered newborn care. There was no difference in the average checklist scores of pre-course observations collected before (43%, N=32) and after (43%, N=24) the in-person training, suggesting that the government training program did not focus on or influence the behaviors we assessed in this study.

Observer effects are another potential confounder and cause for concern. Birth attendants might have been more likely to implement the practices they learned in the course because they were aware of being observed.

Finally, some features of the intervention tested in this study group may not be replicable at scale. The facilities studied may not be representative of Nigeria as a whole, because it was an initial study conducted in a peri-urban area. Moreover, most HWs who took the Newborn Care Foundations course this year were not part of a partner-led WhatsApp group, and they did not receive data stipends. HWs who complete the course independently of their coworkers may be less influenced by the social and normative factors that reinforce practice change. 

While the current study has many limitations, there are also plausible arguments for greater impact. For example, the general population of (largely self-selected) course completers may be more motivated to learn and change their practice than health workers, such as those included in this evaluation, who are financially incentivized to take the course. Moreover, the birth attendants observed in this study had higher pre-test scores than the population of all course completers and all course completers who report attending births. It seems likely that the study participants already had higher knowledge and better baseline practice than the average PHC worker in Nigeria. 

Suggestions for future evaluations

Future evaluations should include facilities from urban and rural areas across Nigeria’s geopolitical zones. We were not able to reach the initial target of 30 paired observations due to low volume of deliveries and challenges with ensuring sufficient coverage of well-trained observers for all deliveries. Future work may need to include more high-volume facilities, and observers may need regular feedback to ensure that they collect consistent and high quality data. We learned a great deal in this evaluation about ways to improve data collection and even the observation checklist itself; those lessons will feed forward into future studies. 

Conclusions

These preliminary results suggest that the Newborn Care Foundations course significantly improves clinical practice and key indicators (such as early initiation of breastfeeding) that are associated with substantial reductions in mortality. We have trained over 5,000 participants in Nigeria in the ~9-month period since the course was launched. Extrapolating from the results of this clinical evaluation to the full population of trainees, the program is plausibly cost-effective to date, and potentially even more cost-effective at a higher scale.

We are keen to improve the program in ways that specifically target the key clinical behaviors identified in this evaluation and conduct further evaluations of the program at scale.

Acknowledgements

This work would not have been possible without the leadership and support of:

• The FMOHSW, especially the Family Health Department.

• The Newborn Care Unit and Newborn Subcommittee of FMOHSW.

• The National Primary Health Care Development Agency (NPHCDA).

• Brooks Insights, our evaluation partner.

• Our implementing partners: Thriving Up Initiative (in FCT), The Taimaka Project (in Gombe State), and iDevPro (in Kano State), as well as subnational governments who supported course dissemination.

• The Global Health Media Project, who created the videos used in the course and gave us permission to use them.

• Our advisors, staff, and volunteers, who have all played critical roles in this work.

• Colleagues at Ambitious Impact and Resolve to Save Lives, who have been supportive of our work from the start.

• Our generous and visionary funders and donors.

• Colleagues who read this report and gave constructive feedback.

• All of the health workers who participated in the course and evaluation.