An Update on HealthLearn’s Impact Model and Next Steps

Executive Summary

HealthLearn provides accredited, engaging, mobile-optimized online courses for health workers in Nigeria and Uganda. Our focus is on lifesaving clinical skills that are simple to implement.

A recent evaluation of our Newborn Care Foundations course showed significant improvements in birth attendants’ clinical practices and key birth outcomes. Early initiation of breastfeeding, strongly linked to reduced newborn mortality, improved significantly in the evaluation.

After applying large (>10X) discounts, we estimate the course is ~24 times more cost-effective than GiveWell’s cash transfer benchmark. We are uncertain about the precise magnitude of impact, but a sensitivity analysis suggests that the program is cost-effective under a wide range of plausible scenarios.

Our already-low unit costs should decline as we scale up. This is likely to increase or at least maintain the program’s cost-effectiveness, even if the impact per trainee is lower than our current point estimate. We also earn revenue by hosting courses for another NGO, which covers a portion of our core team costs and increases cost-effectiveness per philanthropic dollar spent.

We have identified key uncertainties in evidence strength, sustainability of clinical practice change, and intervention reach. We plan to improve our monitoring and evaluation to assess these uncertainties and develop more precise estimates of impact per trainee. We will continue our work to improve and scale up the Newborn Care Foundations course, while also developing new courses addressing other gaps in clinical practices where impactful interventions are needed.

Background

HealthLearn is a nonprofit that develops and provides engaging, accredited, case-based, mobile-optimized online courses for health workers (HWs) in Nigeria and Uganda. This includes one HealthLearn course (Newborn Care Foundations) and two courses (focused on epidemic preparedness and hypertension diagnosis and management) from another NGO. HealthLearn is distinct from many other digital health solutions (for example, data collection tools, chatbots, or clinical decision support apps) in the sense that our intervention is specifically designed to deliver in-service professional development for facility-based health workers who are using their own devices. 

While this post focuses on our newborn care work in Nigeria, our overall objective is to develop a highly scalable tool applicable across clinical areas and countries, targeting critical skills to save lives. 

Newborn Care Foundations

The Newborn Care Foundations course teaches the basic care that every newborn should receive, following Nigeria’s national guidelines and international evidence-based best clinical practices. The training places special emphasis on practices with a proven impact on key health outcomes (newborn and maternal mortality). For example, in every uncomplicated delivery, HWs should:

1. Give mothers a medication that reduces their risk of severe bleeding after the baby is born,

2. Wait 1-3 minutes to clamp and cut the umbilical cord (this substantially reduces preterm neonatal mortality), and

3. Promote early initiation of breastfeeding by keeping the baby and mother in skin-to-skin contact for the first hour after birth and coaching the mother on breastfeeding. Delayed breastfeeding (>1 hour after birth) is associated with a 33% increase in newborn mortality.

It’s worth noting that skin-to-skin care in the first hour after birth is not the same as kangaroo care, which should be provided specifically to low birthweight infants over several days or even weeks. Our course covers kangaroo care and other perinatal practices, but it mainly emphasizes the universal care that every newborn should receive in the first 90 minutes after birth. If you’re interested in learning more about kangaroo care, we’d recommend checking out the excellent work of ansh!

Our Clinical Practice Evaluation

We recently concluded our first clinical practice evaluation, which demonstrates that health workers significantly improve two of the three practices and outcomes enumerated above after taking the course, while the third, delayed cord clamping, already had very high compliance at baseline. 

Although there are a number of limitations to our evaluation (described in more detail under "Discussion" here), these results provide us with an opportunity to update our estimates of HealthLearn’s cost-effectiveness.

All Models Are Wrong – Some Are Useful

Before we delve into the modeling, we want to clarify that our objective here is to assess whether HealthLearn has been plausibly cost-effective to date. While the modeling suggests that the program has been cost-effective, we do not think a model (on its own) can be considered “proof.” 

To build our model, we had to make a number of simplifications and assumptions. 

• We assess the benefits of the improvement we observed in one specific outcome – early initiation of breastfeeding – even though the Newborn Care Foundations course targets multiple clinical practices.

• We extrapolate from the effects observed in a small-scale evaluation to the full population of >5,000 Newborn Care Foundations course completers who collectively report attending tens of thousands of births per month.

• We apply discounts to the estimated impacts due to potentially limited external and internal validity of our evaluation and the supporting evidence we use for modeling. These discounts cumulatively amount to a >10-fold reduction in the original observed effect.

Even after discounting, our model suggests the program is ~24 times more cost-effective than the GiveWell benchmark of pre-2024 cash transfers (comparable to the cost-effectiveness of recognized top charities in global health). A simple sensitivity analysis suggests that the program is more cost-effective than cash transfers under a broad range of scenarios (and discounts) that we consider to be plausible. The results of this simplified model are in overall alignment with a more fully featured probabilistic impact model. 

Figure 1: Distribution of endline cost-effectiveness relative to cash transfers from a probabilistic model. This model has a similar logic and structure as the simple model described above, but accounts for a broad range of possible values for variables that determine impact.

We account for the total lifetime costs of HealthLearn (from founding to November, 2024) in the above analysis. However, there is a wrinkle: HealthLearn earns revenue by hosting and supporting courses for another NGO. Some of that revenue goes towards the time we invest to work closely with them to develop and improve their courses, and the rest contributes to the “shared services” costs of developing and maintaining the online platform that hosts all of these courses. The above estimates are “per philanthropic dollar” (that is, per dollar donated to HealthLearn that we have spent to date). We think it is most appropriate to assess cost-effectiveness this way, but a more conservative approach would measure cost-effectiveness per dollar spent on all of HealthLearn, in which case we would estimate our cost-effectiveness to date to be ~9 times that of the GiveWell benchmark. 

Substantial Uncertainties

This modeling exercise and sensitivity analysis highlight what we consider to be the three key uncertainties associated with our impact evaluation. These are, in order of cost and complexity to evaluate (with the first being the most expensive and challenging to address):

1. Questions about the evidence base: the evidence linking early breastfeeding to lower mortality is observational, not experimental. We think it is plausible that healthier neonates are more likely to begin breastfeeding earlier, which could complicate how we interpret causal linkages.

2. Questions about the true extent and duration of clinical practice change: direct observation could put workers on their best behavior, we have not conducted follow-up assessments to assess practice change months or years after the intervention, and we are uncertain whether our evaluation is representative of all HWs in Nigeria. 

3. Questions about the reach of the intervention: this primarily encompasses the number of births attended by the HWs we train. We based this analysis on HWs’ responses to a survey question in the course and we think this estimate should be confirmed with other approaches.

We have applied discounts to the current model due to these uncertainties. They also provide a roadmap for future evaluation work. 

Modeling Impact at Scale

One of the major benefits of a tech-enabled intervention is excellent economies of scale. While our program does have certain fixed costs (primarily personnel) that are required for our operations, the marginal cost of each additional trainee is comparatively low. The overall cost-effectiveness of our intervention is driven by two key factors: impact per trainee and cost per trainee (Figure 2).

Figure 2: The unit economics of our impact. Lines represent particular benefit-to-cost ratios. Any point above a given line represents an intervention that is more cost-effective than points below the line. If costs per trainee are high, the impact per trainee must also be high to maintain cost-effectiveness. Conversely, the program is cost-effective at a broad range of impacts per trainee as long as the cost per marginal trainee is low. (MV - GiveWell’s units of moral value; note that the cash transfer multipliers given here are relative to GiveWell’s pre-2024 estimate of the impacts of cash transfers in Kenya).

Notably, we are much more uncertain about the impact per trainee than the cost per trainee, which we can calculate directly from our platform analytics and financials. The average philanthropic cost per trainee is ~$14 to date, whereas marginal cost per trainee is a mixture of simple unit costs (e.g., WhatsApp and SMS messaging fees, costs of acquisition) and personnel costs that are more step-like (e.g., the costs of hiring additional software engineers). As we scale up, we anticipate that we will be able to drive down marginal cost per trainee to as low as $1. As marginal costs decline, the intervention will still be cost-effective even if the impact per trainee is substantially lower than the (already discounted) impact we estimate in our current evaluation. 

This relationship between cost and impact per trainee is a linchpin of our strategy. While it may be challenging to measure a precise impact per trainee, if we can establish a plausible range of impacts per trainee while scaling up and keeping unit costs low, we will increase our confidence in the cost-effectiveness of the intervention. Cost per trainee is a simple “North Star metric” that we will use in our planning and ongoing evaluation. 

This analysis also highlights a key challenge we face in evaluating our program: the program as a whole can be very cost-effective even with a relatively small benefit per trainee, so future evaluations of endline impacts (such as reductions in newborn mortality) may require complex study designs and large sample sizes. In this sense, the program may have more in common with policy and mass media interventions than other “direct” global health programs. 

Discussion

We believe the HealthLearn platform could become a cross-cutting tool for improving clinical practice at scale. These impact estimates only account for improvements in early initiation of breastfeeding, but other checklist indicators also improved in our evaluation. It is likely that this increases the impact of the Newborn Care Foundations course, but at this time we do not feel we have sufficient evidence to directly model the cost-effectiveness of other clinical practice improvements. Similarly, the courses we host for other NGOs are likely to increase the overall impact of the platform, but we think that these benefits should primarily be attributed to those organizations rather than HealthLearn and we do not consider them here.

Looking beyond newborn care, there are clinical practices we could cover in future courses that have proven impacts and suboptimal levels of adherence. Notably, most cases of diarrhea are effectively cured with lifesaving and inexpensive oral rehydration therapy, but only 60-70% of children in Nigeria seen in primary health centers for diarrhea receive this treatment. While there are excellent interventions to address this problem at the community level, we’re not aware of programs that effectively and scalably improve the diarrhea diagnosis and management practices of facility-based health workers. We believe that much of the potential impact of our approach could come from developing more high-impact clinical training that covers a variety of topics.

We have addressed the key uncertainties identified in this modeling exercise by applying discounts. While we think this is appropriate given the small scale of our evaluation and gaps in the available research literature, the true impact may be higher than these discounted estimates imply. To give two examples: this model does not account for clinical practice changes that last for more than a year, and the effect of training may be larger for users outside of our evaluation cohort due to lower baseline knowledge. We hope that future evaluations will allow us to assess these uncertainties.

Broader Implications

Unlike many other clinical practices, the skills of coaching the mother on breastfeeding and leaving the baby in skin-to-skin care for the first hour after birth do not require special commodities or equipment. In our evaluation, we observed a low baseline level of early breastfeeding and low adherence to the practices that support it. This aligns with a recent study indicating that just 36% of neonates in Nigeria begin breastfeeding within an hour of birth. Another recent analysis finds that only ~55% of neonates in 35 sub-Saharan African countries were encouraged to breastfeed within the first hour after delivery. In its assessment of facility-based (in-person) health worker training in maternal and neonatal health, GiveWell found that such programs are likely to be cost effective, but also have “very heterogeneous training content.” The specific outcome of early breastfeeding may be a simple, underappreciated, and shared mechanism contributing to the benefits of these newborn care training programs. We wonder whether concerted efforts to improve early breastfeeding via mass media campaigns, policy initiatives, supportive supervision, and/or health worker training might effectively save many lives. 

The association between early initiation of breastfeeding and lower mortality has been studied in many contexts and with large pooled sample sizes, but this evidence is observational: babies who begin breastfeeding earlier are at lower risk of mortality. It seems plausible (even likely) that healthier neonates are more likely to begin breastfeeding earlier, which raises concerns about reverse causality. However, we think the available evidence is consistent with a direct benefit of early breastfeeding. For example, one study found that early initiation of breastfeeding had similar mortality effect sizes even when unwell and premature babies were excluded. In our modeling, we only considered the mortality reduction associated with breastfeeding within the first hour as opposed between 1-24 hours after birth. Babies who begin breastfeeding >24 hours after birth have an even greater risk of mortality (and, we suspect, are somewhat more likely to have underlying health problems that cause the delay).

In a rapid search of the literature, we were not able to find experimental studies that directly assess the impact of behavioral determinants of early breastfeeding (such as coaching the mother and continuous skin-to-skin care) on mortality. Experimental studies do show that early skin-to-skin care increases exclusive breastfeeding (feeding only breastmilk for the first six months of life), and other experimental studies show that exclusive breastfeeding reduces newborn mortality. Nonetheless, more experimental research is needed to better assess the causal relationship between early initiation of breastfeeding and neonatal mortality.

Conclusions and Next Steps

Our modeling indicates that the Newborn Care Foundations course has been plausibly cost-effective to date, even when we apply large (>90%) discounts. The course could potentially be much more cost-effective at scale. The results directly inform our strategy for the next year. We plan to:

• Make improvements to the Newborn Care Foundations course: we are keen to place greater emphasis on the biggest gaps identified in this evaluation and incorporate more behavioral insights to target and reinforce the specific clinical practices that increase early initiation of breastfeeding. 

• Improve our evaluation and impact estimates: we aim to update our monitoring and evaluation based on what we’ve learned. In particular, we aim to continue to assess the three key uncertainties (described above) that influence impact per trainee in our model.

• Build and test pathways to scale: our platform is ready for high traffic, but we still need to learn more about the best ways to recruit and retain learners in our courses at low cost. This will likely involve a combination of product improvements, marketing, linkages to programs run by government and other NGOs, and advocacy work. We aim to scale up enough to keep the average cost per trainee down while also developing better estimates of marginal unit costs.

How You Can Help

• Reach out to us and share your ideas, questions, and feedback. We are seeking input from diverse stakeholders who work in global health, government, philanthropy, and research.

• Collaborate with us. We are particularly keen to work with:

  • NGOs and governments who are interested in developing new courses that focus on high-impact clinical practices,

  • Experts in behavioral science and newborn care who can help us to build and assess effective and scalable tools to reinforce the clinical practices we target,

  • Volunteers with deep experience and expertise in software development, NGO operations, and public health, and

  • Grantmakers who are eager to help us continue exploring cost-effective ways to improve clinical care at scale.

Get in touch at: contact@healthlearn.org

Acknowledgements

This work would not have been possible without the leadership and support of:

• The Federal Ministry of Health and Social Welfare (FMOHSW), especially the Family Health Department.

• The Newborn Care Unit and Newborn Subcommittee of FMOHSW.

• The National Primary Health Care Development Agency (NPHCDA).

• Brooks Insights, our evaluation partner.

• Our implementing partners: Thriving Up Initiative (in the Federal Capital Territory), The Taimaka Project (in Gombe State), and iDevPro (in Kano State), as well as subnational governments who supported course dissemination.

• The Global Health Media Project, who created the videos used in the course and gave us permission to use them.

• Our advisors, staff, and volunteers, who have all played critical roles in this work.

• Colleagues at Ambitious Impact and Resolve to Save Lives, who have been supportive of our work from the start.

• Our generous and visionary funders and donors.

• Colleagues who read this report and gave constructive feedback.

• All of the health workers who participated in the course and evaluation.