Estimating ITN entomological efficacy from experimental hut trial data

Experimental hut trial data

Data from eight EHT, conducted in Tanzania, Côte d’Ivoire and Benin, are used. The main characteristics of each included trial arm are presented in Table 1 and the reader is referred to the original publications for further details (or the Supplementary Tables 6.1 and 6.2 for unpublished trials). We only retained the arms involving Interceptor® G2, Olyset® Plus and pyrethroid-only ITNs. Additionally, we only retained products for which measurements were available for both unwashed/new and washed/operationally-aged nets and we only kept the measurements with the highest number of washes or duration of field use.

In all trials, untreated nets are taken as the control. Each trial reports the number of mosquitoes found unfed-alive (UA), unfed-dead (UD), fed-alive (FA) and fed-dead (FD) immediately after collection and then held in a temperature-controlled facility to assess delayed mortality at 24 h (all trials except Nguessan et al.⁴⁸ and Sovegnon et al.⁴⁹) and 72 h (all trials except BIT055 and Odufuwa et al.⁴⁶). This information is summarised over all nights for each experiment in Supplementary Tables 2.1 and 2.2. The analyses are restricted to the dominant vector species collected (Anopheles arabiensis for Kibondo et al.⁴⁵, BIT080, Assenga et al.⁴⁴, BIT055 and Odufuwa et al.⁴⁶ and Anopheles gambiae s.l. for Assenga et al.⁴⁴, Nguessan et al.⁴⁸ and Sovegnon et al.⁴⁹), due to low total numbers collected for other Anopheles species. In Martin et al.⁴⁷, the analysis was conducted on both Anopheles gambiae s.l. and Anopheles funestus. For each trial, the main analysis used the longest holding time to assess mortality (72 h for Kibondo et al.⁴⁵, BIT080 and Assenga et al.⁴⁴, Martin et al.⁴⁷, Nguessan et al.⁴⁸ and Sovegnon et al.⁴⁹) and 24 h for BIT055 and Odufuwa et al.⁴⁶. As a robustness check, the analysis using 24 h for all trials where the outcome was measured is presented in Supplementary Fig. 7.

Statistical model

The reduction in host availability (deterrence), the pre-prandial killing effect and the post-prandial killing effect are estimated using a Bayesian hierarchical model, adapted from Denz et al.³⁸ (see Fig. 5). In this model, each night \(t\) and in each trial arm \(i\), mosquitoes have the possibility to feed with rate \({\alpha }_{{{{\rm{i}}}}}\) or die during host-seeking with rate \({\mu }_{{{{\rm{i}}}}}\). Those who have fed have then a probability \({{{{{\rm{p}}}}}_{{{{\rm{C}}}}}}_{{{{\rm{i}}}}}\) to survive after feeding. At the end of the night, mosquitoes are found in the four categories ‘UA’ (\({{{\mathrm{UA}}}}_{{{{\rm{i,t}}}}}\)), ‘UD’ (\({{{\mathrm{UD}}}}_{{{{\rm{i,t}}}}}\)), ‘FA’ (\({{{\mathrm{FA}}}}_{{{{\rm{i,t}}}}}\)) and ‘FD’ (\({{{\mathrm{FD}}}}_{{{{\rm{i,t}}}}}\)), with respective probabilities \(\left({{{{{\rm{p}}}}}_{{{\mathrm{UA}}}}}_{{{{\rm{i}}}}},{{{{{\rm{p}}}}}_{{{\mathrm{UD}}}}}_{{{{\rm{i}}}}},{{{{{\rm{p}}}}}_{{{\mathrm{FA}}}}}_{{{{\rm{i}}}}},{{{{{\rm{p}}}}}_{{{\mathrm{FD}}}}}_{{{{\rm{i}}}}}\right)\), with \({{{{{\rm{p}}}}}_{{{\mathrm{UA}}}}}_{{{{\rm{i}}}}}+{{{{{\rm{p}}}}}_{{{\mathrm{UD}}}}}_{{{{\rm{i}}}}}+{{{{{\rm{p}}}}}_{{{\mathrm{FA}}}}}_{{{{\rm{i}}}}}+{{{{{\rm{p}}}}}_{{{\mathrm{FD}}}}}_{{{{\rm{i}}}}}=1\). The model can thus be written with the following equations (all notations are indicated in Table 2):

with

The effect of a given intervention is measured by comparing the rates \({\alpha }_{{{{\rm{i}}}}}\) (feeding rate) and \({\mu }_{{{{\rm{i}}}}}\) (mortality rate while host-seeking) and the survival probability after biting \({{{{{\rm{p}}}}}_{{{{\rm{C}}}}}}_{{{{\rm{i}}}}}\) between the intervention arm (pyrethroid-only, chlorfenapyr or PBO net) and the control arm (untreated net, denoted with \(i=0\)).

The parameter \({\pi }_{{{{\rm{i}}}}}\) corresponds to the reduction in host availability for protected hosts in intervention arm \(i\) and thus represents the deterrence effect. It is defined as follows:

When \({\pi }_{{{{\rm{i}}}}}=0\), the intervention has no deterrence effect and when \({\pi }_{{{{\rm{i}}}}}=1\), protected hosts escape mosquito bites entirely.

The parameter \({\kappa }_{{{{\rm{i}}}}}\) corresponds to an increase in the killing rate while the mosquito is host-seeking in intervention arm \(i\) and it is defined as follows:

When \({\kappa }_{i}=0\), the intervention has no effect on host-seeking mortality and when \({\kappa }_{{{{\rm{i}}}}} > 0\) mosquitoes have a higher mortality while host-seeking than in the control. A related indicator is the pre-prandial killing effect \({\phi }_{{{{\rm{i}}}}}\), namely the reduction in the probability \({{{{\rm{p}}}}}_{{{{{\rm{S}}}}}_{{{{\rm{i}}}}}}\) of surviving before biting (where \({{{{\rm{p}}}}}_{{{{{\rm{S}}}}}_{{{{\rm{i}}}}}}=1-{{{{\rm{p}}}}}_{{{{{\rm{UD}}}}}_{{{{\rm{i}}}},}}\)). The pre-prandial killing effect can be obtained via the following formula (which can be sampled while fitting the Bayesian model):

This can be rewritten as

When \({\phi }_{{{{\rm{i}}}}}=0\), the intervention has no effect on the probability that mosquitoes survive and when \({\phi }_{{{{\rm{i}}}}}=1\), all mosquitoes die and none can feed the protected host. Because \({{{{\rm{p}}}}}_{{{{{\rm{S}}}}}_{{{{\rm{i}}}}}}\) is a function of \({\alpha }_{{{{\rm{i}}}}}\) and \({\mu }_{{{{\rm{i}}}}}\), which themselves depend on \({\pi }_{{{{\rm{i}}}}}\) and \({\kappa }_{{{{\rm{i}}}}}\), the pre-prandial killing effect \({\phi }_{{{{\rm{i}}}}}\) is itself a function of both \({\pi }_{{{{\rm{i}}}}}\) and \({\kappa }_{{{{\rm{i}}}}}\). It is required for subsequent mathematical modelling with OpenMalaria and will be used in the model to represent the potential of the intervention to kill mosquitoes before feeding.

Finally the parameter \({\xi }_{{{{\rm{i}}}}}\) corresponds to the post-prandial killing effect in intervention arm \(i\), whereby the probability of mosquito survival after biting is decreased by the intervention.

When \({\xi }_{{{{\rm{i}}}}}=0\), the intervention has no effect on post-prandial survival and when \({\xi }_{{{{\rm{i}}}}}=1\) no mosquito survives after biting when in contact with the intervention.

The model is fitted with Stan (version 2.32.2) using the Rstan interface (https://cran.r-project.org/web/packages/rstan, version 2.32.7), using 3 MCMC chains with 6000 iterations each, including 3000 warm-ups. Prior distributions and definition ranges for each parameter are indicated in Table 2. The model is fitted independently for each EHT and for unwashed/new and 20 times washed/aged nets separately. Convergence diagnostic plots are presented in the Supplementary Figs. 8.1–8.10.

Vectorial capacity reduction

In the model by Chitnis et al.⁶⁵, which is the non-periodic analogue of the model used in OpenMalaria, the vectorial capacity can be calculated either in the absence or in the presence of ITNs in the population to quantify the reduction in vectorial capacity attributable to the ITN intervention²⁰. A theoretical ideal vector control tool that would entirely remove the mosquito population corresponds to a 100% reduction in vectorial capacity, while a completely ineffective intervention corresponds to a 0% reduction. Vectorial capacity is computed using the AnophelesModel R package²⁰ (https://github.com/SwissTPH/AnophelesModel, version 1.1.0).

In order to account for the decay in ITN effectiveness over time, the model’s equilibrium vectorial capacity is calculated for different time steps over a 3-year period. At each time step, effective coverage is updated to reflect functional survival and entomological efficacy is updated to reflect insecticidal durability. The reductions in vectorial capacity calculated for each time step are averaged to obtain a single summary value over the 3-year time frame.

In order to propagate the fitting uncertainty of the entomological efficacy estimates, vectorial capacity is calculated for 1000 random samples from the posterior distribution for each EHT. The posterior mean and the 95% credible intervals in the vectorial capacity estimates for each EHT were selected. Vectorial capacity computation is deterministic and the only source of uncertainty propagated is the posterior estimation of the reduction in host availability and the pre- and post-prandial killing effects.

In order to reflect entomological efficacy only (in Fig. 1), vectorial capacity is computed under ideal conditions, namely, ITN usage is assumed to be 100%, functional survival is assumed to be 3 years, insecticide decay is assumed to be absent and in-bed exposure is assumed to be 100%. Other conditions are explored subsequently in the cascade of effectiveness and the associated methodology is detailed below.

In order to formally assess the difference in entomological efficacy between products tested within the same EHT, we computed the difference between vectorial capacity estimates for 1000 parameter sets from the posterior distribution. We computed the 2.5 and 97.5 quantiles of the resulting distribution and assessed if this interval contained the value 0 or not (displayed in Supplementary Table 3).

Reproducing in silico randomized control trial with a mathematical model of malaria transmission

Four cluster RCTs comparing the efficacy of pyrethroid-only nets with next-generation nets including PBO and chlorfenapyr are considered in this analysis: Protopopoff et al.^50,51 and Mosha et al.^11,12, conducted in Tanzania, Staedke et al.⁵² conducted in Uganda and Accrombessi et al.^13,14 conducted in Benin. They are reproduced in silico using the OpenMalaria simulation platform (https://github.com/SwissTPH/openmalaria)⁴³. This individual-based model accounts for the main specificities of malaria transmission dynamics in the human host (including immunity and superinfection) and the mosquito vector and it has been described extensively elsewhere⁴³.

The RCTs are modelled by adapting OpenMalaria to the setting-specific conditions of each trial location, using parameters as indicated in Table 3. The mathematical model’s transmission intensity parameter entomological inoculation rate (EIR) is adjusted to represent the pre-intervention prevalence in each trial arm, following the methodology by Lemant et al.³⁵. This approach relies on maximum likelihood to find the point estimate and profile likelihood methods to infer uncertainty ranges (see Supplementary material 9 for details). Resulting EIR estimates may vary across trial arms due to differing baseline prevalence and intervention coverages. RCT observations on post-intervention prevalence are used for validation. The simulations are conducted with OpenMalaria v44 and the OpenMalariaUtilities R package (https://github.com/SwissTPH/r-openMalariaUtilities, version 23.2), using a population of 10,000 individuals and 10 stochastic replicates for each scenario.

The agreement between model predictions and observations was assessed as in ref. ⁴⁰ by computing a linear regression between the model predicted values (X) and the observed values (Y):

(\({Y}_{{{{\rm{i}}}}}=m{X}_{{{{\rm{i}}}}}+0\), where i are individual repeated measures for each trial) and reporting the coefficient m and the adjusted R². In the R software (as in ref. ⁴⁰), the R² for that particular regression model is computed as \({R}^{2}=1-\frac{{\sum ({y}_{{{{\rm{i}}}}}-\hat{{y}_{{{{\rm{i}}}}}})}^{2}}{{\sum {y}_{{{{\rm{i}}}}}}^{2}}\), based on the assumption that \({{{\rm{E}}}}\left[{{{\rm{Y}}}}\right]=0\) when \({{{\bf{X}}}}=0\). This indicator was computed for prevalence measures, for effect sizes (relative difference between prevalence in the intervention arm and the pyrethroid-only control arm, i.e. \(100\,{{{\rm{x}}}}\,\frac{{{{\rm{P}}}}{{{{\rm{R}}}}}_{0}-{{{\rm{P}}}}{{{{\rm{R}}}}}_{{{{\rm{i}}}}}}{{{{\rm{P}}}}{{{{\rm{R}}}}}_{0}}\), where \({{{\rm{P}}}}{{{{\rm{R}}}}}_{0}\) is the prevalence in the control arm and \({{{\mathrm{PR}}}}_{{{{\rm{i}}}}}\) is the prevalence in the intervention arm) and for the relative change in prevalence compared to baseline (defined as \(100\,{{{\rm{x}}}}\,\frac{{{{\rm{P}}}}{{{{\rm{R}}}}}_{{{{\rm{t}}}}=0}-{{{\rm{P}}}}{{{{\rm{R}}}}}_{{{{\rm{t}}}}}}{{{{\rm{P}}}}{{{{\rm{R}}}}}_{{{{\rm{t}}}}=0}}\), where \({{{\rm{P}}}}{{{{\rm{R}}}}}_{{{{\rm{t}}}}=0}\) is the prevalence at baseline (pre-intervention) and \({{{\rm{P}}}}{{{{\rm{R}}}}}_{{{{\rm{t}}}}}\) is the prevalence post-intervention).

Entomological efficacy of ITNs in OpenMalaria

Within OpenMalaria, the effectiveness of ITNs is calculated using an Anopheles life-cycle model^65,66. In this model, the numbers of host-seeking, infected and infectious mosquitoes are simulated using deterministic difference equations that quantify their survival probability across five stages of the feeding cycle (host-seeking, feeding, searching for a resting place, resting and ovipositing), when they attack protected or unprotected human hosts.

ITNs are included in the model as a ‘generic vector interventions’ with the previously fitted values for \(\pi\), \(\phi\) and \(\xi\) taken as deterrence, pre-prandial and post-prandial killing effects, respectively. For each EHT, the set of parameters with the highest posterior density is used.

For RCTs conducted in East Africa (Mosha et al.^11,12, Protopopoff et al.^50,51 and Staedke et al.⁵²), EHT estimates from Tanzania are used. For the RCT conducted in Benin (Accrombessi et al.^13,14), EHT estimates from West Africa are used. Uncertainty in the effect size is propagated by using an ensemble of simulations based on parameters from the different EHTs. For each EHT, the parameter set with the highest posterior is used and we report the median and the envelope over all simulations across all EHT.

Therefore, the final uncertainty range includes (i) model stochasticity (10 seeds), (ii) uncertainty in initial transmission intensity resulting from profile likelihood methods³⁵ (see Supplementary material 9), (iii) variability across EHT datasets.

Epidemiological effectiveness of ITNs in OpenMalaria

The OpenMalaria parameterisation includes various factors that influence ITN effectiveness beyond the sole entomological efficacy.

Firstly, the effectiveness of the ITNs depends on the proportion of individuals having access to and using them. In the model, the proportion of protected and unprotected hosts is informed by the proportion of individuals effectively using a net, inferred from RCT survey data on the proportion of individuals reporting using a net the night before the survey, as explained in the Supplementary material (section 10).

Secondly, the durability of ITNs over the 3-year period between distribution campaigns is accounted for via two mechanisms, namely insecticidal durability and functional survival. Insecticidal durability is informed by the entomological estimates for 20× washed (or aged) and unwashed nets. The estimates obtained for unwashed/new nets were used to represent the initial entomological efficacy. The estimates obtained for nets washed 20 times were used to represent the entomological efficacy after 3 years⁸, assuming a linear decay over the 3-year time interval. The estimates obtained for operationally-aged nets were used to represent the entomological efficacy after 2 or 3 years, depending on the outcome measured in the trial, assuming a linear decay over the time interval. Functional survival is included by reducing the effective ITN usage over time. It is assumed that all individuals initially possessed study nets, which were then lost over time according to a decay function derived from the observed data. The decay distribution was assumed to be a Weibull function with half-life and shape parameter fitted to RCT data on ITN usage as explained in the Supplementary material (section 10), assuming half-life cannot exceed 3 years.

Thirdly, because insecticide treated nets can only protect hosts that are bitten while in bed, published data on activity rhythms for both humans and mosquitoes from Golumbeanu et al.²⁰ (see Supplementary Fig. 11) are used to quantify the exposure to mosquito bites while in bed (denoted \(\epsilon\)). The effective usage of ITNs is thus multiplied by \(\epsilon\). In each case, all datasets originated from the country in which the RCT was conducted are averaged. If the database²⁰ does not contain data for the country, the average across all datasets originating from Africa is used instead.

Cascades of ITN effectiveness

The cascade of effectiveness relies on the concept of vectorial capacity, defined and calculated as explained previously.

The effectiveness cascade includes 5 different factors in this order: entomological efficacy, functional survival, imperfect usage at distribution, insecticidal durability and activity rhythms. For each of these factors, we compute a realistic estimate (derived previously to reproduce the RCT by Mosha et al.^11,12) and a perfect comparator corresponding to an ideal setting where there would be no loss of effectiveness due to the factor. For example, for ITN usage, realistic values from the trial are between 70 and 80%, while the perfect comparator would be 100%. All these estimates and their perfect comparator counterpart are detailed in Table 4.

To derive the effectiveness cascade, the vectorial capacity is first calculated considering the perfect comparator for each factor and then recalculated, replacing sequentially each factor by its realistic value.

In order to propagate the fitting uncertainty of the entomological efficacy estimates, vectorial capacity is calculated for 1000 random samples from the posterior distribution for each EHT. The posterior mean and the 95% credible intervals in the vectorial capacity across the samples of all EHT for each net type were selected (posterior mean and 95% credible interval for each EHT are presented in Supplementary Fig. 4).

The cascades presented in the main text are parameterised to represent the setting in Mosha et al.^11,12), considering Anopheles gambiae vectors. Additionally, a dashboard is available to the reader to adapt the cascades to their own data.

Because there is no natural order between the different factors which all influence the model in non-linear ways, we conducted a sensitivity analysis by testing all possible ordering of the four factors (usage at distribution, functional survival, insecticidal durability and in-bed exposure). Under each permutation, the decrease in percentage points associated with each factor was computed, alongside the rank from 1 to 4 among the four factors.

Ethics approval

Ethical approval from Ifakara Health Institute-Institutional Review Board and National Institute of Medical Research Tanzania were obtained, respectively IHI/IRB/No: 29-2020 and NIMR/HQ/R.8a/Vol.IX/3521 for BIT055 and IHI/IRB/No: 35-2021, NIMR/HQ/R.8a/Vol.IX/3957 for BIT080. For Assenga et al.⁴⁴, ethical approval was and IHI/IRB/No: 21–2023 and NIMR/HQ/R.8a/Vol.IX/4558 for Tanzania and Ref 081-22/MSHPCMU/CNESVS-km for Côte d’Ivoire. Volunteers were recruited on written informed consent. For all other studies, the analysis is based on previously published data.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.