Trial data
We sought cluster-level malaria outcome data from corresponding authors of published CRTs identified in our previous systematic review (PROSPERO: CRD42022315741)17. Authors were initially approached by email, with a second follow-up email to non-responders. The initial review included 71 malaria CRTs that qualified for inclusion if they measured malaria-specific, epidemiological outcomes (prevalence or incidence) and randomised at least six geographical clusters to study arms. For trials measuring prevalence, we requested the number of malaria-positive individuals and total tested per cluster, study arm, and survey (Supplementary Table 8). For incidence measured via active case detection (ACD), we requested new malaria cases and total person-years at risk, and for passive case detection (PCD), new malaria cases and the population at risk, stratified by cluster, study arm, and trial year (Supplementary Table 9, 10). These data were supplemented with covariates from published articles, including diagnostic method and age range tested. Malaria prevalence in this study referred to the number of individuals tested positive for malaria over the total number of individuals tested for malaria. Malaria incidence in this study referred to the number of new cases divided by the total person years at risk (ACD) or total population in each cluster at risk (PCD). We classified trial endemicity based on control-arm prevalence or incidence averaged during the entire trial. Trial endemicity according to prevalence was categorised as high (>40%), medium (10–40%), or low (0–10%). For incidence, trial endemicity was categorised by malaria cases per person-year (py) as high (>0.8/py), medium (0.2–0.8/py), or low (0–0.2/py).
Trial prevalence data were further supplemented with requested intervention coverage/usage data (number of intervention users or individuals covered by interventions/total number surveyed) stratified by cluster, arm and survey. According to the month interventions were deployed and survey dates, we calculated the months since intervention(s) were introduced for each survey and categorised surveys as pre/post-intervention. We categorised all trial surveys as malaria season surveys if they were conducted within the publication-stated rainy season, plus one month to account for the delay between rains and vector propagation22,41. Surveys administered outside this range were considered non-malaria season surveys.
Between-cluster heterogeneity estimation
Methods-of-moments and mixed effects regression modelling approaches were used to estimate empirical values of prevalence k and ICC at the survey-arm level and incidence k at the study year-arm level according to methods described by Hayes and Moulton12,32. We refrained from estimating incidence ICC values, as rates with person-time denominators lack a clearly defined unit of observation36.
For the methods-of-moments approach, we computed the empirical variance (s2) of each survey arm for cluster-level prevalence (Eq. 1) and each study year-arm for cluster-level incidence (Eq. 2) according to:
$${Prevalence}\,\ {s}^{2}=\,\frac{\sum {\left({p}_{i}-\bar{p}\right)}^{2}}{c-1}$$
(1)
$${Incidence}\,\ {s}^{2}=\,\frac{\sum {({r}_{i}-\bar{r})}^{2}}{c-1}$$
(2)
where c refers to the total number of clusters, pi is the malaria prevalence in the ith cluster and \(\bar{p}\) represents the mean cluster prevalence(\(\sum {p}_{i}/c\)). For incidence, ri represents the annual malaria incidence per person in the ith cluster and \(\bar{r}\) represents mean incidence across clusters (\(\sum {r}_{i}/c\)).
To estimate the true between-cluster variance \({\hat{\sigma }}_{B}^{2}\) for each survey arm for prevalence (Eq. 3) or study year-arm for incidence (Eq. 4), we subtracted the random sampling error from the empirical variance s2 as follows:
$${{Prevalence}\,\hat{\sigma }}_{B}^{2}=\,{s}_{{prev}}^{2}-\frac{p\left(1-p\right)}{{\bar{n}}_{H}}$$
(3)
$${Incidence}\,{\hat{\sigma }}_{B}^{2}=\,{s}_{{inci}}^{2}-\frac{r}{{\bar{f}}_{H}}$$
(4)
where p refers to the overall survey-arm malaria prevalence, \({\bar{n}}_{{H}}\) is the harmonic mean of the total number of individuals \({n}_{i}\) tested per cluster (\(c/\sum \left(\frac{1}{{n}_{i}}\right)\)), r refers to the overall study year-arm malaria incidence and \({\bar{f}}_{H}\) is the harmonic mean of the total follow up time in years \({y}_{i}\) per cluster (\(c/\sum \left(\frac{1}{{y}_{i}}\right)\)).
We then estimated prevalence k for each survey-arm (Eq. 5) and incidence k for each study year arm (Eq. 6) according to:
$${Prevalence}\,\hat{k}=\,\frac{{\hat{\sigma }}_{{B\; prev}}\,}{p}$$
(5)
$${Incidence}\,\hat{k}=\,\frac{{\hat{\sigma }}_{{B\; inci}}\,}{r}$$
(6)
In addition to the methods-of-moments approach, random effects regression models without predictors were used to estimate prevalence k at the survey-arm level and incidence k at the study year-arm level. For the prevalence k, the mean prevalence and between-cluster variance were estimated for each survey arm \(s\) using the following model:
$${x}_{{ijs}}={\alpha }_{s}+{\upsilon }_{{js}}+{e}_{{ijs}}$$
(7)
where \({x}_{{ijs}}\) is the observed malaria status (positive or negative) of ith individual in the jth cluster of survey arm \(s\). The term \({\alpha }_{s}\) denotes the overall mean prevalence in survey arm \(s\), while \({\upsilon }_{{js}}\) is the effect of the jth cluster on prevalence in survey arm \(s\), and \({e}_{{ijs}}\) is the individual-level variation. The cluster effects \({\upsilon }_{{js}}\) follow a normal distribution with a mean 0 variance \({\sigma }_{{B}_{s}}^{2}\). Prevalence k and corresponding 95% confidence intervals for each survey arm were calculated from model outputs as follows:
$${Prevalence}\,{\hat{k}}_{s}=\,\frac{{\hat{\sigma }}_{{B}_{s}}}{{\hat{\alpha }}_{s}}$$
(8)
Corresponding 95% confidence intervals for \({Prevalence}\,{\hat{k}}_{s} \) were calculated based on the model-derived variance and its standard error:
$$95\%{CI\; for\; Prevalence}\,{\hat{k}}_{s}=\frac{\sqrt{{\hat{\sigma }}_{{B}_{s}}^{2}\pm {Z}_{\alpha /2}\times {SE}({\hat{\sigma }}_{{B}_{s}}^{2})}}{{\hat{\alpha }}_{s}}$$
(9)
where \({Z}_{\alpha /2}\) is the critical value from the standard normal distribution.
Using the same model output components, we estimated the survey-arm prevalence ICC, which quantifies the proportion of total variance (i.e. between-cluster and within cluster variation) attributable to between-cluster variation:
$${Prevalence}\,{\widehat{{ICC}}}_{s}\,=\,\frac{{\hat{\sigma }}_{{B}_{s}}^{2}}{{\hat{\sigma }}_{{B}_{s}}^{2}+{\hat{\sigma }}_{{E}_{s}}^{2}}$$
(10)
where \({\hat{\sigma }}_{{E}_{S}}^{2}\) represents within-cluster (residual) variance derived from the individual-level error term \({e}_{{ijs}}\). Corresponding 95%CIs were obtained using the “estat icc” command in STATA (v. 18) according to:
$$95\%{CI\; for}\,{\widehat{{ICC}}}_{s}={\widehat{{ICC}}}_{s}\pm {Z}_{\alpha /2}\times {SE}({\widehat{{ICC}}}_{s})$$
(11)
where SE(\({\widehat{{ICC}}}_{s}\)) is the standard error of the ICC estimated via the delta method.
For the estimation of incidence k at the study year level \(s\), we used a Poisson regression model with cluster-level random effects and no predictors to estimate the overall study-year arm incidence and variance between clusters according to:
$${\lambda }_{{ijs}}={{exp}} \left({\alpha }_{s}\right)\times {v}_{{js}}$$
(12)
where \({\lambda }_{{ijs}}\) corresponds to the observed malaria status (positive or negative) of the ith individual in the jth cluster of study year arm \(s\). Parameter \({\hat{\alpha }}_{s}\) represents the overall mean incidence across all clusters in study year \(s\) and \({v}_{{js}}\) is the random effect of cluster j on incidence. The \({v}_{{js}}\) effects assume a gamma distribution with a mean of 1 and a variance of \({\hat{\alpha }}_{s}^{{\prime} }\). Based on this distribution, the standard deviation of lambda across clusters in study year arm \(s\) can be estimated according to:
$${SD}({\lambda }_{{js}})={{exp}} \left({\hat{\alpha }}_{s}\right)\times {SD}\left({v}_{{js}}\right)=\,{\hat{\alpha }}_{s}\,\times \,\sqrt{{\hat{\alpha }}_{s}^{{\prime} }}$$
(13)
and the incidence coefficient of variation (k) in each study year arm \(s\) is then estimated as:
$${Incidence}\,{\hat{k}}_{s}=\,\frac{{SD}({\lambda }_{{js}})}{{\hat{\alpha }}_{s}}=\,\frac{{\hat{\alpha }}_{s}\,\times \,\sqrt{{\hat{\alpha }}_{s}^{{\prime} }}}{{\hat{\alpha }}_{s}}=\,\sqrt{{\hat{\alpha }}_{s}^{{\prime} }}$$
(14)
The 95% confidence interval for incidence k was derived using the standard error of the variance parameter \({\hat{\alpha }}_{s}^{{\prime} }\) as follows:
$$95\%{CI\; for\; incidence}\,{\hat{k}}_{s}=\,\sqrt{{\hat{\alpha }}_{s}^{{\prime} }\pm \,{Z}_{\alpha /2}\times {SE}({\hat{\alpha }}_{s}^{{\prime} })}$$
(15)
STATA (v.18) do file code used estimate k is included in Supplementary data 1. STATA do file code used to estimate ICC is included in Supplementary data 2. Code is accompanied with simulated cluster-level prevalence data (Supplementary data 3) and incidence data (Supplementary data 4).
Data analysis
Using unmatched methods described in12,32, we calculated each trial’s predicted study power (%) according to original predictions of k and control-arm prevalence/incidence using the STATA command “clustersampsi” (v.18). Using empirical estimates of k and control-arm incidence and prevalence, we recalculated observed study power for each trial year and survey, respectively. For both predicted and observed power calculations, all additional parameters remained identical: significance level, cluster size, cluster numbers and desired effect size (anticipated % relative reduction between arms).
We further explored the impact of between-cluster heterogeneity on study power at the 5% significance level for a hypothetical trial with 20 clusters per arm, a cluster size of 50 and an assumed control prevalence of either 10%, 50% or 90%. Using varying k estimates (range: 0.3-1.5) and effect sizes (1-prevalence ratio, range: 0-1) we calculated corresponding study power (%) and sample size (required clusters per arm).
Using trial data, we investigated the impact of observed between-cluster heterogeneity on observed effect sizes between study arms during the intervention periods of trials. For each post-intervention prevalence survey and incidence year, we compared observed k estimates with observed cluster mean prevalence and rate ratios, respectively, along with corresponding 95% confidence intervals. Effect sizes were estimated as follows:
$${Effect\; size}=\,\frac{{\bar{T}}_{1}}{{\bar{T}}_{0}}$$
(16)
where \(\bar{T}\) represents the mean, cluster-level, estimate of prevalence or incidence in the intervention (1) arm and control (0) arm. To estimate corresponding 95%CIs, we multiplied and divided effect size estimates by t-distributed error factors estimated according to:
$${Error\; factor}=exp \left({t}_{v,0.025}\times \sqrt{V}\right)$$
(17)
where \(V\) represents the variance of the prevalence or rate ratios:
$$V=\frac{{S}_{1}^{2}}{{c}_{1}{\bar{T}}_{1}^{2}}+\frac{{S}_{0}^{2}}{{c}_{0}{\bar{T}}_{0}^{2}}$$
(18)
where \({S}^{2}\) corresponds to the within study arm variance and \(c\) signifies the number of clusters per arm.
In addition to arm-level effect sizes, we estimated cluster-level effect sizes for each post-intervention survey for prevalence and each trial year for incidence. Cluster-level prevalence ratios were estimated by dividing each intervention cluster prevalence values by the mean control-arm prevalence in the corresponding survey. Cluster-level incidence ratios were similarly estimated by dividing each intervention cluster incidence value by the mean incidence in the corresponding study year control arm.
Upon estimating k for each trial survey-arm, we investigated factors associated with elevated k. Random effects logistic regression models were used to generate odds ratios to estimate associations with elevated prevalence k surveys (k > 0.5). This threshold was chosen to dichotomise k as an estimate of 0.5 is considered conservative32 and can result in large numbers of clusters in sample size estimations. Explanatory variables included overall survey-arm prevalence (<10%, 10–40%, >40%), mean cluster size (<60, >60), clusters per arm (<15, >15), study arm (control, intervention), season (malaria, non-malaria) and diagnostic (PCR, RDT, microscopy)). These explanatory variables were chosen as they represent key design considerations and were available for all included trials. All models were fit using maximum likelihood and included trial-level random effects as trials had multiple surveys. A multivariate model, constructed in a forward stepwise manner according to superior model fit (LRT < 0.05), was additionally used to generate adjusted odds ratios associated with elevated k surveys to account for potential confounding by the above stated factors.
To further characterise the non-linear relationship between overall survey prevalence or study-year incidence and k, we conducted linear regression analyses on log-transformed values of k. In the prevalence models, the log-transformed k estimates at the survey-arm level served as the dependent variable, while overall survey-arm prevalence was the independent variable. For the incidence models, the dependent variable was the log-transformed study-year arm k estimate, with overall study-year incidence as the independent variable. Predicted k estimates were presented along with 95% confidence intervals (95% CIs) and 95% prediction intervals (95%PIs). The 95% CIs indicate the uncertainty around the linear prediction, whereas the 95%PIs capture the uncertainty around individual survey-arm or study-year arm observations. Data analyses were conducted in STATA version 18 (StataCorp, College Station, TX, USA).
Reporting summary
Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.
