HELP computerized computerized decision support system

The HELP CDSS assigns patients to one of two arms: one for CoNS bacteremia (including Staphylococcus intermedius) or one for S. aureus bacteremia (including Staphylococcus lugdunensis). In the CoNS arm, the CDSS checks the number of independent blood cultures. If only one positive blood culture is available without evidence of catheter infection by the same pathogen (clinical judgment by the physician in charge), the CDSS recommends a follow-up blood culture to rule out contamination before starting antibiotics. If two separate blood cultures yield positive results for the same CoNS species with matching antibiotic susceptibility profiles, the CDSS suggests investigating a possible source of infection and considering antibiotic therapy. Blood cultures were considered separate if they were collected from different sites and at different times (2 h to 5 days). A negative follow-up blood culture suggested contamination, prompting a reevaluation of antibiotic therapy if it was already initiated.

The HELP CDSS for S. aureus bacteremia is based on treatment recommendations published by Hagel et al.²⁰. It was developed by a panel of IDS and medical microbiologists in collaboration with medical informaticians from all study sites. A simplified schematic representation is shown in Fig. 5.

Rough overview of the decision algorithm that was the basis for the HELP CDSS.

For SAB, the CDSS distinguishes between uncomplicated and complicated cases based on criteria such as positive follow-up blood cultures, permanent foreign bodies, prolonged fever, or vasopressor use. Treatment of complex SAB exceeds the capabilities of the CDSS, so immediate IDS consultation is advised. This also applies to polymicrobial bacteremia when more than one pathogen is detected in addition to staphylococci. For uncomplicated SAB treatment, guidance is offered, including source control, diagnostic imaging, and appropriate antibiotics.

Except for complicated SAB episodes, the CDSS can recommend treatment options independently of an IDS consultation. However, all hospitals maintained an IDS consultation service as part of their SOC, which remained operational throughout the treatment phase. Thus, the main difference between the two phases was that the CDSS provided supplementary information and acted as a bridge when immediate IDS consultation was not available. The CDSS was originally designed for seamless integration with hospital clinical information systems, but the development of this system faced challenges due to the European Medical Device Regulation (MDR)²¹, which was introduced in May 2021 and classified the original CDSS as a medical device requiring a highly complex and time-consuming certification process that was not compatible with the schedule. To ensure viability within our trial setting, the CDSS was transformed into a user input-reliant version that avoids automated patient-specific conclusions, functioning as an interactive decision tree accessible on smartphones, desktops, or laptops; an archived version of the CDSS is available online²².

Study design

The HELP trial was designed as a stepped-wedge cluster randomized trial (SW-CRT)²³ conducted at five German university hospitals: Jena, Leipzig, Aachen, Halle, and Essen. Following the methods outlined by Hussey and Hughes^24, we calculated a sample size of 135 wards enrolling at least 2700 patients based on a 90-day mortality of 30% among SAB patients, as reported by Mejer et al.²⁵. For details, see the published trial protocol²⁶. However, only 134 wards met the inclusion criteria described later and were included at randomization. All 134 wards started in the control phase (SOC) and transitioned to the treatment phase (HELP CDSS application) in a stepwise manner (Supplementary Fig. 1). The timing of crossover was determined by randomization, which was stratified by hospital and ward type (critical care units vs. general wards). Randomization lists generated using R²⁷ were integrated into the technical infrastructure deploying the CDSS. This ensured that only physicians in the intervention wards could access the CDSS. There were nine randomization steps, with 2-month intervals for Jena, Leipzig, and Aachen and 1.5-month intervals for Halle and Essen due to legal difficulties in the pilot phase that necessitated a later start and thus a shorter overall data collection period. The trial adhered to Good Clinical Practice (where applicable) and the Declaration of Helsinki²⁸, with approval from Jena University’s Ethics Committee (2018-1264_3-BO) and the respective center-specific ethics committees (Aachen University Ethics Committee, Halle University Ethics Committee, Essen University Ethics Committee, Leipzig University Ethics Committee.) The trial was registered at the German clinical trials register (www.drks.de/DRKS00014320) on June 6th 2019.

Recruitment of wards and patients and CDSS rollout

We included all sufficiently technically equipped wards at each site, excluding maternity, psychiatry, and pediatric units. Physicians in intervention wards were informed about the impending CDSS rollout and received microbiology reports containing references to the CDSS, while wards in the control phase did not have CDSS access. We included all adult patients with a positive blood culture for S. aureus/S. lugdunensis or CoNS, except for CoNS patients who passed away within 72 h of the initial positive blood culture. Patients were included as new patients if they were discharged and then readmitted to the hospital or met the inclusion criteria more than 30 days after their initial inclusion. Study nurses monitored blood cultures and completed an electronic case report form (eCRF) for each HELP patient. Routine documentation collection did not require informed consent, except for SAB patients, who were scheduled for a 90-day telephone follow-up, with prior notification and the option to decline participation.

Outcomes

The coprimary outcomes were hospital mortality for all patients, 90-day mortality/relapse for SAB patients and cumulative vancomycin use in milligrams for CoNS patients. Relapse was defined as the recurrence of S. aureus bacteremia or the occurrence of any related secondary complication within 90 days of the onset of initial SAB. We chose a pragmatic definition of relapse because without genotyping, it is difficult to ascertain whether a second infection is due to the same strain or infectious focus (i.e., relapse) or whether it is a reinfection with a different strain. Most S. aureus reinfections within 90 days are due to the same strain, suggesting the relapse of endogenous infection^29,30. Secondary outcomes included the occurrence of acute renal dysfunction according to the Kidney Disease: Improving Global Outcomes (KDIGO)³¹ criteria, transesophageal echocardiography (TEE) usage and the use of seven additional antibiotics (Supplementary Table 1).

Data collection

We implemented hybrid data collection, combining documentation in eCRFs with electronic health record (EHR) data collected in the five participating DIC. To enable secondary data use, DIC integrate EHR data from different clinical information systems and transform them into Fast Healthcare Interoperability Resources (HL7® FHIR®). This approach allowed the HELP trial to blend the characteristics of traditional randomized controlled trials using data from eCRFs with the practicality of being entirely embedded in the clinical routine. While this setup places restrictions on data availability and quality, it is much closer to a real-world clinical scenario. In addition, this approach has the potential to reduce the documentation burden associated with traditional trials, offering a more pragmatic study design within a learning healthcare system.

To identify patients with positive Staphylococcus blood cultures from participating wards in the DIC data, we sourced admission, discharge, and transfer (ADT) data and integrated them with data from the laboratory information system (LIS). Subsequently, all other data needed for downstream statistical analyses were extracted. To protect patient privacy, we used a distributed analysis approach in which R scripts using the fhircrackr³² package were sent to the DIC for preprocessing and local data aggregation, followed by central statistical analysis in Jena. Figure 6 shows the extraction process in more detail: we provided R code via GitLab to each of the five Data Integration Centers (DIC), tasked with extracting FHIR data from the FHIR server and converting it into tabular format. To ensure patient privacy, all identifying data were removed, and information relevant to clinical descriptions was aggregated. These anonymized data, depicted in the upper part of the figure, were subsequently sent to Jena. The creation of individual R scripts for each site necessitated extensive collaboration between the data analyst and the DIC employees, as outlined in the lower part of the figure. This process included verifying the executability of R scripts in the local IT environment, ensuring the availability and plausibility of FHIR data, and mapping relevant microbiology reports from the Laboratory Information System (LIS) to the associated electronic Case Report Form (eCRF) and admission, discharge, and transfer (ADT) data. The patient cohort was defined by selecting individuals with positive Staphylococcus blood cultures who were admitted to a HELP study ward when the first preliminary microbiology report was issued. Subsequently, data for this patient cohort were extracted from the FHIR server, checked for plausibility, and transmitted to Jena. Each of these steps required multiple iterations to ensure accuracy and functionality. After the study, we conducted an online survey incorporating the “System Usability Scale“³³ among participating physicians to assess HELP CDSS usage and user satisfaction (see the Supplementary Materials).

Starting with FHIR data available in each DIC, each step of the data extraction required iterative adaptions until the local analysis result was ready to be sent to the central analysis site.

Statistical analysis

The coprimary outcomes were tested in a hierarchical, confirmatory fashion, meaning that each hypothesis test could only be interpreted as confirmatory if the null hypothesis for the preceding endpoint was rejected. For mortality outcomes, we used a noninferiority hypothesis with a 5% margin, indicating that our expectation was not to exceed a 5% increase in mortality in the CDSS vs. SOC phase. We employed binomial generalized linear mixed models (GLMMs) to estimate death/relapse probabilities and two-sided 90% confidence intervals (CIs). Regarding vancomycin use, our superiority hypothesis anticipated lower antibiotic usage in the CDSS phase. Extending the protocol descriptions, we used a two-part regression model. This included a binomial GLMM for vancomycin administration probability and a lognormal GLMM for cumulative dosage per patient. Analysis of the secondary outcomes followed the same modeling approach. In all the models, “treatment” (HELPS-CDSS vs. SOC) and “time since study initiation” were included as fixed effects; “ward” (cluster) was included as a random effect. To address potential site effects, we performed sensitivity analyses by adding “site” as a fixed effect in all the models and “coronavirus disease (COVID-19) diagnosis” as a fixed effect in all the mortality outcome models. Complying with privacy protection rules, we aggregated the variables used for the description of additional patient characteristics locally at each site and combined them using the meta-analytic random effects inverse variance model. All the statistical analyses were carried out with R version 4.2.1 using the R packages lme4³⁴, marginaleffects³⁵ and meta³⁶.