Setting and study population
This study was conducted at Södersjukhuset (Stockholm South General Hospital), Sweden, which has one of the largest EDs in the Nordic countries. The study covers a period from May 1st, 2016, to April 30th, 2018. During these two years 229,195 adult visits were registered to the ED and out of these, 60,213 were admitted to the hospital.
Electronic health records
We used the EHR system TakeCare (TC) (used for hospital and outpatient records) as well as Clinisoft (CCC), a patient data management system used in ICUs. Both systems store and retrieve clinical information, including patient history, vital signs, laboratory results, drug administration, other treatments provided, and diagnosis codes.
Application development
A software engineer developed the application in the QlickView (QV) software (Qlik Technologies Inc, King of Prussia, PA, USA.), enabling QV to retrieve and store data from TC, as well as from CCC. Data was generated, coded, and analyzed from a population that consisted of all adult patient visits (≥ 18 years of age) to the ED. The patients were monitored for the initial 48-h period or until they were discharged or passed away, whichever occurred first.
To identify patients likely to have community-onset sepsis according to the Sepsis-3 criteria, the QV application was programmed to identify patients who (1) had blood cultures taken within 48 h from the arrival to the ED, (2) received two doses of antibiotics used for sepsis, and (3) obtained a SOFA score ≥ 2 during at least one 24 h window within 48 h from arrival to the ED. “Time zero”, i.e. the exact time when a patient was considered to have sepsis was defined as the first time a patient obtained a SOFA score ≥ 2. Antibiotics used for sepsis were defined by the recommendations of the Swedish Society of Infectious Diseases (SILF) for treating community onset sepsis in Sweden, namely benzylpenicillin, cloxacillin, piperacillin + tazobactam, cefotaxime, ceftazidime, meropenem, imipenem + cilastatin, tobramycin, gentamicin and amikacin16.
Two doses of antibiotics used for sepsis were required; the first should be administered within 48 h from the patients’ registration at the ED, aiming to identify community-acquired sepsis14. The second dose had to be administered within 18 h from the first dose. The time frame was selected to ensure the inclusion of patients on a twice-daily regimen, even if there were delays in administering the second dose, as all antibiotics recommended by SILF for sepsis treatment require at least two doses per day. Also, the time between prescription and administration may be delayed due to the nurses’ workload.
Patients reaching a SOFA score ≥ 2 for the first time later than 48 h from admission were not included since these were considered as having a hospital acquired sepsis15. The SOFA score’s respiratory component was calculated using the Ellis equation17, converting the peripheral capillary oxygen saturation (SpO2) to partial pressure of oxygen (PaO2) when PaO2 from arterial blood was not available. GCS was not used as a SOFA score criterion for intubated (and thus sedated) patients admitted to the ICU. Urine output was not used as a criterion due to a lack of data. If a SOFA score component was lacking, the value was considered to be zero.
The application did not incorporate the calculation of the patient’s baseline SOFA score. This decision was made due to the potential risk of missing information from previous hospital contacts and the possibility of patients being admitted to other hospitals that do not utilize TC or CCC as their EHR systems.
Validation process
The QV software application was used for the entire cohort of the 60,213 consecutive visits to the emergency department that led to hospitalization. For validation of the application with medical record review, a stratified random sampling was made from three groups18. Stratification was made to ensure a sufficient number of patients from the relatively small groups A and B. From group A, consisting of 7027 ED visits considered to have sepsis according to the application, (i.e. blood cultures taken, antibiotics administered and a SOFA score of ≥ 2) a random sample of 140 visits was selected. From group B, consisting of 1 958 visits that had a suspicion of a serious infection but not sepsis according to the application (i.e. blood cultures taken, antibiotics administered but a SOFA score of < 2) a random sample of 143 visits was selected. Finally, from group C, consisting of 51 228 visits with low likelihood of sepsis or serious infection (i.e. not fulfilling the criteria for groups A and B) a random sample of 143 visits was selected. Thus, a total of 426 ED visits were selected by stratified random sampling from the total cohort and included for medical record review (Fig. 1).
Data selection and categorization for Groups A, B, and C.
Three emergency physicians, one licensed physician from the UK, and one ICU physician were tasked with reviewing whether patients met the Sepsis-3 clinical criteria. They conducted this assessment independently.
The application’s accuracy in detecting community-onset sepsis was compared to manual examination of the EHR.
The reviewers manually calculated baseline SOFA scores. For each component of the SOFA score, the baseline was determined as the best value measured within 90 days prior to arrival at the ED. If the value was unknown, it was assumed to be zero. The extent of missing data for the various SOFA components is detailed in Table 2. The manual examination revealed only one discrepancy between the missing data from the script and the missing data in the manual examination. In group A, the application identified five patients as having fewer points in the respiration SOFA compared to the manual examination, and the same occurred with one patient in group B. This discrepancy arose because the implementation of oxygen in some patients was only recorded manually by physicians or nurses, which the application could not detect in free text. However, in the manual examination, it was observed that all the six patients had a higher SOFA point when the manually recorded implementation of oxygen was added. The rest of the missing data did not present any discrepancy between the reviewers and the application.
Admissions that met the Sepsis-3 clinical criteria of having an increase in SOFA score of ≥ 2, as determined by physician review, were further evaluated for the likelihood of infection. For this purpose, the reviewers scrutinized clinical, microbiological and radiological findings of each patient. The episodes were classified into four categories: no infection, possible infection, probable infection, and definite infection, in accordance with the definitions outlined by Klouwenberg et al.19. Additionally, infections from an unknown source were included based on criteria established by Valik et al., defined as symptoms indicating an infection according to the attending physician, and the patient receiving a full course of anti-infective treatment, but no source could be determined. This category could only be classified as a possible infection. To be considered a true sepsis case in the assessment of the application’s sensitivity and specificity, patients had to meet Sepsis-3 clinical criteria (increase in SOFA score and the presence of possible, probable, or definite infection)7.
Statistical analyses
Weighted estimates of sensitivity, specificity, Positive Predicted Value, Negative Predictive Value, and Youden’s index were used to account for bias caused by the stratified sampling scheme.
Confidence intervals with a 95% confidence level were calculated from 10 000 bootstrap samples18. Furthermore, the positive and negative likelihood ratios were calculated. Data handling and statistical analyses were conducted using R 4.1.1.
Large language models (LLM), ChatGPT
A large language model, ChatGPT, was used solely for the purpose of correcting the English language in some parts of this manuscript; no content or images were created by the LLM.
