The N3C data enclave included 5.6 million COVID-19 positive individuals across 75 centers at the time of analysis. Of those, 493,940 were hospitalized in a range of 3 days prior to and 10 days after test positivity (Fig. 1).
Cohort construction within the N3C data enclave, inclusion and exclusion criteria, and resultant sample size.
A total of 442,351 of these individuals were aged 18 or older at the time of diagnosis. Patients on invasive mechanical ventilation or extracorporeal membrane oxygenation prior to admission were excluded (7,545 and 549, respectively) as data from transferring hospitals is not available in the data enclave. After selecting for patients who received dexamethasone within 24 h of admission, 61,013 patients remained. A total of 48,595 patients remained hospitalized for 3 or more days. Within the cohort of hospitalized patients, 9,708 (20%) had complete data for risk variables and were included in analyses. Randomization of the 9,708 participants in the analytic cohort in a 3:1 ratio yielded a training cohort (n = 7,281) and testing cohort (n = 2,427) for the model. The demographic features, risk variables, and hospital characteristics were similar in the randomly partitioned training and testing cohorts (Table 1).
The mean age of patients was 62.3 (standard deviation 15.8), and 43% were female. Patients represented diverse racial and ethnic backgrounds but were predominantly white and non-Hispanic (68% and 76%, respectively). Mortality at day 28 in this cohort of patients with COVID-19 hospitalized with hypoxemia was 15.4%. Comorbidities associated with COVID-19 morbidity were well represented in the cohort with diabetes in 46% of patients, chronic obstructive pulmonary disease in 19%, coronary artery disease in 9%, chronic kidney disease in 25%, and metastatic cancer in 3% of patients.
The 28-day mortality in the training cohort was 15.4% (1,123 deaths). Bivariable analysis with rank-sum testing of age, day 3 CRP, day 3 neutrophils, day 3 lymphocytes, and day 3 neutrophil to lymphocyte ratio demonstrated statistically significant differences in all variables between participants alive and deceased at day 28 with p-values of < 0.001 for all comparisons (Table 2).
This analysis indicates greater age (71 vs. 61 years), higher day 3 CRP (57.3 vs. 30.4 mg/L), and higher day 3 neutrophil to lymphocyte ratio (13.6 vs. 6.8) in those deceased at day 28. Logistic regression on the continuous variables of age, day 3 CRP, and day 3 neutrophil to lymphocyte ratio for prediction of day 28 mortality showed statistically significant odds ratios for all components of the regression model (S1 in the online data supplement) with a p-values < 0.001 for all odds ratios. The odds ratio for mortality for every 10-year increase in age was 1.51 (1.44–1.59), 10 unit increase in day 3 neutrophil to lymphocyte ratio was 1.32 (1.25–1.40), and 25 unit increase in day 3 CRP 1.22 (1.19–1.25).
The regression model from the training cohort was then applied to the testing cohort for validation. The area under the receiver-operator characteristic curve was 0.77 (95% CI 0.74–0.79). This model is a statistically significant improvement over individual variables that contribute to the model (age, day 3 neutrophil to lymphocyte ratio, and day 3 CRP), with p-values < 0.01 for all comparisons (Fig. 2).
ARC model performance in reserved validation (test) cohort.
The Hosmer-Lemeshow goodness-of-fit statistic showed a non-statistically significant p-value of 0.14 for 4 groups, indicating no evidence of poor model fit. A proposed dichotomization of variables for age of ≥ 70, CRP ≥ 70 mg/L, and neutrophil to lymphocyte ratio ≥ 10 allows for a more rapid application of the prognostic score at the bedside. The inflection points in the relationship between the age and laboratory values and mortality at day 28 in the training data set determined these cut-points. This dichotomization of risk variables assigns 1 point for each value above the cut point and determines the ARC score totaling the number of points an individual accumulates. This dichotomization does not significantly change the AUC of the model in the test set compared to continuous variables (p = 0.10) and simplifies score for bedside deployment (S2). Mortality by ARC score for the training and test cohorts is displayed in Table 3. In the testing cohort, increasing ARC score was associated with increasing 28-day mortality, as follows: 0, 3.9% mortality; 1, 14.7% mortality; 2, 29.2% mortality; 3, 48.9% mortality.
A sensitivity analysis with the participants excluded for incomplete day 3 laboratory data compared demographic, comorbid, and hospitalization specific variables to participants included in the cohorts. Although several variables were statistically different, clinically meaningful differences (e.g. 28-day mortality) were not observed between the analytic cohort and those excluded for missing variables (S3 in the online data supplement). Time-based analysis for broad classifications of variants of concern (Alpha, Delta, and Omicron) was determined using data from the Center for Disease Control and Prevention’s genomic surveillance dataset established through the National SARS-CoV-2 Strain Surveillance program and sequencing by commercial and academic laboratories. Strain identification was assigned if a single circulating strain accounted for greater than 90% of the samples sequenced in the CDC genomic surveillance set at the time of the positive test. The established timeframes for the Alpha, Delta, and Omicron variants were January 1, 2020, to May 29, 2021; July 23, 2021, to November 22, 2021; and January 8, 2022, onward, respectively. SARS-CoV-2 variant-specific analysis resulted in a statistically significantly decreased performance of the model for the Delta strain but not for the widely circulating Omicron strain, using Alpha as the reference (Fig. 3).
ARC model performance in Alpha, Delta, and Omicron variants.
The area under the ROC remains robust (0.78 for Alpha, 0.73 for Delta, and 0.77 for Omicron), suggesting good predictive value across the evolution of viral variants, vaccination, and inpatient treatment paradigms. The model performed less well in a sub-group analysis of patients receiving dexamethasone plus an additional immune modulator therapeutic (tocilizumab or baricitinib) with an area under the ROC of 0.72 but continued to outperform any of the individual elements in the score in this group of 1,637 individuals (S4 in the online data supplement). Introducing regression splines with a single knot to the neutrophil-to-lymphocyte ratio and CRP variables to account for inflection points in the relationship between these variables and mortality did not improve the performance of the model (data not shown).
Equity subgroup analysis was performed on the training dataset, evaluating the performance of the ARC model across sex, race, and ethnicity (S5). There were no statistically significant differences in model performance across sex with an area under the ROC of 0.77 (0.73–0.80) amongst males and 0.77 (0.73–0.81) amonst females (p = 0.94). Additionally, there were no statistically significant differences in model performance between those identified as white (area under the ROC of 0.76 [0.73–0.79]) and those identified as black or African American (0.72 [0.64–0.79]), (p = 0.37). Pairwise comparisons with other racial identities were not performed because of low sample sizes. Finally, there were no statistically significant differences in model performance across ethnicity with an area under the ROC of 0.76 (0.73–0.79) amongst non-Hispanic individuals and 0.82 (0.76–0.88) amonst Hispanic individuals (p = 0.07). Smoothed regression of continuous variables included in the model against mortality was performed in order to ensure a monotonic relationship between the variables and outcomes (S6).
