Chua, H. et al. The use of test-negative controls to monitor vaccine effectiveness: a systematic review of methodology. Epidemiology 31, 43 (2020).
Doll, M. K., Pettigrew, S. M., Ma, J. & Verma, A. Effects of confounding bias in coronavirus disease 2019 (COVID-19) and influenza vaccine effectiveness test-negative designs due to correlated influenza and COVID-19 vaccination behaviors. Clin. Infect. Dis. 75, e564–e571 (2022).
Crowcroft, N. S. & Klein, N. P. A framework for research on vaccine effectiveness. Vaccine 36, 7286–7293 (2018).
Hungerford, D. et al. Mitigating bias in observational vaccine effectiveness studies using simulated comparator populations: Application to rotavirus vaccination in the UK. Vaccine 36, 6674–6682 (2018).
Li, K. Q., Shi, X., Miao, W. & Tchetgen, E. T. Double negative control inference in test-negative design studies of vaccine effectiveness. J. Am. Stat. Assoc. 119, 1859–1870 (2024).
Westreich, D. & Hudgens, M. G. Invited commentary: beware the test-negative design. Am. J. Epidemiol. 184, 354–356 (2016).
Sullivan, S. G., Tchetgen Tchetgen, E. J. & Cowling, B. J. Theoretical basis of the test-negative study design for assessment of influenza vaccine effectiveness. Am. J. Epidemiol. 184, 345–353 (2016).
Lipsitch, M., Jha, A. & Simonsen, L. Observational studies and the difficult quest for causality: lessons from vaccine effectiveness and impact studies. Int. J. Epidemiol. 45, 2060–2074 (2016).
Wratil, P. R. et al. Vaccine-hesitant individuals accumulate additional COVID-19 risk due to divergent perception and behaviors related to SARS-CoV-2 testing: a population-based, cross-sectional study. Infection 51, 909–919 (2023).
Glasziou, P. et al. Testing behaviour may bias observational studies of vaccine effectiveness. J. Assoc. Med. Microbiol. Infect. Dis. Can. 7, 242–246 (2022).
Simpson, C. R., Ritchie, L. D., Robertson, C., Sheikh, A. & McMenamin, J. Effectiveness of H1N1 vaccine for the prevention of pandemic influenza in Scotland, UK: a retrospective observational cohort study. Lancet Infect. Dis. 12, 696–702 (2012).
Huang, Z. et al. Effectiveness of inactivated COVID-19 vaccines among older adults in Shanghai: retrospective cohort study. Nat. Commun. 14, 2009 (2023).
Rennert, L., Ma, Z., McMahan, C. S. & Dean, D. Covid-19 vaccine effectiveness against general SARS-CoV-2 infection from the omicron variant: A retrospective cohort study. PLoS Glob. Public Health 3, e0001111 (2023).
Ono, S. et al. Comparative effectiveness of BNT162b2 and mRNA-1273 booster dose after BNT162b2 primary vaccination against the Omicron variants: a retrospective cohort study using large-scale population-based registries in Japan. Clin. Infect. Dis. 76, 18–24 (2023).
Wolff Sagy, Y. et al. Real-world effectiveness of a single dose of mpox vaccine in males. Nat. Med. 29, 748–752 (2023).
Cerqueira-Silva, T. et al. Duration of protection of CoronaVac plus heterologous BNT162b2 booster in the Omicron period in Brazil. Nat. Commun. 13, 4154 (2022).
Tseng, H. F. et al. Effectiveness of mRNA-1273 vaccination against SARS-CoV-2 omicron subvariants BA.1, BA.2, BA.2.12.1, BA.4, and BA.5. Nat. Commun. 14, 189 (2023).
van Doorn, E. et al. Influenza vaccine effectiveness estimates in the Dutch population from 2003 to 2014: the test-negative design case-control study with different control groups. Vaccine 35, 2831–2839 (2017).
Buchan, S. A. et al. Estimated effectiveness of COVID-19 vaccines against Omicron or Delta symptomatic infection and severe outcomes. JAMA Netw. Open 5, e2232760 (2022).
Jackson, M. L. et al. Influenza vaccine effectiveness in the United States during the 2015–2016 season. N. Engl. J. Med. 377, 534–543 (2017).
Ostropolets, A. & Hripcsak, G. COVID-19 vaccination effectiveness rates by week and sources of bias: a retrospective cohort study. BMJ Open 12, e061126 (2022).
Ioannidis, J. P. A. Factors influencing estimated effectiveness of COVID-19 vaccines in non-randomised studies. BMJ Evid. Based Med. 27, 324–329 (2022).
Fukushima, W. & Hirota, Y. Basic principles of test-negative design in evaluating influenza vaccine effectiveness. Vaccine 35, 4796–4800 (2017).
Jackson, M. L. & Nelson, J. C. The test-negative design for estimating influenza vaccine effectiveness. Vaccine 31, 2165–2168 (2013).
Lewnard, J. A., Tedijanto, C., Cowling, B. J. & Lipsitch, M. Measurement of vaccine direct effects under the test-negative design. Am. J. Epidemiol. 187, 2686–2697 (2018).
Andrews, N. et al. Covid-19 vaccine effectiveness against the Omicron (B.1.1.529) variant. N. Engl. J. Med. 386, 1532–1546 (2022).
Chemaitelly, H. et al. Duration of mRNA vaccine protection against SARS-CoV-2 Omicron BA.1 and BA.2 subvariants in Qatar. Nat. Commun. 13, 3082 (2022).
Suah, J. L., Bar-Zeev, N. & Knoll, M. D. How important are study designs? A simulation assessment of vaccine effectiveness estimation bias with time-varying vaccine coverage, and heterogeneous testing and baseline attack rates. Preprint at medRxiv https://doi.org/10.1101/2022.08.25.22279235 (2022)
Campbell, F. et al. Increased transmissibility and global spread of SARS-CoV-2 variants of concern as at June 2021. Eurosurveillance 26, 2100509 (2021).
Jalali, N. et al. Increased household transmission and immune escape of the SARS-CoV-2 Omicron compared to Delta variants. Nat. Commun. 13, 5706 (2022).
Halloran, M. E., Haber, M. & Longini, I. M. Interpretation and estimation of vaccine efficacy under heterogeneity. Am. J. Epidemiol. 136, 328–343 (1992).
Markov, P. V. et al. The evolution of SARS-CoV-2. Nat. Rev. Microbiol. 21, 361–379 (2023).
Wu, J., Scarabel, F., McCarthy, Z., Xiao, Y. & Ogden, N. H. A window of opportunity for intensifying testing and tracing efforts to prevent new COVID-19 outbreaks due to more transmissible variants. Can. Commun. Rep. 47, 329–338 (2021).
Gabriele-Rivet, V. et al. Modelling the impact of age-stratified public health measures on SARS-CoV-2 transmission in Canada. R. Soc. Open Sci. 8, 210834 (2021).
Cui, Q. et al. Dynamic variations in COVID-19 with the SARS-CoV-2 Omicron variant in Kazakhstan and Pakistan. Infect. Dis. Poverty 12, 18 (2023).
Yuan, P. et al. Assessing the mechanism of citywide test-trace-isolate Zero-COVID policy and exit strategy of COVID-19 pandemic. Infect. Dis. Poverty 11, 104 (2022).
Bodner, K., Knight, J., Hamilton, M. A. & Mishra, S. Testing whether higher contact among the vaccinated can be a mechanism for observed negative vaccine effectiveness. Am. J. Epidemiol. 192, 1335–1340 (2023).
Lyngse, F. P. et al. Household transmission of the SARS-CoV-2 Omicron variant in Denmark. Nat. Commun. 13, 5573 (2022).
Park, A. W. et al. Quantifying the impact of immune escape on transmission dynamics of influenza. Science 326, 726–728 (2009).
Moriyama, M., Hugentobler, W. J. & Iwasaki, A. Seasonality of respiratory viral infections. Annu. Rev. Virol. 7, 83–101 (2020).
Townsend, J. P. et al. Seasonality of endemic COVID-19. mBio 14, e01426-23 (2023).
Koltai, M. et al. Determinants of RSV epidemiology following suppression through pandemic contact restrictions. Epidemics 40, 100614 (2022).
Gail, M. H. et al. Design choices for observational studies of the effect of exposure on disease incidence. BMJ Open 9, e031031 (2019).
Lewnard, J. A. et al. Theoretical framework for retrospective studies of the effectiveness of SARS-CoV-2 vaccines. Epidemiol. Camb. Mass 32, 508–517 (2021).
Lash, T. L. et al. Good practices for quantitative bias analysis. Int. J. Epidemiol. 43, 1969–1985 (2014).
Bodner, K., Irvine, M. A., Kwong, J. C. & Mishra, S. Observed negative vaccine effectiveness could be the canary in the coal mine for biases in observational COVID-19 studies. Int. J. Infect. Dis. 131, 111–114 (2023).
Freedman, A. S. et al. Inferring COVID-19 testing and vaccination behavior from New Jersey testing data. Proc. Natl Acad. Sci. USA 121, e2314357121 (2024).
Nordström, P., Ballin, M. & Nordström, A. Risk of SARS-CoV-2 reinfection and COVID-19 hospitalisation in individuals with natural and hybrid immunity: a retrospective, total population cohort study in Sweden. Lancet Infect. Dis. 22, 781–790 (2022).
Pilz, S., Theiler-Schwetz, V., Trummer, C., Krause, R. & Ioannidis, J. P. A. SARS-CoV-2 reinfections: overview of efficacy and duration of natural and hybrid immunity. Environ. Res. 209, 112911 (2022).
Tsang, T. K. et al. Prior infections and effectiveness of SARS-CoV-2 vaccine in test-negative studies: a systematic review and meta-analysis. Am. J. Epidemiol. https://doi.org/10.1093/aje/kwae142 (2024).
Al Kaabi, N. et al. Effect of 2 inactivated SARS-CoV-2 vaccines on symptomatic COVID-19 infection in adults: a randomized clinical trial. JAMA 326, 35–45 (2021).
Zhu, X. et al. Dynamics of inflammatory responses after SARS-CoV-2 infection by vaccination status in the USA: a prospective cohort study. Lancet Microbe 4, e692–e703 (2023).
Mohr, N. M. et al. Presence of symptoms 6 weeks after COVID-19 among vaccinated and unvaccinated US healthcare personnel: a prospective cohort study. BMJ Open 13, e063141 (2023).
Kuitunen, I., Uimonen, M., Seppälä, S. J. & Ponkilainen, V. T. COVID‐19 vaccination status and testing rates in Finland—a potential cause for bias in observational vaccine effectiveness analysis. Influenza Other Respir. Viruses 16, 842–845 (2022).
Shim, E. & Galvani, A. P. Distinguishing vaccine efficacy and effectiveness. Vaccine 30, 6700–6705 (2012).
Kahn, R., Schrag, S. J., Verani, J. R. & Lipsitch, M. Identifying and alleviating bias due to differential depletion of susceptible people in postmarketing evaluations of COVID-19 vaccines. Am. J. Epidemiol. 191, 800–811 (2022).
Public Health Agency of Canada. COVID-19 signs, symptoms and severity of disease: a clinician guide. https://www.canada.ca/en/public-health/services/diseases/2019-novel-coronavirus-infection/guidance-documents/signs-symptoms-severity.html (2022).
Venkatramanan, S. et al. Using data-driven agent-based models for forecasting emerging infectious diseases. Epidemics 22, 43–49 (2018).
Erdös, P. & Rényi, A. On random graphs I. Publ. Math. Debr. 6, 290–297 (1959).
Liu, C. et al. Rapid review of social contact patterns during the COVID-19 pandemic. Epidemiol. Camb. Mass 32, 781–791 (2021).
Durrett, R. Random Graph Dynamics (Cambridge Univ. Press, Cambridge, U.K., 2007).
National Health Interview Survey. QuickStats: percentage of persons who had a cold in the past 2 weeks, by age group and calendar quarter — National Health Interview Survey, United States. MMWR Morb. Mortal. Wkly. Rep. 69, 429 (2020).
Centers for Disease Control and Prevention. Suffering from a cold? Centers for Disease Control and Prevention – Treatment for Common Illnesses https://www.cdc.gov/antibiotic-use/colds.html (2023).
HealthLink BC. Facts about influenza (the flu). https://www.healthlinkbc.ca/healthlinkbc-files/facts-about-influenza-flu (2022).
Jeter, L. COVID-19 vaccines rollout nationwide: December 2021. National Association of Attorneys General https://www.naag.org/attorney-general-journal/current-status-of-covid-19-vaccines-rollout-nationwide-december-2021-update-for-the-attorney-general-community/ (2021).
Zeng, B., Gao, L., Zhou, Q., Yu, K. & Sun, F. Effectiveness of COVID-19 vaccines against SARS-CoV-2 variants of concern: a systematic review and meta-analysis. BMC Med. 20, 200 (2022).
Menegale, F. et al. Evaluation of waning of SARS-CoV-2 vaccine–induced immunity: a systematic review and meta-analysis. JAMA Netw. Open 6, e2310650 (2023).
Braeye, T. et al. Vaccine effectiveness against transmission of alpha, delta and omicron SARS-COV-2-infection, Belgian contact tracing, 2021–2022. Vaccine 41, 3292–3300 (2023).
Endo, A., Funk, S. & Kucharski, A. J. Bias correction methods for test-negative designs in the presence of misclassification. Epidemiol. Infect. 148, e216 (2020).
R Core Team. R: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, 2023).
Loken, C. et al. SciNet: lessons learned from building a power-efficient top-20 system and data centre. J. Phys. Conf. Ser. 256, 012026 (2010).
Ponce, M. et al. Deploying a top-100 supercomputer for large parallel workloads: the Niagara Supercomputer. In Proc. Practice and Experience in Advanced Research Computing on Rise of the Machines (Learning) 1–8 (Association for Computing Machinery, New York, NY, USA, 2019).
Bodner, K. et al. Code and simulated data for “Impact of unequal testing on vaccine effectiveness estimates across two study designs: a simulation study”. GitHub https://github.com/mishra-lab/testing_bias/releases/tag/v1.0.0 (2025).
Guo, Z. et al. Comparing the incubation period, serial interval, and infectiousness profile between SARS-CoV-2 Omicron and Delta variants. J. Med. Virol. 95, e28648 (2023).
Li, R. et al. Clinical characteristics and risk factors analysis of viral shedding time in mildly symptomatic and asymptomatic patients with SARS-CoV-2 Omicron variant infection in Shanghai. Front. Public Health 10, 1073387 (2022).
Sah, P. et al. Asymptomatic SARS-CoV-2 infection: a systematic review and meta-analysis. Proc. Natl Acad. Sci. USA 118, e2109229118 (2021).
