PURPOSE: The U.S. Food and Drug Administration (FDA) defined disease severity criteria to assist clinical development of medical products for management of COVID-19. These definitions were translated to code-based algorithms for use in real-world data. We validated the algorithms' performance in ambulatory settings at three regional integrated healthcare delivery systems contributing data to FDA's Sentinel System.
METHODS: We identified cohorts of individuals ≥  18 years that met the algorithms' criteria for mild, moderate, and severe COVID-19 at incident COVID-19 diagnosis or positive SARS-CoV-2 test, and separately, at incident COVID-19 treatment, from January 2022 through April 2023. We validated the algorithms via chart review of a random sample to calculate positive predictive values (PPVs) and 95% CIs.
RESULTS: The algorithms identified 33 071 patients at COVID-19 diagnosis or positive test; 26 985 mild (49 chart reviewed), 5180 moderate (55 reviewed), and 906 severe (56 reviewed). A total of 4512 patients were identified at COVID-19 treatment; 3474 mild (56 reviewed), 848 moderate (60 reviewed), and 190 severe (46 reviewed). The PPVs (1) at COVID-19 diagnosis or positive test were: mild 57% (95% CI: 43%-71%), moderate 58% (95% CI: 45%-71%), and severe 54% (95% CI: 41%-67%), and (2) at COVID-19 treatment: mild 57% (95% CI: 44%-70%), moderate 70% (95% CI: 58%-82%) and severe 72% (95% CI: 59%-85%).
CONCLUSION: The algorithms had low-to-moderate performance in classifying COVID-19 severity in ambulatory settings, depending on assessment at diagnosis or treatment. Researchers should consider the performance of the algorithm when using real-world data to assess COVID-19 severity.
Real‐world data encompasses information from health insurance billing and electronic medical records and is routinely used to study the safety and effectiveness of drugs and vaccines. Combinations of information in these data (such as diagnoses, lab tests, and medications)—or “algorithms”—are used to define medical conditions. The U.S. Food and Drug Administration developed algorithms to classify patients within COVID‐19 severity levels, from mild symptoms (like cough) to more severe disease (like requiring oxygen). This chart validation study examined whether these algorithms correctly classified COVID‐19 severity at the time of diagnosis or treatment for COVID‐19. From January 2022 through April 2023, we identified > 37 000 patients who appeared to have mild, moderate, or severe COVID‐19, based on the algorithms criteria. We reviewed the medical records of 322 random patients to confirm if the algorithms correctly categorized patients based on their severity level. Overall, the algorithms correctly categorized one‐half to two‐thirds of the patients, mostly depending on whether patients received a COVID‐19 medication. The algorithms can be useful to researchers in future studies, but they will need to be thoughtful of algorithm performance and make analytic adjustments.