OMARI Seminar Series, Seminar II

BEYOND THE AUC: DEVELOPING AND VALIDATING CLINICAL PREDICTION MODELS - A CASE STUDY IN METABOLIC GESTATIONAL AGE ESTIMATION This Fall, the Ottawa Medical AI Research Institute (OMARI) is launching a new OMARI Seminar Series, bringing together, and in person, researchers, clinicians, medical students, faculty and others interested in the rapidly evolving field of AI in medicine. Join us for the second seminar in the series! This talk is titled Beyond the AUC: Developing and Validating Clinical Prediction Models, A Case Study in Metabolic Gestational Age Estimation. A model can have seemingly impressive AUC and other performance metrics and still give badly wrong individual-level estimates, fail in a new population, or offer little or no clinical benefit. This two-hour deep-dive shows how to develop and evaluate a prediction model properly and honestly and how to interpret model performance metrics reported in scientific papers. The case study we will use is a set of machine learning models that estimate gestational age from routine newborn screening blood tests. These models were developed using Ontario data and externally validated internationally. Using the real models and data, the session shows why performance changes when a model travels, and what to do about it. It also includes a candid look at design choices and possible improvements with the benefit of hindsight. Participants get companion R code and open data to rerun every example. The session is suitable for researchers, trainees and clinicians comfortable with basic statistics; no engineering or coding background is needed.