Artificial Intelligence in Medicine: The Promise, the Pitfalls, and What Patients Should Know

Table of Contents

• Key Points
• Why This Research Matters
• A Century of Statistics in Medicine
• How AI Learns: The Power of Automated Pattern Detection
• Prediction vs. Understanding: Two Different Goals
• Will AI Work in the Real World? Generalizability and Interpretation
• Keeping AI Honest: Stability and Statistical Guarantees
• When AI Gets It Wrong: Real-World Bias
• What This Means for Patients
• Limitations of This Review
• Recommendations for the Future
• Frequently Asked Questions
• Source Information

Key Points

• AI is already used in medicine for mammograms, disease risk prediction, and electronic health records, but it has statistical vulnerabilities.
• AI's automated pattern discovery can amplify hidden biases; an algorithm applied to 200 million Americans discriminated against Black patients.
• AI models are often hard to interpret and verify, requiring transparency and human oversight for safe medical use.
• There is a gap between AI's ability to predict individual outcomes and understanding disease in the broader population.
• Releasing code, prespecified analysis plans, combining AI with conventional statistics, and human judgment are recommended for responsible AI use.

Why This Research Matters

Artificial intelligence is no longer a distant concept in healthcare—it is already here. AI systems are being used to read mammograms, predict disease risk, analyze electronic health records, and even assist with medical note-taking. But how do we know these systems are safe, accurate, and fair?

This review article, written by David J. Hunter, M.B., B.S., from the Nuffield Department of Population Health at the University of Oxford, and Christopher Holmes, Ph.D., from the Department of Statistics and the Alan Turing Institute in London, tackles a critical question: what happens when the rigorous world of medical statistics meets the powerful but often opaque world of artificial intelligence?

The authors describe a fundamental paradox. AI's ability to automatically discover patterns in enormous datasets makes it an incredibly valuable medical research tool—but the very same features make it statistically vulnerable. Techniques adequate for targeted advertising to voters or consumers may not meet the rigorous demands of risk prediction or diagnosis in medicine. This article explores that tension and what it means for the future of patient care.

A Century of Statistics in Medicine

Statistics emerged as a distinct discipline around the beginning of the 20th century. During this period, fundamental concepts were developed that would transform medical research, including the use of randomization in clinical trials, hypothesis testing, likelihood-based inference, P values, and Bayesian analysis and decision theory.

Statistics quickly became essential to applied sciences. In fact, in 2000, the editors of the New England Journal of Medicine cited "Application of Statistics to Medicine" as one of the 11 most important developments in medical science over the previous 1000 years.

So what exactly is statistics? The authors define it as reasoning with incomplete information—the rigorous interpretation and communication of scientific findings from data. Statistics includes determining the optimal design of experiments and accurately quantifying uncertainty about conclusions, all expressed through the language of probability.

Now, in the 21st century, artificial intelligence has emerged as a powerful new force in medical research. This development is driven, in part, by enormous expansions in computer power and data availability—but with these advances come new statistical challenges that this review sets out to address.

How AI Learns: The Power of Automated Pattern Detection

The Traditional Approach: Hands-On Statistics

Traditional statistical modeling relies on careful, hands-on selection of measurements and data features to include in an analysis. For example, a statistician must decide which covariates (factors that might influence the outcome) to include in a regression model—a statistical method that examines relationships between variables. They also determine what transformations or standardizations of measurements are needed.

Semiautomated data-reduction techniques such as random forests and forward- or backward-selection stepwise regression have assisted statisticians in this selection process for decades. In this traditional framework, modeling assumptions and features are typically explicit, and the number of parameters in the model is usually known.

The AI Revolution: Automatic Feature Learning

Arguably the most impressive and distinguishing aspect of AI is its automated ability to search and extract arbitrary, complex, task-oriented features from data—a process called feature representation learning. Features are algorithmically engineered from data during a training phase to uncover data transformations that are correct for the learning task.

AI algorithms largely remove the need for analysts to prespecify features or manually curate variable transformations. This is especially beneficial in large, complex data domains such as:

• Image analysis (like mammograms or pathology slides)
• Genomics (studying the complete set of genes in an organism)
• Modeling electronic health records (digital versions of patients' medical histories)

AI models can search through potentially billions of nonlinear covariate transformations to reduce a large number of variables to a smaller set of task-adapted features. Somewhat paradoxically, increasing the complexity of an AI model through additional parameters—which is what happens in deep learning—actually helps the model find richer internal feature sets, provided training methods are suitably tailored.

The Dark Side of the Paradox

Here is the problem: the features AI engineers are often beyond the scope of what humans can create, which is why AI performs so impressively. But those same features are often hard to interpret, are brittle (prone to breaking) when data changes, and lack common sense when it comes to using background knowledge and qualitative checks that statisticians routinely apply.

AI models are often unable to trace the evidence line from data to features, making auditability and verification challenging. This means greater checks and balances are needed to ensure the validity and generalizability of AI-enabled scientific findings.

The authors also briefly comment on the emerging field of generative AI—such as large language models and medical chatbots that might be used for medical note-taking in electronic health records. These "foundation models" use self-supervised learning on vast quantities of undocumented training data, with objective functions trained using trillions of parameters (at the time of writing). This is a stark contrast to "supervised" learning, where training data are known and labeled according to clinical outcomes, and the training objective is clear and targeted. Given the opaqueness of these generative models, the authors urge extra caution for health applications.

Prediction vs. Understanding: Two Different Goals

AI is especially well suited to—and largely designed for—large-scale prediction tasks. The training objective is clear, and predictive accuracy is usually well characterized. A good example is predicting the risk of disease.

However, the ultimate goal of most medical studies is not explicitly to predict risk. Rather, it is to understand some biological mechanism or cause of disease in the wider population, or to assist in developing new therapies. The authors emphasize that there is an evidence gap between a good predictive model that operates at the individual level and the ability to make inferential statements about the population.

Statistics is mainly concerned with population inference—generalizing evidence from one study to a scientific hypothesis about the broader population. Prediction is an important yet simpler task; scientific inference often has a greater influence on mechanistic understanding.

As Hippocrates observed centuries ago: "It is more important to know what sort of person has a disease than to know what sort of disease a person has."

A Real-World Example: The Covid-19 Pandemic

During the Covid-19 pandemic, various prediction tools were developed to determine whether a person had SARS-CoV-2 (the virus that causes Covid-19) infection. However, moving from individual prediction to understanding the population prevalence—and identifying which subgroups in the population were at higher risk—proved much more challenging.

How Do We Measure Predictive Accuracy?

An additional challenge is that there are many ways to measure and report predictive accuracy, including:

• Area under the receiver-operating-characteristic curve (a measure of how well a test distinguishes between groups)
• Precision and recall (measures of exactness and completeness)
• Mean squared error (average of the squares of the errors)
• Positive predictive value (probability that a positive test result is correct)
• Misclassification rate (how often the model gets the answer wrong)
• Net reclassification index (whether a new model improves classification)
• Log probability score (a measure of prediction confidence)

Choosing the measure that is appropriate for the context is vitally important, because accuracy in one measure may not translate to accuracy in another—and may not relate to a clinically meaningful measure of performance or safety.

In contrast, inferential targets for population statistics tend to be less ambiguous, and uncertainty is clearly characterized using P values, confidence intervals, and credible intervals. Even so, robust, accurate AI prediction models indicate that repeatable signals and stable associations in the data are worth investigating further.

Causal Machine Learning: Where AI and Statistics Meet

An interesting meeting point between AI prediction methods and statistical inference is causal machine learning, which pays particular attention to inferential quantities. By adopting structural causal modeling or potential outcomes frameworks—with tools such as directed acyclic graphs (diagrams that show cause-and-effect relationships)—researchers can use domain knowledge to reduce the probability that an AI model will make mistakes. These mistakes include misspecifying the temporal relationship between exposure and outcome, conditioning on a variable that is caused by both exposure and disease (a "collider"), or highlighting spurious associations (like a batch effect in a biomarker study).

Causal inference methods may also be applied to AI to help interpret radiological or pathological images and to support clinical decision making and diagnosis. However, the authors stress that human judgment will likely be necessary for the foreseeable future, if only because different AI algorithms may present us with different conclusions. Moreover, causal analysis from observational data requires assumptions that lie outside what can be learned from the data alone—to avoid bias from ascertainment, mediation, and confounding.

Will AI Work in the Real World? Generalizability and Interpretation

One of the biggest challenges in interpreting AI results is that algorithms for internal feature representation are designed to automatically adapt their complexity to the task at hand, with nearly infinite flexibility in some approaches.

This flexibility is a great strength—but it also requires care to avoid overfitting, which happens when a model learns the noise and random quirks of its training data so well that it performs poorly on new, unseen data.

The Limits of Traditional Statistical Guarantees

The use of regularization (techniques that prevent overfitting by penalizing model complexity) and controlled stochastic optimization of model parameters during training can help prevent overfitting. But these techniques also mean AI algorithms have poorly defined notions of statistical degrees of freedom and the number of free parameters. Traditional statistical guarantees against overoptimism cannot be used in the AI context.

Instead, researchers must substitute techniques such as cross-validation and held-out samples to mimic true out-of-sample performance. The trade-off is that the amount of data available for discovery is reduced. Taken together, these factors create a real risk of overinterpreting the generalizability and reproducibility of results.

A Notable Example: AI and Breast Cancer Screening

The authors highlight a well-known case involving a study by McKinney et al. on using AI to predict breast cancer based on mammograms. The study showed high potential for AI in breast cancer screening. However, Haibe-Kains and colleagues publicly called for greater transparency, stating: "In their study, McKinney et al. showed the high potential of AI for breast cancer screening. However, the lack of details of the methods and algorithm code undermines its scientific value."

This illustrates why clear reporting of results and availability of code are essential for external replication and refinement by other research groups—though this may be limited by a tendency to seek intellectual property rights for commercial AI products.

Using AI to Winnow Down Big Data

AI approaches can be helpful in reducing a dataset with a very large number of features—such as "-omic" datasets (metabolomic, proteomic, or genomic)—into a smaller number of features that can then be tested with conventional statistical methods.

Popular AI methods that provide "feature relevance" rankings of covariates include:

• Random forests (an ensemble of decision trees)
• XGBoost (a gradient-boosting algorithm)
• Bayesian additive regression trees

Statistical methods such as the least absolute shrinkage and selection operator (LASSO) use explicit variable selection as part of model fitting. Feature reduction helps the human analyst examine the data more effectively and apply constraints based on previous subject knowledge. For example, a researcher might know that feature A is often confounded by feature X, or that a latency period of several years between exposure to feature A and the disease outcome means no relationship is expected in early follow-up.

AI vs. Conventional Statistics: A Side-by-Side Comparison

The article provides a detailed comparison of AI methods and conventional statistics. Key differences include:

• Prior hypotheses: AI is agnostic or very general about hypotheses; conventional statistics specifies hypotheses as primary, secondary, or exploratory.
• Techniques: AI uses random forests, neural networks, and XGBoost; conventional statistics uses parametric and nonparametric comparisons, regression, and survival models with linear predictors.
• Stability: AI analyses are more prone to instability due to application domains and user choices in algorithm specification; conventional statistics follow a prespecified analysis plan with minimal user-defined choices.
• Applications: AI excels at images, monitor outputs, electronic health records, and natural language processing; conventional statistics suits data with fewer predictors, tabular data, and randomized trials.
• Purpose: AI focuses on pattern discovery, automatic feature representation, feature reduction, and prediction; conventional statistics focuses on inference, testing specific factors, controlling confounding, and quantifying uncertainty.
• Reproducibility: AI often provides internal reproducibility (cross-validation or split samples); conventional statistics ideally provides external reproducibility with new data.
• Barriers: AI faces proprietary algorithms not available to other researchers and unclear reporting; conventional statistics faces slow progress in sharing primary data.
• Interpretability: AI is often a black box; conventional statistics has explicit features and clear degrees of freedom.
• Equity: AI's data-driven feature learning is susceptible to biases in data, compounding health inequities; conventional statistics models are more easily checked for equity when relevant data are available.

Keeping AI Honest: Stability and Statistical Guarantees

Medical science is an iterative process of observation and hypothesis refinement—cycles of experimentation, analysis, and conjecture that lead to further experiments and ultimately to a level of evidence that refutes existing theories or supports new therapies and lifestyle recommendations.

Randomized trials of investigational drugs have historically been held to a high standard of rigor. Concerns about overinterpretation of secondary end-point and subgroup analyses have led to an even stronger focus on prespecified description of primary hypotheses and control of the familywise error rate (the probability of making at least one false positive finding among multiple tests) to limit false positive results.

Protocols now often specify:

• The precise estimands (the exact quantities being measured)
• The methods of analysis that will be used to obtain P values
• The covariates to be controlled for
• Even the dummy tables that will be filled in once data are complete

Analyses in observational studies are usually less rigorously prespecified, although a statistical analysis plan established before data analysis begins is increasingly expected as supplementary material in published reports.

The AI Problem: Patterns Without Prespecification

AI approaches often seek patterns in data that are not prespecified—which is one of their strengths. But the downside is that the potential for false positive results increases unless rigorous procedures for assessing reproducibility are incorporated.

New reporting guidelines and recommendations for AI in medical science have been established to ensure greater trust and generalizability of conclusions. The authors also note that highly adaptive AI algorithms inherit all the biases and unrepresentativeness that might be present in the training data. When using black-box AI prediction tools, it can be difficult to judge whether predictive signals arise from confounding due to hidden biases in the data.

Methods from the field of explainable AI (XAI) can help counter opaque feature representation learning. However, for applications in which safety is a critical issue, the black-box nature of AI models warrants careful consideration and justification.

When AI Gets It Wrong: Real-World Bias

One of the most striking examples of AI bias in healthcare comes from research by Obermeyer and colleagues. The team described an AI-informed algorithm that was applied to a population of 200 million persons in the United States each year to identify patients at the highest risk for incurring substantial health care costs and to refer them to "high-risk care management programs."

The analysis suggested that the algorithm unintentionally discriminated against Black patients. The reason? At every level of health care expenditure and age, Black patients had more coexisting conditions than White patients. Because the algorithm was trained on health care spending as a proxy for health needs—and because systemic factors lead to lower spending on Black patients even when they are sicker—the algorithm systematically underestimated the health needs of Black patients.

This example dramatically illustrates a core concern the authors raise: AI systems trained on real-world data can perpetuate and even amplify existing inequities in health care.

What This Means for Patients

For patients, the message is both hopeful and cautionary. AI has the potential to transform medicine in powerful ways:

• Reading and interpreting medical images like mammograms with high accuracy
• Predicting disease risk from electronic health records
• Analyzing genomic and other "-omic" data to uncover disease mechanisms
• Reducing massive datasets to the most relevant features for clinical questions

But patients should also be aware of the limitations:

• AI results can be difficult to interpret and verify
• AI models may not generalize to different populations than what they were trained on
• Biases in training data can lead to unfair or inaccurate results for certain groups
• The lack of transparency in commercial algorithms makes independent validation harder

Patients can advocate for themselves by asking questions about how AI is being used in their care. Willingness by health care institutions to provide clear answers about the validation, limitations, and oversight of AI tools is a good sign of responsible implementation.

The authors note that using conventional statistical prediction methods alongside interpretable AI methods can help provide an understanding of prediction signals and can mitigate nonsensical associations. Combining the best of both approaches—AI's power to handle vast data and statistics' rigor in interpretation—is likely the safest path forward.

Limitations of This Review

As a review article rather than an original research study, this paper synthesizes existing knowledge rather than presenting new experimental data. The authors acknowledge that space constraints precluded a detailed discussion of the important area of AI and experimental design, and they only briefly comment on the emerging area of generative AI and medical chatbots rather than providing a deep dive.

The article reflects the perspectives of two researchers based in the United Kingdom—one from population health at Oxford and one from statistics and medicine at Oxford and the Alan Turing Institute—and draws heavily on examples from the U.S. and U.K. health care contexts. The field of AI in medicine is evolving rapidly, and some specifics (such as the scale of foundation models with trillions of parameters) may change quickly over time.

Recommendations for the Future

Based on their analysis, the authors offer several practical recommendations for medical scientists and health care institutions:

1. Release all code. Sharing code and providing clear statements on model fitting and held-out data used for reporting accuracy facilitates external assessment of reproducibility.
2. Be transparent about data. Clear reporting of results and availability of code add to the potential for external replication and refinement by other groups.
3. Use prespecified analysis plans. For observational studies, a statistical analysis plan established before data analysis should be expected, just as it is for randomized trials.
4. Combine AI with conventional methods. Using traditional statistical prediction methods alongside interpretable AI methods can clarify prediction signals and mitigate nonsensical associations.
5. Employ feature reduction wisely. AI can reduce high-dimensional data to a smaller set of features that can then be tested with conventional statistics, allowing human analysts to apply subject-specific knowledge.
6. Maintain human oversight. Human judgment will be necessary for the foreseeable future, especially because different AI algorithms may present different conclusions, and causal inference requires assumptions beyond what the data alone can reveal.
7. Follow reporting guidelines. New reporting guidelines and recommendations for AI in medical science are being established to ensure greater trust and generalizability of conclusions.
8. Assess internal reproducibility. Data should be partitioned into discovery and test sets, and generalizability to other datasets must be carefully evaluated.

The authors emphasize that approaching AI in medicine with appropriate rigor and safeguards is not just a technical concern—it is essential for patient safety, scientific integrity, and health equity.

As the article's framing makes clear, a technique that works for targeted advertising or weather prediction may not meet the rigorous demands of medical diagnosis and risk prediction. The stakes are simply too high.

Frequently Asked Questions

What is artificial intelligence (AI) currently used for in medicine?

AI is already used in healthcare to read mammograms, predict disease risk, analyze electronic health records, and assist with medical note-taking. It can also help analyze genomic data and reduce massive datasets to the most relevant features for clinical questions. These applications show AI's potential, but they come with important limitations to understand.

How is AI different from traditional statistics in medical research?

Traditional statistics relies on careful, hands-on selection of measurements and features, with explicit modeling assumptions. AI automatically searches for patterns in vast datasets, learning features on its own. This makes AI powerful for prediction but can make it hard to interpret, verify, and check for hidden biases compared to conventional statistical methods.

Can AI be biased or make mistakes in healthcare?

Yes. A real-world AI algorithm applied to 200 million Americans to identify high-risk patients unintentionally discriminated against Black patients. Because it was trained on healthcare spending, and systemic factors lead to lower spending on Black patients even when sicker, it underestimated their health needs. AI can inherit and amplify existing biases in training data.

Why is it hard to verify AI's decisions in medicine?

AI models often use internal feature representation learning, which creates features that are hard to interpret and lack common sense. They are often unable to trace the evidence line from data to features, making auditability and verification challenging. This is why transparency, clear reporting, and human oversight are essential when AI is used in patient care.

What should patients ask about AI used in their care?

Patients can ask how AI is being used in their care, how it was validated, what its limitations are, and who oversees it. A health institution that gives clear answers about these topics is showing responsible implementation. Patients should also be aware that AI results may not generalize to different populations and can contain biases.

Can AI be used to understand disease causes, not just predict risk?

AI is especially good at large-scale prediction, like predicting disease risk. However, understanding disease mechanisms in the wider population is harder. There is an evidence gap between predicting outcomes for an individual and making inferential statements about the population. Combining AI with conventional statistics and human judgment is the safest path forward.

What does the future of AI in medicine look like for patients?

The future holds both promise and caution. AI has potential to transform medicine, but patients should be aware of limitations like difficult interpretation, poor generalizability, and bias. Responsible implementation includes releasing code, transparency about data, using prespecified analysis plans, and human oversight. Combination of AI with conventional methods is likely the safest approach.

Should I get a second opinion if AI is used in my medical diagnosis or treatment plan?

Yes, a second opinion can be valuable when AI tools are involved in your care. AI systems can be biased, as shown by an algorithm that unintentionally discriminated against Black patients, and their results can be hard to interpret or verify. A second opinion from an independent expert can help ensure your diagnosis or treatment plan is accurate and appropriate, especially if you have concerns about how AI was used. Diagnostic Detectives Network provides independent expert second opinions.

Source Information

Original article title: Where Medical Statistics Meets Artificial Intelligence

Authors: David J. Hunter, M.B., B.S., and Christopher Holmes, Ph.D.

Journal: The New England Journal of Medicine, 2023; volume 389, pages 1211–1219

Publication date: September 28, 2023

DOI: 10.1056/NEJMra2212850

Article type: Review Article in the "AI in Medicine" series, edited by Jeffrey M. Drazen, M.D., with guest editors Isaac S. Kohane, M.D., Ph.D., and Tze-Yun Leong, Ph.D.

Copyright: © 2023 Massachusetts Medical Society. All rights reserved.

This patient-friendly article is based on peer-reviewed research and is intended for educational purposes. It does not constitute medical advice.