In my previous article, I discussed how healthcare datasets might be suitable for real-world evidence (RWE) studies, artificial intelligence (AI) applications, or both. The overlap between these use cases is real, but the technical process used to evaluate a dataset is often very different.
For RWE, many professionals are familiar with structured frameworks for fit-for-purpose assessment. Common examples include SPIFD (Structured Process to Identify Fit-for-Purpose Data), which provides a stepwise approach for determining whether a real-world data (RWD) source can support a specific regulatory or decision-grade study, SURF (Structured Unified Review Framework), which supports systematic evaluation of variable availability, provenance, and data relevance for a defined study objective or REQueST (Registry Evaluation and Quality Standards Tool), developed by EUnetHTA for assessing whether patient registries are suitable for HTA and regulatory use.
These frameworks all start from the same principle: the dataset is not assessed in isolation, but against the explicit study design requirements and minimum variables needed to answer the research question. In AI, however, dataset assessment follows a different logic. The objective is not simply to verify whether variables exist, but whether the data can support training a model that performs reliably in real-world deployment. This distinction has important implications for healthcare organizations that want to reuse data assets across evidence generation and AI development.
AI data feasibility starts with the intended tool, not the dataset
One of the biggest differences from RWE is that AI data assessment is rarely performed as a generic source evaluation. It begins with the exact AI use case.
A team developing a hospital readmission model, for example, first defines the machine learning task precisely: the input window, prediction target, decision point, and intended user. This is aligned with internationally converging regulatory principles for AI-based medical devices, including the FDA Good Machine Learning Practice (GMLP) guidance, the EU AI Act data governance requirements, and PMDA expectations for AI-enabled software as a medical device.
This means a dataset cannot be labeled as “AI-ready” in absolute terms. It can only be considered fit relative to a specific model specification. A hospital EHR repository may be suitable for a sepsis prediction model but inadequate for an AI system that relies on continuous physiological signal processing, such as clinical deterioration detection, when time-resolved waveform data (e.g., ECG or arterial pressure waveforms) is unavailable.
The first technical layer: feature availability in context
The closest equivalent to variable assessment in RWE is feature auditing. AI teams map the required model inputs to the actual data source and quantify whether those inputs are present, complete, and usable.
Unlike RWE, however, availability alone is not sufficient. A feature must also be available at the correct time relative to prediction. This is known in operational ML practice as temporal feature validation. If a diagnosis code is entered after discharge but the model is intended to predict readmission at discharge, the variable cannot be used, even if it exists in retrospective records.
This issue is common in healthcare AI because many electronic health record (EHR) fields are entered asynchronously for billing or administrative reasons rather than in real time. As a result, retrospective datasets often contain variables that create data leakage if used naïvely in training.
Label assessment is often the decisive step
In many healthcare AI projects, the limiting factor is not the input data but the labels used for training.
A label in AI is the target outcome the model learns to predict. For clinical applications, labels are often much harder to validate than in RWE studies.
An RWE study may accept a proxy endpoint derived from claims or coded records. AI models often require a higher-confidence ground truth because noisy labels directly affect model performance. For this reason, mature teams perform dedicated label audits before deciding whether a dataset is trainable.
This usually involves reviewing a sampled subset of records and measuring annotation quality. In imaging projects, radiology teams may manually review hundreds of studies to estimate disagreement between source reports and true findings. In EHR-based models, chart review may be performed to validate outcome definitions.
Companies supporting AI projects rely heavily on curated labels from expert clinical abstraction, especially in oncology where endpoints such as progression or treatment response are not consistently available as structured fields.
Statistical readiness is a separate assessment
A dataset may contain all required features and labels yet still fail as a training source because it lacks statistical learnability. This concept has no direct equivalent in traditional RWE feasibility.
AI teams typically examine class prevalence, sample size per outcome class, and subgroup distribution before proceeding. For example, a dataset may include a target adverse event, but if it occurs in only 0.2% of patients, the available signal may be too sparse for robust model training unless the dataset is substantially expanded.
A recent published framework called DREAMER (Data Readiness Evaluation for Machine Learning) formalizes this type of assessment (Ahangaran et al.). It evaluates datasets across dimensions such as completeness, class balance, feature correlation, and model-readiness metrics rather than only classical data quality measures.
This is significant because it reflects a shift from asking whether the data is clean to asking whether the data is learnable.
External validity is tested early, not at the end
Perhaps the most distinctive technical step in AI feasibility is early simulation of generalization.
Before committing to full model development, many organizations build a simple pilot model using baseline algorithms such as logistic regression or XGBoost (Christodoulou et al; Chen et al.). The purpose is not product development but dataset validation.
The model is tested across different data splits: internal holdout, temporal and external site holdout.
For example, a model may be trained on data from one hospital for 2021–2023 and tested on a second hospital in 2024.
If performance drops sharply between internal and external validation, the problem often lies in dataset heterogeneity rather than model architecture. This can indicate differences in coding practices, patient mix, device vendors, or local workflow.
FDA guidance for AI-enabled medical devices explicitly highlights the need for data that reflects the intended patient population and use environment, which is why external representativeness is considered during dataset assessment rather than only after model training.
Why AI data assessment resembles software validation more than epidemiology
This is where the distinction becomes most visible.
RWE feasibility is rooted in epidemiology. It asks whether the source supports valid estimation for a predefined question.
AI feasibility is closer to software engineering and product validation. The assessment combines:
- Data engineering
- Label governance
- Machine learning experimentation
- Deployment simulation
In other words, the dataset is not only being evaluated as a source of evidence. It is being evaluated as an operational component of a future software system.
That is why the same hospital database may pass an RWE feasibility assessment but fail an AI feasibility assessment. The data may be scientifically valid for retrospective analysis but operationally unsuitable for building a generalizable model.
The emerging convergence
Interestingly, healthcare organizations are beginning to combine both perspectives.
As more institutions invest in enterprise data platforms, they increasingly want datasets that support multiple downstream uses: RWE studies, AI training, clinical operations, and regulatory submissions. This has led to a growing interest in AI-ready data quality frameworks, which extend beyond classical fit-for-purpose models.
The practical implication is that healthcare data strategy is no longer just about variable completeness. It is about understanding whether the same source can support both evidence generation and algorithmic deployment. That requires separate but complementary feasibility assessments.
REFERENCES:
Ahangaran M, Zhu H, Li R, Yin L, Jang J, Chaudhry AP, Farrer LA, Au R, Kolachalama VB. DREAMER: a data readiness evaluation framework for machine learning in healthcare. BMC Med Inform Decis Mak. 2024;24:Article 149. doi:10.1186/s12911-024-02544-w.
Chen T, Guestrin C. XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2016:785–794. doi:10.1145/2939672.2939785.
Christodoulou E, Ma J, Collins GS, Steyerberg EW, Verbakel JY, Van Calster B. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol. 2019;110:12–22. doi:10.1016/j.jclinepi.2019.02.004.
U.S. Food and Drug Administration, Health Canada, Medicines and Healthcare products Regulatory Agency. Good machine learning practice for medical device development: guiding principles. 2021. Available from: https://www.fda.gov/medical-devices/software-medical-device-samd/good-machine-learning-practice-medical-device-development-guiding-principles
U.S. Food and Drug Administration. Transparency for machine learning-enabled medical devices: guiding principles. 2021. Available from: https://www.fda.gov/medical-devices/software-medical-device-samd/transparency-machine-learning-enabled-medical-devices-guiding-principles
