Who is medical data labeling and dataset procurement for?
Teams training or validating clinical models who need the right data, the right to use it, and labels they can defend. That includes startups building AI features, pharma groups validating digital biomarkers, and academic labs preparing datasets for publication or regulatory use.
What does medical data labeling and dataset procurement include?
- Dataset sourcing and licensing — identifying sources (health systems, registries, public datasets, vendors), negotiating access, and documenting rights.
- De-identification — Safe Harbor or Expert Determination, coordinated with the data holder, with re-identification risk documented.
- Data governance and lineage — who may use what, for which purpose, and a record from source to training set.
- Labeling protocol design — label definitions, edge cases, and adjudication rules written with clinicians before anyone labels.
- Clinical labeler recruitment and training — physicians, nurses, or specialists matched to the task, trained against the protocol.
- Quality control — inter-rater reliability, adjudication, drift checks, and audit samples.
- Dataset documentation — datasheets and model-card inputs a regulator or reviewer expects.
How does a medical data labeling and dataset procurement engagement run?
- RequirementsThe model or study, the labels it needs, and the volume and provenance required.
- Source and secureDataset acquisition, agreements, de-identification, and governance set-up.
- LabelProtocol, labeler onboarding, tooling, and quality control in production.
- Deliver and documentThe dataset, labels, reliability metrics, and documentation, with a handoff for ongoing labeling if needed.
Getting PHI to annotators: four HIPAA options
Before any labeling starts, you need a lawful way to show clinical data to the people labeling it. HIPAA gives four practical routes. The right one depends on how much identifying detail the labels need: dates and free-text notes, for example, often can't be fully de-identified without losing the signal.
| Route | What it allows | What must be removed | Who signs off | Best-fit labeling job | HHS source |
|---|---|---|---|---|---|
| De-identification: Safe Harbor | Data is no longer PHI, so HIPAA no longer restricts its use or disclosure | All 18 listed identifiers, including most dates and geography below the state level, and no actual knowledge that what remains could identify someone | The covered entity | Imaging and waveform data (ECG, EEG, accelerometry) where the labels don't need dates or locations | HHS de-identification guidance; 45 CFR 164.514(b) |
| De-identification: Expert Determination | Data is no longer PHI | Whatever a qualified expert decides is needed to make the risk of re-identification very small; some dates or details may stay | A qualified statistical expert, who documents the methods and results | Longitudinal or time-stamped data where Safe Harbor would strip out the signal | HHS de-identification guidance |
| Limited data set with a data use agreement | Use for research, public health or health care operations only | Direct identifiers such as names, street addresses, phone numbers and record numbers; dates, city, state and ZIP code may stay | The covered entity and the recipient, who sign a data use agreement | Research labeling projects that need dates of service or event timing | HHS research guidance; 45 CFR 164.514(e) |
| Business Associate Agreement with the labeling vendor | The vendor handles full PHI on the covered entity's behalf, only as the agreement permits | Nothing has to be removed, though the minimum-necessary standard still applies | The covered entity and the labeling vendor, who sign a BAA | Free-text clinical notes and chart review, where removing identifiers is impractical | HHS sample BAA provisions |
We usually recommend the least identifiable route that still supports the label. For label-quality methods, pilot scope and the data types we cover, see expert clinical labeling.
Labels for FDA submissions: reference standards
If the labels will train or test an AI-enabled medical device, they become part of your regulatory evidence. FDA's Good Machine Learning Practice guiding principles (FDA, Health Canada and the UK MHRA, 2021, since built on by an IMDRF final document in January 2025) call for training data sets that are independent of test sets and reference datasets based on the best available methods.
FDA's draft guidance on AI-enabled device software functions (January 2025, not yet final) recommends that marketing submissions describe data collection, the independence of development and test data, reference standards and representativeness, including the criteria and expertise used to set the clinical reference standard. In practice that means:
- Document who labeled what: annotator qualifications, the labeling protocol and how disagreements were adjudicated.
- Keep the test set separate: label it separately, ideally from different sites or patients than the training data, and lock it before model evaluation.
- Measure agreement: report inter-rater agreement so the reference standard's own uncertainty is known.
- Check representativeness: record the demographics and clinical mix of the labeled data against the intended-use population.
For regulatory strategy around the submission itself, see SaMD consulting.
Frequently asked questions about medical data labeling and dataset procurement
Do you label the data yourselves?
We design the protocol, recruit and train clinical labelers, run quality control, and manage the tooling. For large volumes we work with vetted annotation partners under our protocol and QC rather than a generic crowd. For done-for-you labeling of EEG, neurology, and wearable signals, see expert clinical labeling.
Can you source data for a specific condition?
Usually. The path depends on the condition, the modality, and whether identifiable data is required. We tell you early which sources are realistic and what the agreements will cost in time.
How is PHI handled?
Under a Business Associate Agreement where PHI is unavoidable, with de-identification as early in the pipeline as the use case allows and governance that records every access.
Can annotators see PHI when labeling medical data?
Yes, if the right HIPAA route is in place. Data de-identified by Safe Harbor or Expert Determination is no longer PHI. A limited data set can be shared under a data use agreement for research, public health or operations, and a labeling vendor can handle full PHI under a Business Associate Agreement.
What is the difference between Safe Harbor and Expert Determination?
Safe Harbor removes 18 specific identifiers, which is simple but often strips out dates. Expert Determination has a qualified expert assess and document that the risk of re-identification is very small, which can keep more detail, such as some dates, when the analysis supports it.
What does FDA expect from labels used for an AI medical device?
FDA's Good Machine Learning Practice principles call for test data independent of training data and reference datasets based on the best available methods. FDA's January 2025 draft guidance on AI-enabled devices asks submissions to describe the reference standard, including the criteria and expertise used to set it.
Related services
Scope it in 30 minutes
Book a scoping call and we'll name the smallest engagement that moves you forward. Most teams start with the fixed-scope Initial Audit.
Book a 30-min scoping callPrefer to write? Send a short note.