This tool gives general planning estimates based on editable assumptions and published methods. It is not medical, legal or regulatory advice, it is not a quote, and it does not determine what FDA or any regulator will accept. Talk to qualified regulatory counsel about your submission.
All calculation happens in your browser. Nothing you enter is sent anywhere.
How it works
| Data / label type | 2 labelers | 3 labelers |
|---|---|---|
| Yes/no or categories | Cohen's kappa (S1) | Fleiss' kappa (S3) |
| Ordered grades or rubric | Weighted kappa, quadratic (S2) | Krippendorff's alpha, ordinal (S4) |
| Numeric score | ICC, two-way absolute agreement (S5) | |
| Image segmentation | Dice (+ IoU) per structure (S7, S8) | Mean pairwise Dice |
| EEG/ECG/PPG events | Event-level F1 with a stated tolerance window (S11, adapted) | Mean pairwise event F1 |
| Sleep staging | Cohen's kappa, epoch-level (S1, S12) | Fleiss' kappa (S3) |
Also report raw percent agreement and per-class agreement: kappa can look low when one class dominates (S6). Landis & Koch bands (S13) are commonly cited bands, not a pass/fail threshold; for ICC see Koo & Li (S5).
Frequently asked questions
How much of my data should be double-labeled?
There is no single published rule. Small or high-stakes sets are often fully double-labeled; large sets often double-label a sample big enough to estimate agreement reliably. This tool's percentages are editable starting assumptions; reference-standard sets for FDA submissions typically use multiple qualified readers per case.
Which agreement metric should I use?
It depends on the label: Cohen's kappa for two labelers on categories, Fleiss' kappa for three or more, weighted kappa for ordered grades, ICC for numeric scores, Dice for outlined regions, and event-level F1 for EEG or ECG events. Report raw agreement too, because kappa can look low when one class is rare.
Is the hours estimate a quote?
No. It's a rough range from editable assumptions about minutes per item and disagreement rates. Real throughput depends on your data and label spec and is measured in a pilot. For a scoped pilot, contact us for pricing.
Methods and sources
- S1 Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20(1):37–46. doi:10.1177/001316446002000104
- S2 Cohen J. Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychol Bull. 1968;70(4):213–220. doi:10.1037/h0026256
- S3 Fleiss JL. Measuring nominal scale agreement among many raters. Psychol Bull. 1971;76(5):378–382. doi:10.1037/h0031619
- S4 Hayes AF, Krippendorff K. Answering the call for a standard reliability measure for coding data. Commun Methods Meas. 2007;1(1):77–89. doi:10.1080/19312450709336664
- S5 Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155–163. doi:10.1016/j.jcm.2016.02.012
- S6 Feinstein AR, Cicchetti DV. High agreement but low kappa: I. The problems of two paradoxes. J Clin Epidemiol. 1990;43(6):543–549. doi:10.1016/0895-4356(90)90158-L
- S7 Taha AA, Hanbury A. Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Med Imaging. 2015;15:29. doi:10.1186/s12880-015-0068-x
- S8 Maier-Hein L, et al. Metrics reloaded: recommendations for image analysis validation. Nat Methods. 2024;21:195–212. doi:10.1038/s41592-023-02151-z
- S9 Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther. 2005;85(3):257–268. doi:10.1093/ptj/85.3.257
- S10 FDA. Clinical Performance Assessment: Considerations for Computer-Assisted Detection Devices Applied to Radiology Images and Radiology Device Data in Premarket Notification (510(k)) Submissions. Guidance. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/clinical-performance-assessment-considerations-computer-assisted-detection-devices-applied-radiology
- S11 Dan J, Pale U, Amirshahi A, Cappelletti W, et al. SzCORE: Seizure Community Open-Source Research Evaluation framework for the validation of electroencephalography-based automated seizure detection algorithms. Epilepsia. 2025;66(Suppl 3):14–24. doi:10.1111/epi.18113 (PMID 39292446; open access PMC12489712). It defines event-based scoring for EEG seizure events with overlap-based matching, event-merging rules and pre/post tolerance windows, and reports sensitivity, precision and F1, comparing an algorithm against a reference annotation. *Scope note:* it does not address agreement between human labelers or ECG/PPG events. Using event F1 pairwise between labelers, and for non-EEG signals, is a PAHL adaptation of this method, not a claim made by the source.
- S12 American Academy of Sleep Medicine. AASM Manual for the Scoring of Sleep and Associated Events (current version). https://aasm.org/clinical-resources/scoring-manual/
- S13 Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–174. doi:10.2307/2529310
- S14 FDA, Health Canada, MHRA. Good Machine Learning Practice for Medical Device Development: Guiding Principles (2021). https://www.fda.gov/medical-devices/software-medical-device-samd/good-machine-learning-practice-medical-device-development-guiding-principles
- S15 HHS OCR. Guidance Regarding Methods for De-identification of Protected Health Information in Accordance with the HIPAA Privacy Rule (Safe Harbor and Expert Determination). https://www.hhs.gov/hipaa/for-professionals/special-topics/de-identification/index.html
Assumptions you can edit
- Double-label share: 100% up to 500 items, 25% for 501–5,000, 10% above 5,000, with a floor of 200 items (assumption, editable).
- Complex tasks double the share, capped at 100% (assumption, editable).
- Minutes per item by data type and complexity (assumption, editable).
- Disagreement rate 15% (low) to 30% (high); adjudication minutes equal labeling minutes (assumption, editable).
- Setup overhead 16 (low) to 40 (high) hours, ×1.5 for FDA mode (assumption, editable).