SNOMED International has validation tests and KPIs that track SNOMED CT content quality, largely focused on structural aspects such as RF2 compliance, template adherence and avoidance of specifically identified “patterns” such as role group crossovers.
These measures are valuable for ensuring coherence and conformance, but they do not fully address what we might mean by clinical or semantic quality.
In particular, they do not directly test whether:
-
Concept definitions are medically appropriate or clinically valid
-
Parent/child relationships reflect true clinical meaning (not just modelling structure)
-
Content aligns with real-world clinical interpretation and usage
In general those aspects of quality are considered by terminology authors as part of content review, but this approach does not lend itself to quantitative metrics, or anything that could be demonstrated.
This also raises a broader question: are we measuring the right things when we talk about “quality”?
Other domains, particularly software engineering, have developed broader quality frameworks that extend beyond structural validation. For example, standards and approaches such as:
-
ISO/IEC 25010 (SQuaRE model) for software product quality
-
OQuaRE (Ontology Quality Requirements and Evaluation - which adapts software quality dimensions to ontologies)
-
Biomedical ontology evaluation frameworks that separate structural, semantic, and pragmatic quality (eg OEF, OntoQA, SQM)
-
Competency-question based evaluation approaches used in ontology engineering
These approaches consistently highlight a similar gap: structural correctness is necessary, but not sufficient to capture semantic correctness or real-world fitness for purpose.
I’m interested in views on how we might extend or complement current structural KPIs with measures that better reflect clinical appropriateness, semantic validity, and real-world usability. Ideally in a way that is scalable and actionable.
A websearch for work previously done in this area resulted in the list below. I also note that the Member Forum was asked a related question in September here: https://forums.snomed.org/t/preparation-for-member-forum-workshop-content-quality-in-practice/335 (tagging @ahoejen)
- Zhang & Bodenreider (2010) — Structural auditing of SNOMED CT using Formal Concept Analysis
https://pmc.ncbi.nlm.nih.gov/articles/PMC3041382/
Analyses SNOMED CT hierarchy structure to detect subsumption irregularities and structural anomalies using lattice-based methods. - Mikroyannidi et al. (2012) — Syntactic regularities and irregularities in SNOMED CT
https://jbiomedsem.biomedcentral.com/articles/10.1186/2041-1480-3-8
Examines modelling consistency in SNOMED CT by identifying recurring structural patterns and deviations. - Abad-Navarro et al. (2020) — Readability and structural accuracy of SNOMED CT
https://link.springer.com/article/10.1186/s12911-020-01291-y
Evaluates lexical readability and structural accuracy of SNOMED CT content across multiple releases. - SNOMED CT usage and mapping evaluation studies (e.g. NLP and coding accuracy assessments)
https://arxiv.org/abs/2311.10856
Assesses SNOMED CT quality indirectly via real-world performance in clinical coding and automated mapping tasks.
@jcase I thought this would be a good topic to include at our Joint Advisory Group session planned for October, but I would - of course - welcome any thoughts you have in this area in the meantime.