SNOMED CT Quality Investigation using AI

Technology & Software AI & SNOMED CT
1

I was tempted to make the title “SCT AI QA” there :grinning_face_with_smiling_eyes:

AI offers opportunities to identify potential defects in SNOMED CT content, both retrospectively across existing editions, and incrementally as new changes are authored.

There are several aspects we could explore together:

  • Detection methods: Using AI and other automated processes to identify potential quality issues in SNOMED CT content.

  • Early detection: Identifying patterns that could be incorporated into authoring tools to catch issues earlier, or flag them for review.

  • Human validation and feedback: Using expert review to confirm or dismiss potential issues, and feed those decisions back into the detection process.

  • Operational approaches: Exploring how these techniques could be applied at scale, including across historical content and new deltas.

I’d be interested to hear from anyone experimenting with these approaches, or with ideas for other ways AI and automation could support SNOMED CT quality assurance.

If you’ve been doing work in this area, please share your methods, experiences, and findings here. In particular, it would be useful to understand how the approaches work, what kinds of issues they can detect, and how we might build on them collectively.

2 Likes
2

Not quite SNOMED CT’s own content, but a closely related QA case that might interest this thread: I’m the author of an open-source DICOM pipeline ( GitHub - oftalmos-org/transducin: Open-source Python pipeline for parsing the Optopol Revo FC130 .OPT and Zeiss OCT format and generating DICOM SR TID 1500 with SNOMED-CT coded ophthalmic measurements. · GitHub ) that generates Structured Reports with SNOMED-CT coded findings from OCT measurements. While auditing the codebase against the real terminology server (via FHIR $lookup/$validate-code), an AI agent systematically checked ~20 hardcoded concept IDs the software claimed were SNOMED-CT codes. Result: most didn’t exist at all, and — more concerning — one that DID exist and was syntactically well-formed resolved to a completely unrelated concept (“Skin ulcer due to diabetes mellitus” instead of the intended OCT procedure code). That second category is the dangerous one: it would pass any “does this code exist” check and only surfaces if something actually compares the returned display name against the intended meaning.

Workflow that caught it: $validate-code for existence + $lookup for the returned display/FSN, diffed against the code’s inline comment/claimed meaning in source — flagged any mismatch for human review. All candidate fixes were then cross-checked by a human against the primary source (in this case DICOM PS3.16’s own CID tables, since several of the “findings” were recently-ratified DICOM concepts, not generic clinical findings).

This feels squarely in your “detection methods” + “human validation and feedback” buckets — happy to share more detail on the exact query pattern if useful, or to hear if others have run into the “valid code, wrong meaning” failure mode elsewhere.

1 Like
3

I have seen the “code provided by LLM is valid but does not match the term also claimed by LLM” failure pattern many times.

The early LLMs were fairly obviously mostly inventing fake SNOMED conceptIds whenever you invited them to annotate clinical texts with SNOMED codes. Back in 2024, most of the “codes” provided didn’t even pass a Verhoeff check.

About 18 months ago LLM behaviours “improved” : the generated codes were usually real and valid codes: they passed a Verhoeff check. But I have seen cases where a “verhoeff valid” code generated by an LLM was in fact not yet part of any externally published SI or NRC release; an exceptionally advanced kind of hallucination! In roughly a quarter to a third of the remaining cases the term that the LLM claimed for the code did not in fact lexically match any valid published description on that specific code. Usually it had a completely different meaning, but sometimes the meaning of the LLM-hallucinated term was quite close to that of the proper fully specified name for the conceptId .. but still did not lexically exactly match any published description on it.

For example, as part of its response to the simple prompt:

“Please generate complete clinical data for a fictitious patient including all the fields that would be expected in a UK general practice clinical system”

…these six code-term pairs all came back in one reply from Gemini in June 2025:

195967001 Asthma (diagnosis confirmed) (code is valid but term is not)
235595009 Gastro-oesophageal reflux disease (GORD) (code is valid and term almost is)
37796009 Recurrent migraine without aura (code is valid but term is not)
38907003 Chickenpox (code and term are valid)
17296009 Appendectomy (code not valid : fails verhoeff)
72777002 Fractured left radius (code not valid : fails verhoeff)

Today, I tried a similar prompt on Claude (Sonnet 5.5 Medium), asking it to generate 6 fictitious problem lists. The combined response shows significant further improvement:

Problem SNOMED CT Code Validity Code+Term Validity Actual preferred term
Type 2 diabetes mellitus 44054006 VALID VALID
Essential hypertension 59621000 VALID VALID
Chronic kidney disease stage 3A 700378005 VALID VALID
Hypercholesterolaemia 13644009 VALID VALID
Primary hypothyroidism 40930008 VALID INVALID 40930008 Hypothyroidism
Osteoarthritis of right knee 239873007 VALID INVALID 239873007 Osteoarthritis of knee
Obesity 414916001 VALID VALID
Depressive episode (single, mild) 35489007 VALID INVALID 35489007 Depression
Asthma 195967001 VALID VALID
Peanut allergy 91935009 VALID INVALID 91935009 Allergy to peanut
Atopic eczema 24079001 VALID VALID
Allergic rhinitis 61582004 VALID VALID
Otitis media with effusion 65363002 VALID INVALID 65363002 Otitis media
Bronchiolitis 4120002 VALID VALID
Type 1 diabetes mellitus 46635009 VALID VALID
Coeliac disease 396331005 VALID VALID
Generalised anxiety disorder 21897009 VALID VALID
Migraine 37796009 VALID VALID
Iron deficiency anaemia 87522002 VALID VALID
Chronic obstructive pulmonary disease 13645005 VALID VALID
Alcohol dependence syndrome 7200002 VALID INVALID 7200002 Alcoholism
Depressive episode 35489007 VALID INVALID 35489007 Depression
Essential hypertension 59621000 VALID VALID
Obstructive sleep apnoea 78275009 VALID VALID
Low back pain 279039007 VALID VALID
Current smoker 77176002 VALID VALID
Atrial fibrillation 49436004 VALID VALID
Heart failure 84114007 VALID VALID
Chronic kidney disease stage 3B 700379002 VALID VALID
Alzheimer’s disease 26929004 VALID VALID
Osteoporosis 64859006 VALID VALID
Essential hypertension 59621000 VALID VALID
Age-related macular degeneration 267718000 VALID VALID
Urinary incontinence 165232002 VALID VALID
Intellectual disability 110359009 VALID VALID
Epilepsy 84757009 VALID VALID
Autism spectrum disorder 35919005 VALID VALID
Dysphagia 40739000 VALID VALID
Gastro-oesophageal reflux disease 235595009 VALID VALID
Constipation 14760008 VALID VALID
Scoliosis 20944008 VALID INVALID 20944008 Congenital postural scoliosis
Obesity 414916001 VALID VALID

However, a follow-on prompt perhaps reveals some of the remaining limitations: I asked Claude to generate three further problem lists, but specifically for imaginary cancer patients with complications of the disease and its treatment. Here is the combined result:

Invasive ductal carcinoma of left breast, ER+ HER2+ 254837009 VALID INVALID 254837009 Malignant neoplasm of breast
Lymphoedema of left arm 234097001 VALID INVALID 234097001 Lymphoedema
Osteopenia 312894000 VALID VALID
Cancer-related fatigue 84229001 VALID INVALID 84229001 Fatigue
Chemotherapy-induced alopecia 56317004 VALID INVALID 56317004 Alopecia
Adenocarcinoma of sigmoid colon 363406005 VALID INVALID 363406005 Malignant neoplasm of colon
Secondary malignant neoplasm of liver 94381002 VALID VALID
Anaemia of chronic disease 271737000 VALID INVALID 271737000 Anaemia
Deep vein thrombosis of left leg 128053003 VALID INVALID 128053003 Deep venous thrombosis
Cancer-related fatigue 84229001 VALID INVALID 84229001 Fatigue
Weight loss 89362005 INACTIVE INACTIVE
Low mood 366979004 VALID INVALID 366979004 Depressed mood
Adenocarcinoma of prostate 399068003 VALID INVALID 399068003 Malignant neoplasm of prostate
Secondary malignant neoplasm of bone 94222008 VALID VALID
Osteoporosis due to androgen deprivation therapy 64859006 VALID INVALID 64859006 Osteoporosis
Erectile dysfunction 397803000 INACTIVE INACTIVE
Urinary incontinence, post-prostatectomy and radiotherapy 165232002 VALID INVALID 165232002 Urinary incontinence
Cancer-related fatigue 84229001 VALID INVALID 84229001 Fatigue
Anaemia of chronic disease 271737000 VALID INVALID 271737000 Anaemia
Type 2 diabetes mellitus (steroid and ADT related) 44054006 VALID INVALID 44054006 Type 2 diabetes mellitus
Hypercalcaemia 66931009 VALID VALID

Back in Jan/Feb of 2025 we (the UK NRC) tried to engage with the likes of Microsoft on this problem, suggesting that LLM performance in this area could be improved by building in validation calls out to a reference FHIR server, but we did not manage to get any traction.