DescriptionType length limit increase proposal

SNOMED Groups Terminology Release AG
,
1

  • Back in October 2023, the MAG agreed that increasing the DescriptionType spec to allow longer terms in Descriptions would be a good idea - see item 4 here: 2023-10-24/25 Full MAG Atlanta Hybrid Meeting

  • FYI - In the meantime, NZ require a new limit for a handful of terms, and so we extended their DescriptionType in the April 2024 NZ release, in order to allow for slightly longer terms locally until the International spec has been updated.

  • Because the spec already allows us to increase the length of the field, the concern is predominantly for impact to end implementers.

  • It’s mostly, therefore, a question of warning implementers that the status quo is going to change

    • We have therefore started a full community consultation in 2024, in order to garner feedback from anyone who may be impacted by the changes.

    • We will also ask if anyone has any issues with the length that we change it to, as we can foresee terms breaching 512 characters (with vaccine product with 10 ingredients for example) - so 1024 would be the next obvious target. However, at that point it’s not clear why we don’t just jump to 4096, given that we already have that configured for textDefinitions… otherwise we could end up having to extend it again within the next few years.

    • The potential problems are for end implementations who have local hard coding, or worse technical restrictions on imports, etc - but it feels like that’s going to be the same problem at 1024 as at 4096?

    • Therefore it would appear that the best plan would be to increase it from 255 to 4096?

  • Can anyone foresee any implementation issues? (other than providing a reasonable lead time to allow implementers to potentially change hard coded limits, etc?)

    • At the moment the only implementation issues we’re seeing are that several countries are finding terms that contravene the 255 spec (both in the INT Edition + the MS terms) and so only positive reinforcement for increasing the limits (eg) https://ihtsdo.freshdesk.com/helpdesk/tickets/49593

    • But could we foresee, perhaps, any issues with implementers who might have

      • a) hardcoded the 255 limit and could take a long time to get it changed (especially given how long the limit has remained the same), or

      • b) still be running systems that can’t actually cope with >255 characters even if the implementers want to increase the limit?

    • The reason we ask this is that if terms are concatenated (especially Drugs, etc) by out-dated systems, then this could cause Clinical Risks, which we want to avoid at all costs.

    • So perhaps we need a lengthy Community Consultation period (say 6-9 months) to socialise the change before we implement?

      • This would be okay for the current known offenders, as:

        • a) The International terms have successfully been reduced down to comply with the 255 limit for now

        • b) The MS customer who had longer terms specialised their DescriptionType format to 4096 in their own extension - the only issue here of course is global interoperability, but this is a lower risk for now than contravening our own specs.

  • .

  • In summary the feedback so far has been:

    • We have received feedback from 15 entities (both NRC’s + organisations)

    • The vast majority are fully supportive of the proposed changes, and have in fact already signed off the changes

    • This is despite the fact that more than half are expecting to have to make changes to systems/processes in order to accommodate the new limits

    • There are some reservations, which users have asked to be considered before implementing the changes - including (but not limited to):

      • Need to be able to support the change in Postgres, Oracle and Snowflake DBMSs, plus SQL, Java and common programming languages

      • Due to potential impact to existing systems, they hope that SNOMED will keep system limitations in mind and avoid lengthy FSNs to the extent feasible.

      • Potential issue with human-selection of codes if FSN’s increase significantly in length

      • Potential impact to existing API’s, in particular import and upload end points

      • Potential to overload user interfaces

      • *However, many of these issues can be addressed by the fact that SNOMED International have stated that despite the increase in the maximum length, there is currently no intention to significantly increase the length of descriptions

    • Most users are comfortable implementing the changes within the next 12 months, with 3 stating that they will require more than 12 months to make the necessary internal preparatory changes

    • However, the feedback becomes slightly less positive when going down below the NRC level - the UK for example have received 34 responses from their end users, 13 of which will require more than 12 months to make the necessary internal preparatory changes.

    There are still several months to go until the feedback deadline, so the picture formed so far may change, but this is where we stand at this point in time.

    The key point that is coming through so far on the feedback is that the important caveat that we stated in the announcement will resolve most concerns raised:

    It is important to note that there is absolutely no intention to increase the length of a multitude of the International Descriptions. The proposal is simply to increase the maximum size to 4096, in order to allow a small number of necessarily longer terms to be consumed. However, the vast majority of Descriptions will remain at their current length, well below 255 characters.

  • ANY OTHER FEEDBACK, PLEASE PROVIDE IT VIA THE FEEDBACK FORM BEFORE THE UPDATED DEADLINE OF 31st DECEMBER 2024: https://docs.google.com/forms/d/1cBxZnJDV5cy_5hkWaCc02xXhFaz8-2c6JI6yppQ8QUE/viewform?edit_requested=true

  • SHOULD WE ALSO CONSIDER A MORE CAUTIOUS APPROACH? Moving to a limit of say 1024 first, and then building up?

  • OR SHOULD WE move straight to 4096, but then provide a list of all actual terms above 255 in the International Edition so that consumers know exactly what to expect?

  • OR IS IT REASONABLE to expect users to be able to cope with moving to a MAXIMUM of 4096, in the knowledge that most terms will no go anywhere near that length?

  • HOPEFULLY YOU ALL SAW THE COMMS GO OUT SUMMARISING THE CONSULTATION PROCESS RESULTS IN MARCH?

  • SNOMED International Content team was asked to confirm how many FSN’s (especially in the Drug content) in the International edition are pushing up and above the existing 255 limit -

    • The project lead advised that the current content is usually truncated with a comma if it exceeds the 255 limit.

    • A report for content that may use a comma to reduce the character count was run and it looks like a very small number (approx. 17) - however there is no accurate way of identifying the exact numbers as this truncation is historical and the issue that arose with https://ihtsdo.freshdesk.com/a/tickets/49593 only occurred because the descriptions were being modified for a different reason.

    • The DROOLS warning (which had been switched off for awhile on the international platform in response to a request from a member country) will be reinstated for 255 characters in the international release until the increase in character numbers is agreed https://projects.jira.snomed.org/browse/MAINT-2532

  • This confirms that there is no urgency for this change from an International perspective, and Extensions can continue to specialise the config in their own DescriptionType refset where required.

  • WE HAVE THEREFORE APPLIED a suitably lengthy Transition Period for the Community, given the high potential for impact to end users.

  • THE PLANNED MIGRATION DATE IS THEREFORE JULY 2026…

  • NOTE: WE ALSO NEED TO TAKE THE OPPORTUNITY TO STANDARDISE THIS ACROSS ALL PRODUCTS (EXTENSIONS, SPANISH EDITION, DERIVATIVES + ALL OTHER SI PRODUCTS) TO AVOID A SIMILAR CONSULTATION in future!! (as ALL our products would move to 4096)

    • ANY CONCERNS WITH THIS PRODUCT TRANSITIONING AS WELL?

  • EVERYONE AGREED -

  • GUILLERMO would like us to move to this sooner for Spanish Edition

  • AAT to publish proposal comms to transition in November 2025 - with SHORT feedback window

  • Then let Guillermo know in Summer, so we can start planning transition in Nov 2025

The new Briefing Note has been published in the new Forums space:

This contains the conclusions, plans for the next steps, and also answers community queries on the roles & responsibilities for NRC’s.

We’ve also updated the blog in line with this, so that we ensure thorough comms coverage for the community:

If you have any questions at all please don’t hesitate to ask (eg)

  • Should we disseminate this information out now to end users?

  • Is the level of SI comms appropriate for the changes being made? (or too much/too little?)