On this research, we developed an NLP pipeline to automate referral prioritization for sufferers with musculoskeletal problems based mostly on their GP letters. Utilizing solely the contents of the letters, we precisely predicted sufferers with RA, osteoarthritis, and fibromyalgia previous to their first secondary care go to on the rheumatology outpatient clinic. We validated the ML-models throughout a number of hospitals, displaying the fashions for particular person prognosis carry out effectively no matter every hospital-specific construction. Solely the heterogenous group of sufferers with persistent follow-up proved to be difficult to foretell. Nonetheless, all of the ML fashions improved the referral order in comparison with the guide referral system, putting the high-risk sufferers to the entrance of the road.
We imagine this AI-assisted affected person triage exhibits important promise for decreasing wait instances and assuaging clinician workload. In lots of Western nations, together with the Netherlands, an growing older inhabitants and fewer specialists – particularly in peripheral areas – exacerbate delays4,5. Moreover, many sufferers are referred to incorrect care pathways, additional delaying entry to rheumatologists. Consequently, sufferers with MSK signs typically really feel trapped in a “limbo” throughout the healthcare system18.
Healthcare suppliers might implement this pipeline19 to enhance the prioritisation of referral letters based mostly on estimated urgency. Particularly, the RA mannequin could possibly be used to create a “fast-track” pathway to the specialist for early intervention with immunosuppressants. In the meantime, non-autoimmune circumstances like fibromyalgia and osteoarthritis could possibly be directed to various care pathways, reminiscent of occupational- or physiotherapy.
Apart from secondary care, this pipeline additionally exhibits promise for main care. The mannequin’s skill to foretell the right prognosis, even when the GP didn’t explicitly point out it within the referral cause, demonstrates its worth as a supportive software for GPs. Based mostly on the outline of the affected person’s situation it might assist GPs determine when to order further assessments or prioritize referrals. Nevertheless, future analysis ought to consider the mannequin’s efficiency on all sufferers thought-about for referral, not simply these really referred. Since we chosen circumstances based mostly on ultimate diagnoses to make sure satisfactory illustration, we couldn’t assess GPs’ general predictive accuracy.
A key energy of our research is its use of referral letters, that are routinely collected in lots of healthcare methods as a part of normal care9. These letters supply important insights right into a affected person’s situation at early phases of the illness, enabling well timed detection and intervention. That is notably essential for RA, the place early therapy through the “window of alternative” can considerably enhance affected person outcomes20,21.
Moreover, our fashions present chances for a number of ailments, providing flexibility to align with the particular triage priorities of various clinics. The RA mannequin, with sturdy calibration (Brier rating = 0.06), is particularly interpretable, as its thresholds carefully mirror true RA prevalence. Whereas these fashions will not be designed to exchange a specialist’s prognosis, they function helpful instruments to help clinicians by guiding diagnostic and therapeutic selections based mostly on their outputs.
Pure language processing and enormous language fashions stay underutilized within the discipline of rheumatology, regardless of their potential to analyse the huge quantities of scientific textual content knowledge13,22. Whereas Krusche et al. utilized LLMs for triage14, they didn’t study the worth of referral letters. Wider adoption of LLMs could also be hindered by issues about knowledge privateness and lack of infrastructure for internet hosting23. That mentioned, referral letters have been analysed with superior NLP methods previously—for instance, to distinguish inflammatory from non-inflammatory ailments16,17 or to triage sufferers with decrease again ache15. Nevertheless, these research didn’t embody exterior validation, limiting their generalizability.
Our challenge is the primary to create a generalizable NLP pipeline for predicting and prioritizing diagnoses from referral letters for the rheumatology outpatient clinics. The SHAP-analysis exhibits that detailed data on suspected diagnoses, serology, lab outcomes, therapies, and particular symptom descriptions, reminiscent of joint places, are probably the most helpful options for classification. This aligns with the structured referral letter framework proposed by Engels et al.24. However, we do wish to emphasize the significance of validating and optimizing fashions regionally to make sure optimum efficiency on the native referral letters25. In idea, our pipeline must be adaptable throughout languages, because it makes use of methods (such because the BERT transformers) out there for many languages. Nevertheless, additional investigation is required to judge the transferability of our strategy throughout completely different healthcare methods, given the doable variations in GP roles and triage protocols between nations9.
You will need to acknowledge that we didn’t examine various machine studying methods or carry out sensitivity analyses on this research. Whereas we targeted on fine-tuning XGBoost on account of its established effectiveness in textual content classification26,27,28, exploring different strategies or textual content processing approaches, reminiscent of lemmatization or stemming, might doubtlessly improve efficiency29.
We goal to combine ML classifiers into a choice assist software for triaging sufferers and aiding GPs in choosing applicable referral targets. Constructing a choice assist software enters into the transdisciplinary discipline of human centred AI methods30, requiring collaboration with all stakeholders (sufferers, clinicians, GPs). Quite than optimising solely on metrics like AUC-ROC or the F1-score, the main focus will lie on optimising human centric outcomes (reminiscent of person expertise, explainability, transparency and equity). Ideally, the software will embody visualizations to boost clinicians’ understanding of mannequin reasoning—for instance, highlighting sections of the narrative indicating musculoskeletal complaints (by way of SHAP evaluation) or evaluating a affected person’s narrative with these of previous sufferers to seek out similarities. We additionally suggest evaluating common wait instances for various affected person teams and screening effectivity in a pilot research.
Figuring out persistent sufferers—these requiring ongoing rheumatologist care—was difficult, doubtless because of the group’s heterogeneity. In comparison with GPs, the ML mannequin struggled extra to foretell sufferers needing follow-up care past three months. Nevertheless, since GPs can request rheumatologists to take over care, their increased efficiency is considerably anticipated. Moreover, our findings counsel that GPs alter their writing type when transferring look after fibromyalgia sufferers. SHAP evaluation revealed that GPs typically use extra formal and courteous language (e.g., “collegial regards” or “finest regards”) in these referral letters.
Attributable to low illness prevalence, we used AUC-PRC alongside AUC-ROC, as AUC-ROC alone will be deceptive for imbalanced datasets31. AUC-PRC exhibits how exact the choice is at particular sensitivity ranges. For RA, which has a low prevalence of 8-10%, the mannequin constantly achieved the next AUC-PRC in comparison with the identified prevalence, indicating that it prioritized precise circumstances extra successfully than random probability would counsel. In distinction, the “persistent” class additionally achieved the next AUC-PRC, however this was much less spectacular on account of its increased prevalence within the completely different hospitals (53–55%), which inflates the AUC.
The truth that the ML fashions reached a very good efficiency doesn’t per definition translate to an enchancment in the actual world affected person triaging, as that is all the time depending on varied elements aside from efficiency. First, the binarized ML classifier strategy might not mirror real-world situations the place sufferers typically have a number of diagnoses. Second, it’s unclear how customers will work together with automated triage methods or how a lot management they may need over them.
We acknowledge a number of limitations with the info. First, the efficiency might doubtless be elevated if we had extra knowledge, provided that GP letter high quality varies on account of inconsistent formatting12,32 you will need to have a big consultant pattern dimension. Whereas we lacked GP-specific knowledge, our research included referrals from ~300 practices throughout a number of Dutch provinces, most likely capturing an honest vary of letter high quality.
Second, the codified diagnoses used as outcomes could also be misclassified. Though the respective ICD codes usually stay constant all through a affected person’s follow-up, we discovered that roughly 5% of recognized circumstances finally obtain a second codified prognosis, suggesting both twin circumstances or preliminary misclassification. Guide knowledge cleansing might have improved diagnostic accuracy and mannequin efficiency. Lastly, our research doesn’t considerably handle diagnostic delays for uncommon ailments33. Whereas our fashions improve effectivity for widespread musculoskeletal complaints, coaching uncommon illness fashions was infeasible on account of inadequate knowledge.
In conclusion, utilizing GPs’ referral letters solely, we constructed AI-tools that would enhance prioritisation for sufferers with RA, Osteoarthritis and Fibromyalgia, earlier than they even visited a specialist. For all classification duties, we have been in a position to enhance go to prioritisation in comparison with the present ready time. This demonstrates its potential to optimise care effectivity. Using ML based mostly referral prioritisation might each cut back workload of clinicians in addition to facilitate early devoted care presumably in specified care pathways. Future analysis is required to translate the fashions into a whole determination assist software for scientific use.
