Identifying Developmental Intellectual Disability as a SUDEP Risk Marker from Epilepsy Monitoring Unit Reports Using Large Language Models

Abstract

The recent advancement of Large Language Models (LLMs) provides great potential in Natural Language Processing (NLP) tasks in biomedicine. Sudden Unexpected Death in Epilepsy (SUDEP) could risk the lives of epilepsy patients, making the accurate identification of risk markers essential for epilepsy treatment and management. In this study, we developed a pipeline that employs LLMs to automatically identify the SUDEP risk marker: "Developmental Intellectual Disability, I.Q. < 70, or too impaired to test" from unstructured text in Epilepsy Monitoring Unit (EMU) reports. We experimented with four recent LLMs - GPT-4o, LLaMA 3 70B Instruct, Mixtral 8x22B Instruct, and Qwen2 72B Instruct - for this task. We applied the approach to 1,030 EMU reports originating from three different sites. The results showed that Qwen2 far outperformed the other LLMs with a Pseudo F1-score of 90.2% while LLaMA 3, Mixtral, and GPT-4o achieved 66.6%, 62%, and 54.3% respectively. The results indicated that LLMs are effective in this particular clinical Natural Language Processing task. The findings of this work can be leveraged to assess the SUDEP risk of patients so that preventive measures can be implemented early, personalized care plans can be developed, and overall patient outcomes can be improved.