AI Models Improve Patient Understanding Post-Hospital Stay

Patients often struggle to understand what happens during their hospital stays and what they need to do after leaving. Doctors and healthcare staff typically have limited time and resources to explain everything. This study looks into how large language models, which are AI tools, might help create summaries for patients based on doctors' notes. We also examine how different types of training data affect the accuracy and quality of these summaries.

#The Problem with Patient Understanding

After being in the hospital, many patients find it hard to remember their diagnosis and what follow-up appointments they need. Research showed that less than 60% of patients could correctly explain their diagnosis, and even fewer knew the details of their follow-up care. Better communication about discharge instructions can help reduce readmission to hospitals and improve patients' adherence to treatment plans. That's where patient summaries come in-they're meant to communicate important information clearly and simply.

However, writing good summaries isn't easy, and healthcare professionals often have heavy workloads. Large language models have shown promise in summarizing medical information but can produce incorrect or misleading information, known as "hallucinations." This is especially problematic in healthcare, where patient data is often fragmented and may not provide a complete picture.

#Goals of This Study

In this research, we focus on finding ways to produce better patient summaries using AI while minimizing the chances of Inaccuracies. We developed a labeling system to identify mistakes in the summaries and had medical experts review both real and AI-generated summaries.

#Key Contributions

1. We created a dataset of patient summaries with notes taken from doctors.
2. We introduced a method for labeling inaccuracies in summaries and conducted evaluations on both real and AI-generated summaries.
3. We demonstrated that training AI models on cleaned data where inaccuracies were removed can reduce these mistakes while still keeping important information intact.
4. We performed a quality assessment showing that one of the AI models, GPT-4, often produced better summaries than human-made ones.

#Related Work

The demand for automated clinical summaries has increased due to the repetitive nature of medical documentation. Various studies have explored how AI can enhance clinical summarization. Findings indicate that models like GPT-4 are preferred over human-generated summaries in terms of accuracy. However, the issue of inaccurate or unsupported facts remains a concern.

Several methods for tackling inaccuracies have been investigated. One approach involves detecting errors after they have been made, while another focuses on improving the data used for training. Our study aims to address the problem by refining a small number of training examples to ensure higher quality output.

#Overview of Our Dataset

We created a dataset called MIMIC-IV-Note-DI from real patient summaries and corresponding doctor notes. This dataset includes around 100,175 hospital courses and patient summaries. We focused on the "Discharge Instructions" section as it provides crucial information for patients.

To improve the dataset's quality, we filtered out poor summaries and irrelevant content, resulting in two versions of the dataset: one with full context and another with a shorter narrative.

#Annotating Hallucinations

For our study, we examined how frequently incorrect or unsupported information appeared in patient summaries. We analyzed 100 real summaries, labeling a total of 286 inaccuracies. Most were unsupported facts, indicating a significant presence of errors when using the Short Context.

We also looked at the AI-generated summaries and found problems similar to those in real ones. This shows that the challenge of providing accurate information is widespread, regardless of whether it comes from humans or machines.

#Training the Models

We experimented with three AI models for creating patient summaries:

1. LED: A model designed for processing long documents. It was trained on the entire MIMIC-IV dataset but required significant resources.
2. Llama 2: We used two variations of this model to see how well it could summarize patient information after fine-tuning on cleaned data.
3. GPT-4: This model is recognized for producing high-quality summaries and was tested in two ways: using examples from our data and with no training examples.

#Evaluating Model Performance

We assessed each model's summaries based on various factors, including accuracy and quality. We used metrics such as ROUGE for measuring overlap between generated summaries and real ones.

The evaluations highlighted that LED performed the best in quantitative assessments, but GPT-4 excelled in qualitative aspects, particularly in delivering coherent and understandable summaries.

#Qualitative Assessment of Summaries

The generated summaries were examined for various quality measures:

• Relevance: How well the summary captured the important details.
• Consistency: Whether the summary contained accurate information per the original notes.
• Simplification: If the language used was easy to understand for patients.
• Fluency: The grammatical correctness of the sentences.
• Coherence: How naturally the sentences fit together as a whole.

The findings indicated GPT-4 produced summaries that were not only accurate but also more understandable for patients compared to the other models.

#Automatic Hallucination Detection

We also tested whether the models could automatically identify inaccuracies in summaries. The use of AI to spot errors is promising but presents challenges, as the models may struggle to recognize complex or subtle inaccuracies. While GPT-4 showed better results in this area, further improvements are necessary for fully reliable detection.

#Conclusion

This research highlights the potential of large language models to assist in creating patient summaries that are accurate and easy to understand. The results indicate that careful training with curated data can significantly reduce the number of inaccuracies while maintaining essential details. GPT-4 emerged as a strong candidate for generating high-quality summaries that can improve patient understanding and engagement.

Going forward, more research is needed on how to better incorporate patient feedback into summary generation and further explore the effectiveness of these summaries in clinical settings. A multidimensional approach that combines the strengths of AI and human expertise can lead to advances in patient communication and care.

#Future Work

Future studies should test these models across different formats and situations, as well as explore other AI models. Clinical evidence around the effectiveness of these patient summaries will also be essential in validating their use in real-world applications. Furthermore, expanding the research to include the patients' perspectives could lead to even more effective patient communication strategies.

This study demonstrates that, with the right data and methods, AI can play a crucial role in improving patient understanding of their medical situations, ultimately leading to better health outcomes.