Using Language Models to Summarize PET Reports

Radiologists create reports to explain the results of medical imaging tests. These reports are crucial for sharing important information about a patient's condition with other doctors and the healthcare team. Among various imaging tests, whole-body PET scans are known for being longer and more complex. In a PET report, the Findings section lists many observations from the scan, while the impression section gives a summary of the most important points. Since other doctors rely heavily on the impression section for treatment decisions, it must be both accurate and complete. However, writing these Impressions can take a lot of time and may lead to mistakes. Large language models (LLMs) offer a new way to speed up this process by automatically writing impressions based on the findings.

#Background

While LLMs have been used to summarize findings from various imaging tests, they have not been widely applied to whole-body PET reports. PET reports are significantly longer than those for other tests, often containing 250 to 500 words in the findings section. This length brings challenges, as there is a higher chance of missing key information during impression generation. Moreover, individual doctors have different styles of reporting that need to be taken into account for more personalized results. Adapting LLMs to summarize PET reports involves specific challenges.

Evaluating the success of LLMs in producing these impressions is also tricky, as there can be many valid ways to summarize the same information. Expert Evaluation is considered the best way to assess quality, but it is not practical for doctors to review the output of every model. To tackle this, recent studies have developed evaluation metrics to measure how well these models summarize medical documents. However, it has not been determined how effective these metrics are when it comes to PET impressions and how closely they align with the views of Physicians.

#The Study

The aim of this study was to see if LLMs trained on a large number of PET reports could accurately summarize findings and create impressions for practical use. Researchers trained 12 different language models using a Dataset of PET reports and assessed their performance using various evaluation metrics. The best-performing model was then tested for its ability to produce clinically useful impressions.

#Dataset Collection

A total of 37,370 PET reports collected from one hospital between 2010 and 2022 were used in the study. These reports were anonymized to protect patient information. The data was divided into groups for training, validation, and testing. An additional 100 reports from a different source were also collected for external testing.

#Report Preprocessing

Two types of language models were tested: encoder-decoder models and decoder-only models. The first set required specific formatting, where the first lines included details about the scan and the doctor's identity. The second type used a different approach, starting with an instruction that asked the model to generate the impression based on the given report. The actual clinical impressions from the reports were used for model training and evaluation.

#Language Models for PET Reports

The study focused on summarization, wherein the models are expected to interpret findings instead of just repeating parts of the findings section. Researchers trained multiple encoder-decoder models and decoder-only models to see which ones performed best in generating accurate impressions. The fine-tuned models were then assessed using various evaluation metrics to identify the model that performed the best for expert assessment.

#Evaluating Performance

To determine which evaluation metrics correlated best with physician preferences, the researchers presented model-generated impressions to two physicians who rated them. The metrics with the strongest correlation were used to select the top-performing model.

In the expert evaluation phase, three nuclear medicine physicians reviewed a total of 24 reports, assessing the quality of model-generated impressions. They used specific criteria to rate these impressions and also compared them to impressions originally written by themselves and other physicians.

#Results

The study found that two metrics, named BARTScore and PEGASUSScore, had the highest correlations with physician preferences. The PEGASUS model was identified as the top performer. When physicians reviewed the impressions generated by PEGASUS in their own style, a significant number were considered clinically acceptable, suggesting that this model can produce useful outcomes for real-world applications.

When physicians evaluated the impressions generated for their own reports, 89% were scored as clinically acceptable. However, the average utility score was slightly lower than the impressions they originally wrote. This difference was attributed to areas needing improvement, such as factual correctness and clarity.

Furthermore, when evaluating impressions from other physicians, the scores were again lower than those of their own work, highlighting the strong preference that physicians have for their unique reporting styles. Despite the slight differences in satisfaction levels, the overall utility of PEGASUS-generated impressions was considered comparable to those from other physicians.

#Challenges Faced by the Model

While the majority of the impressions generated by PEGASUS were acceptable, some common issues were identified. Factual inaccuracies were a frequent problem, with instances of misinterpretation evident. Additionally, the model sometimes produced diagnoses that were overly confident without sufficient evidence. Recommendations made by the model could also be vague, making it difficult for physicians to use them in clinical practice. These issues emphasize the need for thorough review and editing by physicians before finalizing reports.

#Limitations of the Study

Several limitations were identified throughout the study. For example, when fine-tuning some models, only a simple method of domain adaptation was used due to limited computational resources. The study also only manipulated one element of the input to adjust the style of generated impressions, leaving other potential methods unexplored.

External testing showed a noticeable drop in evaluation scores, suggesting that differences in reporting styles between the internal training group and external physicians affected performance. Lastly, since the dataset came from a single institution, future research should aim to involve multiple institutions to enhance the findings.

#Conclusion

This study examined how large language models could automate the generation of impressions for whole-body PET reports. The results indicated that the best-performing model, PEGASUS, can create personalized and clinically useful impressions in most cases. Given its performance, the model could be integrated into clinical settings to help speed up PET reporting by automatically preparing initial impressions based on the findings available.

The study acknowledges support from various funding sources, while also making clear that the views expressed in the work are those of the authors and do not necessarily reflect the positions of any sponsoring organization.

In conclusion, while challenges remain, the potential for LLMs to improve the process of creating medical reports is promising and can lead to better efficiency in healthcare settings.