AI in Polish Healthcare: Examining LLM Performance

In recent years, artificial intelligence (AI) has made significant strides in various fields, including healthcare. Large Language Models (LLMs) are particularly noteworthy for their ability to address complex tasks. However, much of the existing research emphasizes English-language contexts, leaving a gap in understanding AI's performance in other languages, particularly in specialized areas like medicine.

To bridge this gap, a new benchmark dataset was created based on medical licensing and specialty exams in Poland. This dataset consists of various medical exams that assess the knowledge of medical doctor candidates and practicing doctors pursuing further qualifications. It aims to evaluate LLMs' capabilities in understanding medical questions in Polish and to facilitate Cross-lingual medical knowledge transfer.

#What Are Polish Medical Exams?

Poland conducts several exams for physicians and dentists, including:

1. LEK (Lekarski Egzamin Końcowy) - Medical Final Examination
2. LDEK (Lekarsko-Dentystyczny Egzamin Końcowy) - Dental Final Examination
3. LEW (Lekarski Egzamin Weryfikacyjny) - Medical Verification Examination
4. LDEW (Lekarsko-Dentystyczny Egzamin Weryfikacyjny) - Dental Verification Examination
5. PES (Państwowy Egzamin Specjalizacyjny) - National Specialization Examination

These exams are crucial for graduates to obtain medical licenses and ensure they have the necessary knowledge and skills to practice medicine safely and effectively.

#Dataset Content

The newly created dataset comprises over 24,000 questions from LEK, LDEK, and PES exams. The questions cover a wide range of medical topics and specialties, making it a comprehensive resource for evaluating LLMs. Some of the questions are also available in English, having been translated by professionals for foreign candidates.

These questions were collected from publicly accessible resources offered by the Medical Examination Center and the Chief Medical Chamber. The dataset underwent a thorough cleaning process to ensure its quality and relevance for LLM evaluation.

#Assessing LLM Performance

A systematic evaluation was conducted on various LLMs, including general-purpose and Polish-specific models. The aim was to compare their performance against human medical students.

#Key Findings

One noteworthy finding is that models like GPT-4o performed almost as well as human students. However, challenges remain, especially in cross-lingual translation and domain-specific knowledge in medicine. This emphasizes the importance of understanding the limitations and Ethical concerns surrounding the use of LLMs in healthcare.

#Why Language Matters

LLMs trained in multilingual Datasets often perform better when given prompts in English than in other languages. This can lead to discrepancies in their ability to handle medical scenarios that may be common in one language but not in another. For instance, medical training in Poland may focus on conditions prevalent in the local population, which could vary widely from those in English-speaking countries.

#Local Considerations

Medical education is often tailored to the health issues affecting the local community. For example, a medical student in Poland might learn extensively about tuberculosis, which is prevalent, while a student in another country might focus more on chronic diseases. This localized training can affect an LLM's ability to provide accurate medical insights when dealing with questions from different cultural and epidemiological contexts.

#Data Collection Methods

The data for this project was collected using web scraping techniques from the Medical Examination Center and the Supreme Medical Chamber. A combination of automated tools was employed to extract the exam questions in both HTML and PDF formats, and to process this data for analysis.

#Challenges Encountered

Collecting data came with its own set of challenges. PDF files, for example, posed difficulties as their structure could vary greatly. Some were well-formed and easily readable, while others resembled scanned documents and required extra effort to extract text.

Moreover, certain resources had incomplete data, which necessitated extensive filtering to ensure that the questions used for the dataset were reliable and relevant.

#Performance Comparison

The models were tested on various exam questions, and their results were expressed as a percentage of correct answers. The models were grouped into categories, such as medical LLMs and general-purpose multilingual LLMs.

#Notable Performers

GPT-4o was identified as the top performer in the evaluated models. However, it was found that general-purpose models often outperformed medical-specific models, possibly due to the latter being primarily trained on English medical data.

In terms of Polish-specific LLMs, performance varied, but they were generally less effective than the top general-purpose models.

#Specialty Performance Insights

An interesting aspect of evaluating these models was the discovery of which medical specialties presented more of a challenge. For example, models struggled with dental specialties like orthodontics, while performing better in areas such as laboratory diagnostics. This discrepancy highlights that while LLMs can be useful, they are not perfect.

#Cross-Lingual Performance

The analysis of LLM performance revealed that they generally performed better on English versions of exam questions than on their Polish counterparts. This highlights a persistent issue in the field and emphasizes the critical need for better multilingual training approaches.

#Polish vs. English: The Results

In side-by-side comparisons, models typically outperformed on English questions. For instance, a model that barely passed a Polish exam might ace the equivalent English version. However, some models showed promising developments, as the gap between Polish and English performance narrowed with advancements in technology.

#Comparison with Human Results

To further validate the findings, the performance of LLMs was compared against human students' results from recent LEK and LDEK sessions. The models’ scores were evaluated in relation to the average student scores to see how well they matched up.

#Key Takeaways

Overall, while many models performed well, it was evident that LLMs cannot currently replace comprehensive medical training and clinical experience. The nuances of patient care extend far beyond multiple-choice exams, demanding deeper understanding and human interaction that AI cannot fully replicate.

#Ethical Considerations

With the promise of LLMs comes a responsibility to ensure they are used ethically in a medical context. The potential risks, such as misinformation and misdiagnosis, are serious concerns. Consequently, any application of LLMs in healthcare requires careful oversight by qualified professionals to ensure that the information provided is accurate and reliable.

#Conclusion

The development of this Polish medical exam dataset is a significant step forward in understanding AI's capabilities in healthcare. This research not only sheds light on how well LLMs can perform on medical questions but also highlights the areas that need further improvement. While they can provide valuable support, LLMs should not be viewed as replacements for human doctors but rather as helpful tools that can assist medical professionals in their work.

In the evolving landscape of healthcare, where technology and human expertise need to coexist, there's a lot of room for growth, collaboration, and maybe even a touch of humor—after all, laughter is good medicine!