Adapting DeBERTa for Electronic Health Records

In recent times, there has been a lot of work done on how to better use language Models to help with tasks involving electronic health records (EHRs). Our focus is on how we can adapt a specific language model called DeBERTa to work with EHR tasks. We want to see if this can improve our ability to predict Outcomes for patients in emergency departments.

#The DeBERTa Model and Datasets

To start, we trained a smaller version of the DeBERTa model on a dataset made up of discharge summaries, clinical notes, radiology reports, and medical abstracts. This dataset was sourced from MIMIC-III, which contains a wealth of health records. We compared the performance of our model to another similar model called MeDeBERTa, which had been pretrained on clinical texts from our health institution. We also compared it with XGBoost, which is another model commonly used for tabular Data.

We evaluated the models on three key tasks related to patient outcomes in emergency departments. This evaluation was done using a different dataset known as MIMIC-IV-ED. Before creating these models, we had to convert the data into a text format. During this process, we created four different versions of the original datasets to see how the way we processed the data could affect the model's performance.

#Performance and Results

Our results showed that the model we proposed did better than the others in two of the three tasks, and it performed similarly in the third task. One key finding was that when we used clearer column names in our data, the performance improved compared to using original names.

The use of tabular data is critical in many real-world situations. Tables are common for organizing data like internet traffic, scientific experiments, and information from clinical settings. Traditional machine learning techniques often struggle with unstructured data, which has led to the creation of methods to convert this data into tables.

However, while converting unstructured data into tabular formats, some important information may be lost. For instance, in healthcare, data might include free text notes about medications, diseases, and lab results. When this information is processed into tables, it risks losing the complete context that free text provides.

#Free Text and Tabular Data

In our approach, we looked into whether keeping the original free text data could enhance the performance of our models when predicting outcomes. We also examined various strategies for managing numeric data.

We noticed that many studies have started to look at how language models like BERT can be adapted for tabular data by treating the data as strings of text. Several recent models have shown promising results using this method, and we are trying to build on this foundation.

Our work also addressed known limitations in using language models with numeric data. Some earlier findings suggested that language models trained to recognize numbers can only do so accurately within certain ranges. This limitation can lead to significant errors when they encounter numbers outside their training range.

#Model Training and Evaluation

To evaluate our model's effectiveness, we created benchmark tasks designed to predict patient outcomes. For instance, we wanted to find out if a patient would be admitted to the hospital after visiting the emergency department or if they would need urgent care.

Each task involved fine-tuning the models separately, which allowed us to measure how well they performed. We trained the models over 20 epochs, saving the best versions based on their performance against a validation set.

The models were assessed using specific metrics that indicate their prediction accuracy. We calculated the area under the receiver operating characteristic curve (AUC) to measure how well each model performed. We also examined the impact of different data processing techniques on model performance.

#Importance of Data Processing

Our findings highlighted the importance of how we process data. Using descriptive column names and keeping free text data improved the model's ability to make correct predictions. This is particularly crucial in medical settings where the details in patient records can significantly impact their care.

By incorporating various forms of data, we can get a more complete understanding of the factors affecting patient outcomes. This combination of free text and structured table data can lead to better predictions.

#Clinical Applications

The implications of our work are quite significant. We demonstrated that even small language models can compete with larger ones, making them suitable for settings like hospitals where computing resources may be limited. A large model such as GPT-J requires a lot of memory, while our adapted DeBERTa model needs much less.

In terms of clinical value, understanding which features in the data are most influential can provide insights into patient care. For example, our analysis showed that the free text notes about patients were crucial in predicting hospitalization outcomes.

This information can help healthcare professionals focus on the right aspects of a patient's health to make better decisions about their care. Identifying key risk factors through our models can also lead to improved treatments for patients, particularly in managing medications and understanding their medical history.

#Moving Forward

Despite the positive findings, there are still limitations to our approach. We haven't yet tested it across a wide variety of tasks or compared it directly to much larger models. Future work will involve testing our methods on more tasks and against larger models to fully assess its capabilities.

Our work lays the groundwork for future research in adapting language models for tasks related to electronic health records. We hope that more effective prediction models can lead to better outcomes for patients in hospitals.

#Conclusion

In summary, our study shows that we can successfully adapt the DeBERTa model for tasks using electronic health records. We've demonstrated that our approach performs well in predicting outcomes in emergency departments and highlights the importance of how we prepare the data.

Keeping free text data and using clearer column names can lead to better predictions, emphasizing the need for thorough data processing. This work presents a promising step forward in improving health care through advanced machine learning techniques tailored for real-world challenges faced in medical settings.