Advancing Operations Research with Open-Source LLMs

Large language models (LLMs) are becoming important tools for solving complex problems in operations research (OR). They help automate the process of creating optimization models. However, many current methods depend on proprietary models and specific ways of asking questions, which can lead to concerns about data privacy. This can limit the use of these models in real-world industries.

To address these issues, we suggest training Open-source LLMs specifically for optimization modeling. We have identified four key needs for the training data for these models and developed a process called OR-Instruct. This process helps create synthetic data that meets these needs. We also present the IndustryOR benchmark-the first benchmark for evaluating LLMs on real-world OR problems. By applying the OR-Instruct data to various open-source LLMs, we have greatly improved their ability to handle optimization tasks. Our best-performing model has shown excellent results on major benchmarks.

#The Role of Large Language Models in Operations Research

LLMs are effective in automating optimization modeling, which is critical in areas like logistics, healthcare, and finance. These models can interpret descriptions of problems and generate mathematical models and code with accuracy and speed. By integrating LLMs into various industries, companies can enhance their decision-making processes. This is especially important in fast-moving environments where conditions change frequently, as traditional methods may struggle to keep up.

As LLMs continue to develop, the connection between optimization modeling and these models is expected to foster significant advancements in industry practices.

#Challenges with Current Approaches

Research has often focused on using pre-trained language models (PLMs) to formulate mathematical models for optimization problems. Past methods, like NL4OPT, break the task into several parts, such as identifying semantic entities before creating mathematical models. While this approach can be effective, it often leads to errors and has limited generalization due to the smaller size of the models.

With the arrival of LLMs like ChatGPT, researchers are now able to generate complete solutions-including programs-by prompting these models directly. Some prompting techniques involve collaboration among multiple agents, working together to refine both the mathematical models and the programs. However, these techniques are mostly reliant on proprietary LLMs, which often require sharing sensitive data. This reliance raises significant privacy issues for businesses that must protect their data.

#Our Proposed Solution: Open-Source LLMs for Optimization

To overcome the limitations of current methods, we propose training open-source LLMs for optimization modeling. To ensure effectiveness and reliability, we have identified four crucial requirements that the training dataset must fulfill.

1. Diverse Coverage: The dataset should feature a variety of scenarios, problem types, and levels of complexity to build a robust model.
2. Adaptability: The dataset must reflect changes in objectives and constraints due to shifting business goals or market conditions.
3. Linguistic Variety: Various ways of describing similar problems should be included to prepare the model for real-world communication.
4. Modeling Techniques: The dataset should showcase different methods for solving the same problem, enabling the model to learn various approaches.

However, gathering data that meets all these criteria is challenging since much of it exists only in private business settings.

#Introducing OR-Instruct

To tackle the data collection challenge, we developed OR-Instruct-a semi-automated method for generating synthetic data tailored to specific needs. This process starts with real-world industry cases and expands the dataset through iterative methods.

Initially, we collected a set of seed cases and used GPT-4 to expand the scenarios and question types, significantly increasing the size of the dataset. While this expansion addressed some requirements, it still lacked sufficient variety in terms of complexity and adaptability.

To fill in the gaps, we implemented augmentations such as altering objectives and constraints, rephrasing questions, and introducing multiple modeling techniques. This approach aims to create a more diverse range of problem-solution examples. To ensure quality, we apply filtering techniques to remove low-quality data, resulting in a refined dataset ready for training.

#The IndustryOR Benchmark

To assess the effectiveness of our OR-Instruct method, we created the IndustryOR benchmark. This benchmark uses data from various industries and includes multiple question types and levels of difficulty.

We applied the OR-Instruct data to train several open-source LLMs sized around 7 billion parameters. The resulting models, referred to as ORLMs, exhibited a marked improvement in their optimization capabilities. Our top-performing model achieved state-of-the-art results on several key benchmarks, outperforming previous methods.

#Key Contributions and Findings

Our research contributes significantly to the field of operations research and LLMs:

1. We are the first to train open-source LLMs specifically for optimization modeling in real-world scenarios.
2. We established four critical requirements for the training dataset and designed OR-Instruct to efficiently generate fitting synthetic data.
3. We introduced the IndustryOR benchmark for evaluating LLMs in real-world optimization tasks.
4. Our best-performing ORLM has surpassed existing models in various benchmarks.

#Defining Optimization Modeling

Optimization modeling in operations research involves taking a real-world problem described in natural language and translating it into a mathematical form. This process also includes converting that mathematical model into a program that can be executed by a solver. The ultimate goal is to find the best solution among feasible options while adhering to specific constraints.

An example of an optimization modeling task might involve a company selecting transportation options for shipments. The company must consider costs, capacities, and exclusivity constraints. A mathematical model would define variables representing the selected transportation methods and establish objectives and constraints related to costs and capacities.

#Desiderata for Training LLMs

For effective training of open-source LLMs in optimization modeling, these models must be relevant, efficient, and adaptable to real-world needs. The critical requirements for the training dataset include:

• Comprehensive Coverage: The dataset should encompass diverse scenarios, question types, and varying difficulty levels to ensure broad applicability.
• Environmental Adaptability: The dataset must reflect real-world changes that might affect objectives and constraints.
• Linguistic Diversity: The varying ways in which problems can be described need to be captured to improve the model's understanding.
• Solution Variability: The dataset should include multiple approaches to solving similar problems to enhance the model's learning capabilities.

#The Data Generation Process

OR-Instruct consists of two primary strategies: expansion and augmentation.

1. Expansion: This strategy builds a larger dataset by generating new examples based on existing seed cases. GPT-4 assists in this task, producing a wide variety of scenarios and questions.
2. Augmentation: After gathering an expanded dataset, we enhance it by modifying objectives and constraints, rephrasing questions for linguistic accuracy, and incorporating various modeling techniques.

Each of these strategies is executed iteratively, enabling a more refined collection of high-quality training data.

#Evaluating OR-Instruct and IndustryOR

To evaluate our proposed methods, we employed the IndustryOR benchmark, featuring real-world cases from various industries. This benchmark is designed to test the models on diverse types of questions and varying complexity levels.

We applied the data generated from OR-Instruct to multiple open-source LLMs, revealing significant improvements in their optimization modeling skills. The results demonstrated that our top-performing ORLM could effectively solve complex OR problems, confirming the utility of our approaches.

In summary, our work lays the groundwork for improving how large language models can address optimization tasks in operations research, highlighting the advantages of open-source solutions and robust training methodologies. Looking forward, we aim to expand our efforts to train open-source agents, further contributing to advancements in this field.

#The Importance of Addressing Bias

While our study focused on improving LLM performance in optimization, it is crucial to recognize and mitigate potential biases in these models. Aligning models with societal values requires ongoing evaluations that encompass both technical effectiveness and ethical considerations.

#Conclusion

In conclusion, training open-source LLMs for optimization modeling addresses limitations in current approaches while meeting the needs of real-world applications. With the introduction of OR-Instruct and the IndustryOR benchmark, we facilitate a deeper integration of LLMs in decision-making processes across various industries, paving the way for future advancements and widespread adoption.