Natural Language Commands for Robot Teams

We introduce a new way to help multiple robots follow instructions given in Natural Language. This method lets robots understand and carry out tasks like "go to the left corner" or "pick up the can" without needing special training or complex setups.

We use powerful Language Models, which are tools designed to process language, to help our robots understand instructions. Our robots can learn from just 20 minutes of data collected randomly without relying on simulations or detailed environmental maps. We tested our method with a team of five real robots, and they showed they could handle commands they hadn't seen before, proving that they can grasp the language model's information effectively.

This approach is exciting because it allows us to create fast Control Policies that can be put straight into real robots without needing adjustments. We also share videos of our robot experiments.

#The Importance of Natural Language for Robots

Using natural language to instruct robots creates an easier and more intuitive way to communicate tasks. This method is more straightforward than giving specific coordinates or complex configurations. It allows operators to issue commands in a more conversational style without requiring special training.

Recent research highlights uses of large pretrained models for language processing and robot control. These models take tasks and observations and produce actions or sequences of actions. However, there are limitations with using these models. They can be slow, which is a problem if robots must react quickly in dynamic environments, especially in multi-agent situations where quick adjustments are needed based on the actions of other robots.

Finding ways for many robots to work together quickly with the help of large language models is a significant challenge.

#Our New Method

We introduce a new method that connects high-level language commands directly to the actions performed by a group of robots. We first translate the natural language instructions into a simplified form using a pretrained Language Model. Then, we train our control policies based on these simplified instructions. This setup allows us to achieve real-time control while keeping the language model separate from the immediate decision-making process.

To create a large dataset for training, we randomly collect real-world actions from a single robot. We then train our policies on this dataset through offline Reinforcement Learning. The advantage of using real data is that we can deploy our learned policies right away without any adjustments.

We claim the following major contributions of our work:

• A new structure that supports fast control for multiple robots based on natural language commands.
• A way to create vast amounts of training data from one robot's actions.
• Evidence that even a small change can significantly improve training stability in offline learning.
• Proof that our methods can handle commands they've never seen before based just on value estimates.
• The first test of offline Multi-agent Learning with real robots.

#How Our Robots Work Together

Our robots show they can work together effectively while following natural language tasks. Each robot receives an assigned task and has to navigate towards a goal while avoiding collisions. Each robot's path is color-coded.

In one test, three robots were trying to reach their individual goals but initially blocked each other. Through cooperative behavior, they managed to yield and let others pass, demonstrating an effective way of navigating around obstacles.

#Related Work

Other models like GPT and LLMs like LLaMa and Mistral exhibit strong reasoning abilities. They connect input and output tokens through a special architecture called a transformer. Although these models often generate text outputs, recent studies have started to use them for robotic tasks due to their reasoning strengths. Some work has shown that LLMs can help navigate towards visual targets using text outputs that translate to physical actions.

However, many existing methods still face challenges when it comes to real-time control, especially in multi-robot systems. Most studies have been conducted in simulated environments, which differ from real-world applications.

#Task-Conditioned Policies

Different names exist for what we call task-conditioned reinforcement learning. This involves adding a task or goal directly into the reward and value functions. Therefore, instead of learning for a single task, we build one that can be used across a range of tasks.

Our primary goal is to train many robots to follow natural language navigation tasks. Our process includes two main parts: creating the dataset and then training the model.

To gather data, we record the actions of a single robot as it performs tasks. We collect many natural language commands to match these actions. By combining these tasks and corresponding actions, we create a large dataset for multiple robots.

#Creating Our Dataset

For our experiments, we use a robot called DJI RoboMaster, which can operate holonomically with four wheels. We gather data by logging actions over time, resulting in thousands of action-state pairs. The information we collect includes position and velocity data, with each action corresponding to different movement directions.

Each task in our setup consists of a natural language command that instructs a robot to reach a specific target. We prepare a training set of tasks while also reserving some tasks for testing the robots' abilities.

#Combining Data from Multiple Robots

Instead of collecting data from multiple robots directly, we can use one robot's data to create a larger dataset by organizing its actions into scenarios involving multiple robots. This strategy allows us to artificially expand our dataset without requiring extensive physical testing with multiple robots, which would take an impractical amount of time.

#Designing Rewards and Ending Conditions

For each robot, we construct a reward structure aligned with its assigned tasks. This setup not only encourages reaching the goal but also discourages collisions with other robots or walls.

By establishing clear rewards for achieving goals and penalties for collisions, we help ensure that each robot learns to act efficiently and safely.

#Training Our Models

Our multi-robot model architecture means that each robot receives its own set of tasks and observations. After summarizing these tasks into a simplified representation, we use this data to train a local policy for each robot.

The policy learning occurs entirely based on the dataset we've gathered, meaning our robots can act quickly. While many existing training approaches focus on single-agent scenarios, we adapt our model to suit the needs of multiple robots operating together.

Through our training, we decide to utilize a new approach called Expected SARSA, which helps minimize errors during the learning process. Our approach can address overestimation issues that can arise during training, leading to a more stable learning experience.

#Testing and Results

Our tests aim to answer four primary questions:

1. Can our policy generalize to the language model's latent space?
2. What is the best loss function for training our policy?
3. How much data do we need to train a functional policy?
4. How well does our policy perform on real robots?

#Checking Latent Space

In our first experiment, we want to see if the policy can generalize across the language model's representations. We train a decoder to convert these representations back into goal coordinates. If the decoder correctly predicts values for new commands, it's a sign that it has learned well.

Through various tests, we find that some language models work better than others for our needs. We select one particular model for further experiments based on its performance.

#Simulation Tests

While our approach does not depend on simulation for training, simulations can help analyze performance. We build a simple model to simulate robot behaviors based on the gathered data. This gives us insight into how different objectives affect the robots' decision-making.

#Evaluating Objectives

We look at different training methods and compare results. By examining the performance of various policies, we report metrics on how well they complete unseen tasks. Certain methods yield better results, showing that the right objective can greatly enhance robot performance.

#Data Efficiency

We check how well our policy performs as we decrease the amount of training data. Surprisingly, performance remains strong even with minimal data collection, suggesting that our methods effectively leverage the available tasks.

#Real-World Tests

We conduct real-world navigation tests, where each robot is given a new task every 30 seconds. We track how far they move from their assigned goals. Our findings indicate that the robots can successfully adapt to tasks they've never encountered before.

The robots trained with specific loss functions consistently succeed in reaching their goals, displaying no collision incidents during testing.

#Limitations and Future Directions

Given the complexity of merging offline reinforcement learning, language models, and multi-robot systems, we limit our focus to navigation tasks for now. Future research could expand our methods to more complex scenarios.

We are optimistic about the potential for applying our strategies to broader tasks, but certain complexities would need to be addressed.

In conclusion, we have shown a new way to map tasks expressed in natural language to actions for multiple robots. By harnessing large language models alongside offline reinforcement learning, we can create datasets from single-agent experiences and train efficient policies that generalize to new commands without requiring adjustments when implemented in real-world environments.