Pepper Robot Enhanced for Better Human Interaction

Pepper the robot is known for its ability to interact with people. To make it even better at understanding and responding to human actions, we have made some important changes to its hardware and software. This means adding new tools that help Pepper see and understand people around it in a smarter way.

#Upgrades to the Robot's Hardware

The first part of our work involved upgrading the physical parts of the Pepper robot. Pepper has some limitations when it comes to processing information and recognizing what is happening around it. These limitations made it harder for Pepper to perform more complex tasks that require understanding human behaviors.

To address these issues, we added a powerful NVIDIA Jetson Orin Nano GPU. This new computer allows the robot to handle more tasks at once and to analyze information faster. The Jetson is small enough to fit on Pepper and does not use a lot of energy, which is helpful since robots need to conserve battery power.

Additionally, we installed an Intel RealSense D435i camera. This camera can capture depth images, which means it can understand how far away things are. The combination of the GPU and the camera gives Pepper a much better view of its surroundings and the people in them.

Finally, we added an external battery to provide power for at least 70 minutes. Since the new hardware could not easily connect to Pepper’s existing battery, this change was necessary to ensure that the robot could operate effectively.

#Designing Parts for the New Hardware

To fit the new hardware onto Pepper, we designed custom 3D printed parts. These parts help hold the GPU, camera, and battery securely in place. The designs ensure that Pepper can still function well without any disruptions caused by the new equipment.

1. Custom Box: The box holds the GPU and the battery, separated by a partition. It has air vents to keep everything cool, which is important for the GPU's performance.

2. Camera Mount: The mount for the camera is strong and compact, reducing vibrations. It allows the camera to be easily calibrated since it has a fixed position.

3. Overall Structure: We made sure that all new components could withstand the demands of Pepper's movements and tasks.

#Enhancing Pepper's Understanding of Humans

While Pepper had some ability to track humans, it could not determine their exact locations or how they were facing. Understanding a person's body position and gaze is important for a robot to interact well with humans.

To improve this capability, we developed software that works with the hardware we added. This software helps Pepper detect humans, estimate their poses, and understand where they are looking.

We used a system called ROS (Robot Operating System) to implement these new features. This software allows Pepper to communicate better with its hardware and run the new programs smoothly.

#Detecting Humans and Estimating Positions

A key part of our software is a model called YOLOv8, which helps Pepper detect humans in real-time. This model can identify where people are and how they are positioned. By using the RealSense camera's depth information, we combine 2D data from the model with 3D data from the camera to get a clear picture of where each person's body parts are located.

To refine this process, we developed a way to filter out background noise, allowing the robot to focus on the human it is interacting with. We also use a method called the Kalman Filter to smooth out any inconsistencies in the robot’s understanding of where a person is looking or how they are positioned, especially when they are not directly facing the camera.

#Motion Capture Dataset for Testing

To make sure our improvements work effectively, we created a dataset using motion capture technology. This involves using multiple cameras to track a person’s movements in a controlled space. By capturing various actions in a consistent environment, we can create a baseline to test and enhance the robot’s perception abilities.

The dataset includes various types of movements, such as walking normally, walking with arms crossed, and moving in a zigzag pattern. Each session lasts around two minutes with one person at a time. This setup allows us to assess how well Pepper can use the new information it gathers.

#How the New System Works Together

With the new hardware and software, Pepper can now understand human behavior more effectively. The powerful GPU allows for quick processing of images and data, while the RealSense camera gives Pepper a detailed view of its surroundings.

When a person enters Pepper's field of view, the robot uses its software to detect the human, estimate their pose, and determine where they are looking. This synergy between hardware and software is key to improving interactions between people and the robot.

#Impacts of the Enhancements

These improvements mean that Pepper has a better chance of understanding what people want and how they behave. In various fields like healthcare, education, and entertainment, Pepper's enhanced interaction capabilities can be truly valuable.

For example, in healthcare, Pepper can assist patients by recognizing their emotions and engaging them in conversation. In classrooms, it can better respond to students' needs, becoming an interactive learning tool. In entertainment, it can become more engaging by providing personalized responses to visitors.

#Conclusion

This work has focused on upgrading the Pepper robot to improve its interaction with humans. By integrating advanced hardware like the NVIDIA GPU and RealSense camera, alongside developing new software for human detection and pose estimation, we have significantly enhanced Pepper's capabilities.

These changes make the robot more adept in real-world environments, allowing for richer and more meaningful experiences for users. With these enhancements, Pepper is better equipped to engage with people in various settings, contributing positively to human-robot interaction.