Introducing AssemblyHands: A New Dataset for Hand Pose Analysis

In recent years, there has been a growing interest in understanding how people perform tasks in everyday life, especially from a first-person view. This interest is particularly relevant in the fields of augmented reality (AR) and virtual reality (VR), where recognizing Hand Movements is crucial for interaction with objects. To support this research, we introduce AssemblyHands, a large dataset designed for studying how people assemble and disassemble toys using their hands.

#What is AssemblyHands?

AssemblyHands is a dataset that contains a lot of images where people interact with objects, focusing on their hand movements. This dataset is unique because it provides high-quality data on the 3D Positions of hand joints, which helps in analyzing how hand poses relate to specific actions. The data is gathered from participants who were filmed as they worked with toys, which includes tasks like putting parts together and taking them apart.

#The Importance of Hand Pose Data

Understanding hand poses is essential because they provide valuable information about what a person is doing. Different hand movements often correspond to specific tasks. For instance, when someone is "screwing" something, their hand movements will differ from when they are "lifting" an object. By analyzing these movements, researchers can gain insights into how people perform tasks and how to improve human-computer interaction in AR and VR applications.

#How We Collected the Data

To create the AssemblyHands dataset, we used a set of cameras to capture images from different angles. This setup allows us to get a comprehensive view of the hand movements from a first-person perspective. The process worked as follows:

1. Participants: We invited several individuals to complete tasks involving take-apart toys. They were filmed while assembling and disassembling the toys.

2. Cameras: A combination of stationary cameras and a wearable camera captured the actions from various perspectives. This approach ensures that we can see both the person's viewpoint and the surroundings.

3. Annotation: We manually marked the positions of key points on the hands in the images. This process involved identifying where each joint in the hand was located during the tasks.

4. Quality Control: To ensure high-quality data, we developed a method to check and refine the Annotations. This involved using automatic techniques to predict hand joint locations and improve the accuracy of our annotations.

#Benefits of AssemblyHands

AssemblyHands offers several advantages:

• High-Quality Annotations: The dataset includes precise 3D hand pose annotations, which make it easier to train models for recognizing hand movements.

• Large Scale: With many images collected from diverse subjects, the dataset provides a broad coverage of hand poses in different contexts.

• Action Classification: The data allows researchers to analyze how hand movements relate to specific actions, which is invaluable for improving AI systems in AR and VR.

#Evaluating Hand Pose Quality

To measure the effectiveness of the hand pose data, we compared our annotations to existing methods. We found that our approach resulted in a significant reduction in errors. This means that our dataset is likely to help build better models for understanding hand movements.

#The Role of Hand Poses in Action Recognition

Recognizing what someone is doing based on their hand movements has been a longstanding goal in computer vision. With our dataset, we can explore how hand poses relate to specific actions. For instance, by observing how someone holds a screwdriver, we can infer that they are likely to be "screwing" something.

#Using AssemblyHands for Action Classification

We took the dataset a step further by using it to classify actions based on hand poses. We focused on six common actions that people perform while assembling and disassembling toys. These actions are crucial for understanding not just what is happening, but how it is done.

#The Actions Studied

1. Pick Up: Lifting an object from a surface.

2. Position: Placing an object in a specific spot.

3. Screw: Turning an object into another one.

4. Put Down: Lowering an object onto a surface.

5. Remove: Taking an object away from another.

6. Unscrew: Turning an object out of another.

These actions are frequently observed in the dataset and provide a foundation for studying how hand movements contribute to object manipulation.

#Comparing Methods

We also compared our new model trained on AssemblyHands to existing models built on other datasets. The results showed that the newer model performed better, indicating that the quality and volume of data in AssemblyHands enhance action recognition capabilities.

#Future Work

While AssemblyHands provides valuable insights into hand movements and actions, there are still areas for improvement. Future research can look into the following:

• Object Interaction: Including more details about the objects being manipulated could further improve understanding.

• Higher Sampling Rates: Gathering more data at higher frequencies would capture even more intricate movements.

• Integrating Object Annotations: Providing object-level information, such as the location of toys, could enhance action recognition.

• Multi-Task Learning: Exploring the relationships between hand movements, objects, and actions could lead to new developments in the field.

#Conclusion

AssemblyHands represents a significant advancement in the study of hand actions during activities. By providing a rich dataset with accurate 3D hand pose annotations, it opens new doors for research in AR and VR. Understanding how hand poses relate to specific tasks will help improve human-computer interaction and contribute to the development of more intuitive systems. We believe this dataset will inspire new methods and insights into recognizing human activities from the first-person perspective.