SoccerNet-GSR: Tracking Athletes with Video Data

Tracking and identifying athletes during a soccer game is important for analyzing various aspects of the game, such as measuring how far players run or figuring out team strategies. This process is essential for representing the game state, which includes the positions and identities of players on a 2D map of the field, or minimap. However, getting this information from videos recorded with just one camera can be tricky. It involves understanding where the players are and how the camera is positioned to accurately locate and identify them on the field.

To tackle this issue, we have developed a project called SoccerNet-GSR, which stands for Game State Reconstruction. This new dataset focuses on soccer videos and contains video sequences that are annotated with detailed information for tracking players. SoccerNet-GSR includes video clips of 30 seconds each, with millions of points marked for pitch localization and camera calibration, as well as athlete positions including their role, team, and jersey number. Additionally, we have created a new way to measure the performance of game state reconstruction methods called GS-HOTA. Finally, we are providing an open-source baseline for game state reconstruction to support further research in this area.

#Importance of Athlete Tracking

Recently, there has been a growing interest from sports organizations in gathering data centered on athletes. A major area of focus is tracking and identifying athletes throughout the game using available video footage. This information can be valuable for many reasons, including:

1. Helping coaches improve team performance and athlete training.
2. Supporting talent scouts in finding new players.
3. Providing useful insights for medical teams working with athletes.
4. Engaging fans with personalized content.

Collecting this data manually is often slow and costly. Some alternatives use sensors that require athletes to wear special equipment, which can be expensive. More recently, systems that use multiple cameras have become popular, but they are costly and difficult to set up, limiting their use to high-profile events like the World Cup.

#Introducing Game State Reconstruction

Our project presents a new task, dataset, evaluation metric, and baseline specifically for Game State Reconstruction. The SoccerNet-GSR dataset offers unique identities for players along with their locations on the pitch for a collection of video sequences. Modern computer vision techniques have opened up new possibilities for extracting accurate data solely from video feeds. Traditional methods, known as Multi-Object Tracking (MOT), have been used in sports video analysis. However, they don’t provide all the necessary information for player identification and can be difficult to interpret without connections to the actual game.

To address these gaps, we propose Game State Reconstruction (GSR), a method that recognizes the state of a soccer game by identifying and tracking all athletes based on input videos from a single camera. The game state can be visually represented on a minimap, offering a clear summary of what is happening during the game.

With the SoccerNet-GSR dataset, we have marked over 9 million points for pitch registration, along with over 2 million athlete positions-complete with details like their role, team, and jersey number. Since existing metrics for evaluating tracking methods do not suit our task, we introduced GS-HOTA, a metric that effectively measures the performance of GSR methods. Additionally, we present GSR-Baseline, the first end-to-end open-source pipeline for game state reconstruction based on advanced tracking methods.

#Challenges in Game State Reconstruction

Our initial experiments highlight that GSR is a tough challenge and opens up new avenues for future research. The dataset and our code are publicly available for others to use in their research.

Tracking athletes during sports events has become an important area of study, particularly as teams seek ways to gather detailed performance data. This task includes tracking athletes throughout the game, and the analytics gathered from this can benefit various areas, such as training, scouting, medical support, and fan engagement.

However, generating this data manually can be time-consuming, and sensor-based solutions can be impractical due to the need for specialized equipment. Recently, automatic tracking systems using multiple cameras have emerged, but these are often too expensive for regular use in sports other than elite competitions.

#The Need for a New Benchmark

In light of these challenges, we introduce a new task called Game State Reconstruction, which combines several sub-tasks into one. The SoccerNet-GSR dataset enables the tracking and identification of athletes along with their respective roles, teams, and jersey numbers. We also created GS-HOTA to measure how well different methods perform in this area.

The goal of GSR is to extract relevant information about the game state from videos. This includes identifying the positions of all athletes, their roles, and their jersey numbers. The resulting data can be visualized in a concise minimap format, giving a clear overview of the game dynamics.

#Components of Game State Reconstruction

Game State Reconstruction consists of several key components:

1. Pitch Localization and Camera Calibration: This involves determining the layout of the soccer field and understanding the camera settings used during recordings.
2. Athlete Detection and Tracking: Recognizing where all players are on the field and keeping track of their movements throughout the game.
3. Role Classification and Recognition: Distinguishing between different roles such as players, goalkeepers, referees, and additional staff.
4. Team and Jersey Number Identification: Assigning each athlete to their respective team and recognizing jersey numbers, when visible.

#Introduction to the SoccerNet-GSR Dataset

The SoccerNet-GSR dataset improves on previous datasets by including a variety of annotations needed for Game State Reconstruction. The dataset consists of a collection of 30-second video clips captured from a single moving camera, where only sections of the soccer field are visible at any time. The data provides valuable information about the pitch layout, camera settings, and athlete positions.

To create this dataset, we annotated the pitch lines manually by placing points along the edges and curves. We also tracked these points over time to maintain consistency. Camera calibration is a crucial part of the process, as it helps connect video images to real-world locations on the field.

#Athlete Identification in SoccerNet-GSR

To identify players during the game, we use a combination of manual annotations that indicate each athlete's role, team, and jersey number. We categorize athletes into roles such as 'player,' 'goalkeeper,' 'referee,' and 'other' for anyone else involved in the game, like coaches or medical staff.

For jersey numbers, players are assigned a number if it is visible in at least one frame, otherwise they receive a 'null' designation. A track ID is also included to help identify athletes when their attributes are not enough.

#The GS-HOTA Evaluation Metric

The GS-HOTA metric works differently from standard evaluation methods used in multi-object tracking. It considers additional attributes to evaluate how well a GSR method performed in tracking and identifying all athletes on the pitch.

Differences from traditional metrics make GS-HOTA uniquely suited for GSR, as it assesses accuracy based on positional data and includes the identification of player attributes. This metric is vital for measuring GSR method performance and provides a thorough analysis of the task at hand.

#Building the GSR-Baseline Framework

To make the Game State Reconstruction task easier to study, we developed GSR-Baseline, a pipeline that processes the video input and generates the complete game state. The framework breaks down the task into smaller parts, utilizing various state-of-the-art methods for each component.

The GSR-Baseline takes the input images and passes them through an object detector and a pitch localization model. The results are processed to produce a final output that tracks and identifies each athlete and their respective details.

#Results and Findings

Through our experiments and analyses, we demonstrate that the GSR-Baseline achieves good performance on the SoccerNet-GSR dataset. We highlight the necessity of each module and show how they impact the overall results. Key components, such as pitch localization and jersey number recognition, were identified as areas requiring further improvement.

The results indicate that while each task within GSR can be challenging on its own, integrating them presents a new layer of complexity that requires ongoing research.

#Conclusion

In summary, our work introduces the first Game State Reconstruction benchmark for identifying and tracking athletes on a soccer pitch. We provide a new dataset, evaluation metric, and an open-source framework aimed at supporting research in this area. By benchmarking a complete pipeline that outputs high-level game data, we aim to aid various applications for coaches, scouts, medical staff, and fans alike.

The complexity of the GSR task, alongside the interconnections among its various components, emphasizes the need for continued efforts to enhance the existing models and develop more efficient systems for real-time analysis. We look forward to seeing how future research builds on this foundation, focusing on improving specific modules, achieving real-time tracking, and employing end-to-end methods for a seamless workflow.