New Protocol Sets Standard for Text-to-Video Evaluation

Text-to-video technology has come a long way recently, making it easier for people to create videos from text. Models like Gen2, Pika, and Sora show exciting progress in this field. However, figuring out how well these models perform is not an easy task. Automatic measurements often fall short, so many researchers lean towards manual assessments. Yet, current manual evaluation methods have their own set of problems with consistency, reliability, and practical use.

To tackle these issues, a new protocol called Text-to-Video Human Evaluation (T2VHE) was created. This protocol is designed to offer a clear and standard way to assess text-to-video models. It includes specific measures to evaluate performance, thorough training for those assessing the videos, and a useful system to streamline the evaluation process.

The results indicate that this new approach not only provides high-quality evaluations but can also cut evaluation costs by nearly half. The entire T2VHE setup, including workflow and interface codes, will be made openly available for others to use and adapt.

Text-to-video technology has gained more interest from various communities in the last few years. Products like Gen2 and Pika have captured the attention of many users. Additionally, Sora, a model from OpenAI, has sparked excitement for text-to-video tools. As a result, evaluating these tools is becoming increasingly important for guiding future improvements and helping users choose the best models.

This work reviews existing evaluations and proposes a new human evaluation protocol for text-to-video models. There are two main ways to evaluate video generation: automatic and human assessments. Many recent studies focus solely on automatic metrics like Inception Score, Frechet Inception Distance, and Video Quality Assessment. While these metrics are useful, they have limitations such as relying on reference videos and not always reflecting how humans perceive quality.

Human Evaluations are seen as more reliable, but they also face reproducibility and practicality challenges. The survey conducted shows that there's little consistency in human evaluation approaches across different papers, with significant differences in metrics, methods, and annotator sources. For example, some studies use Likert scales, while others favor comparisons. Furthermore, many studies have inadequate details on evaluation methods, which complicates replication and further research.

Most papers also rely on authors or their teams to recruit annotators, raising questions about the quality of the assessments. In some cases, the number of annotations needed varies widely, which creates challenges in achieving reliable results without using too many resources.

To establish a more effective way of evaluating text-to-video models, the T2VHE protocol offers a structured approach. It includes well-defined metrics, comprehensive training for annotators, and a user-friendly interface. Additionally, it introduces a dynamic evaluation feature that reduces overall costs.

The T2VHE protocol is based on both objective and subjective Evaluation Metrics. Objective metrics focus on video quality, motion quality, and how well the text aligns with the generated video. Subjective metrics evaluate ethical considerations and general human preference.

Instead of asking annotators to give absolute ratings, the protocol uses a comparison-based method, which is more straightforward and user-friendly. By critiquing the traditional reliance on win rates, the T2VHE protocol adopts a probabilistic model to manage results from side-by-side comparisons, leading to clearer rankings and scores for the models.

Regarding annotators, while many studies rely on non-professional recruits, the T2VHE protocol highlights the importance of proper training. By providing detailed guidelines and examples, it aims to improve the reliability of results. The training leads to better alignment with skilled assessors and enhances overall annotation quality.

The dynamic evaluation module is a key feature that enhances the protocol's efficiency. This module sorts videos based on automatic scores and targets the most deserving pairs for manual evaluation later on. Initial video scores are updated after each round of assessments, helping researchers cut costs while ensuring quality results.

The study reveals several critical findings. Trained annotators, whether from crowdsourcing platforms or internal teams, can produce results that align closely with skilled evaluators. Traditional methods relying on side-by-side comparisons show a notable increase in reliability with fewer annotations needed for ranking.

In terms of model performance, the study compares five leading text-to-video models: Gen2, Pika, TF-T2V, Latte, and Videocrafter. The evaluation shows that closed-source models generally deliver better visual quality. Among the open-source alternatives, TF-T2V stands out in video quality, while Videocrafter demonstrates strong capabilities in generating high-quality outputs. Interestingly, Latte excels in text alignment and ethical aspects, garnering higher human preference rankings despite minor differences in other metrics.

The main contributions of this work lie in introducing a new, standardized evaluation protocol for text-to-video models, offering clear metrics and training resources. Moreover, the dynamic evaluation component allows for significant cost reductions in the evaluation process without compromising quality.

Despite the advances, some limitations remain. The models being evaluated are relatively new, and the presence of closed-source models complicates the analysis. Future research could build on this protocol to gain deeper insights into human evaluations of generative models.

#Related Work

Text-to-video models have been a significant area of research for many years. Various generative models, including GANs and autoregressive systems, have been explored in this field. The focus of text-to-video generation is to create videos based on textual descriptions, reflecting specific actions or scenarios.

Recently, the rise of diffusion models in image creation has stirred interest in adapting these models for video synthesis. Reviewing the evaluation methods used in prior studies reveals a wide range of approaches, but many share common limitations, such as being overly reliant on automated metrics.

The existing evaluation metrics for video models can be split into automated metrics and benchmark methods. Automated metrics like Inception Score and Frechet Inception Distance aim to assess video quality but often fail to capture essential aspects like temporal consistency and human appeal. Benchmarks like VBench and EvalCrafter seek to provide a more comprehensive view but still lack diversity, which is crucial for real-world application.

Given the shortcomings of automated assessments, high-quality human evaluation remains essential. Human reviewers can provide a nuanced understanding that automated methods often overlook, ensuring that the generated videos meet the desired standards in terms of quality and relevance.

The natural language generation field has acknowledged the importance of human evaluations to supplement automated metrics. For instance, some frameworks assess models across various aspects, ensuring a broader evaluation perspective. However, similar comprehensive approaches are still lacking in the text-to-video context, underlining the need for a structured evaluation protocol.

#The T2VHE Protocol for Text-to-Video Models

Our T2VHE framework is built around four primary components: evaluation metrics, evaluation methods, evaluators, and a dynamic evaluation module. The evaluation metrics consist of clear definitions and reference perspectives, enabling a thorough assessment of each video generated by the models.

To facilitate ease of annotation, we employ a comparison-based scoring approach and develop detailed training for evaluators. This training ensures that researchers can secure high-quality results through the use of rigorously prepared annotators.

The dynamic evaluation component serves as an optional feature that allows researchers to achieve reliable results at a lower cost. Utilizing this module enables a more efficient evaluation process, focusing on the most relevant comparisons.

In terms of evaluation metrics, we recognize the need to look beyond standard measures. Previous studies often concentrated solely on video quality and text alignment, neglecting crucial factors like motion dynamics and ethical implications. The T2VHE protocol broadens this view by including multiple metrics that address these aspects.

In final evaluations, our framework provides a comprehensive setup that includes both objective assessments and subjective opinions. The objective metrics require strict adherence to defined perspectives, while subjective metrics allow for personal interpretation, creating a well-rounded method for model evaluation.

#Evaluation Methods

The T2VHE protocol distinguishes between two primary scoring methods: comparative and absolute. The comparative method requires annotators to assess pairs of videos and choose the better one, making it more straightforward. In contrast, absolute scoring involves direct ratings, which can complicate the evaluation process due to its complexity.

Traditional evaluation methods using absolute scores have inherent drawbacks. They often lead to discrepancies in the assessment process and require detailed guidelines to minimize noise in the results. Therefore, we favor the more user-friendly comparative scoring approach.

We also aim to enhance the reliability of the assessment. Instead of relying solely on win ratios, we adopt a more sophisticated model to assess annotations. This approach allows for better handling of the results from comparison-based assessments, leading to clearer rankings and score estimations.

#Evaluators

Training and qualification of evaluators play a pivotal role in the quality of assessments. Many studies have relied on non-professional annotators without proper training or quality assurance, which can bias the results. In contrast, our T2VHE protocol emphasizes comprehensive training, providing guidelines and examples to help annotators make informed judgments.

By engaging in proper training, we ensure that evaluators are familiar with the metrics and can produce results that align closely with skilled human annotators. This leads to more consistent and reliable evaluations across various models.

#Dynamic Evaluation Module

As the number of text-to-video models grows, traditional evaluation methods can become resource-intensive. To address this challenge, we develop a dynamic evaluation module that optimizes the annotation process. This module operates based on two key principles: ensuring the quality proximity of evaluated video pairs and prioritizing based on model strength.

Before annotations begin, each model receives an unbiased strength value, which is updated as the evaluations progress. The goal of this module is to cut down on unnecessary annotations while still delivering reliable rankings for the models being assessed.

Through dynamic evaluation, researchers can better manage their evaluation resources, aiming to achieve more accurate rankings with fewer comparisons. This approach has proven effective in maintaining quality while significantly reducing costs.

#Human Evaluation of Text-to-Video Models

As part of our evaluation process, we assessed five leading text-to-video models: Gen2, Pika, TF-T2V, Latte, and Videocrafter. Each model was evaluated based on various aspects, such as video quality, motion fluidity, and how well the generated videos match the textual prompts.

In our assessments, we took care to standardize the presentation of videos to ensure uniformity for evaluators. This consistency helps facilitate better comparisons among the models, making it easier for annotators to assess without the interference of differing video resolutions or formats.

#Data Preparation

For the evaluation, we carefully selected prompts from different categories to assess the performance of the models. A total of 2,000 video pairs were generated for annotation, and 200 of these were randomly sampled to create a pilot dataset.

We engaged three groups of annotators for the evaluation process. Each group included skilled evaluators and different types of internal annotators, ensuring that the results reflect a balanced perspective. This comprehensive setup allows us to validate the efficacy and reliability of the models assessed.

#Evaluation Results

The results of our evaluation show a clear picture of model performances across various dimensions. Trained annotators, whether from crowdsourcing platforms or internal teams, consistently delivered results that aligned closely with expert evaluators.

When comparing the models, closed-source options like Gen2 generally performed better across most quality metrics. Among open-source alternatives, TF-T2V was recognized for its exceptional video quality, while Latte achieved a standout reputation for its text alignment and ethical robustness.

Contrasting the model performances highlights the strengths and weaknesses of each, demonstrating the need for careful consideration when selecting text-to-video models for various applications.

#Conclusion

Our work addresses the challenges present in current evaluation practices for text-to-video models. By introducing the T2VHE protocol, we provide a clear, structured, and resource-efficient method for assessing these models. The combination of defined metrics, comprehensive training for evaluators, and a dynamic evaluation module enables researchers to achieve high-quality results while minimizing costs.

As text-to-video technology continues to evolve, robust evaluation methods become increasingly crucial. We anticipate that our protocol will serve as a foundation for future research, empowering the community to engage in better assessments of generative models.

Researchers and practitioners alike can leverage the insights and practices outlined in this work to refine their evaluation processes and enhance the development of text-to-video technologies.