Rethinking Evaluation in Design Research

Many approaches to evaluating design research often assume that prototypes are finished and ready for testing. However, assessing designs during earlier stages with mid-fidelity prototypes can also provide valuable insights. This type of assessment can help inform the design process and benefit the wider research community. We propose a new concept called "formative situations" to explore these mid-fidelity prototypes. This approach emphasizes the importance of evaluating designs in progress and encourages open discussions on how to incorporate formative evaluations as a valid part of design research.

#The Need for Formative Situations

In the Human-Computer Interaction (HCI) community, summative evaluations are often prioritized over formative evaluations. Summative evaluations aim to assess how well a design performs, while formative evaluations focus on gathering insights that can guide design improvements. There are two main reasons for this preference: first, many researchers may not fully recognize the valuable knowledge that can come from formative evaluations, and second, reviewers sometimes confuse formative and summative evaluations, applying inappropriate standards to them.

This confusion is partly due to the lack of established methods for evaluating designs during their formative phases. To address this issue, we introduce "formative situations," which involve a detailed examination of mid-fidelity prototypes. These situations are not just preliminary evaluations but provide valuable insights that contribute to the research community.

#Evaluating Prototypes in HCI

Formative evaluations are foundational in HCI research. Textbooks often adequately cover both formative and summative evaluations as vital parts of HCI knowledge. However, many HCI courses tend to emphasize summative evaluations, focusing on field studies or controlled lab experiments. This emphasis on summative evaluation continues in HCI publication venues, where much of the attention goes to studies that assess how well a design works in real settings.

This trend raises an important question: is it appropriate to focus solely on summative evaluations? We argue that much of the design work in HCI often falls into the mid-fidelity prototype category, which would be better assessed through formative evaluations. Many prototypes created by researchers are not fully functioning products. They might be incomplete or unstable, yet they are still evaluated summatively, either in controlled lab settings or by testing them in real-world scenarios.

What if we shifted our focus to evaluating mid-fidelity prototypes in suitable contexts? Formative situations could exist in a space that is part artificial, allowing designers to address prototype functionality, and part situated in real-world contexts. By reframing our approach, we could gather more nuanced Feedback and insights that could benefit Design Processes.

#Framework for Formative Situations

Formative situations offer a flexible evaluation method that can draw upon established techniques typically used in summative evaluations. However, they also allow for provisional measures that fit the specific context and goals of the research. Many HCI projects might benefit from including formative evaluation stages in their design cycles, producing valuable knowledge for the broader community. By requiring summative evaluations for publication, researchers face pressure to quickly conduct these evaluations, often before designs are ready.

To address this, we suggest that formative evaluations be regarded as valid contributions to the research process. Instead of rushing to summative assessments, researchers could take the time to craft formative evaluations that provide meaningful insights. Allowing shareable contributions from formative evaluations could foster collaboration across research communities, as insights generated at different design stages can inform further development.

#Rigor in Scientific Research

The concept of rigor in scientific work has been debated in HCI for years. HCI is an interdisciplinary field, drawing from various domains, including computer science, psychology, design, and sociology. Each sub-discipline has its own accepted methods, leading to various understandings of what constitutes rigorous work. The distinct mechanisms by which knowledge is generated also differ, ranging from theoretical discussions to specific use cases.

Within HCI, contributions can be categorized as empirical, conceptual, or constructive. While empirical research focuses on describing real-world phenomena, conceptual research explains connections among different phenomena, and constructive research produces knowledge about creating interactive artifacts. Many contributions at HCI conferences, such as the ACM CHI Conference, primarily focus on empirical research, which can create a reinforcing cycle. Early-career researchers may assume that empirical work is the gold standard based on the prevalence of such papers.

#Current Evaluation Practices within HCI

HCI evaluation methods take inspiration from various fields. Empirical studies often incorporate the scientific method and experimental psychology techniques. In controlled studies, researchers aim to isolate causality and understand how specific interventions lead to certain behaviors or emotions. Although HCI studies may not involve as many participants as clinical trials in medicine, they often strive for similar rigor.

Field studies provide another avenue for evaluation, allowing researchers to test artifacts in real user contexts. However, field studies come with challenges, as they cannot account for all external factors, making it difficult to establish clear cause-and-effect relationships. In contrast, formative situations focus on engaging with users to gather feedback, without the rigid structure of traditional experiments.

#The Role of Formative Situations

Formative situations serve as a method to generate insights during the design process. Rather than aiming for a finished product, these situations provide a context where participants interact with mid-fidelity prototypes, providing rich feedback. This approach allows for a more nuanced understanding of User Needs, challenges, and experiences, which can be vital for informing future design iterations.

#Proposed Criteria for Formative Situations

To guide the creation and assessment of formative situations, we propose a set of criteria. These criteria are designed to help researchers develop situations that elicit valuable feedback while remaining transparent and adaptable.

1. Well Fitted to the Research Questions: The formative situation should be carefully aligned with the research questions. It should be clearly described so participants experience similar conditions, allowing researchers to contextualize feedback effectively.

2. Draws Upon Related Best Practices: Researchers should outline their design process and clearly define the current phase of the prototype. Transparency is essential, and it is crucial not to assume that all contributions are at the same design stage.

3. Appropriate Weight in Proportion to the Prototype: The situation being studied should match the design resolution of the artifact. If a prototype requires extensive support, it might not be suitable for field-testing.

4. Designer Positioning: Formative situations can involve the designer-researcher more fluidly compared to traditional experimental paradigms. Researchers may adapt the situation in real-time based on participant responses, clarifying points as needed.

5. Active Participant Role: Participants should be encouraged to engage with the design process actively. Their feedback should move beyond basic impressions to provide insights into the design's strengths, weaknesses, and areas for improvement.

#Evaluating Rigor in Formative Situations

While formative situations offer flexibility, there are aspects that should be considered to maintain rigor. Evaluators can use established principles from empirical studies to assess the validity of formative situations.

#Transparency and Alignment

Transparency in the research process is paramount. Researchers should provide enough detail to allow for assessing the design stage and fidelity of the prototype. If insufficient transparency exists, the formative situation may not be appropriately evaluated.

#Real-World Alignment

Formative situations should draw from real-world experiences, whether social or personal. Engaging with participants in contexts that mirror their daily lives can enhance ecological validity. Researchers may create scenarios that reflect real-life situations to gather more meaningful feedback.

#Managing Novelty

Participants should be able to interact with the prototype in a way that encourages genuine responses. If the situation is too novel, participants may struggle to offer insightful feedback. The aim is to create contexts that allow participants to provide feedback about the intended use, informing future design iterations.

#Depth of Feedback

Quality feedback is vital in formative situations. Participants should provide rich feedback that goes beyond surface-level impressions. Designers can facilitate this process by engaging participants in discussions, interviews, or other methods to gain deeper insights.

#Conclusion: A Call to Action

In this discussion of formative situations, we propose an approach that emphasizes the value of assessing mid-fidelity prototypes. By allowing room for formative evaluations, we can foster a design community that values growth and feedback throughout the design process.

Industry practitioners desire practical insights from the research community. Insights gained from formative situations can provide more actionable knowledge than evaluations of unfinished prototypes. It's essential to reconsider the purpose of the design knowledge we produce and the stakeholders it serves.

As a community, we should challenge our bias toward summative evaluations. By accepting formative evaluations of mid-fidelity prototypes as valid contributions, we open the door for more diverse and meaningful research contributions. In the future, we hope to see formative evaluations recognized as rigorous and essential to knowledge generation in the HCI field.

This conversation is just the beginning. Through collaboration and open dialogue, we can shape the landscape of design research, reinforcing the idea that not every design needs to undergo a summative evaluation to yield valuable insights. We encourage researchers to share their formative work and engage with the community to refine our understanding of what constitutes rigorous and impactful design research.