Introducing LEXI: A Tool for HAI Research

Recent advances in Large Language Models (LLMs) are changing how we interact with artificial agents. These agents are becoming more common in many areas of life, and their influence on users continues to grow. However, research on how people interact socially with these LLM-powered agents is still in its early stages. Access to the required technology and data is limited, there are no standardized interfaces, and researchers face challenges when setting up controlled experiments.

To help with these issues, a new tool called LEXI has been developed. LEXI, which stands for Large Language Models Experimentation Interface, is open-source software that allows researchers to create and test artificial agents powered by LLM in social interaction studies. With a user-friendly graphical interface, LEXI makes it easier for researchers to build agents and set up experiments, collect data, and analyze interactions.

#The Need for Better Research Tools

Conversational artificial agents, such as chatbots, are often used in behavioral studies to engage users and evoke emotional responses. These interactions can change how people perceive and behave towards technology, impacting their emotions and overall well-being. The addition of LLMs to these agents marks an important step forward in the research of Human-agent Interactions (HAI). However, there is still a significant gap in the investigation of how users interact with LLM-powered agents.

Many researchers may not have the necessary tools or resources to use these advanced models effectively. They often lack access to the right infrastructure and may struggle to integrate various technical components needed for deployment. This leads to a focus on observing the content produced by these agents rather than studying users' actual interactions. To address this gap, we need more empirical research on how people engage with these advanced agents and the effects of different configurations and prompts on user behavior.

#Introducing LEXI

LEXI is designed specifically for conducting experiments on HAI using LLM technology. The tool provides a graphical interface (GUI) that allows researchers to prompt LLM-powered agents and deploy them in controlled experimental designs. It includes features for gathering data through Questionnaires and annotations, enabling efficient collection and analysis of social interactions.

By using LEXI, researchers can save time and resources, allowing for more effective studies. They can systematically compare prompts, models, and user behaviors without the need to create agents from scratch. This standardization facilitates more robust and replicable experimental methods and findings.

Additionally, LEXI's design mirrors familiar chatbot interfaces, helping maintain ecological validity by reflecting real-world scenarios. It also includes a participant registration system, ensuring long-term interactions between researchers and participants, which enhances user engagement.

#Current State of Chatbot Development

Before LLMs became widespread, various online platforms helped users create simple chatbots. These chatbots mostly operated on predefined scripts and were integrated into existing social media platforms. While these tools catered mainly to small businesses, they did not fully support the academic research community.

Some platforms, like Dialogflow by Google, allowed users to build rule-based chatbots with some customization. These tools gained popularity in research, providing a significant developer community and supporting experimentation. However, with the emergence of LLMs, the focus on chatbots shifted to utilizing powerful LLM capabilities.

Researchers have begun deploying their own LLM-powered chatbots using various APIs, such as the OpenAI API. While these DIY chatbots offer flexibility, they often require technical expertise and can be limited in features and complexity. Many existing platforms, even those aimed at researchers, do not allow for the thorough control needed in empirical studies.

#Challenges in Current Platforms

The current landscape reveals gaps in the ability to conduct rigorous and replicable HAI experiments. Most platforms cater to business needs, lacking the necessary features for managing ongoing experiments or collecting detailed interaction data. Existing tools often focus on single-agent interactions, limiting the ability to study complex experimental conditions.

Also, many platforms impose significant restrictions on Data Collection. Researchers using these platforms often do not have access to interaction logs or self-reported data. Moreover, the DIY approach using LLM APIs, despite being resource-efficient, often results in oversimplified agents that may not meet the requirements for rigorous experimental setups.

#LEXI's Design and Features

LEXI aims to provide a solution for the challenges faced in HAI research. It is designed to improve access to LLM technology while maintaining experimental control. Researchers can prompt LLM-powered agents through a graphical interface, which helps standardize studies and improve the replicability of methods and findings.

The tool is open-source, making it accessible to researchers from various backgrounds, including social and behavioral sciences. It encourages contributions and collaboration within the research community, promoting inclusivity in HAI research.

#Key Features of LEXI

1. User-Friendly Interface: LEXI's GUI is designed to be intuitive, making it easy for researchers to set up and manage experiments without extensive technical skills.

2. Experiment Management: Researchers can create and manage detailed experiments. They can easily set parameters, allocate participants to different conditions, and monitor the status of experiments.

3. Building Agents: Researchers can create and customize LLM-powered agents, specifying how they engage with users based on iterative user input.

4. Forms and Questionnaires: LEXI allows researchers to build forms for collecting self-reported data before and after interactions, streamlining the data collection process.

5. Data Storage and Analysis: Collected data is stored in a MongoDB database, enabling easy access and export in various formats for analysis.

#Usability Testing of LEXI

To assess the usability of LEXI, a test was conducted with researchers from different fields. Participants were asked to complete a series of tasks using LEXI, such as setting up agents and experiments. Their feedback was collected regarding ease of use, time taken for tasks, and mental workload during the process.

Overall, the results showed that participants found LEXI easy to use, with minimal time needed for task completion. Responses indicated high satisfaction with the tool's design and functionality. Researchers expressed excitement about the potential of LEXI to enhance their studies.

#Pilot Study Using LEXI

To validate LEXI's capabilities, a proof-of-concept study was conducted comparing empathetic and neutral agents in how they engaged with users. The study involved 100 participants randomly assigned to interact with either an empathetic agent or a neutral agent powered by GPT-3.5-turbo.

Participants completed initial questionnaires to assess their mood and demographics, then interacted with the agent before reporting their mood again. Interaction logs were collected for further analysis.

#Findings from the Pilot Study

The study revealed that participants interacting with the empathetic agent reported higher levels of emotional engagement, wrote longer messages, and expressed more positive sentiments compared to those interacting with the neutral agent. Additionally, participants reported mood improvements after interacting with the empathetic agent.

These outcomes highlight the importance of agent communication style in shaping user experience and sentiment. LEXI proved to be an effective tool for collecting high-quality data in this context.

#Ethical Considerations in HAI Research

With the rise of LLMs in social settings, it’s crucial to understand the ethical implications of deploying artificial agents. LEXI aims to contribute to the responsible use of LLMs by facilitating rigorous empirical research. Findings from studies using LEXI can inform best practices and guidelines for ethical interaction between humans and AI agents.

Maintaining participant privacy is also essential. As researchers deploy agents with LEXI, they bear the responsibility of safeguarding participants' data, just as they would with any other research tool.

#The Future of LEXI and HAI Research

While LEXI currently provides key features for conducting complex experiments, ongoing development will enhance its capabilities. Future enhancements will include more options for experimental conditions, improved memory, and connections to various LLMs.

The aim is to create a versatile research environment that adapts to the needs of the HAI research community, making it easier for researchers to explore interactions with LLM-powered agents.

#Conclusion

In summary, LEXI is a promising tool for advancing research in social interactions with artificial agents powered by Large Language Models. By providing a user-friendly interface and robust features for managing experiments, LEXI addresses many challenges faced by researchers in this field. As it evolves with ongoing contributions from the research community, LEXI will continue to facilitate innovative studies that deepen our understanding of human-agent interactions and their implications for society.