The Indie Game Development Landscape

The game industry is booming. In Sweden, it makes up about 4.1% of the country's service exports, raking in around 86.5 billion SEK each year. That's a lot of cash! With around 25,000 people working in this field worldwide, and 8,500 of them in 939 companies in Sweden alone, you can see that this is not a small playground. But don't be fooled; it’s not all fun and games. The competition is intense, and companies are under pressure to deliver projects quickly and cheaply. If a game idea flops or a feature doesn’t work out, it can be a big hit to the business.

To tackle this, many software companies, especially in the game industry, have started using a method called Continuous Experimentation (CE). This means they test different ideas regularly before launching a game. By doing this, they hope to avoid releasing something that users don’t want or like. It’s all about getting feedback from players, tweaking ideas, and making sure that what they’re developing is something that people will enjoy.

#Experimentation at Different Stages

Ideally, experimentation should happen from the very start of game development all the way through to after launch. This helps ensure that the final product is engaging and valuable. The time before a game is released is crucial, as most development happens during this phase. As the game comes together, making changes becomes more costly and complicated.

However, conducting experiments before a game is released comes with its own set of challenges. Since the number of players is low at this stage, there isn’t much user data available for feedback. This can be especially tough for smaller indie game companies that don’t have the resources or backing of larger publishers. These indie companies, usually made up of small teams with limited experience, need to find ways to gather user data effectively.

#The Need for Research

There isn’t a lot of research out there about Continuous Experimentation specifically in the indie game world. This study aims to fill that gap by looking at how indie developers can apply CE before releasing their games. We spoke to 10 indie game developers to understand how they do this, and our findings highlight five key areas to think about when planning experiments in game development. In the future, we plan to dig deeper into our framework with real case studies and possibly bring larger game companies into the mix.

#Continuous Experimentation Explained

Continuous Experimentation (CE) is a method where decisions in game development are based on actual experiments with users. This involves figuring out what assumptions the team has about the game, turning those into testable hypotheses, running the tests, and then analyzing what worked and what didn’t. The goal is to find designs that encourage players to engage more, like spending money on in-game purchases.

While game development often involves analytics to manage various game features-like balancing levels and spotting bugs-CE takes this a step further by using user feedback directly to make decisions. Some parts of CE have already been explored in game development, such as testing early prototypes and optimizing features after a game is launched.

User research also plays a significant role in directing game development through player feedback. It recognizes that experimentation practices should match the different stages of making a game. In the early stages, qualitative methods are often the go-to since there isn’t much quantitative data available yet. Play-testing is one of the main methods used early on, while A/B testing often comes into play later in the process.

One challenge game developers face is deciding which features to test. The game must remain fun and engaging while making sure that the experiment can still measure the right performance indicators. Different game genres, like mobile games, may show preferences for various testing approaches, often favoring quantitative methods.

#Research Design and Interviews

To explore how CE can be applied in indie game development before releasing a game, we conducted interviews with indie developers at an indie game conference. We focused on gathering a wide range of experiences and practices from represented companies. The interviews were conducted online and lasted about 30 minutes each. We focused our questions on open-ended topics regarding CE and related practices like user research and game testing.

The conversations were recorded, transcribed, and analyzed to identify themes. We noticed that the data began to show some repetition in the later interviews, meaning that we were reaching a good understanding of the common practices among indie game developers.

#Results: A Framework for Continuous Experimentation

Based on our interviews, we created a framework for CE in game development. This framework consists of five main parts: goal definition, design strategy, experiment object, Sampling Strategy, and execution strategy.

#Goal Definition

The goals of our interviews suggest that the purpose of an experiment can change depending on the stage of development. Early on, the focus is often on defining the main game idea, while later on, it shifts toward refining specific features. A good example is when developers present a few game ideas to a potential publisher, gather their feedback, and decide which one to develop further.

When testing, developers targeted four main aspects of their games: aesthetics (the visual style), mechanics (how the game plays), fun factor (how enjoyable it is), and understandability (how easy it is for players to grasp the game’s logic). Even in the initial stages, it’s crucial to consider these factors to ensure a successful game.

#Design Strategy

Developers use a mix of strategies to design their experiments, which can vary based on the goals, type of game, and available resources. The main methods include split testing (comparing two versions), sequential testing (evaluating one option at a time), and exploratory testing (freely exploring ideas). Split testing can be resource-heavy, so it’s typically used in the early parts of development.

As development continues, the focus may shift toward evaluating specific features. Some developers have even come to use A/B testing based on experience with larger companies. Meanwhile, sequential testing often becomes the preferred method as developers refine and iterate on what works and what doesn’t.

Exploratory testing involves a more open-ended approach, where developers go in without predefined hypotheses but rather with questions in mind. This allows for a broader range of insights from players about their experiences.

#Experiment Object

The experiment object refers to what is being tested. It can include anything from sketches and videos to playable games. Developers must align this object with the goals of the experiment and the stage of game development. Often, they prefer to create a playable game version as early as possible to gather valuable feedback, even if that version is rough around the edges.

#Sampling Strategy

Choosing the right participants for the experiments is crucial and depends on the game’s development stage. Developers often use people from their internal teams, friends and family, publishers, and even larger communities of players. Feedback from these groups provides valuable insights but may come with biases, especially when friends and family are involved.

In the early testing phases, involving about 8-10 participants is usually enough. As development moves closer to a game’s release, testing can involve larger groups, especially when using channels like Discord and external platforms to attract players. However, developers noted challenges in recruiting unbiased participants, as close connections often carry pre-existing biases.

#Execution Strategy

Finally, collecting feedback from the participants during trials can take many forms. Most developers prefer observing players without interrupting them during gameplay. This helps gather more genuine reactions, as people might act differently if they know they’re being watched. Following up with surveys after playtesting can help collect specific feedback on gameplay mechanics, but managing feedback from a large number of players can quickly become overwhelming.

Some developers also use analytics and telemetry data during playtesting to gather quantitative metrics about player engagement and game performance. However, implementing these systems can be challenging, especially for smaller indie teams with limited resources.

#Discussion and Conclusions

Our findings show that indie game development relies heavily on Continuous Experimentation to fine-tune ideas and features before releasing a game. By conducting interviews, we gained insights into best practices across five key areas of experimentation, including defining goals, designing experiments, choosing the right objects to test, picking participants, and executing the strategy.

The challenges faced, such as evaluating the fun factor and understanding player engagement, highlight the importance of effective experimentation practices. While smaller companies generally use qualitative methods for feedback early in development, insight from external platforms is also increasingly valuable.

Indie game developers show considerable creativity and resourcefulness, effectively using limited resources to tackle the demanding pace of the gaming industry. Future research should look into how these practices could be expanded and improved, especially for larger companies and post-release processes.

In summary, while the world of indie game development is filled with challenges, it also offers plenty of opportunities for creativity and innovation. So the next time you fire up your favorite indie game, just think of all the experimentation that went into making it fun and engaging-zero pressure to win, just a lot of passion!