Development Diary: How we got a local LAN party to test our Action RPGEdit

Tuesday, November 2, 2010

Last weekend, we ran a special Path of Exile pre-alpha test event at the Lanholm LAN in West Auckland, New Zealand. 26 players played for 90 minutes, competing for an exclusive shirt and cash prize. The players had fun, and we learned a lot about running this type of event. Pick an appropriate venue

Lanholm is a series of LANs that has been going on for almost a decade. Some of our team members met there many years ago, and it's an important part of our local gaming community. We heard there was a LAN planned for last weekend so we decided that it'd be a great opportunity to get some feedback on Path of Exile.

We decided not to publicise the event on our game forums beforehand. It was important to us that we got a wide sample of gamers and hardware, rather than a group of dedicated action RPG fans. If we didn’t know in advance that the event would have enough people, we’d definitely have had to advertise it a bit. Offer some prizes

We decided to give the winner of our competition $50 and an exclusive Path of Exile shirt (the last one of the batch we had made for team members). Second place got $20 and the knowledge that he just didn’t kill enough monsters in time. Standings were based on an experience “ladder” that we had displayed on one screen during the competition.

We were impressed by how the prizes encouraged people to try the game out. We knew that a core 15 or so friends and RPG players would be interested in playing, but most of the other attendees played for the entire competition duration because they were interested in winning the money. It pleased us that many of them continued to play even after the competition was over. One guy told us that he was going to have to cancel going to a Halloween party because he wanted to compete for the $50.

Plan the competition to be fun and intenseEdit

Some people at the event were friends who are on our internal alpha testing team. They’re familiar with the game, so we told them in advance that they were excluded from prizes. They were disappointed, but understood that we wanted to keep the competition fair for new players so that more people were interested and we would get better feedback.

We found that the “ladder rush” format worked really well for a 90 minute competition. The people at the top of the ladder were very close, and when it came to the end of the competition, they were fighting neck and neck to be placed first. When the time ran out, the winning player was only 5000 experience ahead (roughly 20 monsters).

We made sure to balance the amount of content available with the time limit that we had decided on. We were given a 90 minute slot in the LAN’s tournament schedule, so we enabled gameplay up to the end of the Brutus fight in the prison. This meant that the fastest players were just running out of new areas as the competition came to an end. Prepare a stable build in advance

Before allowing people to see or play a build of a game, it’s important to make sure that it’s stable and well-tested. Because of this, it’s really risky to allow players to play the mainline “trunk” version of the game. It’s much safer to tag a known stable version and test that one, fixing any issues in that specific version rather than constantly updating to the cutting edge of what the team is working on.

We tagged 0.6.8 (the version created for this LAN) on the day before the event, giving us only one day to fix minor issues in that build. Tagging earlier would have allowed us to polish more, but would have resulted in the build being less up to date.

It would have been good to get more than a few hours of testing done on this build before taking it to the LAN. Although there were no major problems, half a dozen polish issues could have been corrected if we had noticed them beforehand. Next time we’ll try to finalise the build earlier than the day that we have to use it. In hindsight it would have been sensible to turn off some of the weird debug commands. Someone pressed F9, which puts the client into movie mode (causing it to run very slowly while taking high resolution screenshots to make movies from). They reported this problem as “Hey man, your game is really laggy”.

Don’t hack up a special build for the eventEdit

use it as an excuse to test your deployment process

One of our main goals for this event was to test our client and server deployment process. We wanted to use all of our existing build pipeline to package the special version for this LAN. It would run from a local server rather than one on the internet, and it’d only work at this one event. We did not want to assemble a “frankenbuild”, as that method had burned us in the past. The deployment worked perfectly, and was a great test of our build system.

We actually deployed our servers to a virtual machine that we took to the LAN. We were interested to know how well it would perform under emulated hardware (turns out there was minimal perceptible overhead), and we didn’t want to rip out any of the server machines from the build farm at our office.

We should have made it easier to enable people’s accounts at the event. Traditionally, our servers allow people to sign up for accounts and then we can add game access on a case by case basis (for example, adding people to the official beta test that starts early next year). We decided to retain this model for the LAN event so that we could control who gets access and also only allow players to log in when we were ready. Overall it was quite a lot of effort to tick the box on each player’s account as they created it, so it would have been easier to just allow unlimited access at the venue.

We made sure that the game client that we gave out was only usable at this one LAN. It’s built so that it can’t be used to connect to our alpha servers, and has to be replaced if we run another event. This was done on purpose because we expected that people would leave the venue without deleting the game client from their machines.

Distribute the game client before the competition startsEdit

45 minutes before we started the competition, we began to distribute the game client to players. We were intending to stagger this so that there weren’t 20+ people copying it from the same server, but it turns out that gigabit Ethernet is more than fast enough.

We let players play on the server for a while before we started the competition. This let them get slightly familiar with the game, and allowed us time to solve any problems people had with their accounts. When the competition itself started, we took down the servers, made an announcement, and then wiped the characters/items databases so that players could start fresh.

Have staff on hand to investigate problems as they occurEdit

All sorts of problems can crop up when testing a pre-alpha game on an arbitrary set of hardware. Out of the 30 people that tried to run the game, 26 succeeded. The other four had strange legacy graphics cards that we haven’t tested on before, and ran into some incompatibilities. We dumped DirectX diagnostic information so that we can get the game to work with these cards back at the office.

We had three staff on hand at the LAN, and they were all busy answering questions throughout the competition. Most technical issues were resolved before the competition began, but players often had questions about the game as they progressed, so we were busy the whole time helping them out.

Get usability and gameplay feedback from the playersEdit

Although it feels really important to help players and explain everything to them, it’s often useful to stand back and watch them struggle with complex features before you explain them. Some things that we thought were obvious were missed by players (an item that says “Right click this item then left click another one to identify it” had people asking “How do I use this?”. Also, some people would walk right through the town area and not talk to the clearly marked NPCs, failing to get some important skill gems). We noted down many areas that could be improved with better explanations or visual cues.

Gather as much data as possibleEdit

with an instrumented game client

In retrospect, we should have instrumented the game client substantially more. Although we paid attention to which machines were performing poorly (and asked what graphics cards those players had), it would have been great to log the average frame rate against the hardware specs of each machine.

We were interested in seeing what proportion of character classes people would pick at the event. There were two available, and we were pleased to see an almost equal split of character chosen.

Try to have fun yourselfEdit

Thankfully, everything went fine for us and people had a lot of fun playing. We received a lot of good feedback and many of the players expressed interest in bribing us for beta access. I imagine it would have been stressful if things were going wrong, but it was a great afternoon hanging out with other gamers and showing off the progress we’ve made on our project so far. Hopefully our beta testers feel the same way early next year!