Why Do Online Only Games Have Such Rocky Starts? September 12, 2016

Anyone who downloaded Pokémon Go when it first launched probably found themselves unable play it most of the time. The application was fundamentally tied into its online component, which was bogged down for weeks by the sheer number of simultaneous user requests. It would take a while for Niantic to iron out all the issues with the game on both the client and server end, but they made the adjustments needed and now the game is (barring the occasional odd crash) pretty playable.

So why did the game have such a bad launch period?  Well, Pokémon Go is hardly alone when it comes to issues with games requiring online services during their launch.  Heck they’re hardly tied to the smartphone platform, because triple-A PC titles like Sim City and Diablo III were downright unplayable at launch, rightfully angering fans who bought them at release.

So why is this happening?  It can’t be for lack of dependable testers, I know for a fact that millions of people all over the world were willing to test Pokémon Go before it launched, and those players could have been used to test the game before it was released. While I’m not totally certain of the exact number of testers the game had during development I can certainly confirm they didn’t accept everyone who applied because I applied and I know for sure I wasn’t picked!

After all these bad game launches, a lot of frustrated players have asked why weren’t these games properly tested before they launched? The truth is they had been tested, but after their poor launches it’s clear they weren’t tested adequately.  How could this be?

I spoke to a friend of mine who had a passing understanding of Google’s online testing methods who shared his thoughts with me. He doesn’t work for Google or Niantic, but he does work in the tech industry and he is familiar with a lot of their testing methods.  While I can’t confirm Niantic (or any other online game developer) uses this method to test their games, his information did make these day one problems gain some sense.

Games are tested in controlled environments before they’re released to the public. We call that QA Testing for Quality Assurance.  You don’t have to read further than The Trenches webcomic to see just how bad QA Testing can get, but what about games that require an online component to function?  Those are tested in what are called “proportional” circumstances. Just like the Mythbusters will test theories in smaller scale conditions before replicating a myth in full size, online game developers traditionally test their games in limited environments with fewer devices. The idea behind it is that if a server with limited bandwidth can remain stable under a proportionally limited test case of players, their servers can handle the expected amount of end users at full bandwidth when the game is finally released.  It’s believed that testing online games proportionally during development is the best possible testing method.

After he finished giving me this information I told him, “Wow after Pokémon Go‘s launch was such a disaster, they must be really rethinking that flawed test method, aren’t they?”

You would have thought I insulted the man’s mother if you could have seen the face he gave me after I made that statement.  After telling me in no short order that there was nothing wrong with that testing method (ignoring the fact it failed miserably when the final games were brought online in many different cases), he told me that I had no idea what I was talking about and I just looked at him like he was completely out of touch with reality.  If proportional testing was the indeed the method Niantic used to test Pokémon Go or EA used to test Sim City, and that testing method had worked, Pokémon Go would not have had the plague of crashes, login failures and random quits for three weeks after it launched, and Sim City would have been playable.

When I grew up testing games on the PC, developers would traditionally hold a “stress test” period where they would get as many simultaneous users as they could to see if their game would break or buckle under the strain of the number of users testing it. Sometimes they would start with fewer testers and add more as time went on, but by the end of the testing period they would usually offer everyone they could the game’s online beta test client as a free download. This testing method is still being used for games like Gears of War 4, Titanfall 2, and Halo Wars 2, whose developers have all offered open online stress testing this year.  From a practical standpoint, this seems like a far more fruitful method of testing a game toward the end of its development cycle.  By offering your game’s test client for free to everyone with even a passing interest in the game, developers can better predict player numbers as high as or higher than a game could expect to get at launch.  It can also help investors shape sales expectations and ensure a smoother launch period.

I sought advice from other peers of mine familiar with the tech industry as I was writing this article and they had plenty they felt needed to be added to this discussion. They argued that hosting an open beta test for a game like Pokémon Go would have been a bad idea, since the normal spectrum of bugs and glitches that players could experience during testing might have had the side effect of giving testers a poor initial impression of the game, and make them lose interest in playing it when the full version was released. While I understand some players could accept this arguement, I do not.  It is reasonably accepted amongst gamers that test clients could have their fair share of bugs and glitches. In fact, every EULA I’ve ever read for a beta game references this, so players are prepared for it. However, nothing turns potential players off a game more than a glitchy launch, and I would argue that it would make more sense to have bugs show up during the game’s test phase then to hold back testing and discover your game has problems only when the game is in the hands of paying customers.

I have not talked to anyone from Niantic and I’m no more familiar with insider information about recent Pokémon Go developments than anyone with access to the company’s Twitter feed. Pokémon Go earned millions of dollars of income in the first few weeks since it was released. There’s no telling how much more money Niantic could have made if they provided a stable platform on day one.  Perhaps if they had done a stress test they would have been better prepared for what they were in for but I guess we’ll never know for sure.