Don't take the ESA's games industry numbers at face value

11 percent of US households own a VR headset... if you include Google Cardboard.

The Entertainment Software Association has issued its 2017 gaming industry snapshot, a report conducted by Ipsos MediaCT which surveyed "more than 4,000 American households" to paint a picture of growth and prosperity for gaming. I enjoy these yearly reports—who doesn't love data—but surveys like the ESA's are often cited uncritically and used to defend dubious positions. And they definitely shouldn't be.

The stated purpose of the ESA is "serving the business and public affairs needs of companies that publish computer and video games." So right away we know that it isn't likely to publish anything critical of its member companies, which include Activision Blizzard, Electronic Arts, Ubisoft and pretty much every other major game publisher. 

It also leaves a lot out of these reports—for instance, a breakdown of the study's procedures—and glosses over the details behind some of its numbers. So though I find the information useful and interesting, I take a skeptical approach to interpreting the ESA's data. To demonstrate why, I've pulled out some of the most interesting results from this year's sheet, some of which ring true, and others not so much.

Gender and age

The ESA's surface-level stats shouldn't lead us toward any hard conclusions about who plays or buys games or why.

Among the "most frequent game purchasers," the ESA found that 63 percent are male and 37 percent are female. Last year's report had a 60-40 split. The term "most frequent game purchasers" isn't clearly defined, but the ESA clarified to me that it refers to a member of the household surveyed who identifies as such. 

No effort is made by the ESA to correlate this statistic to the types of games being released or how they're marketed, and there's little information about the types of households surveyed or what kind of games they're buying, so there's not much insight to be had from this statistic, if you ask me.

Researcher Nick Yee of Quantic Foundry has a similar criticism of the ESA's yearly gender stats, arguing that simply dividing game players and game purchasers by gender can lead to unproductive analysis. And there are far more interesting numbers to look at. For instance, Yee's research found that Dragon Age: Inquisition has nearly double the percentage of female players as other western RPGs. Assassin's Creed Syndicate and Star Wars: The Old Republic are also outliers, with a larger audience of women than similar games.

Why that's the case is a productive line of questioning, while the ESA's surface-level stats shouldn't lead us toward any hard conclusions about who plays or buys games or why.

For one thing, four of the top 20 best-selling games last year were sports games, which Yee found to be the least-popular genre among women. That men bought Madden NFL 17 and NBA 2K17 en masse says nothing about the audience for high fantasy MMOs, 36 percent of which are women, also according to Yee.

A simple percentage breakdown of purchasing by men and women is too reductive to draw any great insight from.

And as a further example of why statistics need scrutiny, Yee's own results about sports games could be mistaken as confirmation that 'women aren't into sports,' which is false. The gender breakdown of sports games players could have many, many causes. For instance, a 2015 Gallup poll found that 76 percent of men in households which earn $75K or more per year describe themselves as sports fans, while only 50 percent of women in the same high income bracket do. So there's a larger group of men who like sports and have high incomes than women who like sports and have high incomes—incomes which one might use to buy things like videogames.

That's just one part of the puzzle, which includes everything from how sports and sports games are marketed (primarily to men) to broad analysis of capitalist culture and gender—which is outside the scope of this article, but suffice it to say that there's a lot going on. A simple percentage breakdown of purchasing by men and women is too reductive to draw any great insight from.

What comes across as more important to the ESA in this year's report is age. The average US gamer, according to the ESA, is 35 years old. Last year's report concluded the same thing. Broken down further, the average male gamer is 33, and the average female gamer is 37. So if any perception that games are for kids remains, the ESA is eager to squash it: it pointed out last year, and again this year, that the number of women older than 35 who play games is equal to the number under 35.

Overestimating VR

According to the ESA's report, 67 percent of US households own a device used to play videogames. That's certainly believable. Pew reported in 2015 that 68 percent of Americans own smartphones, which of course can be used to play games. When it comes to VR headsets, however, some important information is left out.

If that 11 percent statistic is accurate, then somewhere around 13,840,090 US households own VR headsets.

According to the graph above, 11 percent of US households own VR headsets. By the US Census Bureau's estimate, there were 125,819,000 households in the US as of 2016. If that 11 percent statistic is accurate, then somewhere around 13,840,090 US households own VR headsets. That sure sounds like a lot!

In reality, the combined sales of the Oculus Rift, HTC Vive, PSVR, and Gear VR don't come close to that figure, and as I suspected and the ESA confirmed via email, the 11 percent number includes entry-level equipment like the $15 Google Cardboard kits used for viewing 360-degree photos and videos with a smartphone. 

That's not quite the same as a massive adoption of VR gaming setups the 11 percent figure suggests, though according to the ESA's research, "one-third of the most frequent video game players say they will buy a VR headset in the next year." So the number of VR owners should jump dramatically very soon.

It seems a lot of people keep planning on buying VR headsets, but many aren't actually pulling the trigger.

Or not. That self-reported intention requires scrutiny as well. In last year's report, 40 percent of the same group also said they "will likely purchase VR within the next year."  

This year's report claims that 65 percent of US households are home to one of these "frequent gamers"—those who play more than three hours a week. That's over 75 million households. So if 40 percent of frequent gamers had really gone out and bought VR headsets within the last year, somewhere around 30 million headsets would be out there in the US alone.

It seems a lot of people keep planning on buying VR headsets, but many aren't actually pulling the trigger.

Another curious figure claims that 97 percent of households own a "personal computer," which the ESA clarified to me means desktop PCs and laptops. That sounds generous. In 2014, the US Census Bureau reported that "83.8 percent of the nation’s households have a computer (either desktop, laptop, tablet or smartphone)." If all of those devices combined only accounted for 83.8 percent of US households three years ago, what accounts for the number of desktops and laptops alone jumping to 97 percent?

I would expect the number of households which own laptops and desktops to drop, given that many of the functions of a PC can now be done with tablets and phones. If you include Surface tablets as PCs, there was a little bump in US PC sales last year, but the worldwide trend has been one of decline. So either there's been a huge surge of PC purchases in the US, or that 97 percent figure includes Palm Pilots at the bottoms of boxes.

Other results

The ESA's yearly report can be useful for identifying trends, as are other reports by independent market researchers. I've only included data above that I found suspect, or lacking detail—I'm making a point, obviously—but plenty of the other results are interesting and don't require as much scrutiny. For instance, I was surprised to see that, at least for those surveyed, graphical quality is still the most important factor when deciding to purchase a game.

I would have expected price to be more important, though it's equally shocking that price is second and 'interesting story/premise' is third when the best-selling game of 2016 was Call of Duty: Infinite Warfare. Maybe the reasons given don't actually match the behavior, but it's interesting on its own to see how gamers self-reflect on their purchasing decisions.

The most popular games are still shooters, action games, sports games, and RPGs, which isn't especially surprising. But there are quibbles here, too. This year's report doesn't separate 'computer games' from other games, which would produce different results, namely a bigger share for strategy. And genre categorization is a tricky thing. Was Deus Ex: Mankind Divided counted as an action game, RPG, shooter, or all three? What are League of Legends and Dota 2 categorized as?

There's no mention of how many workers are contractors, how many were laid off, how many worked unpaid overtime, and so on.

And like every year, according to the ESA the games industry still makes a lot of money. "In 2016, the industry contributed $11.7 billion in value to US GDP," reads the press release. "This fueled the direct employment of 65,678 Americans and $30.4 billion in consumer spending, compared to $23.5 billion in 2015."

But, of course, there's no mention of how many workers are contractors, how many were laid off, how many worked unpaid overtime, and so on.

It's dangerous to treat the terms "survey results" and "facts" as synonymous, or to ignore who is producing reports. In this case, it's an organization built to support and promote companies in the games industry. So even though lots of the ESA's information is interesting, I hope gamers and developers treat it very critically when applying it to any arguments about the industry. You can read the ESA's full report here.