Diablo 4's lead class designer called trying to test an action RPG made to be played for hundreds of hours "a cursed problem" (referencing a 2019 Riot GDC talk) that would take "more hours than what you would normally expect someone to work at a job," in an interview today with Diablo 4 streamer Raxxanterax.
Jackson was answering a question about how the team incorporates player feedback and internal playtesting into Diablo 4's development. He said the live team regularly monitors feedback from Reddit, streams, and the forums but also spends every day playing the game on their own, keeping a list of things that could be better.
"We have the ability to see data all over the game of how many people are equipping different legendary aspects, how many people are engaging with certain types of content," Jackson says. "We can see what's working, what's not working, what people are actually doing versus what they say."
But player feedback and data don't exist for new systems that haven't been added to the game yet, which is "where things get a lot harder," Jackson says. He calls it a "cursed problem" trying to thoroughly test a game as complex as Diablo 4.
We have to use our guts and existing ideas and telemetry to figure out if the decisions we make are right.
Adam Jackson, lead class designer
"Diablo is a freakin' huge game that takes forever to play and really truly understand because our systems are so interconnected. So there is no substitute for actually going through that 80-hour experience and playing everything," he explained. "The part that makes that really hard, and where I think the community gives us a hard time sometimes … is that to really test our game you need to spend your life playing our game."
That's why they need the help of data analysts and QA testers, he says. "I would love to play season 5 200 hours before it comes out [but] I only get it for 50 hours. We have to use our guts and existing ideas and telemetry to figure out if the decisions we make are right."
As an example, Jackson pointed to Diablo 4's new masterworking crafting system on the season 4 public test realm (PTR) last month. Every single time you upgraded an item, you were forced to watch a short animation of a circle filling up with molten metal. Players immediately revolted at the prospect of having to sit through it hundreds of times as they play.
Blizzard has since fixed that issue by including a skip button, but Jackson says it's an example of the kind of thing that's hard to test internally. "That kind of stuff is where sometimes things fall through the cracks," he said, and that you have to balance the feedback of "doing this thing 2000 times is actually terrible because you gotta wait forever versus well, we want to have a moment of celebration and for the cool thing to be cool."
Now, Jackson says, the team considers what every new feature should be like if you're doing it for 200 hours, while also thinking about the presentation of it for less-experienced players. "The answer is not always so obvious, but we are receptive to when the community tells us clearly 'hey, no, we want it this way,' we're going to do it that way."
Diablo 4 season 4 launches on May 14 with one of its biggest updates ever.
