Valve and Bungie team up to fix Destiny 2's Beaver errors
This one's complicated.
Good news for Destiny 2 players who've been struggling with Beavers recently: Bungie has finally brought the gnawing little beasties to heel, with some able help from Valve.
The Beavers in question, I should probably clarify, are Destiny 2 error codes. Rather than opting for mundane terminology that might hint at what the problem is, Bungie uses nicknames like Buffalo, Saxophone, Chive, Zebra, and Beaver, which has recently been rearing its toothy little head for some players. Beaver errors, along with Flatworm and Leopard, "are caused by a failure to connect your console to another player’s console via the internet," the Bungie help page says.
But figuring out what exactly was causing the spike in Beaver errors has been a real chore, as Valve software developer Fletcher Dunn explained, in great detail, on Twitter. He got involved in the process because Destiny 2 has been making use of peer-to-peer networking technology on Steam that he's been working on.
The tech is software-based routing: route through general-compute linux boxes using custom protocol, with clients exercising significant routing autonomy. Since peers do not share their IP addresses, script kiddies cannot DDoS other players.https://t.co/Uxp8hgThZXJuly 23, 2020
That resistance to DDoS attacks that Dunn describes in the tweet above sounds like an obvious big plus, but Dunn added that the disconnection rate—those pesky Beavers—was higher than expected after it launched.
"Some players would get it often; others never saw it. Restarting the game would often make it go away," he wrote. "We could never reproduce it. (It works on my machine!)"
After a couple of months of fixing bugs, working with Bungie, and bonding with his sons over a lot of Destiny 2, this happened:
The kernel does a heroic amount of work for you to deliver a single UDP packet, most of which is bypassed by XDP. So XDP is *insanely* fast. In our case, the XDP code can process 5x-10x the packets for the same CPU cost as the plain BSD socket code.July 23, 2020
It all gets quite technical, but the short version is that the discovery of an "oddball" relay unfolded into the realization that a bug was causing a communications breakdown under very specific circumstances, which in turn triggered another bug that kept any of it from being logged.
The biggest gaming news, reviews and hardware deals
Keep up to date with the most important stories and the best deals, as picked by the PC Gamer team.
Bungie confirmed that the fix is in place and works, although you may still see Beavers popping up now and then.
"This past week Valve identified hardware configuration issues with 4 relays in their Chicago, Virginia, Stockholm, and Dubai data centers. In each case, the affected relay was unable to send traffic to one other relay in the same data center. If a connection to a peer went through both of those relays, then it would drop. Valve has fixed the configuration issues, and we have confirmed that the rate of disconnections in the affected areas has been reduced significantly," it wrote in the latest This Week at Bungie update.
"These changes have reduced Beaver error codes significantly with players in the central and eastern United States of America being impacted the most. Beaver error codes are an indication of connectivity failure and can be caused by a wide range of issues. Some Beaver error codes are caused by local internet connectivity issues. If you still experiencing a lot of Beaver errors please use follow our Network Troubleshooting Guide and then report any further issues on our Help forum."
As for why it took months to figure all of this out, Dunn summed it up nicely:
Why didn’t monitoring catch this?We do have significant monitoring, especially for problems *between* data centers. But it did not detect problems between hosts in the same data center.Also: networking is complicated.🧵>July 23, 2020
May the Mother Beaver watch over us all.
Andy has been gaming on PCs from the very beginning, starting as a youngster with text adventures and primitive action games on a cassette-based TRS80. From there he graduated to the glory days of Sierra Online adventures and Microprose sims, ran a local BBS, learned how to build PCs, and developed a longstanding love of RPGs, immersive sims, and shooters. He began writing videogame news in 2007 for The Escapist and somehow managed to avoid getting fired until 2014, when he joined the storied ranks of PC Gamer. He covers all aspects of the industry, from new game announcements and patch notes to legal disputes, Twitch beefs, esports, and Henry Cavill. Lots of Henry Cavill.
'Destiny has a long history of reinventing itself in response to feedback': Assistant director teases a Metroidvania-inspired future, talks weapon crafting and vault space, but fails to address the shocking number of bugs
Ballistic, Fortnite's new tactical FPS mode, is a deeply unserious Counter-Strike clone that's going to be huge anyway