Foveated streaming is the genius tech behind Valve's Steam Frame and its seamless wireless gaming performance

The Steam Frame has a heap of new features to dig into, but I want to focus on one here: foveated streaming. It sounds like a medical issue but I assure you, it's all about ensuring the Steam Frame's wireless connection is strong, stable, and appears high quality.

The Steam Frame offers a few ways to play games, both locally on its own Arm chip and streamed from a nearby gaming PC. As the name suggests, foveated streaming is used during the latter.

"So, a lot of people might have heard of foveated rendering," Jeremy Selan, an engineer at Valve working on the Steam Frame, says. "That's where you use the direction you're looking at to optimise what the device is doing. So it only has to do a lot of work in that area. We are doing something similar to that dedicated to the streaming process."

Foveated streaming requires the Steam Frame's new eye tracking feature. This uses two cameras embedded just above the wearer's eyes, through which Valve can very quickly monitor where a user is looking at any one time. The clearest patch of your vision, that's thanks to your fovea, hence the name.

Connecting to a PC via its own wireless adapter and a dedicated 6 GHz connection, the Steam Frame has a relatively high level of bandwidth available to it. But it also uses two 2160 x 2160 screens and can run up to 144 Hz (in an experimental mode), which combined with the audio stream, is a lot of data to beam across even a high-capacity wireless connection.

To get around these limitations, Valve is only sending the highest quality image to the area directly in front of where a user is looking.

"We can send a very, very high-fidelity representation of the source data for where you're looking, and then spend far fewer bits on the surrounding area," Selan says.

"So, you could imagine that if that foveated area, say, represents 10% of the full field of view, it would actually be a 10x multiplication factor in bandwidth, in latency, in robustness."

Rather than send two views (one for each eye), which is usually what's required for VR, Valve is sending four. Two views are sent in low resolution and represent the entire field of view. The other two are only a fragment of the complete picture, but sent in high resolution.

"We send a bunch more bits to that area than we do to the low-resolution imagery," Selan says.

It's like a magnifying glass, offering a better glimpse of whatever it hovers over, and following a user's vision.

"The trick is that the eye tracking is fast enough that it essentially beats where you're going. So, by the time you look to the right, we've already sent the bits at the very high resolution to where you're looking to the right. When we send if you look to the left, we beat your eyes looking there where we already have landed the bits at the full resolution."

I've used the Steam Frame and I never noticed any low-resolution areas in my vision. I had almost forgotten about the foveated streaming until it was mentioned to me. I tried darting my eyes from left to right to try and trick the eye tracking, but it totally kept up.

Image 1 of 2

"We find that even for the most observant viewers, it's very, very hard to see what's happening," Selan says.

I take this as a bit of a challenge, but I still can't see any difference in quality anywhere on the screen. Even when Selan enabled a special mode in Half-Life Alyx that highlighted the high-quality area, where I'm looking, I still couldn't make out the low-quality areas. I don't know if you've ever tried to view something in your peripheral vision without looking at it, but it's actually quite tough. Moreover, that part of your vision isn't very detailed—your brain fills in a lot of the gaps. That's what foveated streaming is taking advantage of.

Foveated rendering, a feature you might be more familiar with, works in a similar way with eye tracking. What's known as quad view rendering is essentially the exact same system. The key difference here is that foveated streaming doesn't lower the performance requirements for the PC that's rendering the VR game to be sent to the Steam Frame—it's working overtime to deliver the full, crisp image.

That's a good and a bad thing. On the one hand, if your gaming PC is struggling to play VR games, foveated streaming won't help. On the other, foveated streaming doesn't require a developer to do anything. Though I wouldn't worry about picking between them: Valve has confirmed you will be able to use both at the same time in games that support foveated rendering.

"Most critically, the software developer, the partners who ship on Steam, don't have to do any work to take advantage of that," Selan says.

Having spent time without a cable to anchor me down, I can see why Valve is pushing to remove wires from the VR experience. It's far more convincing for a lack of it. So too does the streaming experience feel the most convincing part of the Steam Frame, even more so than its new onboard processing capabilities. Not the least bit because the battery life while streaming should be many more hours than playing games on the headset itself. That feels like an easy win, as does foveated streaming.