Stable Diffusion VR is a startling vision of the future of gaming

Scottie Fox's stable diffusion VR
(Image credit: ScottieFoxTTV)
Audio player loading…

A while ago I spotted someone working on real time AI image generation in VR and I had to bring it to your attention because frankly, I cannot express how majestic it is to watch AI-modulated AR shifting the world before us into glorious, emergent dreamscapes. 

Applying AI to augmented or virtual reality isn't a novel concept, but there have been certain limitations in applying it—computing power being one of the major barriers to its practical usage. Stable Diffusion image generation software, however, is a boiled-down algorithm for use on consumer-level hardware and has been released on a Creative ML OpenRAIL-M licence. That means not only can developers use the tech to create and launch programs without renting huge amounts of server silicon, but they're also free to profit from their creations.

I was awoken in the middle of the night to conceptualize this project

Scottie Fox - Stable Diffusion VR dev

ScottieFoxTTV (opens in new tab) is one creator who's been showing off their work with the algorithm in VR on twitter. "I was awoken in the middle of the night to conceptualize this project," he says. As a creator myself, I understand that the Muses enjoy striking at ungodly hours.

What they brought to him was an amalgamation of Stable Diffusion VR and TouchDesigner (opens in new tab) app-building engine, the results of which he refers to as "Real-time immersive latent space." That might sound like some hippie nonsense to some, but latent space is a concept fascinating the world right about now. 

At a base level, it's a phrase that in this context describes the swelling potential that artificial intelligence brings to augmented reality as it pulls ideas together from the vastness of the unknown. While it's an interesting concept, it's one for a feature at a later date. Right now I'm interested in how Stable Diffusion VR manages to work so well in real time without turning any consumer GPU (even the recent RTX 4090 (opens in new tab)) into a smouldering puddle.

See more

Scottie Fox's stable diffusion VR

(Image credit: ScottieFoxTTV)

"Diffusing small pieces into the environment saves on resources," Scotty explains. "Small clips are sent from the engine to be diffused. Once ready, they're queued back into the projection." The blue boxes in the images here show the parts of the image being worked on by the algorithm at any one time. It's a much more efficient way to have it working in real time.

Anyone who's used an online image generation tool will understand that a single image can take up to a minute to create, but even if it does take a little while to work on each individual section, the results still feel like they're happening immediately as you're not focusing waiting for a single image to finish diffusing. And although not at the level of photorealism as they may one day be, the videos Scotty's been posting are utterly breathtaking.

Your next upgrade

(Image credit: Future)

Best CPU for gaming (opens in new tab): The top chips from Intel and AMD
Best gaming motherboard (opens in new tab): The right boards
Best graphics card (opens in new tab): Your perfect pixel-pusher awaits
Best SSD for gaming (opens in new tab): Get into the game ahead of the rest

Flying fish in the living room, ever-shifting interior design ideas, lush forests and nightscapes evolving before your eyes. With AI able to make projections onto our physical world in real time, there is so much potential for use in the gaming space.

Midjourney CEO David Holz describes the potential for games to one day be "dreams" (opens in new tab) and it certainly feels like we're moving hastily in that direction. Though, the important next step is navigating the minefield that is the copyright and data protection issues (opens in new tab) arising around the datasets that algorithms like Stable Diffusion trained on. 

Katie Wickens
Hardware Writer

Screw sports, Katie would rather watch Intel, AMD and Nvidia go at it. Having been obsessed with computers and graphics for three long decades, she took Game Art and Design up to Masters level at uni, and has been demystifying tech and science—rather sarcastically—for two years since. She can be found admiring AI advancements, scrambling for scintillating Raspberry Pi projects, preaching cybersecurity awareness, sighing over semiconductors, and gawping at the latest GPU upgrades. She's been heading the PCG Steam Deck content hike, while waiting patiently for her chance to upload her consciousness into the cloud.