Modder wires ChatGPT into Skyrim VR so NPCs can roleplay and remember past conversations

Spend enough time in any RPG and you'll eventually run out of things to talk about with its characters. But what if they had a never-ending supply of dynamically generated anecdotes? What if you could ask them questions that weren't listed on a menu in front of you? What if they could even remember the experiences they've had with you in the game, and could talk about them at length? Those are the questions modder Art From The Machine is trying to answer in Skyrim VR.

In the video below you can see some scenes from the work-in-progress mod, which uses OpenAI's large language model ChatGPT to generate responses, xVASynth for text-to-speech so the NPCs can be fully voiced, and Whisper for speech-to-text, so players can speak into their mic and the NPCs can understand them. 

That's a lot of different systems being put into play, and the early results are some pretty stilted responses. But at the same time you see a glimmer of the potential of NPCs wired up with AI to create dynamic conversations:

There are a bunch of different conversations and snippets shown in the video, but the most interesting is a chat with Ulfberth War-Bear, star employee of Warmaiden's in Whiterun. Thanks to the mod, Ulfberth knows the shop's hours of operation and can tell them to the player (though awkwardly) and can even calculate how soon the store will close based on the in-game time. 

Yeah, Ulf's pronunciation is pretty awful—though it does sound like his voice, and he's lip synched, which is impressive—but I like the idea of playing a game where a simple question like "How long until you close?" pops into my head and I can just ask a shopkeeper and get a straightforward answer. That'd help make an NPC feel more like a real person.

The modder also picks up a sword and asks Ulfberth to describe it, which he does: "It seems to be a well-crafted iron sword with a soul gem embedded in the hilt. The enchantment on it allows the wielder to capture the souls of their enemies."

It's not exactly scintillating as Ulfberth recites what feels like a few lines from a game wiki, and it's not particularly impressive that he 'knows' details about the sword—the game itself obviously knows what kind of sword it is, too. But it's not this exact conversation that's exciting, it's that it's being generated on the fly (though slowly) and what it could mean in future games. NPCs reacting dynamically to something you just did or said, without having to have that specific dialogue written and recorded for it in advance—that's interesting to think about.

Even more interesting is if these AI-enabled NPCs could remember past events and conversations when you talk to them later, something Art From The Machine is working on in this mod. "I have a basic memory system set up where I ask ChatGPT to summarize the conversation on exit to help condense it down for future prompts," they say on Reddit. "There are much more sophisticated tools out there to handle memory though such as Langchain which I am hoping to implement in the future."

The mod isn't available to try yet—Art From The Machine wants to add more features and do more bug testing before releasing it. But Skyrim isn't the only game being modded with AI tools: Mount & Blade: Bannerlord has a mod that lets its NPCs use AI to answer questions typed by the player. You can see a video of that mod here.

Both of these mods are impressive, though I think apart from the technical challenges of wiring AI into NPCs, the issue for me is this: even if NPCs can endlessly conjure up new dialogue, is all that chatter going to be interesting enough to want to listen to? We recently saw researchers populate a virtual town with AI agents, and even though it's cool that they threw parties and talked with each other, the conversations themselves were… pretty dull. I don't think the tech to make NPCs talk endlessly is going to matter that much if they ultimately don't have anything interesting to say. Hopefully most games and mods in the future will stick with good old-fashioned dialogue written and performed by humans.