Whenere

A dispatch from Rhûn

Whenere (it’s just a portmanteau of “when” and “where”) is an outgrowth of the Virj project. The originator and vision keeper is my longtime creative collaborator Karen Laur. I’ll interview her in a future post. In the meantime here’s how it looks from my point of view as a novelist and a nerd.

Coherent-World Storytelling (CWS)

I read Dune and the Lord of the Rings when I was a teenager, and they changed my life. They have maps. Those are what really hooked me. But they, and other books like them, also have glossaries, historical timelines, and scholarly appendices about made-up languages.

Implied by the alternate-world setting is this: the story you’re reading is only one of many that could be told in this world.

I’m going to call this kind of thing coherent-world storytelling or CWS. It’s most strongly associated with fantasy and science fiction. But the more you look at romance and historical fiction, the more they look like CWS as well.

[ here, in an earlier draft, I typed in paragraphs of statistics trying to explain just how totally the media industry — movies, television, books, and games — is now dominated by CWS. But on re-reading it just seemed too obvious to be worth saying. Of the top 100 grossing movies of all time, 8 are set in anything resembling the real world. Only Titanic, Top Gun, and Oppenheimer really look like the traditional real-world movies that dominated the box office in the 1970s. ]

Why CWS has become the dominant medium

CWS gives the audience the tools and permission to imagine other stories in their favorite worlds.

In my mind, it all started with those maps, which are participatory by nature. You can’t look at a map of a fantasy world without asking yourself “hang on, what’s happening over here around the Sea of Rhûn?” or what have you.

Merely by doing that—by asking yourself that question—you’re venturing beyond the limits of what’s actually written in the book and beginning to create fan fiction in your head.

Fandom, which is only one site where fans of CWS gather to talk about the worlds they love, has hundreds of millions of users. San Diego Comic-Con, which is only one (albeit the largest) fan convention, has an annual attendance exceeding 150,000, with an economic impact in the hundreds of millions of dollars.

CWS and Video Games

Long ago the revenues of the video game industry surpassed those of Hollywood. Many big video games are set in coherent worlds from history, fantasy, or science fiction.

What’s interesting about video games, from a CWS standpoint, is that you can’t even make a video game without modeling the world. That’s what game engines such as Unity and Unreal do.

This world model includes not just visual elements such as buildings and landscapes, but the audio environment, the physics of how things interact, and the logic that makes it seem, well, coherent. It’s populated with characters that move around in it and interact with it in commonsensical ways (they can’t walk through walls. If there’s nothing underneath them, they fall).

Game engines are what you get if you take CWS and just literally implement it as a technology.

As time goes on, more and more functionality gets added to game engines. They can now deliver near-cinematic visual quality. Highly developed AI subsystems are now pre-packaged into them so that developers can imbue their NPCs with complex behaviors without having to code everything from scratch.

But that’s a different kind of AI from what’s been making the news lately.

What we talk about when we talk about AI (in games)

Until a couple of years ago, when we used the term AI in the context of games, it referred to the type of systems that have been used since at least the 1990s to control the behaviors of NPCs - some of them humanoids, but also monsters, animals, cockroaches, or anything capable of making decisions.

Recently, however, AI has exploded into public consciousness in an entirely different form based on large language models and large vision models. This form of AI has basically nothing in common with old-school game engine AI.

As I explained in the Virj post, and as you can learn by browsing the Inworld AI website, large language models (LLMs) have now been jacked into game engines in a way that makes it straightforward to drive NPCs from “brains” that the developer can customize so that they have appropriate, character- and world-specific knowledge and personality traits.

Talking: a new game interface.

When multiplayer games became possible, largely because of the Internet, voice conferencing systems got built into games so that you could have conversations with friends (and enemies) playing in the same virtual space. The words you spoke didn’t actually affect the game, because games weren’t yet smart enough to understand what you were saying. But simply being able to talk to your friends made everything more fun.

As voice became a more important aspect of gameplay, gamers began to invest in better audio gear, such as headsets with built-in microphones. COVID made Zoom ubiquitous and got everyone in the habit of talking to their computers.

What we learned from Virj was that talking is now a fully mature and viable interface for interacting with a game world. Not just talking to other human players, but having conversations with the NPCs that inhabit the world. And doing so with effect. Now, when you talk to an NPC, it actually matters what you say, because there’s an LLM brain at the other end of the line, parsing your speech, changing its state, and responding accordingly.

Brains, Space, and Time

TL;DR we can use LLMs in Coherent-World Storytelling. But we are pragmatic—we only use them for what they’re actually good at!

Brain

We were astonished at how well Virj could understand what we said, and talk back to us realistically. The brain part of it worked.

Space

But Virj didn’t really understand where he was in space. Having a 3D spatial model simply isn’t a feature of LLMs. Fortunately, we have game engines, the whole point of which is to keep track of things in space to an exquisite level of detail. And there’s no reason we can’t combine both kinds of AI: the old-school kind that ships with game engines for free, and LLMs.

Time

Real conversations have a beginning, middle, and end. They progress over time. They have a purpose.

In fiction, many such conversations, as well as other kinds of interactions, are woven together to create a plot that likewise has a beginning, middle, and end.

In theory we could get AI models to generate plots for us—but why? As fantasy writer Joanna Maciejewska recently said on Twitter, to the tune of 3 million views and 100,000 likes:

I want AI to do my laundry and dishes so that I can do art and writing, not for AI to do my art and writing so that I can do my laundry and dishes.

As I said above, coherent-world storytelling has become the world’s dominant medium because it gives its audience the tools and permission to imagine other stories in their favorite worlds. Quite obviously, people like doing that.

The idea behind Whenere is to give those people the power to enter their favorite worlds, to interact with them, and to tell their own stories in them—not just on the page but in beautifully rendered environments.

What we’re building

A platform that supplies built-out worlds in Unreal Engine that look great, pre-populated with characters that look and behave like the ones in the book. Out of the box, they can act out the original story, saying all of the lines of dialog verbatim in realistic voices (currently provided by Eleven Labs). If you have a different opinion as to what a given character looks like, you can change their appearance and attire. You can select different voices. You can rearrange the furniture, or swap out one house for another. And if you want to tell a different story, you can do that by writing a script, using a simple UI we’re building. You just specify who says what and how their emotional state changes. None of this requires learning how to use the game engine. You can output the results to video, so that you and your friends can just watch it on any device. Or, you can just go into the world, playing a role that makes sense in that world, and talk to the characters.

Current status

We have a prototype that demonstrates basic features and that has taught us what we need to do next. We intend to keep developing it, and we are raising money to pay for that.

The Genesis of Whenere

In March of 2023 we showed Virj at GDC, the Game Developers’ Conference. A conversation there between Karen Laur and Inworld AI’s Head of Production, Joana Flor re-activated some ideas Karen had been thinking about for years. Along with Jamil Moledina we fleshed it out over the summer and co-founded the company in August. Subsequently we were joined by Tadhg Kelly. A small team has been cranking on it ever since. Though the platform is intended to support an unlimited number of different coherent-world universes, we’re starting with Jane Austen’s Pride and Prejudice, partly because we love it, and partly because a lot of other people love it too.

Some people we’ve talked to have been surprised that we’re not leading with a Stephenson title, or at least something more cyberpunk. The answers are simple. My more well-known titles are encumbered by various rights deals that have become attached to them over the years. And, though I’m a successful novelist by any standard, my sales figures are microscopic in the big scheme of things. Snow Crash has sold something like a million copies over thirty years. Big hits like the Twilight novels sold tens of millions per year. Pride and Prejudice by some measures is America’s Favorite Book. Seems like an important fact when raising money from investors. And it consists largely of people standing around talking to each other, which is orders of magnitude cheaper to produce in a game engine than complicated sci-fi action sequences.

Up to this point, Whenere has been funded by myself and by a few other investors who see the potential, as well as by a grant from Epic Games. We’re launching an angel round now on WeFunder. As we continue to develop the project, I’ll post the occasional followup here, but we’ll also be active on other socials and on the WeFunder page itself.