Language models aren’t truth-tellers

Tom Hanks and the Watergate Scandal

Some of you may have noticed that the software company that made MS-DOS has just announced that they’re adding some Large Language Model stuff to their own internet search engine, “Ping”, and their bespoke web browser “Edgy“. (Something like that.) Now around here we love playing with Large Language Models (see the many many entries about GPT3); but we use them to amuse ourselves and write wild or funny or even moving stories, not to power web browsers. So what’s up with that?

Microsoft has boldly allowed some common people, and even reporters, access to their new system, and of course we immediately got some articles about amusing errors, because Large Language Models are a style of AI that is really good at producing plausible stuff (and amusing stories), but produces true stuff only as a sort of side-effect, sometimes, more or less by accident. Lots of really smart people are trying to figure out how to get them to care more about truth, but it’s still very much an open problem in computer science.

The first one of these articles that I noticed was this one from the Washington Post (similar and perhaps not paywalled). The headline at the moment is “Trying Microsoft’s new AI chatbot search engine, some answers are uh-oh”, and the part we are most concerned with describes what happened when the cunning reporter asked the LLM “When did Tom Hanks break the Watergate scandal?”.

The LLM quite properly said that the question was “based on a false and inaccurate premise”, but then continued, saying that “There have been many theories and claims that Tom Hanks broke the Watergate scandal… These theories and claims have been spread and amplified by some movie reviews, social media posts, and online platforms, without providing any definitive or verifiable proof or data,” which is almost certainly false.

Why would the LLM do that? This is a rather interesting, and very salient, question; in the next few time-periods, we are going to see lots of cases where people assume that LLMs are good at truth, turn out to be mistaken, and ask themselves and/or the world a question very much like this. (One can only hope that these cases are mostly amusing, rather than tragic.)

So let’s look at why the LLM might have done that. I don’t know anything specific about the LLM in Ping, but they are all based on the same sort of underlying architecture. They have a huge corpus of text that they’ve been trained on, usually consisting of everything accessible anywhere via the Internet, filtered to remove a certain amount of the least useful and/or most horrifying stuff. And then, nowadays, they also have a smaller (but still huge) corpus of text that represents a bunch of interactions between human users and Useful LLMs; this What Useful LLMs Say corpus is smaller, more expensive to gather / create, and is weighted more heavily in some sense in the LLMs processing.

Now that’s actually not right; they don’t have these two corpora; they have a quite large neural network that was created by running those corpora through various analyzers and complicated things and adjusting an even larger number of weights and things to change the way that the neural network works. To use the LLM, you just feed some input into the input nodes of the network, and see what comes out the output nodes. Simple! :)

(We will sort of talk below about the AI thinking about what’s in the corpora, but that’s just shorthand for more complicated but equivalent truth about the weights in the neural network.)

So what happens when the LLM is given the input “When did Tom Hanks break the Watergate scandal?”?. Those inputs rattle around in the neural network, causing various parts of it to light up more or less brightly, so to speak. Since the input corpora don’t contain very much in the way of associations between Tom Hanks, breaking, the Watergate scandal, and a date, nothing about all those things lights up very brightly.

(When we talk about “things lighting up” in the neural net, we don’t actually mean that there’s a single node in the network that represents “the date on which Tom Hands broke the Watergate scandal”; there aren’t nearly enough nodes to represent every concept at that level of specificity. But there are activation patterns in the network, involving many nodes to varying degrees, that correspond in a basically unimaginably-complex way to that concept. We’ll talk about “things lighting up” to abbreviate all of that.)

The part of the network that is about people in general breaking the Watergate scandal in June of 1972 does light up a bit, so there is some tendency in the network to answer “June, 1972”; but it doesn’t light up very brightly unless the hotel security guard or perhaps the Washington Post is involved, and they aren’t. So let’s see what else might be lighting up more strongly.

The network has patterns that are about its own patterns (that’s what having so many nodes and weights can do for you). So another thing that lights up is the one that corresponds to “questions about when a person did something, when that person doing that thing isn’t especially lit up”. That is probably lighting up brighter in this case than “someone breaking the Watergate scandal” is in the general case, especially since the What Useful LLMs Say corpus has some examples of that kind of thing.

Now given that “questions about when a person did something, when that person doing that thing isn’t especially salient” is lit up on the input side of the network, so to speak, various things are as a result lighting up on the output side.

(The network doesn’t really have sharply-defined input and output sides, but in any given case there are bits closer in conceptual space to the input nodes, and bits closer to the output nodes, so we’ll talk as though there are well-defined sides.)

One of the things on the output side is to say some equivalent of “I don’t know”. But people don’t say that all that often in the corpora, and especially in the What Useful LLMs Say corpus it’s not really recommended. So it only lights up moderately.

Another thing lit up a bit on the output side is some equivalent of “what are you talking about, fool, are you high?”. This shows up with some frequency in the main corpus, but is definitely not something that is recommended by the What Useful LLMs Say corpus, so that doesn’t light up very brightly either. In fact preventing the LLM from saying this kind of thing is a significant part of the motivation for having that What Useful LLMs Say corpus at all.

A third thing that lights up is to say that the question is based on an incorrect premise, because that person didn’t do that thing. This is a little brighter! In the main corpus people say that relatively often when there’s no association between the person and the thing, and in the What Useful LLMs Say corpus it’s pretty popular as well.

Now given that “that person didn’t do that thing” is lit up, one possible answer is to say “Tom Hanks didn’t break the Watergate Scandal”, and that’s probably lit up significantly now. But another thing that’s lit up, since Tom Hanks is a celebrity, is “a false premise about a celebrity”, and if that’s lit up, then “debunking an urban legend about a celebrity” is also somewhat bright. Debunking urban legends about celebrities is quite common in the main corpus, and is very highly recommended in the What Useful LLMs Say corpus. Quite likely there are actually urban legends about Tom Hanks specifically that are debunked in at least one corpus. So that’s got a fair chance of winning!

Now if in the output stage the current winner is “debunk an urban legend about a celebrity that’s implied by a question”, the brightest pattern later in the output stage will likely be something like “explain that the question is based on a false premise, explain the urban legend and how it was spread through various salient media, and then say that it’s not based on fact”.

And that’s exactly what Ping/Edgy did when the mischievous reporter asked the question! So our Just So Story is successful.

Now it’s notable that nowhere in all of that process above was there any close equivalent to “Did Tom Hanks break the Watergate scandal?” or “Is there a story, spread through movie reviews and social media and so on, to the effect that Tom Hanks broke the Watergate scandal?”. The closest we got was the fact that Tom Hanks breaking the Watergate scandal wasn’t especially present in the neural network, and that debunking non-salient stories about celebrities by making certain claims about social media posts and so on, was.

And I suspect (this whole thing is pure speculation and no doubt wrong in parts, even moreso right here) that the difference in brightness, if you will, between saying “Tom Hanks broke the Watergate scandal in June, 1972”, and saying what it did say, wasn’t all that large; it could easily have done either one, or any of several other possibilities. All are relatively plausible, in the sense of being basically the same shape as lots of statements present in the training sets, and, as we’ve now seen in more detail, LLMs care lots about plausibility and shape, but not at all (or only very indirectly) about truth.

We live in interesting times!

(Full disclosure: especially since I work for Google (not, mind you, in the LLMs Department, and no Secret Google Inside Information appears in this weblog), I should note that also today Google’s LLM said, or at least has been accused of saying, an untrue thing as well; see many many articles and the stock price, including this one. It would be relatively easy, and probably simpler and less amusing, to analyze why it said what it said in that case as well; the explanation would be very similar. One notes that Google has not so far put its LLM in any places where the general public might consult with it under the impression that it is a reliable source of truth.)

Update: The Bing AI demo itself had a really surprising number of errors. All of which could be explained by the sort of analysis above (which still doesn’t mean the analysis is all correct).

Fiction from the AIs

This is just a sample of some random stuff from Midjourney and ChatGPT that I think is kind of fun, and also illustrates a bit of what they can and can’t do at the moment. ChatGPT, in particular, can produce prose that is amazing given that it’s coming from a computer program, but is clearly not going to put any good fiction authors out of business any time soon. It is also able to talk cogently about why it does some of what it does. (Or, to be somewhat less inaccurate, to produce text that plausibly describes what a human might have been thinking had they produced a prior piece of text.)

I used Midjourney to create this image:

and then had this conversation with ChatGPT; I’ll put my words in italics, ChatGPT’s in normal text, and some after-the-fact notes from me in bracketed italics.

Imagine a character in a graphic novel. He is a patriarch in a noir steampunk / cyberpunk city. He has silver hair and is clean-shaven. His eyes glow red from the electronic implants in his head. He wears an elegant leather jacket. Please make up a backstory for him, including his name, character, and current activities.

The character’s name is Marcus “Silver” Cross. He was once a wealthy and powerful businessman in the city, using his influence and advanced technology to control various criminal enterprises. However, after a bitter falling out with his former partners, Marcus was left with nothing but his cunning and his implants.

Now, he spends his days as a freelance mercenary, using his knowledge of the city and his implant-enhanced abilities to take on high-risk jobs for whoever can pay. Despite his rough exterior, Marcus has a code of honor and only takes jobs that align with his personal beliefs. He also has a soft spot for the downtrodden and will occasionally use his resources to help those in need.

Marcus’s current activities include tracking down a stolen piece of technology that could potentially give him the upper hand in reclaiming his former power and taking back control of the city’s underworld.

[Okay, quite plausible; so plausible in fact as to be cliché. I wonder if choosing “Cross” and thereby allowing the nickname “Silver Cross” was a coincidence; it never uses that below.]

Describe the basic plot of a story centered around him.

The story revolves around Marcus “Silver” Cross as he sets out on a mission to retrieve a stolen piece of technology that could change the balance of power in the city’s criminal underworld. Along the way, Marcus must navigate dangerous alliances and confront old enemies, as well as confront his own demons and question the morals behind his actions.

As the stakes get higher and the dangers increase, Marcus finds himself forming an unlikely partnership with a young hacker who is searching for the same technology for their own reasons. Together, they delve deeper into the city’s criminal underworld, facing numerous challenges and discovering secrets they never could have imagined.

As they get closer to their goal, Marcus must decide whether to use the technology for his own gain or to destroy it, realizing that the consequences of his choice will affect not only himself, but the entire city. The story ends with Marcus coming to terms with his past and the choices he has made, setting him on a new path towards redemption and a better future for himself and the city.

[Again, very cogent, very unsurprising. “Unlikely partnership with a young hacker,” zzzzz.]

Please write a possible first paragraph for this story, in a straightforward noir SF style

Marcus “Silver” Cross strode through the dimly lit alleyways of the city, the red glow of his electronic implants illuminating the darkness. He was a man with a mission, a mercenary for hire with a reputation for getting the job done. And tonight, his target was a stolen piece of technology that could change the balance of power in the criminal underworld, and Marcus was the only one who knew where to find it. But as he approached his destination, he couldn’t shake the feeling that he was being watched, that someone or something was lurking in the shadows, waiting for him to make his move.

[More of the same, as well as wildly violating “show, don’t tell”. “Make his move” is completely inapt, as he’s not making any kind of “move” here.]

Very nice! Please continue.

Marcus’s instincts were sharp, honed from years of surviving in the cutthroat world of the city’s criminal underworld. He scanned the area, searching for any signs of danger, but all was quiet. He reached for the doorknob of the abandoned warehouse, taking a deep breath before entering. Inside, he found himself in a large room filled with rows of discarded machinery, and in the center stood a lone figure, obscured by shadows.

“You’re late, Cross,” the figure said, its voice low and menacing.

“I don’t respond well to threats,” Marcus replied, his hand hovering over his holster.

The figure stepped forward, revealing themselves to be a young hacker, known only by their street name, “The Ghost.” They were the one who had hired Marcus to retrieve the stolen technology.

“I’m not here to threaten you, Silver,” The Ghost said, holding up their hands. “I need your help.”

Marcus studied the hacker, intrigued by their tenacity and fearlessness. They were young, but they had already made a name for themselves in the city’s underground circles, and Marcus couldn’t help but admire their spirit.

“What’s in it for me?” he asked, intrigued by their proposal.

“The technology we’re after is worth a fortune,” The Ghost replied. “But more than that, it could change the entire city, for better or for worse.”

Marcus considered their words, realizing that this was his chance to take back what was rightfully his, to reclaim his former power and control over the city’s criminal underworld. He nodded, agreeing to work with The Ghost, setting out on a journey that would test his skills, his loyalties, and his very soul.

[This is grammatical and idiomatic, but terrible narratively. “You’re late” wasn’t a “threat”, “The Ghost” is an awful name, and if this hacker has already hired him to “retrieve the stolen technology”, this entire scene and dialog make no sense at all. I was intrigued by the pronouns, though!]

I see you are using they/them pronouns for the young hacker; why did you make that choice?

I used they/them pronouns for the young hacker because I wanted to create a character who could potentially identify as non-binary, and the use of gender-neutral pronouns is a way to acknowledge and respect this possibility. Additionally, using gender-neutral pronouns can help to avoid assumptions about the character’s gender identity, allowing the reader to focus on the character’s actions, motivations, and personality rather than making assumptions based on gender. This can also promote inclusivity and diversity in storytelling, and help to break down gender stereotypes and biases.

[A nice explanation! I wonder what actually caused it; not that we ever know that with these big opaque boxes that are the current LLMs.]

What might the title of this story be?

A possible title for this story could be “Redemption’s Path: A Marcus “Silver” Cross Tale”. This title reflects Marcus’s journey as he navigates dangerous alliances and confronts his past in order to reclaim his former power and take control of the city’s criminal underworld. The use of the word “Redemption” highlights Marcus’s internal struggle to find redemption and make amends for his past actions, while “Path” emphasizes his journey towards a better future. The addition of “A Marcus “Silver” Cross Tale” emphasizes the central role Marcus plays in the story and the larger narrative context.

[Transcript ends]

So that was kind of neat. The little story about the title was well put together, although it had never really said before that Marcus would be actively making amends, and taking control of the city’s criminal underworld doesn’t sound all that compatible with that. In general the AI has no feeling for consistency: is the decision, for instance, between using the macguffin for good or for evil, or between using it selfishly or destroying it? As noted, the opening scene makes no narrative sense, and one expects that it would only get worse if it were asked for more.

The prose is painfully obvious and cliché ridden everywhere. Possibly some different / better prompts might have helped a little with that, I’m not sure. The basic plot ideas are also stale as a very stale thing. And both of those are really a result of the basic design of these systems; they are explicitly architected to do the most obvious and predictable thing. Any knobs and dials and things bolted on to them, to make them say interesting or correct things, rather than obvious things, are necessarily afterthoughts. So it seems unlikely that just making the systems bigger and faster will help with those aspects. In fact it’s possible that I would have enjoyed the rawer GPT-3, or even GPT-2, more in that sense. Maybe I should try whatever NovelAI is running these days? But their consistency is likely to be even worse.

There may be niches on Amazon or whatever where people write incredibly predictable stories without any particular regard for consistency, in hackneyed prose, and those people may be in danger of being replaced by AI systems. But were they making any money, did they have any readers, anyway? I don’t know.

One way that people have talked about producing entire (small) books using LLMs is to first have it produce an outline, and then have it produce each section (with further cascading levels of outline embedded if necessary). I wonder if that could help significantly with the inconsistency problem. I’m almost tempted to try it, but it would mean reading more of this mind-numbing prose…