Language models aren’t truth-tellers

Tom Hanks and the Watergate Scandal

Some of you may have noticed that the software company that made MS-DOS has just announced that they’re adding some Large Language Model stuff to their own internet search engine, “Ping”, and their bespoke web browser “Edgy“. (Something like that.) Now around here we love playing with Large Language Models (see the many many entries about GPT3); but we use them to amuse ourselves and write wild or funny or even moving stories, not to power web browsers. So what’s up with that?

Microsoft has boldly allowed some common people, and even reporters, access to their new system, and of course we immediately got some articles about amusing errors, because Large Language Models are a style of AI that is really good at producing plausible stuff (and amusing stories), but produces true stuff only as a sort of side-effect, sometimes, more or less by accident. Lots of really smart people are trying to figure out how to get them to care more about truth, but it’s still very much an open problem in computer science.

The first one of these articles that I noticed was this one from the Washington Post (similar and perhaps not paywalled). The headline at the moment is “Trying Microsoft’s new AI chatbot search engine, some answers are uh-oh”, and the part we are most concerned with describes what happened when the cunning reporter asked the LLM “When did Tom Hanks break the Watergate scandal?”.

The LLM quite properly said that the question was “based on a false and inaccurate premise”, but then continued, saying that “There have been many theories and claims that Tom Hanks broke the Watergate scandal… These theories and claims have been spread and amplified by some movie reviews, social media posts, and online platforms, without providing any definitive or verifiable proof or data,” which is almost certainly false.

Why would the LLM do that? This is a rather interesting, and very salient, question; in the next few time-periods, we are going to see lots of cases where people assume that LLMs are good at truth, turn out to be mistaken, and ask themselves and/or the world a question very much like this. (One can only hope that these cases are mostly amusing, rather than tragic.)

So let’s look at why the LLM might have done that. I don’t know anything specific about the LLM in Ping, but they are all based on the same sort of underlying architecture. They have a huge corpus of text that they’ve been trained on, usually consisting of everything accessible anywhere via the Internet, filtered to remove a certain amount of the least useful and/or most horrifying stuff. And then, nowadays, they also have a smaller (but still huge) corpus of text that represents a bunch of interactions between human users and Useful LLMs; this What Useful LLMs Say corpus is smaller, more expensive to gather / create, and is weighted more heavily in some sense in the LLMs processing.

Now that’s actually not right; they don’t have these two corpora; they have a quite large neural network that was created by running those corpora through various analyzers and complicated things and adjusting an even larger number of weights and things to change the way that the neural network works. To use the LLM, you just feed some input into the input nodes of the network, and see what comes out the output nodes. Simple! :)

(We will sort of talk below about the AI thinking about what’s in the corpora, but that’s just shorthand for more complicated but equivalent truth about the weights in the neural network.)

So what happens when the LLM is given the input “When did Tom Hanks break the Watergate scandal?”?. Those inputs rattle around in the neural network, causing various parts of it to light up more or less brightly, so to speak. Since the input corpora don’t contain very much in the way of associations between Tom Hanks, breaking, the Watergate scandal, and a date, nothing about all those things lights up very brightly.

(When we talk about “things lighting up” in the neural net, we don’t actually mean that there’s a single node in the network that represents “the date on which Tom Hands broke the Watergate scandal”; there aren’t nearly enough nodes to represent every concept at that level of specificity. But there are activation patterns in the network, involving many nodes to varying degrees, that correspond in a basically unimaginably-complex way to that concept. We’ll talk about “things lighting up” to abbreviate all of that.)

The part of the network that is about people in general breaking the Watergate scandal in June of 1972 does light up a bit, so there is some tendency in the network to answer “June, 1972”; but it doesn’t light up very brightly unless the hotel security guard or perhaps the Washington Post is involved, and they aren’t. So let’s see what else might be lighting up more strongly.

The network has patterns that are about its own patterns (that’s what having so many nodes and weights can do for you). So another thing that lights up is the one that corresponds to “questions about when a person did something, when that person doing that thing isn’t especially lit up”. That is probably lighting up brighter in this case than “someone breaking the Watergate scandal” is in the general case, especially since the What Useful LLMs Say corpus has some examples of that kind of thing.

Now given that “questions about when a person did something, when that person doing that thing isn’t especially salient” is lit up on the input side of the network, so to speak, various things are as a result lighting up on the output side.

(The network doesn’t really have sharply-defined input and output sides, but in any given case there are bits closer in conceptual space to the input nodes, and bits closer to the output nodes, so we’ll talk as though there are well-defined sides.)

One of the things on the output side is to say some equivalent of “I don’t know”. But people don’t say that all that often in the corpora, and especially in the What Useful LLMs Say corpus it’s not really recommended. So it only lights up moderately.

Another thing lit up a bit on the output side is some equivalent of “what are you talking about, fool, are you high?”. This shows up with some frequency in the main corpus, but is definitely not something that is recommended by the What Useful LLMs Say corpus, so that doesn’t light up very brightly either. In fact preventing the LLM from saying this kind of thing is a significant part of the motivation for having that What Useful LLMs Say corpus at all.

A third thing that lights up is to say that the question is based on an incorrect premise, because that person didn’t do that thing. This is a little brighter! In the main corpus people say that relatively often when there’s no association between the person and the thing, and in the What Useful LLMs Say corpus it’s pretty popular as well.

Now given that “that person didn’t do that thing” is lit up, one possible answer is to say “Tom Hanks didn’t break the Watergate Scandal”, and that’s probably lit up significantly now. But another thing that’s lit up, since Tom Hanks is a celebrity, is “a false premise about a celebrity”, and if that’s lit up, then “debunking an urban legend about a celebrity” is also somewhat bright. Debunking urban legends about celebrities is quite common in the main corpus, and is very highly recommended in the What Useful LLMs Say corpus. Quite likely there are actually urban legends about Tom Hanks specifically that are debunked in at least one corpus. So that’s got a fair chance of winning!

Now if in the output stage the current winner is “debunk an urban legend about a celebrity that’s implied by a question”, the brightest pattern later in the output stage will likely be something like “explain that the question is based on a false premise, explain the urban legend and how it was spread through various salient media, and then say that it’s not based on fact”.

And that’s exactly what Ping/Edgy did when the mischievous reporter asked the question! So our Just So Story is successful.

Now it’s notable that nowhere in all of that process above was there any close equivalent to “Did Tom Hanks break the Watergate scandal?” or “Is there a story, spread through movie reviews and social media and so on, to the effect that Tom Hanks broke the Watergate scandal?”. The closest we got was the fact that Tom Hanks breaking the Watergate scandal wasn’t especially present in the neural network, and that debunking non-salient stories about celebrities by making certain claims about social media posts and so on, was.

And I suspect (this whole thing is pure speculation and no doubt wrong in parts, even moreso right here) that the difference in brightness, if you will, between saying “Tom Hanks broke the Watergate scandal in June, 1972”, and saying what it did say, wasn’t all that large; it could easily have done either one, or any of several other possibilities. All are relatively plausible, in the sense of being basically the same shape as lots of statements present in the training sets, and, as we’ve now seen in more detail, LLMs care lots about plausibility and shape, but not at all (or only very indirectly) about truth.

We live in interesting times!

(Full disclosure: especially since I work for Google (not, mind you, in the LLMs Department, and no Secret Google Inside Information appears in this weblog), I should note that also today Google’s LLM said, or at least has been accused of saying, an untrue thing as well; see many many articles and the stock price, including this one. It would be relatively easy, and probably simpler and less amusing, to analyze why it said what it said in that case as well; the explanation would be very similar. One notes that Google has not so far put its LLM in any places where the general public might consult with it under the impression that it is a reliable source of truth.)

Update: The Bing AI demo itself had a really surprising number of errors. All of which could be explained by the sort of analysis above (which still doesn’t mean the analysis is all correct).

Microsoft Hates This One Weird Trick To Escape OneDrive!

(Actually I have no idea whether Microsoft hates it; what would it mean for a corporation to experience hatred? Can an entity hate something without experiencing hatred? Some thorny questions there! But for now, on to the trick.)

This is a followup to yesterday’s whinging about how Windows 10 decided to store my “desktop” and “documents” and “pictures” in their teeny little cloud drive (“OneDrive”), and then nag me for money once it got full. I have (I think) a solution! But first I will talk about what didn’t work, because this is a weblog.

First, what not to do, or at least what to undo quickly if you do it: One thing that the Microsoft site and various web searches will recommend if you ask the question in certain ways, is to just go into OneDrive setup, and turn off syncing (or backup or something) of certain directories. If you read the fine print before doing this, what it actually tells Windows is “I don’t want to be able to see these directories from my computer, just keep them up there in the cloud.”

This is exactly the opposite of what I want, which is to have these directories actually physically present on my computer, and not in the cloud at all (“There is no cloud; it’s just someone else’s computer”) except to the extent that I elect to put them there explicitly.

To accomplish that thing that I do want, it turns out that you want to (here is the trick) “unlink” the computer from OneDrive (or perhaps unlink OneDrive from the computer). The instructions for doing it are on the obvious Microsoft Help Page, but they cleverly don’t say what it actually does! And what they do say (“You won’t lose files or data by unlinking OneDrive from your computer. You can always access your files by signing in to OneDrive.com.”) makes it sound like it will do what turning off syncing does, which is to make the files invisible from the computer.

But in fact what it does (I’m pretty sure; this is what it appears to have done for me anyway!) is point the Desktop, Pictures, and Documents at local directories (without the dreaded “\OneDrive” in their names), put all of their content into those directories, and then stop sending everything that you put into them up into the cloud, where they will cause OneDrive to fill up the tiny 5G space and start begging you for money. Copies of the files will (probably?) still exist in OneDrive, getting slowly out of date, unless you take action to clean them up (and who has time for that, really?).

(One caveat: among the fine print as you do this, is a note that any files (or folders) that are “online only”, will disappear from your computer. Do I have any “online only” folders or files? Beats me! Is there any way to tell, short of looking at every one? I don’t know! But I assumed, and so far apparently correctly, that I don’t have any of those. With luck, you don’t either.)

(Important update: In fact if you are like me, lots of files might be “online only” without your realizing it! Like, in my case, at least all of the screenshots prior to May of 2020! So you’ll probably want to check in whatever way you prefer (haha) to make sure that there are at least the same number and/or size of files in the newly-local directories (“folders”) as there are up on onedrive.com or whatever it is. Arggh!)

It’s possible that some of the behavior I’m seeing here is because the directories (“folders”) involved are the special “personal folders”: Desktop, Pictures, and Documents. (You might think that all of the folders on a Personal computer would be Personal folders, but in fact Microsoft considers everything outside of Desktop, Pictures, and Documents to belong to Microsoft Corporation, and don’t you forget it!)

One thing that I’m considering doing is creating like a “d:\onedrive_cloud_data” directory, and telling Windows to “link” and/or “sync” with just that. Then I could store stuff in OneDrive only when I wanted to, by putting it in there. But I kind of doubt that I’ll do that, because it’s just too likely that if OneDrive is turned on at all Microsoft will start putting the Desktop etc. there again just for fun and revenue.

So now you know! Assuming I’m not completely wrong.

Afterward: While poking around inside Windows for this trick and these weblog entries, I ran into various reminders of why I try to avoid poking around inside Windows (and why I’m so happy that I don’t rely on Windows for anything beyond playing entirely optional games).

For instance, while I was poking around for disk information, I went into the “System and Security” part of “Control Panel”, and found that in there there is a subsection called “Backup and Restore (Windows 7)”. I’m sure I don’t want to know why that is there.

There is another subsection called “Flash Player (32-bit)”, which I wish wasn’t there because it suggests that I have Flash Player installed, which is a security nightmare and now I have to go find it and uninstall it. One also wonders in what sense the Flash Player is “System and Security”, but I suppose calling the section “System and Security Nightmares” would have been bad for marketing.