Bing also hallucinates, even with footnotes

Large Language Models (LLMs) like ChatGPT are well-known to hallucinate – to make up answers that sound pretty plausible, but have no relation to reality.  That of course is because they’re designed to produce text that sounds about right given a prompt.  What sounds kind of right may or may not be right, however.   ChatGPT-3 made up a hilariously bad answer to a Kierkagaard prompt I gave it and put a bunch of words into Sartre’s mouth.  It also fabricated a medical journal article to support a fabricated risk to oral contraception. ChatGPT-4 kept right on making up cites for me.  It has also defamed an Australian mayor and an American law professor.  Let’s call this a known problem.  You might even suggest, following Harry Frankfurt, that it’s not so much hallucinating as it is bullshitting.

Microsoft’s Bing chatbot-assisted search puts footnotes in its answers.  So it makes sense to wonder if it also hallucinates, or if it does better.  I started with ChatGPT today and asked it to name some articles by “Gordon Hull the philosopher.”  I’ll spare you the details, but suffice it to say it produced a list of six things that I did not write.  When I asked it where I might read one of them, it gave me a reference to an issue of TCS that included neither an article by me nor an article of that title.

So Bing doesn’t have to be spectacular to do better!  I asked Bing the same question and got the following: