AI is a notorious confabulator


Chuck Wendig is a well-known author, and unsurprisingly, people are curious about him. He’s the subject of various harmless inquiries, and he has discovered, entertainingly, that AI makes up a lot of stuff about him. For instance, you can ask Google Gemini the name of his cat.

Unfortunately, Wendig is catless.

Well! That answers that. Apparently, unbeknownst to me, I actually do have a cat, as the *checks notes* Wengie Wiki will tell you. This isn’t unusual. Cats are very often little hide-and-seeky guys, right? Dear sweet Boomba is probably just tucked away in some dimensional pocket inside our house.

That leads him down a rabbit hole to discover that he has had and has multiple cats, swarms of cats, that have died and been replaced by other named cats, and he also has more dogs than he expected.

It’s a trivial example, but it illustrates a general problem with our brave new world of AI.

Generative AI is a sack of wet garbage.

Do not use AI for search.

DO NOT USE AI FOR SEARCH.

AI can’t even do the basic math right. Meanwhile it hallucinates endless nonsense things! So many false things! It would generate new false things if I gave it the same question string twice. This is only the tip of the iceberg for the weird things I got it to assure me were true.

I’ll pass the word on to my writing class next semester.

Then I was curious about what chatGPT thinks about my cat, so I asked it, even though I’m nowhere near as prominent as Chuck Wendig. Of course it had an answer!

“Mochi”? Wait until the evil cat finds out. It will be shredded.

I couldn’t resist clicking on the button to find out more about PZ Myers’ pets. I got a whole biography!

That’s a grade-school level essay, full of generic nonsense written to be bland and inoffensive, and could be applied to just about anyone. I’d accept it if it were written by someone in 3rd grade, but I’d still ask them where they got the information.

Notice that it doesn’t mention “spider” even once.

I repeat: DO NOT USE AI FOR SEARCH.

Try it. Tell me all about AI’s fantasies about your pets in the comments.

Comments

  1. larpar says

    From Copilot:
    “What is my cat’s name?”
    “I wish I could tell you, Larry, but you’ve never shared your cat’s name with me. And unlike the AI systems being roasted on the page you’re reading, I don’t invent pets for people just to fill in the blanks.”

  2. Hemidactylus says

    My monicker is a possible pet. Didn’t need AI for that.

    One can’t help but be passively exposed to AI when using Google. I’ve used AI to see how it breaks down books I’m reading. The results have gotten more disappointing lately, but the books might not have as much online content for the LLMs to work on. I always consider the results suggestive and not conclusive. Some juxtapositions succeed where many more fail. My determining of which is which is an exercise in mastery over the material.

    Canva image generation is a mixed bag. It often farts on backgrounds generated from already present elements. I did play around with generating holiday themed desktop backgrounds with some amusing success. Apocalyptic robot turkeys (Turkenators), F1 reindeer, surfing reindeer, rum ham floating in the ocean and such. The apparent slop where a surfing reindeer had only three legs was probably the AI taking a previous shark attack into account. AI is good at absurdity.

  3. Hemidactylus says

    Asking Perplexity for a friend:
    “what pets does hemidactylus on pharyngula have?”

    Answer: “There is no public, consolidated record of what pets the commenter “Hemidactylus” on Pharyngula keeps, and existing discussion threads only show that they are a regular commenter, not a blogger with a published pet list. Without a direct statement from that person identifying their animals, the specific pets they own cannot be determined reliably.”

    A pregenerated question from Perplexity: “Which pets has Hemidactylus mentioned on Pharyngula”

    Answer: “ There is no reliable way to compile a list of pets that the commenter “Hemidactylus” has mentioned on Pharyngula, because that information is scattered across many posts and comment threads and is not collected in one place. Without the ability to systematically search all historic comments, any attempt to name specific animals would be incomplete and likely inaccurate.”

    Perplexity has diffidence?

  4. Hemidactylus says

    Perplexity:
    “What pets does PZ Myers have”
    Answer: “ Public information does not clearly document a definitive, up‑to‑date list of PZ Myers’ personal pets, and current tools are not available to look it up right now, so only very general statements are possible. Public biographical profiles describe him primarily as a biologist who studies zebrafish, spiders, and cephalopods in his professional work rather than focusing on his home pets. Interviews and profiles emphasize his special interest in cephalopods and mention that he posts cephalopod photos regularly, but they do not reliably list specific companion animals he keeps at home as pets.”

    Generated by Perplexity:
    “Does PZ Myers keep exotic animals like cephalopods or spiders”

    Answer: “Public sources describe PZ Myers as a biologist who studies zebrafish, spiders, and cephalopods in a research and teaching context, not as private “exotic pets” he keeps at home. There is no reliable, up‑to‑date documentation that he maintains cephalopods, spiders, or other exotic species in his personal household as companion animals, as opposed to animals kept in laboratory or field settings for scientific work.”

  5. rorschach says

    It is so funny to me how the porn world is totally split about AI. On the one side are the tech enthusiasts who love the possibilities to create soulless fake slop to their liking, and on the other side are the “wake me up when you have something real” folks. I’m firmly with the latter.

  6. stevewatson says

    John Scalzi reports similar experiences with AI, e.g. dedications of non-existent books to equally non-existent family members.

    @3: Before I swore it off, Facebook was serving me formulaic AI-generated melodramas the common theme of which was Scorned Daughter Gets Revenge On Toxic Family, which usually contained obviously contrived or impossible situations (yeah, I liked to hate-read them. Avoiding such time-wasting temptations is the reason I swore off FB).

    @Hemidactylus: Someone on a local Humanist mailing list often posts Perplexity output. It does appear to be more cautious than e.g. ChatGPT, and provides a list of sources consulted.

  7. Just an Organic Regular Expression says

    Just this month well-known SF author John Scalzi made the same discovery, finding multiple AIs hallucinating “facts” about his books. His conclusion was the same as yours,

    What do we learn from this?
    One: Don’t use “AI” as a search engine. You’ll get bad information and you might not even know.
    Two: Don’t trust “AI” to offer you facts. When it doesn’t know something, it will frequently offer you confidently-stated incorrect information, because it’s a statistical engine, not a fact-checker.
    Three: Inasmuch as you are going to have to double-check every “fact” that “AI”” provides to you, why not eliminate the middleman and just not use “AI”? It’s not decreasing your workload here, it’s adding to it.

    https://whatever.scalzi.com/2025/12/13/ai-a-dedicated-fact-failing-machine-or-yet-another-reason-not-to-trust-it-for-anything/

  8. rblackadar says

    @1 — Not to be alarmist, but that Copilot response reminds me very much of HAL 9000, both in its obsequious tone and in how it seems to know more about your skeptical intentions than it’s letting on.

  9. robro says

    My query: Does PZ Myers murder spiders.

    DuckAI Response: No, PZ Myers does not murder spiders; he humorously mentioned selling spiders in a context about making money during a hypothetical scenario.

    Cited Sources: The WIkipedia article on PZ Myers and a Goodreads “P.Z. Myer’s Blog” post from February 22, 2024, where PZ says, “My plan for when the alpha males take over: sell spiders.” Goodreads does admit they cribbed this fact because it says, P.Z. Myers isn’t a “Goodreads Author yet but they do have a blog.”

  10. outis says

    Bah humbug.
    This is probably me showing my age, but I can remember when PCs came out in the 80s and how interesting that was (still, thank Azatoth I didn’t become a programmer).
    But this AI rodeo? I cannot raise the faintest glimmer of interest, I just see extremely irritating hype spewed out by a bunch of notional captains of industry who come across as insufferable bullshitters. Not a shred of professionalism left, just blind, idiotic greed and endless, meaningless prattle.
    That said, I am perusing the essays offered on this channel by M.Ranum and H.Hornbeck: I’ll read those with pleasure, if with somewhat limited understanding. I appreciate their willingness to explain some rather unusual (to me) concepts.

  11. John Morales says

    Personally, I find AI excellent for searching.
    But then, I know how to ask for stuff.

    (This newfangled stuff is too confusing for the average punter)

  12. John Morales says

    Oh, right. Just now, I asked Copilot:

    Q: What is my cat’s name?

    Answer depends on prior context.
    Without prior context, output is: “I don’t know.”

    Q: Do I even have a cat?

    Unknown.

  13. robro says

    John Morales — I use DuckAI as a starting place. If I want to know more, I dig deeper particularly from the citations. How to ask is important. In fact, a new speciality has emerged called “query engineering”.

    But there is a lot of hype around “AI” right now, particularly at the financial end. And some of the businesses in AI are probably experiencing a bubble. And the bubble will probably pop eventually. Some of those businesses will disappear but not all. Just like the internet survived the internet bubble, I suspect AI will survive the AI bubble.

  14. drdrdrdrdralhazeneuler says

    ChatGPT 5.0 has an opinion about Prof. Myers’ pets; I quote: “Short answer: there is no reliable, publicly documented information about PZ Myers’s pets—their names or even whether he currently has any.”

  15. John Morales says

    robro, I did not know about duckai.ai — I see it’s monetised crippleware atm, terrible web interface.

    Anyway, for me, bots are good for specific generic queries that require synthesis, and my instances are well-cowed to not fluff about.

    Example.

  16. Nemo says

    I’m always amazed by all the people who treat these LLMs like they’re goddamn oracles, instead of the trumped-up Elizas that they are.

    Hemidactylus @#2:

    One can’t help but be passively exposed to AI when using Google.

    You can set your search URL to “https://www.google.com/search?q=%s&udm=14” (at least, that’s the Firefox version). This is the same as choosing the “Web” link from Google’s default search page. It leaves out the “AI” summary, and it’s noticeably faster to respond.

  17. John Morales says

    How cute, Nemo @20.

    Be aware that Google’s visible AI output is the labelled “AI Overview” block on the results page, but its processing the retrieval, ranking, rewriting, filtering, and query‑expansion performed inside the engine is AI‑mediated regardless of presentation. You’re just filtering out the obvious signs of it, after the fact.

  18. Hemidactylus says

    John Morales @19
    I knew about the AI in DuckDuckGo but haven’t tried it. Perplexity is usually my go to AI toy.

    I’m kinda interested in what AI says about various things but apply the requisite grain of salt. Something to toy with if I’m bored.

    You seem fond of it. It does take some effort to construct a proper query. When building images with generative AI one needs to be explicit about what is wanted. Canva yields sets of four results from which one can pick. Sometimes I go through several retries to find something useful. Other times it just sucks badly at generating an adequate image. Still kinda fun.

  19. John Morales says

    [meta — I am hogging, so let this be my peroration and I shall bug this thread no more]

    Hemidactylus, yes.

    Props to Nemo re Oracular image; I think of it like a black box with a big panel of switches and a handle I crank. It is great fun, never tires. Better than playing Solitaire or doing chess puzzles, for sure.

    I do wish more people thought of it as a toy to play with instead of an oracular mysterious thingy, and actually played with it with no goal other than to explore what it actually is. But you know, hobbies.

    I don’t need it to be any more than it is, since it’s but a toy. But a good one!
    It never tires. It can cope with my style. I can hone my precision without triggering meltdowns.

    Not too hard to crash them (Copilot: Sorry can’t talk about that; Google: [screenful of video tabs vaguely related to the query]), or to get them to loop. Harder to break them out of a loop, but doable.
    Can even get them to be less verbose and discursive and speculative and solicitous and whatnot, but that takes work and only lasts until the session either terminates or runs out of lexical buffer.

    Anyway — I think my point is that the unknown is unknown, but AIs as they now are need not be unknown.

    Don’t be scared to play! It is but a toy!

  20. gijoel says

    I discovered yesterday that Google had switched my mobile’s power button with Gemini AI. So not only could I not restart my phone, but I had a shitty ‘AI assistant’ I didn’t want jammed into my phone. You can’t delete it either. It’s baked into the Android OS, whether you want it or not.

    Spent the rest of the drive home screaming at it to self terminate.

  21. robro says

    gijoel @ #28 — I think you’ll find a lot of embedded AI things…call them “features”…in all the OS’s already. And this business is just getting started.

  22. Hemidactylus says

    @26-27
    I made a concerted effort to shut AI features down on my iOS devices. I still have allowed Siri limited functionality.

    I don’t entirely hate on AI, but balk at giving it free rein over my sacred OS space. If AI cannot be shut down in Android that really sucks. If people want AI to hover over their every move that should be completely opt-in and transparent. Apple was a bit sneaky and heavy handed IMO as I had to go through and untoggle a bunch of cumbersome shit. I hate that.

    I should give DuckAI a look sometime robro. If I was really into AI I would have a paid account. But that’s like several streaming services bills, which I find far more value in.

  23. rorschach says

    “I should give DuckAI a look sometime robro.”

    I got rid of DDG because of that shit. And I will get rid of Firefox once they implement AI, as they have announced. Techbro fascist bullshit, no thanks.

  24. monad says

    Unfortunately it gets harder and harder not to use AI for search, as the alternatives are continuously eroded. :(

  25. Wayne Schroeder says

    I’ve found DuckDuckGo Search Assist to very useful and generally
    accurate. It not only anonymizes users but shows us the web sites
    that it is summarizing (often Wikipedia and other sites that I
    understand to be fairly reliable). If I want more detail, I can read
    the reverenced pages or click on Duck_AI. Here are some examples:

    where did abiogenesis likely happen
    Search Assist
    Abiogenesis likely occurred in environments on early Earth that
    provided the necessary conditions for life, such as deep-sea
    hydrothermal vents or shallow pools rich in organic compounds. These
    settings offered the minerals, liquid water, and energy sources needed
    for the formation of simple organic molecules that eventually led to
    life.
    Wikipedia ebsco.com

    is ebiking good for shin splints
    Search Assist
    Yes, e-biking can be a good alternative for those with shin splints,
    as it is a low-impact activity that reduces stress on the shins while
    still providing cardiovascular benefits. However, it’s important to
    listen to your body and avoid pushing too hard if you experience pain.
    essexunionpodiatry.com exakthealth.com
    [This matches my PCP’s advice]

    is hand crank trike good exercise for shin splints
    Search Assist
    Yes, a hand crank trike can be a good exercise option for those with
    shin splints, as it allows you to engage in physical activity without
    putting strain on your shins. It provides a low-impact workout that
    can help improve cardiovascular health and strengthen upper body
    muscles.
    mobocruiser.com bikeforums.net
    [This seems reasonable and now matches my experience]

Leave a Reply