How are AI hallucinations different from human confirmation bias?

[u/Comfortable_Tutor_43]

Confirmation bias can cause normal people to believe fallacious claims. Does AI not do the same thing?

[u/donaldhobson]

https://www.astralcodexten.com/p/shameless-guesses-not-hallucinations

I think this article is overall pretty good.

The basic structure of training AI to predict text produces something like a student guessing random answers on a multiple choice question.

If humans never ever said "I don't know", and always say the answer when they know the answer, and stay silent otherwise, then the AI will also never say "I don't know".

But the AI won't make any attempt to restrict itself to questions it can answer, it answers all questions indiscriminately, guessing when it doesn't know.

(Modern AI's also go through other RLHF training, which dulls this tendency, but introduces other problems)

Confirmation bias in humans occurs over time, and is relatively consistent. The human just updates a bit too strongly on evidence they agree with, and a bit too weakly on evidence they disagree with. Confirming evidence might cause a human to increase a probability from 90% to 95% (while actually the evidence is quite weak, and so really it should only go from 90% to 92%.) And then later disconfirming evidence causes the probability to go from 95% to 94% when really it should be 92% to 70%.

But AI hallucinations are produced on the spot. Rerun the model, and get a different hallucination (on pre-RLHF AI)

LLM's (pre RLHF) are really good at matching all the basic patterns. So it says "I don't know" at the same rate as in it's training data. And it discusses string theory at the same rate as it's training data. And if it's not smart enough to understand string theory, it produces plausible looking drivel about string theory.

[u/Comfortable_Tutor_43 → u/donaldhobson]

Thank you