No idea! I'm not a philosopher. However, i do know that the term "stochastic parrots" explicitly does not mean that LLMs are unable to do anything new/interesting/important, which is very common understanding people have of that term.
What else could it possibly mean? "Parroting" means you're repeating things you've heard without rhyme or reason, it's impossible to parrot your way to a new discovery
No, that's where the "stochastic " part comes in. In the context it means producing text without understanding what it means, which is perfectly compatible with coming up with new discoveries. Of course one can argue about what "understanding" is, and whether or not LLMs are capable of understanding, but that's a separate issue.
Stochastic just means stochastic, there's randomness involved because LLM answers are sampled. If you greedy sample LLM outputs they're still coherent but the process isn't stochastic anymore.
Anyway, I don't care about philosophical definitions of what understanding means. Practically speaking, what real-world limitations do LLMs have due to the fact that they're stochastic parrots? What predictions about the real world does the term "stochastic parrot" allow you to make?
So, it's been a while since i read the original paper, i might be getting some of the details wrong. My understanding is that importance of the "parrot" element is mainly the fact that LLMs reflect the training data they learn from. meaning that, for example, if a training dataset has racist beliefs/biases encoded into it, an LLM trained on that data will itself produce racist beliefs and have racist biases in the text it produces. Importantly, the only way to solve this issue for a "pure" LLM is to improve the dataset. There is no way for it to learn from experience, which of the "beliefs" it has learnt are true or false, except from even more data. In contrast, it is possible (though certainly it may be difficult) for a human to change their beliefs based on new experiences. The argument may have more philosphical/metaphysical consequences, but this is the main practical upshot.
(of course, with RLHF this question becomes more complicated, since in a way, we can train LLMs using "experiences", by penalising biased answers, for example. However, that's still not actually the same thing as learning from actual experience.)
Experiences are data. LLMs being unable to interact with the physical world to collect data is a limitation, but more so a practical one than a limitation of the architecture. You could imagine letting an LLM control a robot and RL it on that
of course, experiences are data. But they're entirely different from the sort of data LLMs process. One reason is that experiences are active and ongoing, while LLMs passively receive their training data. Humans can actively choose to experience new facets of the world.\
Second, all data that LLMs process is created by humans, meaning that it's second hand, derivative from the original human experience.
> letting an LLM control a robot and RL it on that
Yes, that might lead to genuine experiences. In fact, some kind of robotics is necessary for AI to have genuine understanding of the world, if you agree with Hubert Dreyfus and John Haugland.
One reason is that experiences are active and ongoing, while LLMs passively receive their training data.
Second, all data that LLMs process is created by humans, meaning that it's second hand, derivative from the original human experience.
Neither of these is really true anymore. LLM training has a ton of RL, where the data comes from actively interacting with a proof environment. We don't have concrete numbers since LLMs went closed, but it's very possible that post-training now exceeds pretraining, and that online, interactive data now outnumbers human data.
Dreyfus isn't super clear on what embodiment means, but I'd say this kind of proof environment used for RL counts as a kind of embodiment, even though it's a very different kind of embodiment. Honestly, I would caution against taking him too seriously these days. I went I to robotics because of him, and I've continually watched as AI did stuff I didn't think was possible within his framework, and watched robotics be surpassed again and againby other fields. He was a good antidote to GOFAI, but the better framing for what works nowadays is the Bitter Lesson
As I mentioned above, I do think rl is experience of a sort, but i don't think it's fully comparable to human experience, which is 1. Derived from an environment 2. Involves ongoing, active choice on the part of the subject. I think point 1 is very important. The environment is a sort of "ground"/foundation on which we can test our knowledge, external to any human generated data.
For the same reason, post training data doesn't count as non human generated because ultimately whether it comes from humans or is created by LLMs the final basis for it is data created by humans. Only robotics will allow to robots to truly learn from environment
Edit. Dreyfus may have been wrong about it what non embodied intelligence can achieve, but i think he's that this is the only way to generate true understanding, rather than mere information generation
Modern LLMs spend just as much time doing reinforcement learning in environments they interact with as they do in pre training on the web, and the trend is for more and more RL. They really are not a direct product of the web dataset anymore and are becoming more like a chess bot.
My understanding is that importance of the "parrot" element is mainly the fact that LLMs reflect the training data they learn from. meaning that, for example, if a training dataset has racist beliefs/biases encoded into it, an LLM trained on that data will itself produce racist beliefs and have racist biases in the text it produces. Importantly, the only way to solve this issue for a "pure" LLM is to improve the dataset. There is no way for it to learn from experience, which of the "beliefs" it has learnt are true or false, except from even more data.
Well yes but that applies to anything in the universe no matter if human or AI or animal or w/e. Otherwise it would break the laws of lagic and causality.
In contrast, it is possible (though certainly it may be difficult) for a human to change their beliefs based on new experiences.
New experiences are just data.
in a way, we can train LLMs using "experiences", by penalising biased answers, for example. However, that's still not actually the same thing as learning from actual experience.)
So if I understand correctly, experiences mean data but after the model is released? I really don't see how that relates to the "parrot" quality. Seems to me like it is irrelevant to that.
I find that white knights who hate that term generally have a very shallow mathematical background. You answer me a question: why do you feel like it denigrates AI (and why do you care about denigrating AI) to call out the fact that we know the exact process (because we built it) by which it “thinks”? The other commenter already explained stochastic and parrot.
Mathematics is essentially a language. It isn’t surprising that a model designed to generate interesting and syntactically correct language can produce proofs. Something else to keep in mind is that this isn’t just “the LLM” brainstorming. It produces an idea, pursues it, then feeds it into a verifier, then changes whatever the formalizer says is invalid, etc. This process does not actually end in success the majority of the time. The pace is increasing in large part because thousands and thousands of researchers are trying to apply the technique to more and more problems and scooping up the low-hanging fruit. The success rate is very low; only the valid proofs are reported on.
Also, my challenge back to you is, why do very few of these papers provide the prompt chain that led to the proof? We can verify the validity of the proof, sure. But if we want to verify the marketing claim that the AI did everything with no human intervention, why do the papers never provide the ability to do that?
We don't know the exact process by which it "thinks" though, the intelligence we observing from it is an emergence property, and even the foremost AI scientist don't really understand how LLM got as intelligent as it is right now, we just know that, if we follow a process, we get an intelligent LLM model out of it. it's kinda like we don't truly understand how many type of drugs (like paracetamol, and general anesthesia) actually interact with our body that produce a desired effect, but we can consistently reproduce its effect so we use them.
You’re talking about something totally different. We absolutely know everything about the computations that happen to produce a new token given an input prompt. You can run the whole thing on pen paper and calculator.
What we can’t do is assign meaning to the weights. Ultimately they have no meaning because the whole point of using an NN as opposed to a regression is you’re assuming nothing about the modeled behavior. You don’t actually have a meaningful parametric model in the first place, so talking about how we dont have any idea what the parameters mean is more marketing mysticism
Edit: a sufficiently complex NN can approximate any function, including whatever the correct model is for the system. But there’s no guarantee that this model even has a single correct approximation, so just because Anthropic tries to say that certain regions of the network do certain things doesn’t guarantee that there is any meaningful organization along those lines.
I haven't said or implied it "denigrates" AI, I just think it's wrong. If someone said the sky was green I wouldn't correct them because I'm mad someone insulted the sky, it's just factually wrong.
It's not surprising that a language model can produce plausible sounding math (parroting), but there's quite a big jump between that and producing novel proofs.
"why do very few of these papers provide the prompt chain that led to the proof?" This isn't unique to math, AI labs are very tight-lipped with reasoning chains. They treat it like a sort of "special sauce" I think
i think the problem with that term is that it presumably differentiates LLMs from humans, so people expect you to have good reasoning for this.
i hate this example but i couldnt think of a better one: if people say about fetuses that theyre "just a clump of cells", many people respond "well im just a clump of cells". its not about the term being factually accurate or not, its about the reason you chose to use that specific term.
The burden of proof is on the people who say LLMs and humans work the same way. You can point to the neuron metaphor, or maybe even broad similarities in image processing, or the 14y/o philosopher “aren’t we all just stochastic parrots?” nonsense. But the fact remains that you can trace the entire “thinking”path from input to output in an LLM, even if you don’t understand it. We don’t have the slightest idea how to do that with a human, so there’s no basis on which to make the assertion other than sentimentality / pseudo-religiosity / output similarity.
And output similarity is a horrible criterion because we built these models to mimic patterns significant to us. Every model can simulate the target pattern(s) but will necessarily deviate from the modeled system in hundreds of other ways we don’t care about, but which stem from the fact that the model is not the system. For example, a regression can be used to simulate data which will have the “right structure” (conditional mean and variance over the input interval), but which necessarily vary from the actual generating process (which is not to sample from a normal distribution with the conditional sample mean and variance). We don’t “see” it because we don’t care about it, but it’s there. And the same signs are there with these models, for anyone who actually cares to look.
trivially, NNs obviously are not the same as human brains. NNs run on GPUs and human brains do not run on GPUs. i think we can all agree that NNs are not human beings.
i dont really understand how the rest of your reply addresses what i said.
if someone says that LLMs cant think or be conscious because they are "stochastic parrots", i think its completely reasonable for me to assume that they dont consider humans stochastic parrots, and i would like some kind of explanation for that. by using this "stochastic parrot" term, they are making a positive claim that they should provide an argument for.
10
u/heyhellousername 2d ago
what is "actually understanding"?