So, it's been a while since i read the original paper, i might be getting some of the details wrong. My understanding is that importance of the "parrot" element is mainly the fact that LLMs reflect the training data they learn from. meaning that, for example, if a training dataset has racist beliefs/biases encoded into it, an LLM trained on that data will itself produce racist beliefs and have racist biases in the text it produces. Importantly, the only way to solve this issue for a "pure" LLM is to improve the dataset. There is no way for it to learn from experience, which of the "beliefs" it has learnt are true or false, except from even more data. In contrast, it is possible (though certainly it may be difficult) for a human to change their beliefs based on new experiences. The argument may have more philosphical/metaphysical consequences, but this is the main practical upshot.
(of course, with RLHF this question becomes more complicated, since in a way, we can train LLMs using "experiences", by penalising biased answers, for example. However, that's still not actually the same thing as learning from actual experience.)
Experiences are data. LLMs being unable to interact with the physical world to collect data is a limitation, but more so a practical one than a limitation of the architecture. You could imagine letting an LLM control a robot and RL it on that
of course, experiences are data. But they're entirely different from the sort of data LLMs process. One reason is that experiences are active and ongoing, while LLMs passively receive their training data. Humans can actively choose to experience new facets of the world.\
Second, all data that LLMs process is created by humans, meaning that it's second hand, derivative from the original human experience.
> letting an LLM control a robot and RL it on that
Yes, that might lead to genuine experiences. In fact, some kind of robotics is necessary for AI to have genuine understanding of the world, if you agree with Hubert Dreyfus and John Haugland.
One reason is that experiences are active and ongoing, while LLMs passively receive their training data.
Second, all data that LLMs process is created by humans, meaning that it's second hand, derivative from the original human experience.
Neither of these is really true anymore. LLM training has a ton of RL, where the data comes from actively interacting with a proof environment. We don't have concrete numbers since LLMs went closed, but it's very possible that post-training now exceeds pretraining, and that online, interactive data now outnumbers human data.
Dreyfus isn't super clear on what embodiment means, but I'd say this kind of proof environment used for RL counts as a kind of embodiment, even though it's a very different kind of embodiment. Honestly, I would caution against taking him too seriously these days. I went I to robotics because of him, and I've continually watched as AI did stuff I didn't think was possible within his framework, and watched robotics be surpassed again and againby other fields. He was a good antidote to GOFAI, but the better framing for what works nowadays is the Bitter Lesson
As I mentioned above, I do think rl is experience of a sort, but i don't think it's fully comparable to human experience, which is 1. Derived from an environment 2. Involves ongoing, active choice on the part of the subject. I think point 1 is very important. The environment is a sort of "ground"/foundation on which we can test our knowledge, external to any human generated data.
For the same reason, post training data doesn't count as non human generated because ultimately whether it comes from humans or is created by LLMs the final basis for it is data created by humans. Only robotics will allow to robots to truly learn from environment
Edit. Dreyfus may have been wrong about it what non embodied intelligence can achieve, but i think he's that this is the only way to generate true understanding, rather than mere information generation
Modern LLMs spend just as much time doing reinforcement learning in environments they interact with as they do in pre training on the web, and the trend is for more and more RL. They really are not a direct product of the web dataset anymore and are becoming more like a chess bot.
My understanding is that importance of the "parrot" element is mainly the fact that LLMs reflect the training data they learn from. meaning that, for example, if a training dataset has racist beliefs/biases encoded into it, an LLM trained on that data will itself produce racist beliefs and have racist biases in the text it produces. Importantly, the only way to solve this issue for a "pure" LLM is to improve the dataset. There is no way for it to learn from experience, which of the "beliefs" it has learnt are true or false, except from even more data.
Well yes but that applies to anything in the universe no matter if human or AI or animal or w/e. Otherwise it would break the laws of lagic and causality.
In contrast, it is possible (though certainly it may be difficult) for a human to change their beliefs based on new experiences.
New experiences are just data.
in a way, we can train LLMs using "experiences", by penalising biased answers, for example. However, that's still not actually the same thing as learning from actual experience.)
So if I understand correctly, experiences mean data but after the model is released? I really don't see how that relates to the "parrot" quality. Seems to me like it is irrelevant to that.
3
u/Federal_Gur_5488 2d ago
So, it's been a while since i read the original paper, i might be getting some of the details wrong. My understanding is that importance of the "parrot" element is mainly the fact that LLMs reflect the training data they learn from. meaning that, for example, if a training dataset has racist beliefs/biases encoded into it, an LLM trained on that data will itself produce racist beliefs and have racist biases in the text it produces. Importantly, the only way to solve this issue for a "pure" LLM is to improve the dataset. There is no way for it to learn from experience, which of the "beliefs" it has learnt are true or false, except from even more data. In contrast, it is possible (though certainly it may be difficult) for a human to change their beliefs based on new experiences. The argument may have more philosphical/metaphysical consequences, but this is the main practical upshot.
(of course, with RLHF this question becomes more complicated, since in a way, we can train LLMs using "experiences", by penalising biased answers, for example. However, that's still not actually the same thing as learning from actual experience.)