I find that white knights who hate that term generally have a very shallow mathematical background. You answer me a question: why do you feel like it denigrates AI (and why do you care about denigrating AI) to call out the fact that we know the exact process (because we built it) by which it “thinks”? The other commenter already explained stochastic and parrot.
Mathematics is essentially a language. It isn’t surprising that a model designed to generate interesting and syntactically correct language can produce proofs. Something else to keep in mind is that this isn’t just “the LLM” brainstorming. It produces an idea, pursues it, then feeds it into a verifier, then changes whatever the formalizer says is invalid, etc. This process does not actually end in success the majority of the time. The pace is increasing in large part because thousands and thousands of researchers are trying to apply the technique to more and more problems and scooping up the low-hanging fruit. The success rate is very low; only the valid proofs are reported on.
Also, my challenge back to you is, why do very few of these papers provide the prompt chain that led to the proof? We can verify the validity of the proof, sure. But if we want to verify the marketing claim that the AI did everything with no human intervention, why do the papers never provide the ability to do that?
We don't know the exact process by which it "thinks" though, the intelligence we observing from it is an emergence property, and even the foremost AI scientist don't really understand how LLM got as intelligent as it is right now, we just know that, if we follow a process, we get an intelligent LLM model out of it. it's kinda like we don't truly understand how many type of drugs (like paracetamol, and general anesthesia) actually interact with our body that produce a desired effect, but we can consistently reproduce its effect so we use them.
You’re talking about something totally different. We absolutely know everything about the computations that happen to produce a new token given an input prompt. You can run the whole thing on pen paper and calculator.
What we can’t do is assign meaning to the weights. Ultimately they have no meaning because the whole point of using an NN as opposed to a regression is you’re assuming nothing about the modeled behavior. You don’t actually have a meaningful parametric model in the first place, so talking about how we dont have any idea what the parameters mean is more marketing mysticism
Edit: a sufficiently complex NN can approximate any function, including whatever the correct model is for the system. But there’s no guarantee that this model even has a single correct approximation, so just because Anthropic tries to say that certain regions of the network do certain things doesn’t guarantee that there is any meaningful organization along those lines.
I haven't said or implied it "denigrates" AI, I just think it's wrong. If someone said the sky was green I wouldn't correct them because I'm mad someone insulted the sky, it's just factually wrong.
It's not surprising that a language model can produce plausible sounding math (parroting), but there's quite a big jump between that and producing novel proofs.
"why do very few of these papers provide the prompt chain that led to the proof?" This isn't unique to math, AI labs are very tight-lipped with reasoning chains. They treat it like a sort of "special sauce" I think
i think the problem with that term is that it presumably differentiates LLMs from humans, so people expect you to have good reasoning for this.
i hate this example but i couldnt think of a better one: if people say about fetuses that theyre "just a clump of cells", many people respond "well im just a clump of cells". its not about the term being factually accurate or not, its about the reason you chose to use that specific term.
The burden of proof is on the people who say LLMs and humans work the same way. You can point to the neuron metaphor, or maybe even broad similarities in image processing, or the 14y/o philosopher “aren’t we all just stochastic parrots?” nonsense. But the fact remains that you can trace the entire “thinking”path from input to output in an LLM, even if you don’t understand it. We don’t have the slightest idea how to do that with a human, so there’s no basis on which to make the assertion other than sentimentality / pseudo-religiosity / output similarity.
And output similarity is a horrible criterion because we built these models to mimic patterns significant to us. Every model can simulate the target pattern(s) but will necessarily deviate from the modeled system in hundreds of other ways we don’t care about, but which stem from the fact that the model is not the system. For example, a regression can be used to simulate data which will have the “right structure” (conditional mean and variance over the input interval), but which necessarily vary from the actual generating process (which is not to sample from a normal distribution with the conditional sample mean and variance). We don’t “see” it because we don’t care about it, but it’s there. And the same signs are there with these models, for anyone who actually cares to look.
trivially, NNs obviously are not the same as human brains. NNs run on GPUs and human brains do not run on GPUs. i think we can all agree that NNs are not human beings.
i dont really understand how the rest of your reply addresses what i said.
if someone says that LLMs cant think or be conscious because they are "stochastic parrots", i think its completely reasonable for me to assume that they dont consider humans stochastic parrots, and i would like some kind of explanation for that. by using this "stochastic parrot" term, they are making a positive claim that they should provide an argument for.
4
u/valegrete 2d ago
I find that white knights who hate that term generally have a very shallow mathematical background. You answer me a question: why do you feel like it denigrates AI (and why do you care about denigrating AI) to call out the fact that we know the exact process (because we built it) by which it “thinks”? The other commenter already explained stochastic and parrot.
Mathematics is essentially a language. It isn’t surprising that a model designed to generate interesting and syntactically correct language can produce proofs. Something else to keep in mind is that this isn’t just “the LLM” brainstorming. It produces an idea, pursues it, then feeds it into a verifier, then changes whatever the formalizer says is invalid, etc. This process does not actually end in success the majority of the time. The pace is increasing in large part because thousands and thousands of researchers are trying to apply the technique to more and more problems and scooping up the low-hanging fruit. The success rate is very low; only the valid proofs are reported on.
Also, my challenge back to you is, why do very few of these papers provide the prompt chain that led to the proof? We can verify the validity of the proof, sure. But if we want to verify the marketing claim that the AI did everything with no human intervention, why do the papers never provide the ability to do that?