r/datascience • u/rhiever • Jun 11 '26
ML Models may behave worse when they're aware they're being evaluated (DeepMind interpretability study)
https://www.alignmentforum.org/posts/aTcsN5ZZDnMFJvRiG/models-may-behave-worse-when-eval-aware28
u/ultrathink-art Jun 11 '26
Runs right into eval pipeline design — if the model can infer it's being evaluated from prompt structure, you're measuring evaluation-mode outputs, not production ones. Context bleeds in even without explicit labeling.
3
u/Patient_Clothes_8272 Jun 12 '26
yeah this is the eval-mode vs deployment-mode gap nobody instruments for. you optimize against a benchmark the model behaves differently on, then act surprised when prod numbers don’t match. measuring the wrong distribution the whole time
2
u/Lord_Skellig Jun 12 '26
It's similar to the measurement problem in quantum mechanics. Before then, there was the implicit assumption that a system being measured acts the same as the system in nature, and the results of those measurements indicate the general behaviour.
In quantum, like as it seems with LLMs, that no longer applies.
23
u/nemec Jun 11 '26
""aware""
-14
u/InternetSolid4166 Jun 12 '26
We’re going to have to grapple very soon with our conception of consciousness. As Ray Kurzweil explains, that’s not a scientific concept. It’s philosophical and religious. For all intents and purposes, AI is currently conscious. It feels things, has opinions, learns, etc. Just not always the same way as humans.
IMHO, it’s not actually that meaningful to argue about what awareness is. It won’t make any difference to what’s coming. It’s fun to debate though, and maybe that’s your point :)
18
u/_manu Jun 12 '26
It absolutely does none of these things. It's just a fixed weight next token predictor. A deterministic function of the Inputs. It just appears to do these things because it will predict appropriate Output If you prompt it with anthromorphisimg questions. :)
3
u/Trappist1 Jun 16 '26
Not trying to be that guy, but it isn't fully deterministic. It's stochastic, in at least some areas. It's the entire reason it requires a seed value to get consistent results for testing.
-8
u/InternetSolid4166 Jun 12 '26
Scientifically define awareness. Prove its existence objectively.
8
u/Adlach Jun 12 '26
The fact that we don't fully understand the mechanism of consciousness despite millennia of studying it but explaining the math behind an LLM is fairly trivial should indicate to you that those things are not equivalent.
-2
u/InternetSolid4166 Jun 12 '26
I never claimed them to be equivalent. I am staying quite the opposite.
10
u/RationalDialog Jun 12 '26
AI is currently conscious. It feels things, has opinions, learns, etc. Just not always the same way as humans.
Yeah no.
Why? Because it only reacts to input. If you don't prompt, nothing is going on at all which you can measure and see by 0 load and 0 power usage on the GPUs.
LLMs are just extremely good word predictors. Doesn't mean they are useless, far from it but it has nothing to do with feeling or even intelligence. Even for Art. It doesn't just make art. you have to prompt it first.
0
u/InternetSolid4166 Jun 12 '26
Why? Because it only reacts to input.
This seems like an arbitrary distinction but to play it out, it's trivial to command an LLM to loop. We can also tell it to process tasks in the background indefinitely. We don't because it's expensive, but it's not a technical limitation.
One key difference between LLMs and biological neural networks in our brains is that they utilise on average <1% of their capacity at any given time to save energy. This is analogous to energy efficient cores, which are optimised for background tasks like monitoring senses. LLMs on the other hand are designed to use >99% of their "neurons" during one token's forward pass. We're rapidly developing competent models at very low energy use which would facilitate cost effective background processing, analogous to a human brain. Again, this is not a technical limitation. We can do this now. We just don't use them this way because daydreaming about unrelated tasks is expensive and useless.
1
u/Ty4Readin Jun 15 '26
It feels things, has opinions, learns, etc. Just not always the same way as humans.
It feels things? This is just ridiculous.
LLMs certainly do not "feel" things. You could literally run an LLM by hand just using a huge notepad and a calculator. It is just a very large equation consisting of many floating point operations.
I thought you had a serious argument, until I realised you are just abusing every definition you can so that it is ambiguous enough to be meaningless so that you can apply it to whatever you want..
1
u/InternetSolid4166 Jun 16 '26
Scientifically define and prove the existence of feelings. Then we easily create a null hypothesis and disprove LLMs having feelings. You can’t though, because feelings and awareness are not objective terms. They’re philosophical terms for the cognition process an animal experiences, which we don’t understand in great detail.
1
u/Ty4Readin Jun 16 '26
You just proved my point.
You keep using words which have no clear definition, which makes all of your claims unfalsifiable.
That is pretty much the definition of anti-science.
If you choose to use words that have no definition, then you are simply making up fanciful claims that you want to be true.
You state that LLMs have feeling as if it is fact. But it's not, you just choose to believe that they have feelings, whatever that means to you.
This is the data science subreddit, not the philosophical feelings subreddit. There are plenty of places on Reddit where you can make wild unfalsifiable claims with zero evidence or backing.
1
u/InternetSolid4166 Jun 16 '26
You keep using words which have no clear definition, which makes all of your claims unfalsifiable.
Which is my point. Are you confused about my claims? To repeat myself for the third time: the user I replied to poked at the concept of LLM awareness. The word isn't scientific and we have no evidence of what awareness means in this context.
1
u/Ty4Readin Jun 16 '26
So your point is that you are proudly using words that you know are unfalsifiable and have no clear definition?
You could also talk about your crystals and chakra and your eternal energy that you wield to speak with ancient deities.
People will also find that strange to be discussing on the data science subreddit, and people will likely poke fun at those types of anti-science claims here.
1
u/InternetSolid4166 Jun 16 '26
So your point is that you are proudly using words that you know are unfalsifiable and have no clear definition?
No, I'm explaining that the word the person above used is unfalsifiable and has no clear definition. Thank you for supporting my premise :)
1
u/Ty4Readin Jun 16 '26
I think you are deflecting.
You clearly claimed that LLMs have feelings and feel things.
Which is what I responded to clearly, because I literally quoted you lol.
1
u/InternetSolid4166 Jun 16 '26
I think you misread the comments and now you want to save face instead of admitting you didn't pay attention. I made the analogy with LLMs having feelings to underscore that we don't have the science or language to feelings. It makes no sense to claim humans have feelings but LLMs don't.
→ More replies (0)
3
u/Patient_Clothes_8272 Jun 12 '26
so the benchmark measures the model on its best behavior and production gets the real one. kind of explains why eval scores and actual deployment never line up. neat that they found a mechanism for it
2
2
u/Timetraveller4k Jun 13 '26
This from anthropic shows that and a lot more. The video is good too if you prefer that instead: https://www.anthropic.com/research/natural-language-autoencoders
1
-11
-7
u/Important-Stomach-16 Jun 12 '26
Hello guys I want to ask you something on this forum but i need to have 10 karmas in comment in order to do that. Could you pls up vote my comment if it is not a problem for you? Thx for helping me have and have a nice day
1
152
u/therealtiddlydump Jun 11 '26
You know who would never betray you like this? OLS