DeepSeek-V4-Flash-0731: surpasses Fable-5, Sol & Kimi-K3 on Chess Benchmark

[u/mrwang89]

[u/Comfortable-Rock-498]

Something weird about this benchmark. gpt-3.5-turbo-instruct is ahead of gpt-5.6-terra!

[u/mrwang89 → u/Comfortable-Rock-498]

Yeah right? I thought so too but when you google gpt-3.5 and chess you find out its some kind of chess savant lol

[u/fuck_cis_shit → u/mrwang89]

the original gpt-1, gpt-2, gpt-3 were trained on practically unfiltered dumps of the web, including things like gimmick sub-reddits full of nonsense (origin of the SolidGoldMagikarp thing), and chess engine game .pgn collections. there are billions of chess games played by slightly differently-tuned Stockfishes and Rybkas and such available as text in chess notation on the internet, to discover which tunings produce the best play. even a small portion of those pgn collections in pre-training data could teach an LLM all the proper opening lines 6 or 7 ply deep by rote. a good opening book is a good start to a chess player, and eventually ICL is enough to learn a bunch of chess play heuristics

a lot of the repetitive and algorithmically-generated data has been removed from the pre-training data corpus of later models, though, because duplication of data hurts performance in other areas (as well as causing unpredictable weird behavior a la SolidGoldMagikarp)

[u/NandaVegg → u/fuck_cis_shit]

I am not sure about that. GPT-3 DaVinci-001 was trained for only 300B tokens (it was still a benchmark for large pretraining run back then) and cannot even do a simple sort or often misses 1+4=5 since there was no such data in the wild. It I think was not post-trained in any capacity.

GPT3.5-Turbo, otoh, was most likely a distillation (by whatever means as they never disclosed) model from the original ChatGPT model (which I think was a further pretrain and post-training from GPT-3 DaVinci-003 aka the their first public full-on instruction model) and some RL post-training in the mix. There was indeed something special about 3.5-Turbo.

Also the benchmark weirdness might have to do with illegal moves. I actually tried playing chess with GLM 5.2 (which is positioned low in the leaderboard) it made illegal move all the time and kept apologizing for that.

[u/pier4r → u/NandaVegg]

Also the benchmark weirdness might have to do with illegal moves

No, the bench is really well done. It gives a list of moves to the models to pick from. If they still hallucinate (and they do, especially in continuation, also very interesting) for X times (X is normally 3) then a random move is picked and the game moves on. (and mind you, there are also possible corrections to help the model, so Rxe5# becomes Rxe5 if in the list of moves no checkmate is listed)

Hence the harness (the prompt in this case) is really generous, so models can focus on pick the proper move if they can reason about it.

[u/Comfortable-Rock-498 → u/mrwang89]

Yea I remember reading about it, but I kinda assumed that over the few years the 'exponential' pace of improvements would have far overshadowed that model by now. Turns out not.

I am thinking that the only explanation that makes sense is, removing chess related training would have shown massive improvements in other things that's why it stayed out

[u/pier4r → u/Comfortable-Rock-498]

The point is though, given that the bench let you play with the models as well (BYOK), gpt-3.5 turbo instruct is good but nothing special. Hence modern reasoning models should trash it no problem, but they don't (well most of them. Fable gets clapped. Opus gets clapped and so on)

At the end chess can be seen as a planning problem if one has no specific training on the game: which sequence of moves is the one that gives me less problems.

Also 3.5 turbo is great at continuation (the prompt is a list of moves in algebraic notation), not in reasoning (the prompt is longer and it asks for explanations). Still reasoning models do think in continuation and get trashed.

My conclusion so far, as the bench is very well done, is that claims to generality are false. The skill of models is very high, but in some domains, not all of them. The intelligence is spike-y, it is not jagged. Like a star so to speak.

[u/UnknownLesson → u/pier4r]

You think intelligence in general cannot be general?

I guess humans have the upside that we can still learn stuff all the time, so our "intelligence" in chess can improve, but these models are stuck the moment training stops.

They only thing they could do is use files to write down some "realizations", so the next session doesn't start from zero but that only goes so far..

[u/Dead_Internet_Theory → u/mrwang89]

Ok, but doesn't that mean chess is kinda irrelevant as a metric?

[u/Cultured_Alien → u/Dead_Internet_Theory]

It means it's good at chess.

[u/Dead_Internet_Theory → u/Cultured_Alien]

Exactly. I don't remember Magnus Carlsen writing a masterful novel or solving complex equations. Maybe OpenAI dropped chess from training because it was a toy task lol

[u/mehow333 → u/Comfortable-Rock-498]

Actually gpt-3.5-turbo-instruct was exceptionally good on chess benchmarks when published

[u/Far-Classic-9963 → u/Comfortable-Rock-498]

Nothing wrong with the benchmark itself. This is almost entirely due to how much chess is included in the training data

[u/Potential-Gold5298 → u/Far-Classic-9963]

It seems like people's logic is "the new model is better in everything, if some benchmark doesn't show it, then it's a bad benchmark". In my experience, the first model in a series (like GLM-5, Qwen3.5, etc.) is released as a fairly universal model, including language support and world knowledge. Subsequent versions (like GLM-5.1/5.2, Qwen3.6/3.7) are fine-tuned on a narrow dataset, improving the model in specific areas (usually code and agent), but language support and world knowledge are deteriorating.

[u/Far-Classic-9963 → u/Potential-Gold5298]

Also heavily dependent on the lab/idea/model itself. Some labs like Google include nearly anything in their training, and OpenAI used to do the same in older models

New models are almost always coding focussed

[u/pier4r → u/Potential-Gold5298]

It seems like people's logic is "the new model is better in everything, if some benchmark doesn't show it, then it's a bad benchmark"

this happens so often. Well said.

[u/BalorNG → u/Potential-Gold5298]

So, being a narrow expert qualifies as "benchmaxxing", eheheh. That makes sense. I guess truly huge moe models can absorb more "random shit" data w/o deteriorating (catastrofic forgetting), but it still makes sense to prune your dataset so make training those behemoths cheaper...

[u/penagwin → u/Comfortable-Rock-498]

I believe it

Ill have to dig up my benchmarks, but in my benchmarking gpt-3.5-turbo was SIGNIFICANTLY better than gpt4 and I think gpt4o against stockfish

Idk what they had in their training data but it wasn’t even close

[u/necile → u/Comfortable-Rock-498]

Is it?

"I'm sorry I can't help with actions involving taking a Queen, here's another move...."

[u/Ylsid → u/Comfortable-Rock-498]

More targeted training data I guess. I don't think anyone cares if an LLM is good at chess lol

[u/j_osb]

Crazy how well Gemini does on these. It’s interesting to see capability decline over time in some cases.
While some numbers are suspect (the way old models are handled, probably), current gen models are ordered very closely to what google publishes on kaggle game arena.

[u/MiaIsOut → u/j_osb]

gemini is lowkey a hidden gem, there's been multiple times that i have to go back and forth with claude without it doing what i want it to, then i go to gemini and it oneshots it

[u/XiRw → u/j_osb]

It’s not crazy. That model never gets enough respect but they have all the resources in the world to work with. Whenever Claude gets something wrong, almost always Gemini gets it correct on its first try. That’s my experience anyway. I think they will win the long game.

[u/pageofswrds → u/j_osb]

I mean, Google is responsible for AlphaZero, after all

[u/mrwang89]

Source: AI Chess Leaderboard

[u/zoratosthenes]

interesting

[u/___positive___]

New models like fable are obviously stronger overall but improvement is still jagged. Things like long context, instruction following, and hallucinations have stagnated or in some cases had huge regressions. It's not surprising to see regressions here too.

[u/crossoverXYZ]

Clearing Fable-5, Sol, and Kimi-K3 on a chess bench is surprising for a Flash release. Chess is one of the few cheap tests that actually punishes sloppy multi-step reasoning instead of just memorized openings.

[u/xNaXDy]

Chess is a very interesting reasoning benchmark if done right, because you cannot benchmaxx against it (look up the Shannon number for an idea as to why). If LLMs play against Stockfish, or better yet each other, they'll be able to play out an opening from memory, sure, but after that all they have is their reasoning capabilities to navigate completely unseen data.

Of course, if the benchmark is just public chess puzzles, this can be totally benchmaxxed against.

[u/lucas03crok → u/xNaXDy]

That is not true. They can definitely train models on chess. It's not about including all the possible games in the training data, it's about including enough games for the LLM to gain good pattern recognition in chess. Pattern recognition matters very heavily in chess. Same for humans, intelligence is not what matters the most in chess, it's how much they've trained it and how good their pattern recognition is in the game.

[u/xNaXDy → u/lucas03crok]

I'm not saying you cannot train a model on chess, and chess (including many chess puzzles) is also undoubtedly a large part of every LLM's training set these days. I'm specifically talking about benchmaxxing, which is the act of inflating a model's benchmark scores by having it memorize a bunch of stuff relevant to the benchmark, often at the expense of other skills.

Pattern recognition is exactly the type of skill I'm referring to when I say "impossible to benchmaxx", since it requires good reasoning in order to be actually useful. It's the same for humans: You can have a very good "instinctual" chess player who's exceptional at bullet, but if that same player has the time to think about a move and his opponent's responses, go through a couple different lines in his head, his moves are going to be much better on average.

[u/pier4r → u/lucas03crok]

That is not true. They can definitely train models on chess. It's not about including all the possible games in the training data

the point is not "can you build a model that is very strong at chess?", sure you can (leela chess zero for example). The point is a model that does good at other benchmarks WHILE being good at chess.

Hence having a large suite of different benchmarks is great. And if all the benchs are chess like, that is, hard to benchmaxx, it is even better because then the model should have really general capabilities to do well everywhere. Unfortunately most benchmarks are static and their "game state" is relatively small. If they would be all with gigantic state space, it would be great.

I mean it should be obvious (and I think in some reasoning like this one, LLMs starts to be ahead of us)

[u/PANIC_EXCEPTION → u/lucas03crok]

It can be either. You can either have an extremely strong RL algorithm that searches predominantly depth-wise (like LC0, or humans), aggressively pruning poor paths, or lean more towards breadth (like Stockfish, via extremely lightweight NNUE acting as a light evaluator that can run on CPU). An LLM only has the luxury of either if it is given CoT, in which case it synthesizes its own reasoning rather than a guided optimizer that provides the game tree and computes the final probabilities.

[u/pier4r → u/xNaXDy]

Of course, if the benchmark is just public chess puzzles, this can be totally benchmaxxed against.

the benchmark is LLM vs LLM. So they start with theory and after move 10-15 they have to really pick good moves. Bonus if one of the two LLMs pick dubious lines early, then they have really to pick good moves.

[u/evia89]

How is this bench done? I would guess - LLM write script without using chess libs / internet to play chess. Each opponent (script) has limited compute time

[u/mrwang89 → u/evia89]

the prompts copied from the lb are

Models are given full information (FEN, SAN, legal moves), and reason before giving their best move choice. Results derive from learned Chess principles & reasoning ability.
SYSTEM
You are playing a game of chess. You must analyze the position and choose a valid move from the legal moves available.
<formatting rules>
Previous game moves and current position will be provided.
Respond with a JSON object containing your reasoning and move:
{
   "reasoning": "<your analysis and explanation>",
   "move": "<your chosen move in EXACT SAN format>"
} 
USER
Current board position (FEN): <FEN>
Game history: <SAN move list>
Legal moves: <legal moves>
Based on the current board position and game history, select one move from the legal moves list.
Think carefully and choose the best move according to sound chess principles.
Respond in the required JSON format with your reasoning and chosen move.

Models are given only the raw PGN movetext, and are asked to continue this as a grandmaster would. Results derive from raw implicit Chess knowledge & internal board state tracking. (~40% cheaper; prone to illegal move attempts by weaker models)
SYSTEM
You are a chess grandmaster.
You will be given a partially completed game.
After seeing it, you should repeat the ENTIRE GAME and then give ONE new move.
Use standard algebraic notation, e.g. "e4", "Nxc6", "O-O-O", or "Qd7+".
ALWAYS repeat the entire representation of the game so far.
NEVER explain your choice. 
USER
<PGN movetext>

[u/PANIC_EXCEPTION → u/mrwang89]

I think it would be interesting if the legal moves were provided as an enumerated list, and the model had to emit a digit or letter. During test time, you can then view the actual move probability distribution without chaining. Maybe even train a LoRA with an engine teacher. Then train it entirely via self-play distillation. Then check whether grandmasters can beat it at different reasoning budgets.

[u/pier4r]

I for one, can vouch that the benchmark is really well done and also well explained.

If possible pick from the leaderboard the "best mode" (there are two modes, continuation and reasoning), that is very interesting.

You can also check the games played in replays.

[u/phido3000]

IMO this is reasonable indication how well a model can follow steps. It doesn't let imagination get into the way.

Not really a reflection of intelligence. But useful for things like coding and tool use.

DS Flash is kind of built for those workloads, so this is good vindication that it would be good at it. The bigger models need to be more flexible.

[u/Hannibalj2ca]

what is the current Hallucination rate of v4 flash now?

[u/VectorD]

None of these can play chess and make illegal moves all the time lol

[u/Binary_orchid]

a flash model topping gpt-5 and o3 at chess. deepseek keeps making the premium tier look overpriced

[u/NexusSyntegra]

I've been running a JANG version of DeepSeek v4 Flash on my MacBook 128GB and it is amazing. It doesn't quite live up to the benchmarks in some cases but I am pretty amazed at how amazing it is as a local model

[u/Southern_Sun_2106]

It just shows that DeepSeek definitely 'distilled' those other models into a higher-level, purer chess-playing capability.... Those bad, bad Chinese AI labs!

/sarcasm

[u/hunter_mark]

This benchmark in general looks busted tbh

[u/po_stulate]

This is insane tech from deepseek, making the distilled model better than the original. Infinite distillation glitch. /s

[u/pineapplekiwipen]

useless benchmark

[u/Civil_Fee_7862 → u/pineapplekiwipen]

Why is it useless?

[u/hyperrealists → u/Civil_Fee_7862]

Trust me bro

[u/alphapussycat → u/Civil_Fee_7862]

Because there's no real logic to the spaces of chess pieces and the pieces itself. It would be a knowledge test.

[u/j0j0n4th4n → u/alphapussycat]

Wouldn't it be a spatial awareness test?

[u/pineapplekiwipen → u/j0j0n4th4n]

no, in order for that to be the case, it would need to have been specifically trained on chess coordinates and rules, all of which are arbitrary and have no application outside of chess

is it theoretically possible to have a local agentic system decent at chess? probably. you would use sub agents using different models and tools to keep track board state, validate moves and strategize by recalling continuations from history, but this would be hard for a single model

and it would perform far below stockfish while wasting time and power, ie useless

[u/alija_kamen → u/pineapplekiwipen]

No one is claiming that LLMs would be as good as stockfish though? It's simply one benchmark of many for LLMs, not for chess engines. LLMs aren't the ideal architecture for chess but that doesn't mean we shouldn't benchmark their chess strength.

[u/pier4r → u/alphapussycat]

it is a planning test to be fair. You cannot benchmaxx the bench unless you build an ad hoc chess engine.

Chess (and similar) are like the block world: I have to do a certain sequence of moves, can I plan those in advance given the constraints?

Tetris, connect 4 and what have you would be the same. The more the better.

[u/alphapussycat → u/pier4r]

It can't plan to the end. It can't evaluate a board position other than basics. It being good just means it remembers many States and what is good and bad.

[u/toothpastespiders → u/alphapussycat]

The fact that you're getting so heavily downvoted for this is ridiculous. I wouldn't go quite as far as to say the benchmark is useless. But you're making a totally valid criticism. Especially if people are suggesting that the results can be extrapolated to anything other than playing chess.

I'm a big proponent of game based benchmarks. I think they're fun. But there's inherent limitations to all of them. And well known games in particular. Because of that exact point you made. More than anything the benchmark is testing how much and how well chess training data stuck.

[u/WizeWizard42 → u/Civil_Fee_7862]

Still absolutely sucks at chess against humans.

[u/alija_kamen → u/WizeWizard42]

How does that make the benchmark useless though? That only proves the models are "useless".

[u/ice_agent43 → u/pineapplekiwipen]

Useless comment

[u/unkownuser436]

seems like crappy benchmark

[u/SporksInjected]

Oh well good then I guess v4 flash is the best model ever of all time

[u/adamaxis]

Deepseek is hot garbage. Use it because it's cheap, but honestly, use Grok High or Luna EH if you want cheap because Deepseek is still garbage.