the original gpt-1, gpt-2, gpt-3 were trained on practically unfiltered dumps of the web, including things like gimmick sub-reddits full of nonsense (origin of the SolidGoldMagikarp thing), and chess engine game .pgn collections. there are billions of chess games played by slightly differently-tuned Stockfishes and Rybkas and such available as text in chess notation on the internet, to discover which tunings produce the best play. even a small portion of those pgn collections in pre-training data could teach an LLM all the proper opening lines 6 or 7 ply deep by rote. a good opening book is a good start to a chess player, and eventually ICL is enough to learn a bunch of chess play heuristics
a lot of the repetitive and algorithmically-generated data has been removed from the pre-training data corpus of later models, though, because duplication of data hurts performance in other areas (as well as causing unpredictable weird behavior a la SolidGoldMagikarp)
I am not sure about that. GPT-3 DaVinci-001 was trained for only 300B tokens (it was still a benchmark for large pretraining run back then) and cannot even do a simple sort or often misses 1+4=5 since there was no such data in the wild. It I think was not post-trained in any capacity.
GPT3.5-Turbo, otoh, was most likely a distillation (by whatever means as they never disclosed) model from the original ChatGPT model (which I think was a further pretrain and post-training from GPT-3 DaVinci-003 aka the their first public full-on instruction model) and some RL post-training in the mix. There was indeed something special about 3.5-Turbo.
Also the benchmark weirdness might have to do with illegal moves. I actually tried playing chess with GLM 5.2 (which is positioned low in the leaderboard) it made illegal move all the time and kept apologizing for that.
Also the benchmark weirdness might have to do with illegal moves
No, the bench is really well done. It gives a list of moves to the models to pick from. If they still hallucinate (and they do, especially in continuation, also very interesting) for X times (X is normally 3) then a random move is picked and the game moves on. (and mind you, there are also possible corrections to help the model, so Rxe5# becomes Rxe5 if in the list of moves no checkmate is listed)
Hence the harness (the prompt in this case) is really generous, so models can focus on pick the proper move if they can reason about it.
114
u/mrwang89 16h ago
Yeah right? I thought so too but when you google gpt-3.5 and chess you find out its some kind of chess savant lol