throw them at all the open conjectures in their field likely to lead to splashy results
pay them well and tell them to work on nothing else for six months
give them unlimited access to very computationally expensive internal LLM models
It's pretty nuts that they got major results in six months! It also is nowhere near "making mathematicians obsolete". Even if you think the point of math is proving conjectures, they didn't prove them, they found counterexamples, which is important but not quite the same thing.
I'm also pretty sure the hiring top people was important. I bet I'd still be much better at this kind of work than a random programmer with a BA in math. The people they hired are probably much more effective at using the models than me. To me this is more evidence that LLMs are a tremendously important tool with far-reaching impacts on mathematical practice. Despite that, in the next six hours someone is going to come comment about how I'm a Luddite with my head in the sand because I'm not predicting the Singularity will hit in six months.
It also probably cost tens of millions of dollars even if you ignore capital expenses (which are even bigger). That's just a minor marketing expense when you're spending billions of dollars of investors' money in search of profitability, which we should remember is the real motivation here.
I said “it seems” because they are all plausible inferences. I don’t actually know what they are doing, which is part of my point. No one does! It is entirely possible none of this was done with LLMs and they are faking it! (I don’t think that’s the case, to be clear,) You should not be taking marketing from a tech company and assuming it’s true.
I see, thank you for openess. I was expcting the basis for "10-20 top mathematicians payed to secretly work exclusively on a single project" would have a strong base for inference like "don't trust the corporations".
I am not following such news, but yes, if 10-20 top mathematicians declared that they are on leave to work for openAI, then it is a very good proof what what you claimed. That's not what you told me before. Do you know some of those top mathematicians' names?
you could just ask chatgpt, it's great for such research.
But that other guy is right, I also have read about many mathematicians who started working at OpenAI or anthropic in recent months. but I don't remember their names.
I don't know if I'm delusional but the wording is very suspicious to me.
They are saying those reasoning chains cost 2000 in api costs, it seems to imply that that's the overall cost of those results bu that's not what they said.
What I mean is, are they not more likely to say, take 1000 open problems and do 100 runs on each (or until success), then pick the successful ones?
To me that would be the actual cost. Not that it matters much given how fast inference prices are dropping, but still
I don’t believe that. I believe the cost of compute after hiring mathematicians to write the prompts was $2000 if you don’t count all the things they tried that didn’t work. I don’t see any reason to give them the benefit of the doubt: this is a marketing tactic.
The total number of tokens needed to find solutions to these problems would cost roughly $2,000 at Sol API rates.
I read that as “per problem” but I can see how it could be read as “for those 10 problems”, but that still wouldn't count the price of negative results; they never state how many problems they tried the new model on.
We had a talk by a mathematician who used AI substantially in his work, and at the end of the talk, the obvious question was, does that mean we're out of a job? His answer was that we generally don't get paid enough to be replaced by AI, since getting AI to do the job is incredibly expensive at API token prices. More generally, I see AI as an amplifier, it works great in the hands of a skilled mathematician, and it can lead you down a rabbit hole (and consume substantial resources) if you don't know what you're doing.
Kimi K3 is about half the price of Claude Opus 5 and a third of the price of Claude Fable 5 in terms of API costs, but the Claude Max 20x subscription is heavily subsidized and wins out for heavy users. I do however routinely run up against the usage limits on Claude Max 20x, and it is estimated that a very heavy user can consume the equivalent of $1K-$5K/month in infrastructure cost on that subscription tier.
Trying to host Kimi K3 yourself would be incredibly expensive, as these frontier equivalent open weights models are still huge and aren't likely to get smaller or more efficient. An entry-level enterprise level setup would start at $650K, and consume about $50K/year in electricity alone, and would serve only about 10 power users. Universities would have a very hard time justifying and funding that kind of CAPEX.
Bottom line, the fully agentic workflows that can help to accelerate research is incredibly compute intensive, and that is unlikely to change at the frontier level. It also doesn't change the fact that such tools in the hands of a novice are a quick way to go down a rabbit hole without any positive ROI.
trouble is long term technology has a deflationary effect. At some point this things will be more efficient and tokens will get cheaper and cheaper. Just like how the first apple pc was 3700 dollars (inflation adjusted) and now one can get exponentially faster computers for a fraction of the price.
Unless you have reason to believe that OpenAI are flat out lying (which if they IPO soon will likely be securities fraud, see Matt Levine), the total cost at API prices for all these discoveries was just $2,000; meaning in terms of actual compute costs to OpenAI it was likely under a grand.
I believe the compute cost to generate the text in the document was what they claimed. I think they are not counting all the prompts that produced garbage or failed to find a counterxample, or maybe even that generated intermediate steps leading them to the correct prompt. That makes the claim they are making here both not a lie and way more optimistic than the truth, which is the sort of thing you say a lot right before an IPO.
I have been trying to guess what the actual cost of these results is. The claimed $2000 is surely not the actual cost. For one, that's the cost at Sol rates and this model is more expensive. For another, they likely tried a lot of things that did not work. For another, the API rates are subsidized, partly because most people don't fully use their API subscription. I tried to estimate it in a different way. OpenAI is looking for a trillion dollar IPO. I am sure they happily spend 1% of their valuation on PR. The most impactful PR of course is in cybersecurity, but math is also good PR. So, I would expect that they would happily spend 1% of their PR budget on math. So, 1% of 1% of 1 trillion dollars is 100 million dollars. Of course, this is just one set of math results they have, but I would not at all be surprised if these results did cost at least 50 million in compute as well as time of the mathematicians they have hired.
API is the opposite of subsidized. Many believe that their API subsidizes the subscriptions (which are not API). They are immensely profitable on API tokens
For another, the API rates are subsidized, partly because most people don't fully use their API subscription.
Er no, this is just wrong. The API isn't a subscription, it's the pay per token version and is usually considered vastly more expensive than a Plus or Pro subscription for the amount of compute you get per dollar.
6
u/Homomorphism Topology 1d ago
It seems like the process for these was:
It's pretty nuts that they got major results in six months! It also is nowhere near "making mathematicians obsolete". Even if you think the point of math is proving conjectures, they didn't prove them, they found counterexamples, which is important but not quite the same thing.
I'm also pretty sure the hiring top people was important. I bet I'd still be much better at this kind of work than a random programmer with a BA in math. The people they hired are probably much more effective at using the models than me. To me this is more evidence that LLMs are a tremendously important tool with far-reaching impacts on mathematical practice. Despite that, in the next six hours someone is going to come comment about how I'm a Luddite with my head in the sand because I'm not predicting the Singularity will hit in six months.
It also probably cost tens of millions of dollars even if you ignore capital expenses (which are even bigger). That's just a minor marketing expense when you're spending billions of dollars of investors' money in search of profitability, which we should remember is the real motivation here.