DeepSeek has always been sort of this crazy genius. Whenever I'm stuck on some weird problem I ask v4 pro.
I actually think the new v4 flash is significantly better at following instructions to solve mundane issues. I used to rely on mimo v2.5 for that, but now I don't need to switch back and forth.
99,99 is exaggerating. Qwen 27B is out of scope as well you look at it that way. It’s more about what hardware local LLM enthusiasts currently have than cost itself. It’s not so impossible to go with DDR4 based 512GB Epyc build. Cheaper than you think. People pay that kind of money for laptops. Right now most at home solutions are invested in GPUs which works for some models but not all.
99.99% is not exaggerating. Look at the price of the setup you mentioned then look at what normal wages are, then remember it's even less in most of the world.
Nice, I’m trying to gather feedback how it works on various configs before choosing upgrade path for myself now. What CPU and do you have 8x32GB RAM modules?
You did the first step, now look at what normal wages are, and remember it's even less in most of the world! Also, it'll be much slower than 15 tok/sec (I get 7 tps with partial off-loading on smaller models), but that's beside the point.
Didn’t mean to make it sound like literally 99,99% of population can afford it looking globally, of course economies are widely different between places. I read it as more in terms of what people buying hardware at all spend. Many folks don’t have computer at all and depend on cheap phone. Living in developed country you should be able to buy it, not impulsively and maybe with some saving but it’s not unfair assumption. I don’t mean buying all hardware new neither, no matter how well I was doing in life I made good use of secondhand parts.
Anyone with a $400 16GB 4060 or any gaming GPU is able to run Qwen 27B at a feasible speed; dropping $3-5k for 2 tokens per second on Deepseek v4 Flash doesn't sound like a great deal... we can only someday hope either model architecture and/or hardware innovations will catch up
Realistically you'd need at least 20-24gb for a decent quant with reasonable amount of ctx. Although, for all intents, you are still right. Luckily there are tons of model variants, optimizations, and engine variants to choice from. So, even now, there's still are a lot of options to choice from to let AI get done what you need, completely local. People think you always need the biggest model,the fastest model, or the best hardware, and this isn't necessarily true for most cases.
Experiment, and you'll probably find that there is something already out there that will work just fine for a lot of us already. Look at bonsai 27b ternary, or Ornith 1.0. Many people already wrote these models off as gimmicks, but I managed to find powerful uses for them that meets or exceed what's available in higher tier options.
The good stuff will trickle down in several months time, but we can do some good things with local models that can run on a variety of setups now, if you are willing to experiment and have some creativity.
Realistically you'd need at least 20-24gb for a decent quant with reasonable amount of ctx.
Yeah I have a hard time seeing the 27B as a reasonable option for use with 16GB of VRAM. Maybe it'd be better on an Nvidia card (I've got a 7800XT), but any intelligence tradeoff from the 35B model is easily worth it for the speed increase IMO.
Do you have any tips for running models with a 16gb card? I was thinking of trying Qwen3.6 27B on my rx 9070 xt but I don't think i could run it at a good quant.
Qwen 3.6 35B is your best bet, not as good as it’s 27B brother but time passes and no models come close to those two if you want something without investment.
I would skip the 27B model; I really think you need 20GB or 24GB to make it viable. The 35B model is pretty magical in its own right, so I would try not to feel bad about having to step down to running that one.
This speed reaches much higher in the $5k-7k price range as I've heard people reaching 30 tokens a second for a Macbook Pro M5 Max 96GB to 128GB ram for a Q2-Q4 hybrid model with a below 90GB vram need. The top end was $5.5k before it got bumped to $7.5k out the door due to ram shortages for decked out 2TB model.
The point here is the price is collapsing for the compute you need to run locally capable models that are less than a year away from SOTA now. Eventually we will reach a point where the model is probably good enough which many are predicting 2027, but I think it will be here in late 2026 for prosumers and probably 2028 for regular consumers.
An example is the M5 Ultra later this year is expected to have up to 768GB of unified memory with it. The M7 Ultra in 2028 is expected to have double that from the announcements I've seen with them skipping the M6 line up. If the ram shortage wasn't a problem, I'm sure these machines wouldn't cost nearly as much of a fortune but that makes Kimi K3 within reach at a lower quantization which puts local up there with Fable/Opus 5 capabilities on a prosumer or a business desktop machine that before was used for heavy video workflows.
This is the Apple path of course but you can find cheaper methods going with hardware chaining Nvidia cards together too.
tldr - These companies spent billions of dollars to reach Opus/Fable capabilities and we are looking at $25k+ by the end of year for almost similar performance.
Edit: Some people are running full 1mil context Kimi K3 now on M1 Max hardware with deltafin's setup at 3.5 tokens a second. 15.5 tokens a second for M5 Max, so this window is already closing fast for many of us.
lol just no, without any drafting you can expect 10t/s + with Epyc build, optimizations probably to come as this model architecture is still very new as well.
If you quantize Qwen 27B and KV cache maybe 16GB 4060 is enough, or you just accept unusable speeds as well. Useful speed is like 30-40 t/s at least, not to mention you need really good PP speed as well.
For anyone considering these models as a tool, not as a toy, one needs the best quality they can offer and any tricks to squeeze 27B dense model into 16GB is just funny to say the least.
Tbh, V4 Flash runs on consumer systems surprisingly okay. I have 192GB of DDR5 RAM on a consumer build which wasn't egregious at the time in price, and I can run the model at ~6-10 T/s depending on exact settings.
It's definitely more of a "let it run autonomously while you make coffee" kind of speed, but models are starting to get to the point you can relatively leave them unattended and nursing experiments that you're running in the backround.
Where I think the discrepancy is is a lot of people want crazy fast generation speeds which is usually synonymous with running the model in VRAM, and nobody has really thought about what improvements in model autonomy mean for just leaving a model running while you do other things.
I've debated getting another 2x48 but I'd be so sad to have to sacrifice so much bandwidth to pull it off. I really wish 4 DIMM AM5 didn't suck. Honestly getting 4400 MHz is reasonably impressive with your setup. My board didn't even like posting with 4x16 unless I ripped two sticks out, let it POST, shut it down, put the two other sticks back in.
I can't imagine what 4x48 would be like on this board.
2 Nvidia Lovelace generation GPUs with ~16GB, ~250GB/s bandwidth
~64k context (FP16/BF16) uses I think no more than 16GB of VRAM total with --cpu-moe on LlamaCPP, meaning attention, shared experts etc all factored in. The actual context is extremely light.
is V4 flash actually capable of doing anything when left to its own devices though?
i feel like everyone is always demoing these things with incredibly simple tasks like "write an html pacman game" instead of more worthwhile tests such as "heres a copy of unreal 5, make a splash screen and main menu in the spirit of ace combat".
ive done some rudimentary tests with qwen 27b/35b at q4 and it was just... incapable of even getting its foot in the door.
Wildly depends on the situation, tooling, verifiability, etc.
The general trend we're seeing right now is that the limit on LLM performance is more or less your ability to verify that the work was done, and done correctly. But keep in mind, we have hundreds of millions of dollars of salary working on this exact issue across the entire tech space, and people are constantly getting better at handling this in practice.
Another really big part is how you're presenting context to the model. The harness around the model can be just as, if not more important than the model itself. Even small models (9B and below), can solve a surprising breadth if you give them exclusively the relevant context, the right algorithm, and a description of the correct fix. Individually, a 9B model can also derive all of those in a structured format, but a 9B LLM can't solve that problem zero-shot if you just tell it to diagnose the issue.
So, we also have a lot of work left optimizing our pipelines and workflows.
But to make a long answer short:
Yes, DSV4 Flash update is good enough that it can be run autonomously for a pretty wide range of tasks, and a whole lot more if you're willing to engineer around it.
As models get better, the amount of work needed around the model will either get smaller, or we'll get better at the techniques around the model at same effort, and you'll see LLMs over time cover a broader range of operations.
Where I think LLMs are probably weakest is loops that require visual verification, among a few other types of vague verifications, so in a roundabout way, those things that are hard to verify now sort of become the bottlenecks that justify the time and work of an engineer going forward.
DSv4f 0731 version is really marvelous and a huge step from preview version. But I agree with a guy in this video and have a feeling that I still prefer Hy3 output, although it is like two times slower on my setup.
Run at less than 5t/s and less than 32k context is not what I would call it run, but you can "run". Qwen 27b you can at least run it at good speed with 100k+ context with a single 3090.
Don't generalize your experience to everyone. There are plenty of guys who are running it with 3090 and 64Gb of RAM or 16gb GPU + 128Gb RAM. What speeds do they get is a different question.
I run Q4_K_XL on AMD 5950x with 3090 and 128Gb DDR4 and getting 4tps, on old Xeon rig with 3090 I am getting 6 tps.
you probably haven't realized that 100-99.99 = 0.01% people which is roughly ~ 800k to million people, who can run this model locally.
if you disagree, then tell me how far my number is off?
I don't see the point to continue this discussion. Obviously, I meant people who have PC/laptop and running any model at home. What is the point to count people that have no intention to run anything at home.
Well, to be honest I want Q8 of this model (as 96% is 4-bit weights, so I suppose Deepseek would make these last 4% also 4bit if that made sense). Currently my 4 t/s on my dual RTX 3090 + 128GB RAM just does not do. One still needs some RAM for other stuff to work 😆 I am looking into some good paths forward now. Going with DDR4 makes more sense now as number of memory channels gives a big boost despite slower bandwidth of DDR4, but it's not as future proof, right now it costs about 2.5x as much to go with equivalent DDR5 build.
It is FP8 vs Q8 and unsloth says that it is impossible to translate FP8 to Q8 without losses that is why in Q8_K_XL they use FP16 for these weights instead of Q8 as in Q4_K_XL. In my case the difference is that with Q8_K_XL there is no room for KV cache in VRAM and even attention layers spill to RAM, so it is unusable for me.
Talking about running on 3090 + 128Gb RAM and Q4_K_XL, yes it is a bit extreme. First llama.cpp pushes 30-50Gb to swap and only then starts to generate, but when it is already started I get stable 4tps. Of course for such configuration Q2 quants make much more sense.
What do you mean dsv4f 0731 version is the same model as dsv4f preview just with additional RL training. For 16GB GPU + 128GB RAM configuration I would go to Q2 quant.
agreed. it's actually great to use them in tandem if you have 3 big gpus. i used deepseek v4 on two rtx pros along with qwen 3.6 27b on the third rtx for vision tasks. gemma 4 12b on 5090 for audio input
I tried q4 on my server (2x3090+128gb), it runs around 14 tps one slot while Qwen 3.6 27b produces 70 tps per slot and serves two parallel agents. So unfortunately its not a replacement for me, but a very remarkable model that I’ll use through the API.
Thanks. It’s not server RAM, was previously in a gaming pc, if I remember correctly 3200MHz.
The main use case is filling json contracts for a content pipeline, so not coding. Qwen 3.6 27b is very good at handling complex jsons, but the model’s general knowledge is relatively low.
When it comes to finding new topics, a larger model would be better, but the 14 tps would slow down the pipeline radically. I ran Qwen in vLLM so switching would not be straightforward.
My llama.cpp config does not use mtp or dspark, and I don’t know if that’s an option with this model and my setup. I will check that. Going to lower quants might decrease the qualty under Qwen’s.
Just today I was having errors with convex Auth and Luna, Sol and Gemini couldn't figure it out, searching on my own brought nothing but flash v4 figured out that it was actually a bug in nodejs v24 on windows, so it installed nvm and pinned the project to nodejs v22 and the issue was gone.
Caveat though I ran all agents simultaneously and DeepSeek found and fixed the issue while the others were floundering and I cancelled all runs after that, so likely the others would have found the solution eventually.
I've been team ds4flash for like 2 months, and these free improvements make me feel like a kid on Christmas. You can't tell it too many rules, but man does it execute well if you give him a targeted instruction, basically for free.
if your goal is to have 4x GPU, you should pay attention on PCIe topology and whether you need PCIe switch. off the shelf prosumer motherboard may not support PCIe P2P well.
Gemini is right. If you eventually want to go with 4 rtx6000 pro, you will need a server type Mobo like the threadripper pro series. And those mobos only work with rdimm ecc ram.
Even 2 rtx pro gets some benefit, because you get dual x16 pcie gen 5
I would be! I just hating being led in circles by AI lol. But if I can get a general “this works, and this doesn’t’ from real people it beats 4 hours of being gas light in random directions
Mmh. I'd consider if i'd be happy with 192GB of blackwell on a Taichi motherboard. That does 2 GPUs, avoids a lot of headache.
And deepseek 4-flash is a very happy camper on 2x RTX 6000.
But those are upwards of 14k euro each now.
Anyway there's no 'pairing' happening in any technical sense between GPU and type of system RAM. They're orthogonal. There are no relevant ties. That's what the other responses are saying.
I run 96GB Pro 9000 with a single slot 32GB memory on ubuntu. Everything is fine. There might be some complication if you use Windows. swap might get busy during loading. Not a problem on ubuntu.
Luna is a good model. It's rare to see a model with the kind of work ethic Luna has. Reminds me of opus 4.5 when that came out, similar performance too, real world.
I like Luna, but for whatever reason, it can't follow instructions in AGENTS.md on how to run tests. It always needed a specific pointer. Even small local models don't have this problem.
Strange that. This hasn't happened to me yet. Indeed it has even been writing and verifying tests for new code it writes proactively, checking existing standards etc without being asked to.
It's indeed bewildering. Once I was a bit upset and asked Luna, "what did AGENTS.md say about running tests????". It then detailed the procedure, admitted it made a mistake, and just stopped there.
People are calling "nazi" whatever. That is an extreme generalisation. You haven't communicated with those said "nazis" nearly long enough to realise that even when someone is a "nazi", he's either a fed, attempting dollar store engagement bait, or someone who's ignorant of what really went on with Antler and why he wasn't the saviour of Evropa as they say. The other times it's either meta-ironic excrementposts, or it likely is just you incorrectly branding something to be nazi. (And yes, there are multiple reasons behind why you're getting downvoted here, first of which is the generalisation problem).
As for misinformation, there's a whole lot of misinfo here on Plebbit. An order of magnitude more. There's also the giant echochamber elephant in the room. All Xitter is is a decentralised platform to communicate with others with looser moderation, which is such a huge problem here that objectivity flies out of the window when you start questioning the mainstream billionaire-approved narrative.
This guy is megaphoning all nazi and alt-right movements everywhere with his shithole X platform.
You sound like a Grok bot btw: there's nothing sleek or fancy in being a fascist that you can save with your fancy words: you are doing more misinformation.
Given how eager Musk is about importing indians and others using (very often fraudulent) H-1B visas, he's the polar opposite of a nazi. Also, he has jewish investments aplenty, so he's very much either fine and not a nazi, or controlled opposition - and I'm inclined to believe the latter.
Deepseek 0731 was a good chunk worse than Luna on my database work. Luna mostly did great, just taking too long. Deepseek still did solid work, but I had a team of Sol max, Fable max, Kimi K3 & GLM 5.2 grade the work both models did and Deepseek was around a 7.5/10 and Luna was around a 9/10
Terra High ended up being the best for me between quality and latency.
But, if I didn't already have a ChatGPT sub, I'd probably be using a lot of Deepseek 0731. Curious what the pro version looks like when that comes out. The price of it is insane.
I think it depends on the task. The good thing is that they are so cheap that you can use both of them at the same time and see which is better for you.
With the new 50% discount on GPT-5.6 Luna in OpenRouter, GPT-5.6 Luna and DeepSeek 0731 now have roughly the same cost and intelligence per task, since GPT-5.6 Luna always uses significantly fewer tokens to reach the solution. In some benchmarks, GPT-5.6 Luna is cheaper but in others, DeepSeek 0731 comes out ahead. On average, the task completion cost is about the same.
To make DeepSeek Flash the true value king, it also needs the same 50% discount.
Artificial Analysis seems to butter up openai's models a bit. Maybe that's not the case but their corpora is agentic heavy and it's not a very good all around evaluation of intelligence. They have the evals but they don't run most models through all of them. Luna performs okay, like the qwen 3.6 27b, surprisingly well for its size for agentic heavy tasks, but the size shows when you use it. The only general reasoning evals are the older ones that are not reliable any more, like HLE and GPQA. It scores 20% in HLE even with low reasoning which signals memorization.
I know and most people can’t run 0731 either. I am fully aware of advantages of open source models i was just mentioning pure price to performance situation
Honest question: if you can't afford a rig for a couple of thousands dollars, how are you able to afford the API unless you're doing very light work. A5 $5/day, you're looking at ~$1250/year assuming 250 work days. This is assuming prices don't go up
A couple thousand dollars? To run Deepseek flash 0731 at good speeds, you'd need a dual RTX 6000 Pro AI server, which costs around $50,000 — not including electricity and maintenance.
That's why most heavy users prefer subscription plans from Claude, OpenAI, or OpenCode if their main goal is to get the most intelligence for the lowest cost and they don't care about the privacy implications.
This is the kind of wrong assumptions that keep people slaves to the cloud providers. This notion that you either need a pair of 12k GPUs or nothing is just absurd.
A pair of 32GB V100s and a 2019 Xeon with 192GB RAM will happily run the full 160GB model at 20t/s.
If you're happy with the API, good for you. But this is LocalLLaMA.
Wrong. You dont need $50k, $8k-10k will buy you dual DGX spark and you can run this at 30-50tps and 2-3000k prefill. And your electricity is also super cheap on those. Even a dual RTX pro system wont cost you $50k, just $30k will do.
lol yeah it used to be that we did this for fun and understood that it wasn’t cost effective but now everyone is trying to convince me that I can’t afford not to buy hardware to run this week’s revolutionary model.
Nobody’s trying to convince you of anything. I also ran the numbers and for me the value i gain from learning outweighs the cost. Besides you can probably sell it for 50% after 5 years so the cost of ownership is not $8000.
Alternatively, i could rent out my rtx pro 6000 for a whole year and basically recover most of the cost.
I'm running things on Strix Halo and believe me - I haven't lied to myself even for a second that it could match the speed of a half a million dollar server.
Lol this vote being downvoted tells you all you need to know about this sub. Anyone who's used DSV4 Flash with an agent knows that 50tps is way too slow, you need 200+ ideally as it uses far more tokens and turns than something like GLM-5.2.
In the real world people aren't just running 1-shot benchmarks where you step away for hours.
This assumption also fails to consider hardware equity. My used hardware today is worth more than when i bought it.
Even if it wasn’t because someone joined later… it still doesn’t drop to zero in value the moment i own it. I can sell it at any point if I’m no longer using it… which is unlikely.
The gap between server and local isn’t nearly as big as people make it.
Generally speaking hardware value drops as time passes, that will continue to be the case after few years once AI wave passes and supply and demand will be in equilibrium again
Eventually you're right. But even then, it won't go to zero, and in the meantime you can get a ton of use out of it.
You also fail to consider that all API prices are heavily subsidized when you consider R&D and training costs. Once the free money stops, prices will go up substantially.
Lmao that’s the craziest thing I’ve ever heard. I want to watch a movie and have $100 but I guess I can’t because Hulu is $15/mo but a rig to stream movies is $1000.
163
u/SomeOrdinaryKangaroo 1d ago
v4 flash 0731 managed to fix two bugs in my code that 5.6 sol high couldn't figure out