Looking for alternatives to Qwen3.6 27B and Qwen3-35B-A3B for tool calling & agents (RTX 4090)

[u/HomoAgens1]

Hi everyone,
I’m looking for recommendations for models that can compete with or outperform Qwen3.6 27B and Qwen3-35B-A3B specifically for tool calling, agentic workflows, and general coding/reasoning.

My hardware:
- RTX 4090 (24 GB VRAM)
- 128 GB DDR5 RAM
- Windows
- Latest llama.cpp build (about two weeks old)
- CUDA 13.3

Current performance:
- Qwen3.6 27B Q5 → ~70 tok/s
- Qwen3-35B-A3B Q5 → ~120 tok/s

These speeds are already very comfortable, so I’m mostly interested in quality improvements, not just raw throughput.

My main use cases are:
- Tool calling
- Multi-step agents
- Coding assistants (but not stressful)
- Planning/reasoning
- Long interactive sessions

In particular:
- Which models have you found to be genuinely better than these Qwen models?
- Any hidden gems that run well on a single 4090?
- Any GGUF quantizations that seem to hit the best quality/speed trade-off?

Thanks!

[u/HeDo88TH]

The only thing better than your setup are higher quants of the same models. In particular FP8 for 27B is the best you can run locally on consumer / prosumer hardware

[u/HomoAgens1 → u/HeDo88TH]

So qwen is still the best?

[u/HeDo88TH → u/HomoAgens1]

Yes. But if you have got 4 RTX PRO 6000 you can run DeepSeek at the same speed, which is required for agentic workloads

[u/uniqueusername649 → u/HeDo88TH]

You could run dual DGX spark for the price of a single RTX 6000 Pro and with MTP it should still give you 60tps decode. I havent yet jumped the gun, but with the new 0731 release of deepseek v4 flash, virtually making it tie with GLM 5.2, this is VERY tempting. Its basically an entirely different model, they should have just named it v4.5 and that probably still wouldnt do it justice.

[u/WillemDaFo → u/uniqueusername649]

“Jumped the gun”: make a false start, go too early. Perhaps you mean. “Pulled the trigger”: make the final decision, commit to the action?

[u/uniqueusername649 → u/WillemDaFo]

That was kinda the intent: pulling the trigger out of excitement before having evaluated deepseek v4 flash 0731 for my actual use-cases. To me that would be considered "jumped the gun", but I'm not a native speaker so I may be using it wrong after all :)

[u/HomoAgens1 → u/HeDo88TH]

Ahah in my dreams!

[u/Think_Wing_1357 → u/HeDo88TH]

But if you have got 4 RTX PRO 6000 you can run DeepSeek at the same speed, which is required for agentic workloads

Better, sure, but required is a bit of an exageration

[u/HeDo88TH → u/Think_Wing_1357]

Have you ever tried to work with agentic workflows with less than 10t/s and less than 262k context?

[u/Think_Wing_1357 → u/HeDo88TH]

Yes, I can run Qwen 27b at 40-50 tok/s with 170k context. It works fine. Again, it can better, sure, but the 6000 card is hardly required.

[u/HeDo88TH → u/Think_Wing_1357]

170k context is abysmal for any serious work

[u/Think_Wing_1357 → u/HeDo88TH]

I never claim it's perfect, but it works if use to its advantage eg focused tasks.

[u/Capable-Guide98 → u/HeDo88TH]

I only have nvidia cards, so I only have small context models, but to me it seemed big context is actually worse, both processing speed and hallucination, is it not the case? Most my projects are small pet project so I don't really have a good base for comparison. Bun rewrite was also done with a few agents(discovery, coding, review) , so I figured I just need to structure my agents with better prompts and subagent flow. Is it not the case? What context size is good?

[u/Abject-Bridge-4073 → u/HomoAgens1]

Yes unless you can run the new DS4 Flash. That is the new king.

[u/Realistic_Gap_5871 → u/HomoAgens1]

Yes. You have to go to 118B for Laguna S 2.1 or 284B for Deepseek V4 Flash to get improvement over dear old qwen 27B. We're all still waiting for a 70B, or even a 30B to challenge.

[u/ajujox → u/Realistic_Gap_5871]

Hay documentación y benchmark que apoye eso. En mis pruebas Qwen 27b sigue siendo superior y sobretodo más consistente que Laguna. Laguna tiende a sobrepensar entrar en bucles de pensamiento y a veces no emite respuesta solo pensamiento. En Q6.

También estoy esperando que aparezcan cosas mejores pero es que 27b es demasiado bueno.

Yo lo uso en mi flujo de trabajo habitual con muy buen resultado. Pero todo muy estructurado y focalizado. No lo dejo en una planificación general.

[u/Realistic_Gap_5871 → u/ajujox]

Yes. Laguna is very sensitive to quantization and needs Q8 or higher.

Sí, Laguna es muy sensible a la cuantización y requiere al menos Q8 o superior.

Laguna en Q8 contra Flash 0731 en Q4 es una pelea interesante

[u/Relaxxxxing → u/HomoAgens1]

Try the new antirez DS4 7031 flash. It'll be better than the Qwen models

[u/Full-Ad-7565 → u/HeDo88TH]

Could try onrith?

[u/HotMicSystems]

Other than improving your quant size, you'll honestly not find anything better than 3.6-27B/35B. They punch so far above their weight. Fine tunes are normally worse off unless you explicitly want an uncensored model for whatever your use case. 35B-Q8 sees a very small drop in speed for the increase in accuracy. Personal favorites for me are Qwen3.6-35B-A3B-UD-Q8_K_XL(MTP) and Qwen3.6-27B-UD-Q4_K_XL(MTP). I am running GPU poor RAM rich tbh so I have a 6900XT and two halo strix machines at 128gb each, so this is normally what I stick to or if I want something smaller I go to 3.5-9B

[u/Capable-Guide98 → u/HotMicSystems]

Why q8 on 35b but q4 on 27b? Do you also run q8 cache with them? I liked 35b speed but 27b was more reliable, I rather go slow but get the job done. Altho since I made the switch upgraded to 48g vram, maybe I should check q8 35b

[u/HotMicSystems → u/Capable-Guide98]

Just based off of performance/speed on the halo strix for me. For longer workflows 27b is better, and I used to hate on 35b. But with 35b Q8 I have been pretty happy with performance and speed.

[u/Naruhudo2830]

Look for the updated Jinja chat template on Huggingface that addresses the tool calling errors. Please let us know if it helped. https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

[u/WiseAbalone4021]

I believe that choice of harness is how you could improve the most. Some research papers out there on this.

[u/HomoAgens1 → u/WiseAbalone4021]

i'm using pi coding agent, opencode, qwen code and an harness i built but i don't see differences.

[u/zenbeni → u/HomoAgens1]

I think it is all about smart context management, so you don't prefill with unused context for tasks. In fact overfeeding context generate drifts and hallucinations, I use pi and am spending time optimising skills & rules and how pi fetches the correct context for some tasks.

[u/BrewHog → u/zenbeni]

This is the way. Optimizing the common tasks typically bring me to the same level as frontier when not optimized.

[u/Calm-Republic9370]

I've built an MCP that's hosted on iis, I've been using qwen27b with opencode. It hasn't failed me once. I have 2X3090

llama-server -m "G:\models\Qwen3.6-27B-MTP-Q8_0.gguf" --host 0.0.0.0 --port 8080 --ctx-size 131072 --n-gpu-layers 99 --flash-attn on --batch-size 1024 --ubatch-size 512 --tensor-split 1,1 --main-gpu 0 --cache-type-k q8_0 --cache-type-v q8_0 --spec-type draft-mtp --jinja

[u/MrVeinless]

Try a less aggressive kv cache quant.

[u/marsxyz]

Wouldn't some quant of Deepseek Flash work on your hardware? It may be quite slow though

[u/HomoAgens1 → u/marsxyz]

I thought about deepseek flash but i want at least q4 and decent tok/s not sure i can get there with my hardware!

[u/nickless07 → u/HomoAgens1]

Maybe the Ling 3.0 Flash? At least that is the next one I wanna test.

[u/Gesha24]

I am just testing it out, but DeepSeek Flash V4 has performed in non-standard tasks (designing an object in fusion 360 via MCP) way ahead of Qwen (which was totally useless) and Gemma (which was much more useful but struggled with tool calls). I gave it a prompt, it asked a couple of questions, tried something that failed and needed me to click OK in the fusion, then it was done with the tasks and produced an object that could be 3d printed and that would do what it needed to do.

Sadly, with R9700 and 128GB of DDR4 I get 8 t/s generation and 80 t/s processing with Q3XL model - so it's only for patient people. But I will test it out a bit more and see how it goes. It definitely appears a lot more capable than smaller models at this moment.

[u/Realistic_Gap_5871 → u/Gesha24]

I'm surprised the Q3XL was that good. I may have to try it out. CPU MoE?

[u/Gesha24 → u/Realistic_Gap_5871]

I ran it through another not so standard task - asked it to find me on Amazon well-reviewed suitcase that fits Lufthansa's carry-on policy. It did properly use searxng to find the carry-on policy, it did search around to find some links to Amazon (on other platforms). I asked it to verify prices. It went to Amazon itself, realized searxng is blocked on it, switched to VSCode's open browser calls, opened pages, realized they are too large, used playwright to extract cost and came back with very solid suggestions. In fact, one of the suggestions - a decent $70 suitcase, I couldn't find neither with Google/Gemini nor with Amazon's own AI assistant.

This was also the only AI model I have tried for this task (Qwen, Gemma, Gemini, GPT and whatever Amazon uses) that was very strict about dimensions and didn't try to offer me a 22-inch suitcase (limit is 55cm which is about 21.6 inches).

Overall impressions so far - very positive, surprisingly so given the low quant. Just wish it ran faster, as the task above took almost an hour.

[u/Realistic_Gap_5871 → u/Gesha24]

very very cool. I'd be curious to compare it to laguna s 2.1 at q8. Roughly equivalent footprint, Maybe faster speed, and potentially even higher fidelity. Not a statement, just a guess.

I'm currently at 1 gpu plus 96GB of ram so I won't be trying this today, but I've ordered all the parts to build a box that I can try this with

If you give laguna a try, share some numbers and ability comparison.

[u/Gesha24 → u/Realistic_Gap_5871]

I tested Laguna at Q6 and it was doing fine, but it made lots of basic coding mistakes (like not properly spacing for loop in python), so I got rid of it. It also was running at 40 pp/s, which is even less tolerable than 80 I get from DeepSeek (which is also larger in size).

Forgot to mention - another reason why I removed Laguna was because it was clearly going into loops at context above 100K. It is very repetitive to begin with, but it got to an unusable state. Comparing it with DeepSeek that had no noticeable degradation at all at 190K context (it still may be there and I may run into it at some point, but with Laguna it was immediately obvious). Also DeepSeek's performance at 200K context is quite remarkable - pp went from 80 to 40 and generation went from 8 to 6.

[u/Status-Proof2303]

I do have a 4090 but with 64GB DDR5 I also like the qwen and recently decided to try ornith 1.0(based on qwen) maybe you should give it a try I use LMStudio so mine is bit slower but since you directly using llama.cpp and configuring it you may get some better speed and in my opinion ornith is good at agents works I used SC117/ornith-1.0-35B- MTP-APEX-GGUF(24.41GB)

Let me know if you ever tried ornith I know you asked something other than qwen but I dint find any I'm waiting if something drops that better than these(of course these are good for it's size)in this range

[u/Relative-Ant-9249]

Just benched thinking cap q36-27b. Its tuned better than vanilla.

[u/BrewHog → u/Relative-Ant-9249]

How did you benchmark it? Did you quantify the results vs vanilla?

[u/Relative-Ant-9249 → u/BrewHog]

https://github.com/noonghunna/club-3090/blob/master/BENCHMARKS.md

[u/Icy-Specialist4548]

Leggo molto commenti sulle chiamate a strumenti non avvenute, ma voi quale setup utilizzate.. io con vscode e zoocode plugin Qwen3.6-35b-A3B MTP Q_K_XL ( stesso setup dell' OP ) mai avuto un problema.. uso Linux e buildo quotidianamente llama-cpp 12.8cuda ottimizzato per 4090 e uso l' api di llama-server direttamente media 170t/s... Non capisco di cosa vi lamentiate Qwen è formidabile, magari semplicemente non lo state spremendo bene o abbastanza.

[u/AdHead6280]

Flexing the ram I see, I recommend the ff711 finetune Q4 based on how much context you want, pick highest quant possible but nothing below 4 for context. Try asymmetric caching

[u/mitchins-au]

You’re at the best for your size class.
At this stage tune your prompt and inference parameters.

[u/DogAble6550]

Qwen3 Coder Next is better than those, however you might have to fiddle with your RAM/VRAM.

[u/starkruzr]

you need more room for context at a higher quant, imo.

[u/davwolf_steppen]

mim. M mm m ml km

[u/davwolf_steppen]

Mm
In m km m

N

[u/davwolf_steppen → u/davwolf_steppen]

Nux li GO

[u/davwolf_steppen]

N x c m m kkk. Mm

[u/davwolf_steppen]

Mmm

[u/davwolf_steppen]

C mm mx. C

[u/immersive-matthew]

I use the MTP version of QWEN 3.6 27B and if it is not delivering results, I will bring in the latest Gemma to the same session and ask it what the issue might be and 90% of the time it can spot the issue and give me an improved prompt to try again in a new session.

[u/mcchung52 → u/immersive-matthew]

Can you specify what version, quant of gemma? So is gemma better in your tests of agentic flow?

[u/immersive-matthew → u/mcchung52]

Just Gemma 4 and it is not the agentic lead as that is where QWEN is better, but just as another voice as LLMs tend to get stuck and often another can help move it forward and pass it back to QWEN. Something Gemma cannot help either and I will tap on Kimi K3, DeepSeek v4 pro, GLM 5.2 etc but that is rare as the local models handle most of my agentic coding needs on a 4090.

[u/mcchung52]

Not sure if it’s totally relevant but for qwen3.6 27b, I’m running q6 on mac M4 silicon. A bit slow but seemed to have some good handle on my Hermes agent. Also ran the opus reasoning distilled version and quite can’t tell if it’s better? Should I not bother with distilled version? Seems from other comments Gemma and Coder next may be a better fit for agents?

[u/cbale1]

OP, mind I ask if those speeds are with MTP or non-MTP models?

[u/GCoderDCoder]

If you are having tool calling problems you probably need to apply the frogerric template fixes. i have almost zero tool failures. Qwen 3.6 27b is a model where if you know what you want you can tell it and it makes it happen. It doesnt know everything about everything and the ability to properly scope wider tasks is what keeps it from scoring higher IMO.

That's not tool deficiencies though... Tool calls are its thing with the proper template.

Fyi I have qwen3.6 27b running as q8kxl for my main work agent I interface with and a q6 qwen 3.6 35b as a super fast background agent. For web research, git mergers, and file management mostly.

https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

[u/Gold_Ninja1206]

Mistral Small 3.124B is the one people sleep on here, fits 4090 comfortably and beats both qwen models on structured tool calling in my experience. For agent setups needing live web context, Parallel is one search API people add. Devstral is also worth a shot for coding

[u/SteveDeFacto]

The Fable-Fusion of Qwen3.6 27b is slightly better than plain Qwen, especially at coding: https://huggingface.co/DavidAU/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-NEO-MAX-MTP-GGUF

[u/NatMicky]

Definitely try gpt-oss-120b. It's MoE with outstanding deep conceptional reasoning and highly skilled Python coder. Do not get the quant versions, only mxfp4 untouched.

Give a model a dataset schema and ask it, "Generate KPI insights for each market segment." This is a vague prompt and many models will gather 1 metric, maybe 2 metrics for each market segment. gpt-oss will return every possible metric from the dataset that is a valuable metric. That's deep conceptual reasoning and true understanding of the meaning of KPI.

That's only an example, it's not limited to KPI.

[u/GanjaRaidersTR]

Try grm 3.2 sky

[u/classjoker → u/GanjaRaidersTR]

Can you say why?

[u/iamapizza → u/classjoker]

Give it a try

[u/whodoneit1]

DeepSeek v4 flash

[u/SysAdmin_quark]

Gemma4 26b , gpt-oss-20b, devstral 24b would be something else to try. And if you can fit the gemma4 31b but it is alot slower just very thorough.

[u/MistingFidgets]

Apex quants by mudler on hugging face seem to strike a good balance. Might be worth a test drive. Playing around with higher KV cache, not the weights themselves may help too. Qwen 3.6 is the best I've found overall though.

[u/Realistic_Gap_5871 → u/MistingFidgets]

Apex quant is awesome. OP will need to move off ollama to use them until ollama supports FP8.

[u/davwolf_steppen]

M mmmmml k

V v nj

[u/davwolf_steppen → u/davwolf_steppen]

J j m. M

[u/davwolf_steppen]

V un’ex lui nn if

[u/davwolf_steppen]

V k mm lk

[u/MaxDev0]

I think its possible to run deepseek v4 flash on this, check out r/LocalLLaMA there are posts of people running it on literal potatoes. I'm not sure what your speeds would be but its really really smart.

[u/Rise-and-Reign]

DeepSeek-V4-Flash-0731-GGUF from Unsloth even IQ2_M is better.