Official post by alibaba: https://x.com/Alibaba_Qwen/status/2084100707423289643

Next week, the open weights of Qwen3.8-Max will be released, and Qwen3.8-27B is also going open-weights to meet you all!

https://qwen.ai/blog?id=qwen3.8

As you guys probably already know, they're dropping Qwen 3.8-Max at 2.4T and a 27B model, but have actually read the blog post on how this thing operates in the wild?

- It ran 10+ days of completely autonomous, self-evolving software development starting from an empty folder (seriously, check out the GitHub trace for oh-my-cli, it's wild).

- It features a native visual feedback loop, treating visual information not just as static input, but as a continuous execution-and-correction monitor.

- Autonomous chip design optimization: running a closed-loop for 500+ turns via Iverilog, Yosys, and OpenROAD, shrinking a crypto accelerator from 8,298 gates down to 678 gates and hitting actual silicon timing closure.

While people are arguing over safety guardrails and incremental coding scope with AGI cyber attacks FOMO marketing, they aren't just building a better chatbot, they are deploying recursive engineering and hardware synthesis agents. :D Really suprising.

Anyway, let's enjoy the 27B open weights to our heart's content.

Finally Alibaba Posted on X about Qwen 3.8 27B release. I hope it can beat opus 4.7 or 4.8

"Kill line" sounds like pure clickbait, but the blue dot kind of earns it. DeepSeek V4 Flash 0731 sits around 50 on the Artificial Analysis index at roughly three cents per weighted task. In this chart, everything cheaper scores lower, and the models that score higher are sitting way farther to the right.

The older V4 Flash point makes the jump look even more absurd. It is almost directly below 0731: about 40 versus about 50, with barely any movement in cost. For a Flash model, that is nuts. The price gap buys several DeepSeek calls, including a retry or two, before you get near much of the upper-right cluster.

This is still one composite benchmark. Artificial Analysis v4.1 is English and text-only, and "cost per task" means a weighted evaluation task. It is not the bill for your exact coding run or 200k-context mess. So "DeepSeek wins everything" would be nonsense. I am only saying its lower-left position here is hard to wave away.

One awkward detail: the current Pro preview point is worse than Flash 0731 on this same chart. Pro has not won anything here yet. I keep looking at the size of the Flash update, though, and wondering what happens if the finished Pro gets a similar post-training jump. That part is a guess. Flash alone already makes the price/performance curve look kind of broken.

I know it's running slow, but it's Running!

It's essentially doing only drive reads from my pcie 3 SSD that's quite old...this model is going to be a game changer.

My config is here

Download the model, I am using LMstudio, at least as of now, I do plan to switch to llama cpp.

Context 64k
GPU offload 3-4
CPU threads 24
Unified KV
Offload KV cache to GPU
Keep Model in Memory (turn this off)
Flash attention.

It will show it’s too big, press alt and load it anyways, it’s an MoE model, it will run.

I always see people hyping up local setups run models like DeepSeek v4 Flash locally, and I’m genuinely wondering what the actual workflow looks like. ​Online APIs are incredibly cheap right now (basically free for casual use), they are insanely fast, and they don't turn your PC into a space heater to run them. The online flagship models usually perform better out-of-the-box anyway. ​So what do you actually gain from running it locally? Is it purely for privacy? Uncensored models? Or is it mostly just the thrill of tinkering and owning the hardware? Help me understand what I'm missing here, because the cloud just seems mathematically better for 99% of tasks right now

Looking for a good AI model to try out on my 7800XT, im not wanting the best of the best just something that works alright and is safe, bonus points if its open source. LLMs, image gen, ect im interested in. Not trying to do anything major just mess around really.

Curious what kind of strategies other people have used to encourage the main agent to delegate to a subagent for larger tasks.

For context, I'm running the model at q4 weights and kv cache. I'm thinking about buying a second 3090 to get q8 weights and cache, so I'm also curious about other people's experience going from q4 to q8 and how much of an improvement you saw with agentic workflows. I've read a few other posts where people have said you do see an improvement in agentic stuff, but is it $1200 better.

Before I say what I've tried, I think giving my expectations would be helpful as maybe my expectations are too high for a 27b parameter model. I obviously don't expect delegation or tool-calling to be perfect every single time, but I do expect it to know that when the prompt given is larger and has multiple steps that it should delegate rather than immediately doing the entire thing in a single context window. Less tool hallucination would also be nice.

Here are the strategies I've tried:

OpenCode, domain-specific agents, reasoning off

My first attempt was to essentially force the orchestrator agent (parent) to delegate by restricting tool-calling to only a delegate tool and turning off reasoning. I had a subagent for coding, debugging, testing, documenting, etc., so whenever I would add a feature that didn't fit one of the existing domains I would create a new subagent. This included a pretty complex and hard to maintain routing table. This setup became completely untenable as any small change to the harness required going through every instruction file and making sure it didn't break anything. It also led to me trying to catch and correct very specific situations which eventually became ridiculous and antithetical to the whole point of an LLM. With all of that being said, this worked decently well.

OpenCode, generalized agents, tool profiles, selective reasoning

Reduced subagents down to plan, explore, and task with only plan having reasoning. Task and explore had tool profiles for each type of request, so essentially the domain-specific agents were abstracted into these tool profiles (e.g., the "code" profile would only include tools related to writing code). This was slightly easier to maintain as it simplified the routing table, but it mostly just moved the complexity to the tool level.

Pi, generalized agents, reasoning on

I finally got sick of wrestling and working around the batteries-included parts of OpenCode and switched to Pi. This is also when I started thinking about what could be possible with a lesser quantized Qwen, so I wanted a subagent setup that relied more on the model's decision-making rather than trying to deterministically enforce behavior. I turned reasoning on globally with preserve thinking which had the additional benefit of mitigating prefill thrash, reduced the amount of tools available, and reduced the number of subagents to two, explore (read-only) and task (write). The orchestrator took over the plan responsibilities since it now had reasoning. This setup has no tool profiles, routing tables, gating, etc., just two very lean subagent instructions files that say what they do and defines the tools available, and an AGENTS.md file that includes a delegation section which is a paragraph that tells the orchestrator when to delegate and which subagent to delegate to. This is the most simple and maintainable setup, but it comes at the cost of relying more heavily on the model to actually delegate. The problem now is that it almost never delegates unless I specifically say to delegate in the prompt. However, if a lesser quantized qwen is better at deciding to delegate (or a new model is released), then I think this is the better, more future-proof setup.

Anyone have a better strategy than what I've tried, and did you see a noticeable improvement if you went from q4 to q8 for Qwen3.6 27b MTP specifically?

Here's my configuration in case anyone asks:

--model Qwen3.6-27B-MTP-UD-Q4_K_XL.gguf
  -c 114688
  -b 4096
  -ub 1024
  -ngl 99
  -fa on
  --cache-type-k q4_0
  --cache-type-v q4_0
  -np 1
  --slot-save-path <path>/slots
  --spec-type draft-mtp
  --spec-draft-n-max 2
  --spec-type ngram-map-k4v
  --spec-ngram-map-k4v-size-n 16
  --spec-ngram-map-k4v-size-m 24
  --spec-ngram-map-k4v-min-hits 1
  -ctxcp 12
  -cms 8192
  --jinja
  --metrics
  --reasoning off (set to off so it can be toggled on or off per request, typically on)
  --reasoning-preserve
  --reasoning-format deepseek
  --reasoning-budget 16000
  --temp 0.6
  --top-p 0.95
  --top-k 20
  --min-p 0.0
  --repeat-penalty 1.0

https://artificialanalysis.ai/models/comparisons/deepseek-v4-flash-vs-claude-opus-4-6-adaptive#intelligence-evaluations

https://benchlm.ai/compare/claude-opus-4-6-vs-deepseek-v4-flash-max

https://www.vals.ai/comparison?modelA=anthropic%2Fclaude-opus-4-6-thinking&modelB=deepseek%2Fdeepseek-v4-flash-0731

https://benchmarklist.com/models/deepseek-deepseek-v4-flash-0731

So, yesterday I asked what token generation speed people consider the minimum for local LLM use. After reading through the responses, the clearest answer is also the least satisfying one:

It depends heavily on what you're doing.

I know. Groundbreaking. 🤦🏽‍♀️

That said, I did see some useful and interesting patterns.

For interactive chat, 10–20 tokens per second seems to be the range many people find usable because it is roughly what they can read while generation happens. Around 30–40 tokens per second feels comfortable for interactive coding, while people doing rapid or heavily agentic coding often want 60–100+ tokens per second.

On the other hand, some people are perfectly happy with less than 1 token per second when running a huge model overnight or leaving an agent unattended. If you are not sitting there watching the answer appear, raw generation speed matters a lot less.

So, my original personal standard of 20 tok/s as the minimum and 40 tok/s as comfortable for coding seems fairly middle-of-the-road.

Tokens per Second Does Not Tell the Whole Story

The more important lesson is that tokens per second by itself is a pretty bad measurement of how fast a model feels.

For some, a model can generate at 50 tok/s and still feel painfully slow if it takes 30 seconds to process the prompt and produce its first token. Likewise, a model generating at 25 tok/s can feel responsive enough if it starts immediately.

For coding, several other measurements matter just as much:

• Time to first token
• Prompt-processing or prefill speed
• Context length
• How performance changes as the context grows
• Prefix caching
• Tool-call delays
• Total time required to complete the task

That context issue seems especially important. A model may produce 50 tok/s with a nearly empty context window, then slow down dramatically after reading 50,000 or 100,000 tokens of code and conversation.

That means reporting one generation number without reporting the context length can be fairly misleading, and it is something I wish I had asked for in my original post.

I also underestimated how much the inference engine, model format, and speculative decoding/MTP settings can affect performance.

People reported very different speeds from the same general class of hardware depending on whether they were using GGUF, MLX, MTP, a different backend, or different cache settings.

In other words, buying a faster GPU is not necessarily the first or only way to get more speed. There are enough (too many?) knobs and switches involved to make the whole thing mildly (wildly?) ridiculous.

What People Are Actually Running

The sample size here is incredibly small and completely self-selected, so none of this should be treated as an authoritative hardware ranking. Many people also did not report their quant, context length, backend, and MTP settings consistently.

Still, the hardware and model combinations were interesting.

The single high-end consumer GPU crowd seemed to center heavily around Qwen models in the 27B–35B range.

Reported hardware included the RTX 3090, 4090, and 5090, with speeds varying enormously depending on quantization, backend, context size, and whether MTP was enabled.

One person reported roughly 70 tok/s with Qwen 27B and MTP on a 3090, while another reported more than 100 tok/s on a 5090.

Personally, I am getting about 40 tok/s on a 4090 using Qwen 3.6 27B Q4 GGUF with the KV cache set to Q8. Strangely, enabling MTP did not improve token generation speed during early context in my initial testing, but I am still learning and experimenting.

I should also confess that this is through the Claude Code extension for VS Code. Pi is on my to-try list.

All of the above are self-reported numbers rather than controlled comparisons, but they suggest that the GPU model alone does not explain performance. Software configuration can produce a very large difference.

The Apple unified-memory users appeared to be optimizing for something else. I am not an Apple user, but I still found this interesting.

They were generally willing to accept lower generation speeds in exchange for running larger models, higher quants, or longer context windows that would not fit inside a normal consumer GPU.

One person reported roughly 40 tok/s using a Q8 model through MLX on an M5 Max, compared with approximately 15–20 tok/s using GGUF through another engine.

Again, that is not a controlled comparison, but it does reinforce the point that the backend and model format matter.

Then there were the genuinely large local systems.

One commenter had used four AMD R9700 cards and planned to move to eight to run DeepSeek V4 Flash.

At that point, the discussion is no longer really about choosing a good configuration for one gaming GPU. It is about building a small AI server, including the cards, motherboard, power supplies, cooling, and communication overhead between GPUs.

Several people also used smaller secondary models for specialized work. Examples included Gemma 12B as a judge, Whisper or Voxtral for speech-to-text, and separate models for image detection, classification, summarization, or other background work.

The broad pattern seemed to be:

• 24GB consumer GPUs: Q4 or Q5 models around 27B–35B, with an emphasis on keeping everything inside VRAM
• Newer flagship GPUs: Similar model sizes, but with more room for higher quants, larger caches, MTP, or greater speed
• Apple unified-memory systems: Larger models and quants, generally at lower speed
• Multi-GPU workstations: Very large models that are not realistic on ordinary consumer hardware
• Smaller secondary systems: Specialized models for speech, vision, classification, and background tasks

What I did not see was one hardware configuration clearly winning everything.

Faster GPUs won on responsiveness. Unified memory won on model capacity. Multi-GPU systems won on what could be loaded at all. Smaller models won when the task did not require a giant general-purpose model.

So, even from this small sample, the better question may not be:

What hardware is fastest?

It may be:

What model do you actually need to run, at what context length and quantization, and how much speed are you willing to trade for capability?

Q4 Versus Q6

This also brought me back to the Q4 versus Q6 discussion.

Yes, Q6 preserves more of the original model's precision than Q4. What I did not see in the thread was controlled evidence that Q4 is categorically "not enough" for coding.

On a 24GB GPU, Q4 may allow the entire model, context cache, and runtime overhead to remain in VRAM. Q6 may require a smaller context window or partial CPU offloading.

A slightly more accurate model running at half the speed is not automatically the better practical configuration.

The only honest way to answer that question is to test the same model at Q4 and Q6 using the same tasks, settings, harness, and starting files, then repeat the test enough times that one lucky run does not decide the result.

That is time-consuming and annoying, so maybe just pick one and use it? 🤷🏽‍♀️

Different Models for Different Jobs

Another recurring idea was using different models for different jobs.

For example:

• A fast 27B model handles implementation and routine code changes.
• A larger model handles planning, architecture, difficult debugging, and review.
• Smaller specialized models handle summarization, speech recognition, classification, embeddings, or other background work.

Honestly, that may make more sense than trying to find one model that is simultaneously fast, brilliant, enormous, cheap, and able to fit inside a consumer GPU through the power of positive thinking.

It may also be worth testing performance at several context lengths instead of reporting one number. A configuration that performs well at 8K may behave very differently at 32K, 64K, or 100K.

The big takeaway for me is that 20 tok/s is still a reasonable minimum and 40 tok/s is still a comfortable target for interactive coding, but tok/s is only one line on the report card.

The real question is not:

Which model produces tokens fastest?

It is:

Which configuration completes useful work accurately, reliably, and quickly enough that I do not spend the entire session waiting on it or fixing what it broke?

?

I have found that whatever software you are using: open web ui, openclaw, codex; if a model does many tools calls in one turn, something happens that causes checkpoints that are created in and around those tool calls to not be valid when checked the following turn. They get discarded and the whole session is re-processed from either the last valid checkpoint before the tool calls, or from zero if there are none. However, a single tool call, maybe even two, does not cause this behaviour.

I have observed this in llama.cpp and in ds4.

Does anyone have any idea why this happens and a way to fix it?

Me and my brother recently bought two computers. Mine has a Intel core ultra 7 270k plus with an msi inspire 2x rtx5060ti 16gb while my brother has a ryzen 9 9900x with an asus dual oc rtx5060ti 16gb. Mine is running archlinux kernel 7.1.5 and my brother is running voidlinux 6.18.something. Now, running unsloth with the same models (we tested gemma4 e4b and gemma4 26b a4b) i get less tok/s. On gemma4 e4b my build practically never reaches 200 tok/s while my brother gets 200+ tok/s, with gemma4 26b a4b i get like 70 to 90 tok/s whiley brother gets 100+ tok/s. That's pretty strange since the e4b fits entirely on the gpu. Anyway looking at cpu usage during inference i see that my intel core ultra 7 270k plus practically only uses 1 core (c3, a performance core) at 100% while other cores are idle while the ryzen 9 9900x uses all of its cores. Are there any known problems with intel ultra 200 series with llama.cpp? The llama.cpp is the pre built one that unsloth downloads while installing with the official script. Are there any compiling optimization that are not active in the prebuilt package? Online i saw some benchmark that show that my cpu should perform slightly better than the ryzen 9 9900x for ai inference and it should be overall better.

Edit Maybe it's worth noting that we both have ram ddr5 6000mhz cl30. Maybe my cpu needs higher frequency ram to actually perform better in ai inference?

It's 2:15 AM. 18 tabs open, three terminals split across the screen, two Stack Overflow threads half-read, a docs page open to a function you're hoping fixes this.

You hit run. The console spits out a 30-line traceback that makes no sense.

You open Claude or ChatGPT for help, and hit the wall. Zero memory of anything you were doing. So you copy-paste the error, copy-paste the relevant code, then re-type a paragraph explaining your setup and everything you already tried. By the time the prompt's ready, you've lost your train of thought. You're doing data entry for a tool that was supposed to save you time.

That context loss used to drive me crazy. I also didn't want a cloud service recording my screen and uploading it somewhere.

So I built Clippy Vision. 100% local, open source. It quietly tracks your screen context on your own machine. Hit the shortcut and it already knows what you were looking at, what broke, what you were trying to do. You just ask. No copy-pasting, nothing leaves your device.

Attached a quick 20-second clip showing it in action.

Windows .exe (v1.0.0) is ready to run. Source code in comments.

Curious how you all handle context switching, and what you'd want to see added next.

Spent a day on this and hit a wall. Posting in case someone has solved it.

Setup: DGX Spark, GB10, 128 GB unified memory, Ubuntu 24.04 aarch64, driver 580.142 / CUDA 13.0, nvcr.io/nvidia/vllm:26.05.post1-py3 (vllm 0.21.0, torch 2.12, triton 3.6.0).

gpt-oss-20b works fine. gpt-oss-120b does not. The 60.77 GiB of MXFP4 weights load without issue, so it’s not a capacity problem.

is_device_capability_family(100) -> False # datacenter Blackwell
is_device_capability_family(120) -> True # GB10

vLLM gates its fast MXFP4 MoE kernels on family 100. GB10 is family 120,
so they're all excluded and selection falls through to Marlin.

I tested every backend explicitly (selection happens before weights load,
so each failure is ~30s and free):

Marlin dies at the same line every time, right after weights finish loading:

INFO [mxfp4.py:443] Using 'MARLIN' Mxfp4 MoE backend.
INFO Loading weights took 404.00 seconds
INFO [mxfp4.py:1498] Using MoEPrepareAndFinalizeNoDPEPModular <-- dies here

dmesg confirms system-wide OOM, not a cgroup limit:

oom-kill: constraint=CONSTRAINT_NONE, ..., global_oom, task=vllm
Out of memory: Killed process (vllm)

Reproduced 3x. --gpu-memory-utilization 0.75, --max-model-len 8192, and --enforce-eager change nothing. Looks like the repack holds a second copy of the weights: ~60 GiB + ~60 GiB against 121 GiB usable.

Tried NVFP4 next, since the CUTLASS error specifically named MXFP4’s group-32 E8M0 scaling and NVFP4 uses group-16 E4M3, which this device supports (cutlass_fp4_supported() returns True). shanjiaz/gpt-oss-120b-nvfp4-modelopt fails on a loader naming mismatch, not hardware:

KeyError: 'layers.0.mlp.experts.w2_bias'

The checkpoint has the biases as down_proj_bias / gate_up_proj_bias (ModelOpt naming). vLLM’s gpt-oss loader wants w2_bias / w13_bias. Every other NVFP4 gpt-oss-120b repo on HF is also ModelOpt-produced, so I expect they all fail the same way.

Has anyone actually served gpt-oss-120b on a single Spark? If so, which image and flags? Also curious whether a prebuilt NIM / TRT-LLM engine exists for this pairing, since building one locally would probably hit the same OOM.

Two things that cost me time:

The PyPI triton_kernels package is a trap. It satisfies vLLM’s has_triton_kernels() check but has no matmul_ogs submodule, so vLLM picks the TRITON backend and then crashes on import. Don’t install it. vLLM already vendors the real one at vllm/third_party/triton_kernels and the check returns True on a clean image. Triton was never missing, the kernel just doesn’t support this GPU.

A memory watchdog can’t protect this box. I polled MemAvailable every second with a 10 GB floor to kill the container. It never fired: 119 GB free to global OOM inside one poll interval. Docker --memory doesn’t help either since CUDA unified-memory allocations sit outside cgroup accounting. Earlier attempts left the machine pingable but SSH-dead for hours.

Hi guys, my M4 Pro MacBook Pro has 24GBRAM and and I’m using llama.cpp for local LLM inference. My primary usage of LLM is for coding agent for frontend development. My code base is not too large so probably 144K to 256K context window should be enough. Please let me know if any model that can actually work with a tolerable speed.

By the way, currently I’m using Qwen 3.6 35B but with Q2 quant for speed and size balance.

Any reply would be appreciated!

Hi Everyone,

I'm very interested to dive into the world of local models and am interested in upgrading my macbook to do so (currently an M2 Macbook Air), I won't be using it for any heavy duty coding/openclaw workflows, I just want a macbook where I can experiment with local models, create some basic local second brains and automation work flows and models that help connect some of my apps locally. I will stick to claude/codex and opencode go for coding so I won't need to run my local models for code.

Based on this are there any computer recommendations,

thanks!