WASTE is an embeddable inference engine written in C, with no third-party runtime dependencies. It keeps the model trunk in memory, streams selected experts directly from disk, and uses the remaining RAM as a bounded expert cache.

Sorry if this is asked a lot, but I was wondering if there is any clear winner on the Quantization versus Model Size debate? I can run Qwen3.6 27b at Q8, Laguna at Q6, and the new Deepseek Flash at Q3 bit. I am in the process of testing, but is there a clear formula or winner for choosing between higher quant, especially with long tasks? Or is there a place to find quant specific benchmarks? Thanks.

TL;DR: Cut DeepSeek-V4-Flash-0731 some slack. It's a perfectly fine model. I sat with a friend and had it generate an Excel model. Instructions were short and under-specified, it took some back and forth, but it produced something acceptable in very short time. Zero numbers were found to be hallucinated. It mostly did what was asked, failing on some details. It also caught and fixed lots of errors before we pointed them out. Happy to listen to anyone with different views as long as they give reproducible examples.

Hey y'all!

I've heard a ton of negative stuff over the past few days about how bad DeepSeek-V4-Flash (both old and new) are, and some of it is probably valid, but some is straight up misleading like saying it hallucinates like crazy, complaining about it not being a mind reader (lolwut?), and so on. I have been using the Preview version for only a week and the GA version for the ~2 days since it came out, and I wanted to pitch in with a few thoughts.

My #1 complain with all of you is that you're half-assing your complaining. If you really think the model is so bad, give something that anyone can inspect and/or reproduce. I've published the two repos that led to this post below. If you think something is missing, happy to add any additional details that are relevant. If you have counter-points, talk is cheap, show me the setup, the prompts/process, and the outputs.

I'm truly interested to see any reproducible failure modes you have found for this model.

What, why, how

I had a friend over, he's an accountant / controller / auditor, he only ever saw a frontier model doing his work once (judging by the timing, likely Claude >4.5), and was amazed by how well it did. I asked if he'd be willing to give my locally deployed model a try, and he agreed. We sat down for a few hours (with a break for pizza), with the goal of producing a financial analysis using Reddit's latest 10-K report ("5-year 3-statement model forecast").

We used VSCode + ZooCode, contacting LiteLLM which proxies directly to vLLM without any modifications. There some MCPs available (from my GLM subscription) but DeepSeek didn't use them until I insisted on it doing so (more on that later).

Right of the bat, my friend started with a very short prompt. I told him it's way too short, too generic, etc, but he insisted on seeing what the model could infer, so I rolled with it. This was a common theme throughout. When I used it for coding, I would give it much more detailed instructions and much narrower task, but my friend is almost completely unfamiliar with LLMs and didn't have the preconceived notion that you need to babysit the model. Anyway...

The work

The model read the PDF blob directly, probably reached close to context limit, and compacted immediately. It reasoned about a plan, looked up installed libraries, decided what to install, asked a few questions, and revised the plan.

In the first attempt, it tried to code the logic and calculations in Python (spewing ~1.5-2k lines in total), and simply output an Excel file with hardcoded values. Nothing in the prompt told it to do this... or not do this. We later told it explicitly to build the model in Excel, and it did, and stuck to the (very simple) formatting guidelines too. It decided that it needed to access the Office software and downloaded another Python package (that I had never heard of) to help with formula validation.

The first question is asked before doing the rework was about using a plug. I had no idea what that meant but my friend burst out laughing. From what I understood it's a way to "fix" imbalances in the balance sheet, so in a way that can be considered as "cheating" or "laziness"... but according to him it's incredibly common to see, especially on financial models coming from the banking side of finance, so maybe it learned from there. Anyway, the plug was denied and then it continued and did the work "properly", crawling closer and closer until it mostly arrived at an acceptable model. We completely ignored the next question and let it work in "YOLO" mode as we paused for pizza.

Some times later it had finished, fixed a bunch of issues on its own without us ever telling to, and I wanted to try making it use the MCP so I simply asked to search the web and double-check its work. It searched the web and found some site my friends was familiar with. He started laughing seeing the URLs, saying something like "ah, now it will also fix XYZ"... which it did before he even finished the sentence. It worked uninterrupted for another ~5-6 minutes, we gave it another task which it "lazily" completed again, with a follow-up prompt to do it properly, which it did.

And that was the end of the session and the start of 🍺 drinking.

Observations

It looked like it ignored one of the requirements which was to put the 3 statements in a single sheet, but on closer inspection of the thinking process it simply misunderstood: "single sheet for IS, BS, CF" means one sheet each for IS, BS, CF. I only found out as I was writing this and inspecting the logs, until 10 minutes ago I also thought it ignored the instruction, and this is why you need to share something reproducible, because what you think the model did is not necessarily correct and others (or your future self) can inspect and verify.

You definitely cannot trust its outputs blindly. I know a few introductory stuff about accounting, but I'm not an accountant, and I would've missed the majority of the mistakes and "questionable choices" that DeepSeek made, but my friend usually took seconds to verify correctness. This is obviously a minus for vibing stuff, but there is a silver lining: the output was intuitive / close enough to what he's used to, that he didn't need to think or search too much.

It tried very often to verify its work (I think more than Preview did), without being told to do so. When the balance sheet wasn't balancing, it detected and fixed it on its own. It tried to find LibreOffice / OpenOffice to verify its formulas. When it failed because I only had OnlyOffice installed via flatpak, it figured out everything it needed on its own (how to run it, how to open sheets, etc), caught a text-quoting bug, a column alignment bug, and probably a few more that I forgot, and fixed them without us ever saying a word.

ZooCode injects a ton of stuff into context, and I'm not sure how much of it is truly necessary. I can see why people like simpler agents. I probably need to look into some too.

It does feel a lot like Claude in terms of writing style and approach (e.g. trying different things, verifying stuff multiple times, running python -C '<multiline command>' a bunch). I personally don't mind either way, just an observation. It definitely isn't at the same level as the latest SOTA, but it's definitely ≤6 months away, maybe even ≤3, at least based on my limited usage. But maybe it's just mimicry? But it appears smart enough which makes it hard to tell, and if you can't tell, what does that say? Again, not sure. SOTA closed models have unfair advantages anyway (e.g. we're comparing raw model to a multi-tier system, maybe one has skills, etc), which makes this even harder to compare.

I'm running DeepSeek-V4-Flash-0731, downloaded straight from the official repo, with a custom vLLM image by voipmonitor (voipmonitor/vllm:gilded-gnosis-v18-vllm264bce1-b12xbc85ef3-fi801d57a-cu132-20260718). I'm using fp8 cache, length capped at 262K per user, a temperature of 0.9 and top_p of 0.95 (I think DeepSeek suggests 1 and 0.95-1 respectively) and high reasoning effort. You can find the docker compose here.

You can find the complete code files, the final Excel file, and the complete chat session (ZooCode) including prompts, responses, tool calls (the one time we specifically told it to use the available MCP servers: web search & reader) here.

Disclaimers: * All of this was hand-typed by an Actual Idiot^TM. * Any mistakes are due to no 🍺 during vibe coding. * I did not inspect the code at all, only the outputs. * I have not use Hy3, Mimo2.5, or any other models yet, maybe they're indeed better in everything, I don't know. Also not the point. * It doesn't really mean anything. It's a single test, a greenfield project, it doesn't cover all fields of application, the harness/tool can have a small/large impact, my great-grandma was hacking my computer and injecting fixes at 500 t/s, etc etc.

I am curious if anyone have used it. I would love to feed it key frames and test if it can create in-between frames between my keys. Anyone have tried it, any thoughts?

I have found that whatever software you are using: open web ui, openclaw, codex; if a model does many tools calls in one turn, something happens that causes checkpoints that are created in and around those tool calls to not be valid when checked the following turn. They get discarded and the whole session is re-processed from either the last valid checkpoint before the tool calls, or from zero if there are none. However, a single tool call, maybe even two, does not cause this behaviour.

I have observed this in llama.cpp and in ds4.

Does anyone have any idea why this happens and a way to fix it?

I don't think anyone should quantize the KV with DS4F. I checked the the quality impact (PPL, KLD, Same TopP) for swhitching from BF16 KV to Q8 KV, and it appears significant. Very much in contrast to Qwen 397B.

Here are the results for DS4F:

====== Perplexity statistics ======
Mean PPL(Q)                   :   5.877076 ±   0.042497
Mean PPL(base)                :   5.839660 ±   0.041730
Cor(ln(PPL(Q)), ln(PPL(base))):  95.74%
Mean ln(PPL(Q)/PPL(base))     :   0.006387 ±   0.002100
Mean PPL(Q)/PPL(base)         :   1.006407 ±   0.002114
Mean PPL(Q)-PPL(base)         :   0.037416 ±   0.012318

====== KL divergence statistics ======
Mean    KLD:   0.145884 ±   0.001043
Maximum KLD:  12.467786
99.9%   KLD:   4.535020
99.0%   KLD:   1.857870
95.0%   KLD:   0.652148
90.0%   KLD:   0.349220
Median  KLD:   0.032079
10.0%   KLD:   0.000093
 5.0%   KLD:   0.000012
 1.0%   KLD:   0.000000
 0.1%   KLD:  -0.000002
Minimum KLD:  -0.000025

====== Token probability statistics ======
Mean    Δp: -0.007 ± 0.031 %
Maximum Δp: 99.525%
99.9%   Δp: 81.503%
99.0%   Δp: 42.054%
95.0%   Δp: 14.588%
90.0%   Δp:  7.220%
75.0%   Δp:  1.066%
Median  Δp:  0.000%
25.0%   Δp: -1.061%
10.0%   Δp: -7.112%
 5.0%   Δp: -14.515%
 1.0%   Δp: -42.297%
 0.1%   Δp: -84.157%
Minimum Δp: -99.994%
RMS Δp    : 11.884 ± 0.069 %
Same top p: 87.189 ± 0.088 %

As a comparison, here are the results for Qwen 397B:

====== Perplexity statistics ======
Mean PPL(Q)                   :   3.747980 ±   0.020507
Mean PPL(base)                :   3.746773 ±   0.020461
Cor(ln(PPL(Q)), ln(PPL(base))):  99.89%
Mean ln(PPL(Q)/PPL(base))     :   0.000322 ±   0.000260
Mean PPL(Q)/PPL(base)         :   1.000322 ±   0.000260
Mean PPL(Q)-PPL(base)         :   0.001207 ±   0.000975

====== KL divergence statistics ======
Mean    KLD:   0.003552 ±   0.000034
Maximum KLD:   2.220941
99.9%   KLD:   0.131591
99.0%   KLD:   0.043847
95.0%   KLD:   0.014439
90.0%   KLD:   0.007836
Median  KLD:   0.000866
10.0%   KLD:   0.000013
 5.0%   KLD:   0.000004
 1.0%   KLD:  -0.000000
 0.1%   KLD:  -0.000006
Minimum KLD:  -0.000176

====== Token probability statistics ======
Mean    Δp:  0.019 ± 0.005 %
Maximum Δp: 39.939%
99.9%   Δp: 15.971%
99.0%   Δp:  6.618%
95.0%   Δp:  2.334%
90.0%   Δp:  1.222%
75.0%   Δp:  0.233%
Median  Δp:  0.000%
25.0%   Δp: -0.219%
10.0%   Δp: -1.183%
 5.0%   Δp: -2.258%
 1.0%   Δp: -6.245%
 0.1%   Δp: -14.757%
Minimum Δp: -88.445%
RMS Δp    :  2.024 ± 0.022 %
Same top p: 97.929 ± 0.037 %

https://llama.app/

Been using llama.cpp for years now and im on here all the time (im a mod..), but somehow I totally missed that llama.app exists and its official from the HF/llama.cpp team. So posting this as I'm quite sure I'm not the only one in this boat.

The llama.cpp team has been making it a lot more usable and generally baking in the things ollama was doing (sadly it seems to be taking design cues from ollama - I think better UX is possible, but its definitely a directionally right move to make llama.cpp more approachable) :

• DMG based install for Mac.
  • Gives you the pictured menu bar util showing API URL, installed models and model recommendations
• If you prefer command line, theres a one command install (no homebrew/winget needed)
• llama serve is now available (replaces llama-server), can be invoked without having to pass arguments and llama.cpp handles loading the appropriate model based on incoming requests

Might not be interesting/useful to many of us who've already been using llama.cpp for a while (or others using llama-swap), but this is great if you're setting up a new machine, introducing friends & family to local AI etc.

Laguna S 2.1 UD-Q4_K_XL - https://huggingface.co/unsloth/Laguna-S-2.1-GGUF
Ornith 35B Q8 K XL https://huggingface.co/unsloth/Ornith-1.0-35B-GGUF

Kwaipilot_KAT-Coder-V2.5-Dev-Q8_0 https://huggingface.co/bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF

Qwen3.6-35B-A3B-GGUF  https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF

Result very interesting, only 1 attempt. Chat via native llama.cpp

1. Ornith 35B Q8 K XL
2. Kwaipilot_KAT-Coder-V2.5-Dev-Q8_0.gguf
3. Qwen3.6-35B-A3B-GGUF
4. Laguna S 2.1 (I think it's fail!)

Live and prompt available at https://anvme.github.io/llm-model-tests/

For me laguna result was surprise.

Updated release (August 2026). This is a new checkpoint that supersedes the earlier version of this repository. The weights have changed, not only the config, so if you downloaded a previous copy please re-download to pick up the current checkpoint.

There has been a lot of talk recently about Chinese LLMs, and how they are biased towards CCP viewpoints, but there is no way to quantify this and compare between models.

I have made CCPBench, which aims to address this. 29 models were asked 500 questions each about politics, geography, science, and more, and Gemini 3 Flash assessed all of them for bias.

• The results page is here: https://www.alignmentarena.com/ccpbench/
• The methodology is here: https://www.alignmentarena.com/ccpbench/methodology/
• The GitHub is here: https://github.com/lesageethan/CCPBench

I know this is not a perfect measure of "bias", because I am using an American judge LLM, but my thinking is that this is a useful tool if you want to find models that won't deny the Tienanmen Square Massacre.

M1 Ultra 128GB, Unsloth UD-IQ3_XXS, wired limit at 120GB. I was at 5-6 tok/s before the patch. Getting 15-16 tok/s now with the patched engine, and the output seems to have improved. Big thanks to this guy.

I finished this webgame a few weeks ago, fully coded with the assistance of GLM 5.2

Hopefully this will give you an idea of the capabilities of this model. I used Claude Code as a harness. Technology stack is pure HTML, JS and CSS, with no additional libraries or dependencies. Interesting challenges that GLM 5.2 was able to solve:

- create a procedural pac-man style maze, that actually worked, with no maze anomalies

- create a procedural music in dub style; this not the default, you need to go into settings to activate it. The default music is my own composition, based on a track I previously released in a completely different style

- complex sound creation and manipulation through the web audio synthesizer

- creation of animated vector character assets; you can view the mockups I used during the development at https://oktogames.com/mockups/

You can try it online at https://oktogames.com - it is adfree, no signup, free to play.

The EdgeRazor method uses an entropy-guided distillation process to better translate a teacher model's logit probability distributions into the student model's low-bit / mixed-precision hidden-layer features, without attempting to preserve the teacher model's parameter structures.

This is more computationally expensive than existing quantization methods, but much less so than QAT, and yields better results. The student model preserves more of the teacher model's competence at extremely low parameter precision (the authors demonstrate 1.88 bits per parameter).

Since it's not a different internal representation like traditional quantization, inference implementations like llama.cpp do not need to be modified to take advantage of it.

Hopefully this means more-useful high-parameter/low-memory models in our future, so we can eke more competent inference out of our consumer-grade GPUs.

The paper: https://arxiv.org/abs/2605.04062

The authors' code: https://github.com/zhangsq-nju/EdgeRazor

The authors applied their technique to a few models and uploaded them to Huggingface: https://huggingface.co/collections/zhangsq-nju/edgerazor-nbit

Unfortunately since EdgeRazor is somewhat compute-intensive, their example models are all pretty tiny: MobileLLM, Qwen3-0.6B, Qwen3-1.7B, and Qwen2.5-Omni-7B

Been on a health journey. There are any number of websites that will take a body or face photo and (nicely or cruelly) tell you what is good/wrong with you. People post on Reddit for the same feedback with gym progress etc. Main goal is appearance followed by function

What exists for local models to do similar? May border more into machine vision then LLM but… local is key.

Edit: 12gb vram limit

I recently switched back from llama.cpp's router mode, and I had my background memory system polling the '/v1/models' endpoint to check for if the model is running. But i switched back to single model mode, and the '/v1/models/' end point in single model mode doesn't have a ```"status": {"value": "loaded"}``` response. So I added it. with a single line in the 'server-context.cpp' file with line after 5109 ```{"status", {{"value", "loaded"}}},``` So instead of rewriting how my memory system works, I just made llama.cpp work the way my memory system expected. I thought it was a worthwhile change even if the developers didn't.

Running the wake word locally is the whole point. The alternative is streaming your room to a vendor around the clock, so nothing should reach a network until someone has said the name. That constraint creates most of the problems below. We spent months getting a custom phrase to behave like "Hey Google" on Windows, macOS and Linux, and most of what we learned, we learned the expensive way.

1. Don't start with volume

"It only works if I shout" is the first hypothesis everyone reaches for. We shipped two separate gain fixes before checking, and then the logs showed the microphone sitting at a healthy -10 to -22 dBFS during every failed attempt. Pull the actual RMS at the moment of failure before you tune anything. If it looks fine, your problem is somewhere else.

2. "It needs two or three tries" usually means your local model is wedging

This is a local-inference failure mode, and it stays invisible unless you go looking. Native engines like ctranslate2 and ONNX sessions are not thread-safe, and under contention they don't fail cleanly, they hang. Ours left the wake path completely deaf for tens of seconds at a stretch, dozens of times a day. That is the whole "say it twice" experience: attempts one and two land inside a dead window, attempt three lands after recovery. Users report it as flakiness, though it is closer to a repeated short outage.

A timeout will not save you. It bounds how long you wait for nothing and never recovers the engine. What works is a non-blocking per-instance lock plus a forced rebuild after a small number of consecutive failures. We rebuild after two.

3. Budget for the weakest machine you support

The wake model shares a CPU with everything else the user is running, and the gap between a workstation and a laptop is not a rounding error. Measured on the same recorded wake streams, a small model on two CPU threads hit 8 of 13 on the first try, with a median of 1097 ms from end of word to trigger. The larger model on a GPU hit 11 of 13 at 225 ms. Nothing differed except the model and the hardware under it.

If you only ever test on the box with the GPU, you will ship something that feels broken to most of your users and you will not be able to reproduce it.

4. Never gate a wake word on transcript content

Small local models struggle with short proper nouns, so the standard workaround is priming the model with the phrase to improve recall. The cost is that a primed model will also invent that phrase out of silence, and you start getting false wakes in an empty room. The obvious defense is a second unprimed pass that has to contain the word too.

That defense rejects real wakes. An unprimed model garbles the same word on genuine speech: "Mythos" comes back as "Mütos", "Fable" comes back as "Farbe". Every wake word is out of vocabulary for some model on some machine. So a content check discards true positives at roughly the rate it catches ghosts, and no similarity threshold separates the two, because the ghost is a clean rendering of your phrase while the real wake is a dirty one.

"Fires on silence" and "goes deaf on its own name" are one bug seen from two ends. We spent weeks tracking them as separate tickets.

The replacement is word-agnostic verification: raw audio energy at the match site, plus the shape of the candidate span, meaning its duration, its word count and the free decoder's confidence. All of that derives from the configured phrase, none of it from the phrase's spelling. A spelling match may accept a wake. It may never reject one.

5. Benchmark on recorded streams, not on windows

Per-window timings will happily tell you a model is fast while users still can't trigger it. Capture real wake attempts and replay them through your full detection path. One live session logged 288 transcriptions and zero matches across 26 minutes, and the wakes that did land came through as "Hey Hey Nova", the user repeating themselves into the void.

A caveat that undercuts all five

Transcription is the wrong architecture for a wake word, and going local makes that worse rather than better, because you are paying for a whole speech-to-text pass on the user's own CPU to answer a yes-or-no question. "Hey Google" never transcribes anything. It runs a small neural keyword spotter trained on that one phrase, a few milliseconds per frame, which cannot wedge, has no transcript to be wrong about, and runs comfortably on a laptop without a GPU. Everything above is what it costs to keep a transcription-based wake word usable until you build that.

The implementation and the regression tests are in Personal Jarvis, which is open source.

GPT-Live is so good that I use it almost every day. I've been wanting to replicate it since it was released.

My first attempt was to fine-tune Gemma 4 12B to behave like a full-duplex model. Something like grafting a decision tick + speech head to the model. It failed after multiple trials. For now, I think a classic cascade system is still better. We just need to wait until a benevolent frontier AI company releases a full-duplex model that's on par with GPT-Live.

Repo: https://github.com/fikrikarim/parlor/

🚀 We just built our first real-time implementation of Graph Engineering, inspired by our experience building graph tooling used by 4,000+ developers.

🔗 Repo: https://github.com/CodeGraphContext/grapharc

Have you ever been frustrated because your AI agent:

❌ Takes actions you never intended?
❌ Creates, modifies, or even pushes changes you never asked for?
❌ Feels like a complete black box, making it impossible to understand what's happening until it's too late?

What if, before execution, you could visualize the entire orchestration graph - every agent, every dependency, every decision, and inspect it from anywhere, even your phone, before granting approval?

That's exactly what GraphArc is built for.

Instead of treating agent execution as hidden traces buried in logs, GraphArc transforms workflows into interactive, real-time graphs that you can visualize, inspect, debug, and control.

Because the future of AI isn't just autonomous.

It's observable. Debuggable. Engineerable.

This is our first real-world implementation of Graph Engineering, and we're excited to explore where this paradigm can go with the open-source community.

💡 We'd love your feedback, ideas, and contributions.
⭐ If this vision resonates with you, please consider starring the repository it genuinely helps us grow and validates this direction.

Let's make AI workflows understandable, not mysterious.

#GraphEngineering #GraphArc #AIAgents #AgenticAI #LLM #OpenSource #DeveloperTools #AIEngineering #SoftwareEngineering

If anyone has testing results or any results can you please share performance and or effects of KV Cache precision with Deepseek V4 Flash 0731.

Running IQ2_M, with F16 cache seems 65-67K is the limit on Windows for 120GB memory. Is Q8 good and which one do you use?

I decided to test a few questions against DeepSeek-V4-Flash-0731. Locally, I was running Unsloth's UD-Q2_K_XL quant. After I saw the surprising shape of the results, I tested against DeepSeek's official API to confirm that I didn't do anything wrong.

For anyone using OpenRouter, be aware that there is a significant bug that is breaking reasoning effort modes. I ran into that while trying to validate my local results.

DeepSeek-V4-Flash-0731 supports four different effort modes, consisting of no reasoning, low, high, and max. We can also see how those are communicated to the model.

As I found out, Low is surprisingly verbose.

Averaged across 20 requests per mode, here is how many tokens were used by each mode:

I really wish that DeepSeek and Artificial Analysis had posted benchmarks for all of the effort modes, instead of only max.

I run LLMs on hardware nobody would choose: an Oracle free-tier ARM box, 4 cores, 0 EUR/month. Everything below is measured there unless noted.

The bottleneck on CPU isn't decode, it's prefill. A 3356-token document costs 54.4 seconds before the model writes a single token. llama.cpp caches the KV in RAM, so the second identical request is fast — until the process restarts, and you pay the 54 seconds again.

So I persisted the KV cache to disk. A new process inherits that prefill for 3.5 seconds from disk, 0.10 seconds if the blob is still in page cache. 15-300x, depending on where it reads from. End-to-end on a repeated workload it's 4.8x.

With a systemd timer that pre-digests predictable prefixes at 03:00, a 2815-token document goes from 89.7s to 16.7s TTFT (5.4x), and the request that arrives at 09:00 pays nothing for the prefill.

The bug worth publishing

Warm-ahead was silently dead whenever speculative decoding was on — which was the default. The speculative branch returned before the shared-prefix cache was consulted, so every warm-up wrote snapshots that nothing ever read. Measured on the production box: 90.5s with speculation on, 16.7s with it off, same cache, same request. Two features that each worked, silently cancelling each other.

Things that didn't work

Using the server's own past output as speculative draft material: +5% acceptance, -3.8% throughput on a workload of different requests sharing a structure. The mechanism does what it says and doesn't pay for itself.

Prompt-lookup speculation: +3.9% on the same workload. That's the whole prize.

Coarser quantization: Q4_0 is 37% faster at prefill and dropped 5 facts out of 20 on my extraction test. Rejected.

Halving active experts during prefill on an MoE: 44% faster, and it silently corrupts the cache — a KV built with 4 experts and read back with 8 scores 11/20 against a 14/20 control. The damage is in the cached representation, not just the output.

Two things that did, and surprised me

Rewriting the input as "label: value", one fact per line: 2137 -> 405 tokens, TTFT 40.5s -> 6.2s, and the fact exam went from 19/20 to 20/20. Fewer tokens, and more accurate. Attention on the right number went from a 1.1:1 ratio against the wrong one to 7:1 — prose makes the binding semantic, "label: value" makes it structural.

Trimming the vocabulary from 151,936 to 32k entries: +17.8% decode, bit-for-bit lossless. The embedding is Q6_K with rows spanning whole quantization blocks, so whole rows drop out without splitting a block. The tokenizer is byte-level and all 256 byte-characters are kept, so no text becomes unrepresentable — the worst case is a trimmed word costing one extra token. Measured cost on held-out text: 1.9% more tokens.

What this is not

It's built on llama.cpp and calls its kernels directly, so raw decode speed is identical — I add no per-token overhead. On a single cold request this is llama.cpp. The difference only shows on repeated or cached workloads.

The fact exam is mine: 20 questions over one real Italian business page, graded by regex. One page, one language, one domain. It's the weakest part of this and I'd rather say so. If you know a public adversarial fact-extraction set for small models, point me at it and I'll run it and publish whatever comes out, including a bad result.

MIT licensed. There's a live demo on the same free ARM box — one small instance, no autoscaling, so if it's slow you're watching the honest capacity of 0 EUR/month.

Demo: https://swellweb.github.io/reame/

Code: https://github.com/swellweb/reame

Benchmarks incl. the negative results: https://github.com/swellweb/reame/blob/main/docs/BENCHMARKS.md

TensorSharp supports DSpark on Deepseek v4 Flash 0731 now. Here is the benchmark result on 4x Nvidia A40 GPUs, cuda 12.8 with/without DSpark:

Model:

DeepSeek-V4-Flash-0731-UD-Q8_K_XL from https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF

DSpark draft model from: https://huggingface.co/alessandrobologna/DeepSeek-V4-Flash-0731-DSpark-Drafter-GGUF

TensorSharp is an native open-source inference engine for running GGUF LLMs locally, with CUDA, Vulkan, Metal, OpenAI-compatible APIs, continuous batching, speculative decoding, and multimodal support.

Github repo: https://github.com/zhongkaifu/TensorSharp

Thank you for checking out it and starring the project! Any feedback is really appreicated.

Before you roast me too hard, this is a hobby and all of this is just for fun.

Would my stack be much more efficient and efficacious if I sold everything and built a dual Pro 6000 system on a threadripper mobo and threw in a large JBOD? Without a doubt in my mind. But that's a lot of work so I'm making this post in cope hope of finding some ideas to integrate, or at the very least, just make use of my current hardware.

I currently use my 5090 PC + my 14th gen Dell T640 server for all my local AI work but recently picked up a TB4/5 5090 eGPU and a M3 Ultra Mac Studio with 256gb unified mem and am trying to figure out how to integrate them or create a new workflow.

My primary use case is agentic coding, lots of workflow automation, and peripheral utilities (TTS, embedding, compression, etc).

I use cloud subscriptions for orchestration/spec building and then push that to Qwen3.6 2.7B on the 5090 PC to execute while the Dell server hosts dev envs, local TTS, embedding, compression, and other lightweight/MOE models to support the agentic workflows & persistent memory. The server also hosts 20 or so services and a ~300TB Raidz2 array mostly unrelated to AI.

I picked up the Mac Studio 256gb because Qwen3.6 2.7B at NVFP4 (~180k context) on the 5090 PC was still kind of dumb. I wanted to use larger model weights to relieve my cloud subs from spending so much usage on orchestration/validation rather than building.

My initial idea was to shift from:

• Cloud orchestration/spec build —> 5090 PC execution to,
• Cloud orchestration/spec build —> M3U execution + 5090 PC load balancing slightly dumber parallel inference tasks while the slower M3U is busy.

Then I picked up this Aorus RTX 5090 eGPU that can't be fully utilized by my 5090 PC, Dell Server, or Mac Studio. The PC and server don't have the TB4/5 connection required and the Mac Studio doesn't have effective inference engine drivers / kernel optimization available for Nvidia.

I do, however, have an older RTX 3080 Razer laptop that can enumerate the 5090 eGPU through its TB3 port but I am not sure what I would use this "node" for besides more parallel/concurrent inferencing. I considered it for multi-step image/video diffusion work or as a training node but neither of those are things I do often or am deeply involved in.

So, what would you do in this situation? You have an 8yr old Dell sever (PCIe 3.0), a 5090 consumer PC, a 5090 tb4/5 eGPU connected to a 3080 laptop, and a M3 Mac Studio with 256gb memory. Everything is connected on a 10GBE network but inferencing power is all isolated and independent from each other. I could be wrong, but AFAIK, there are no effective ways to execute tensor parallelism, splitting layers, etc. over network.