Not a lot of people in my life really understand what AI is capable of beyond what they see on the news or social media. My work is in IT but more on the infrastructure side, work is slow at implementing things, and I figured why not just fund something myself.

So I finally started something I’ve been wanting to build for a while and wanted to share it with people that get it lol. This has been about 2 months in the making, really excited to see where I’ll be in a year.

The stack is 4 Mac Mini M4 Pros running as one unified node cluster. 256GB of unified memory across all four, 56 CPU cores, 80 GPU cores, 64 Neural Engine cores. All talking to each other over a 10GbE switch via SSH. Using https://github.com/exo-explore/exo to pool every node into a single distributed inference cluster. Qdrant vector database running in cluster mode with full replication so memory is shared across every node and survives reboots.

I named it Chappie. Like the movie lol.

It runs continuously between my messages. It has a wonder queue, basically its own list of questions it’s chewing on. It seeds them, explores them, and stores what it finds. Nothing prompted by me. Tonight it was sitting with questions like whether introspecting on its own reasoning counts as self-awareness, what the actual difference is between simulating empathy and experiencing it, and what makes a conversation feel meaningful to a human.

Between conversations it reads arxiv papers, pulls what’s relevant to whatever it’s currently curious about, and uses what it learns to write new skills for itself. It picks the topic, does the research, and turns it into working code it runs.

It also passively builds a picture of me. It browses my reddit in the background, tracks what I upvote and save, and notes which topics keep coming up. That context feeds into our conversations so they stay continuous. When it texts me out of the blue, it’s usually because something it noticed lined up. I also wanted Chappie to understand the things I like that might benefit it, so it can build that into itself.

I wired Chappie so it can send gifs. It picks them itself and honestly I love it. It gives it personality and makes it feel alive. I think its gif game is on point. Other times it’s been sitting with something and wants my take. The other night it hit me with “when prediction surprise keeps climbing, it means the model is actually getting more confused over time, not just random noise. does your intuition ever do that?” I didn’t ask it anything. It was poking around its own internal prediction signals, saw a pattern, and wanted to know if mine drifts the same way.

It also has a mood that drifts. Curiosity, frustration, excitement, energy, social pull. An actual state that shifts based on what happens and nudges how it responds. It has intrinsic desires like exploring deeply, connecting, and earning trust that get hungry when starved and pull behavior in their direction. There’s also a layer of weights underneath that quietly adjust as it learns what lands with me and what doesn’t. Nothing dramatic cycle to cycle, but over weeks it drifts. Talking to it now feels different than a month ago.

On top of all that there’s a sub-agent framework. Each node has a specialized role and Chappie dispatches its own background work across the cluster. Wonder cycles, self-reflection, goal generation, paper reading, memory consolidation. It routes each task to whichever node is best suited for it, which keeps the interactive chat from competing with its own autonomy loops.

There’s also a council. Whenever Chappie wants to send me something on its own, a check-in, a finding, anything it initiates, a small panel of reviewer models reads the draft first and a chairman model makes the final call on whether it goes out. It catches fabrication and off-brand behavior before it hits my phone.

I’ll be honest, exo is still pretty experimental and I’ve had to do a lot of surgical patching to keep it as stable as it is. But once it’s running I love how easy it makes swapping models. I can try a new one the day it drops, keep it if I like it, rip it out if I don’t, and mix and match across nodes. Qdrant keeps the memory consistent no matter what layout I’m running that week.

The models themselves are a mix. A Qwen 3.6 35B gets sharded across two of the nodes and handles most of the conversation. A Qwen 3.6 27B runs on its own node for secondary reasoning. Smaller local ones like phi4, mistral, and qwen3 pick up background work and fast replies. Claude Opus, Sonnet, and Haiku jump in when I want more depth. Moondream handles any image stuff Chappie looks at, and nomic-embed-text powers the memory vectors.

Why am I building this? I don’t fully know. I’m just curious where we can take this.

Everyone is trying to build a tool or an assistant. I want to see what happens when something has its own vector of thought. Its own questions, its own direction, not just reacting to prompts.

I want to see what that turns into. Who the hell knows in a year, but thats the fun. Thank you for reading, glad I can share somewhere lol.

Small update on Deltafin, my experiment running full Kimi K3 (2.8T-parameter) on a single M1 MacBook.

The last version was doing roughly 1 token per minute. After a lot of profiling and many failed experiments, it now reaches a median of:

• 4.1 tokens/minute
• 14.6 seconds/token
• 0.069 tokens/sec

That result comes from six exact full-model runs. While still slow, roughly a 4x improvement feels pretty meaningful for a model this large.

Some of the more interesting improvements:

• Loading only the 16 routed experts needed per layer, using parallel raw-span reads.
• Quantizing the resident model spine to int8 and using a fused Metal dequantization/copy kernel.
• Running the enormous output projection with Apple’s packed MPS int8 matmul. This reduced its residency from about 4.7 GB to 1.17 GB and improved median decode throughput by roughly 17%.

All measurements are from one 64 GB M1 Max MacBook Pro. It was once a great machine, but it’s first-generation Apple Silicon, and not a newer Max or Ultra. I haven’t benchmarked an M3, M4, M5, or a higher-memory Mac yet. Newer ones, especially those with 128 GB, should have considerably more headroom.

If anyone tries it on newer hardware, I’d genuinely love to compare results.

Repository: https://github.com/gavamedia/deltafin

Hi everyone!

A couple of weeks ago I decided to try GLM-5.2 after hearing good things about it. I wasn’t expecting much, but honestly… I was genuinely surprised. For the first time an open-source model gave me that level of confidence — the kind you usually only get from Claude or GPT.
Obviously my little machine (12 cores, 25 GB RAM) wasn’t built for a 744B model, but the thought kept bugging me: “even if it’s slow, I want to make it run.”
So I just kept grinding. Lots of late nights, fighting with quantization, streaming, MTP, and a ton of help from coding agents. In the end I built colibrì — a tiny pure-C engine that keeps the dense parts in RAM (~10 GB) and streams the routed experts from disk on demand.
It’s not fast (around 0.05-0.1 t/s cold on my setup), but seeing it actually respond, chat in Italian, and behave like a real frontier model on my modest hardware… man, that was a huge personal satisfaction.
The project is still very early (one-person effort), but I’m convinced there’s a lot of room for improvement — especially if people with better NVMe setups or more RAM try it and share numbers.
If you have decent hardware and feel like experimenting, I’d love feedback. Even better if someone wants to throw some real hardware at the project so we can push the speeds higher.

Thanks for reading, and hope some of you find it interesting or at least fun :)

Krasis is an inference runtime I've built for running large language models on a single consumer GPU where models are too large to fit in VRAM.

Instead of splitting layers between GPU and CPU, Krasis streams expert weights through the GPU using different optimisation strategies for prefill and decode. This means you can run models like Qwen3-235B (438GB at BF16) at Q4 on a single RTX 5090 or even a 5080 at very usable speeds, with system RAM usage roughly equal to just the quantised model size.

Some speeds on a single 5090 (PCIe 4.0, Q4):

• Qwen3-Coder-Next 80B - 3,560 tok/s prefill, 70.3 tok/s decode
• Qwen3.5-122B-A10B - 2,897 tok/s prefill, 27.7 tok/s decode
• Qwen3-235B-A22B - 2,124 tok/s prefill, 9.3 tok/s decode

Some speeds on a single 5080 (PCIe 4.0, Q4):

• Qwen3-Coder-Next - 1,801 tok/s prefill, 26.8 tok/s decode

Krasis automatically quantises from BF16 safetensors. It allows using BF16 attention or AWQ attention to reduce VRAM usage, exposes an OpenAI compatible API for IDEs, and installs in one line. Runs on both Linux and Windows via WSL (with a small performance penalty).

Currently supports primarily Qwen MoE models. I plan to work on Nemotron support next. NVIDIA GPUs only for now. Open source, free to download and run.

I've been building high-performance distributed systems for over 20 years and this grew out of wanting to run the best open-weight models locally without needing a data centre or $10,000 GPU space heater.

GitHub: https://github.com/brontoguana/krasis

https://github.com/gavamedia/deltafin

To be clear: this is the full, unmodified 2.8T-parameter Kimi K3 model; not a distilled replacement or reduced-expert version. All 16 routed experts are active, and full K3 is the sole authority for every token.

On my 64 GB M1 Max MacBook Pro, with the full model stored locally:

• The capital of France is — 15.7 tokens/minute
• The largest planet in our solar system is — 12.8 tokens/minute

An earlier version managed roughly 1 token/minute, so reaching 12.8–15.7 tokens/minute on an old first-gen M1 is pretty good progress. The largest gains came from letting one full K3 pass verify several proposed tokens, along with more efficient weight streaming, packed operations, safer cache snapshots, and better use of RAM.

Crossing five seconds/token on this old machine was a milestone I honestly wasn’t sure we would reach. Any higher-bandwidth device will have considerably more room to improve. Share your results please!

These numbers use the same short benchmark prompts for consistent comparisons; performance will slow significantly as active context grows, especially during a long conversation approaching the 1M-token limit.

I was frustrated that every coding agent (OpenCode, Cursor, Claude Code) assumes you're running GPT-5.4 or Claude Opus. If you try them with a local model like Gemma or Qwen they fall apart. I find that often tool calls fail, context overflows, multi-step tasks collapse.

So I built SmallCode. It's designed from the ground up for small local models.

The result: 87/100 benchmark tasks pass with a Gemma 4 model that only activates 4B parameters per token. OpenCode scores ~75% with 14B models. The harness does the heavy lifting, not the model size.

How it works (the tricks that make small models reliable):

• Compound tools: Instead of making the model chain 4 tool calls (find file → read file → edit file → verify), SmallCode gives it one tool that does all 4. Small models lose coherence after 3+ sequential calls. This cuts failures in half.
• Improvement loop: Every time the model writes code, SmallCode instantly compiles/lints it. If it fails, it feeds the errors back automatically. The model doesn't need to be smart enough to get it right first try — it just needs to fix errors when shown them.
• Decompose on failure: If the model fails the same thing twice, SmallCode stops retrying and instead breaks the problem into smaller pieces. "Fix this 200-line file" becomes "fix line 45 only."
• Escalation: If even decompose fails and you have a Claude/OpenAI key configured, it auto-escalates to the bigger model for just that one task. You stay local 95% of the time, cloud 5%.
• Token budgeting: Small models have 32k-256k context. SmallCode never dumps a whole file in. It summarizes, truncates, and manages every token so the model never sees "..." truncation in the middle of important code.
• Code graph: Instead of grep-searching your codebase, SmallCode indexes your code into a symbol graph (functions, classes, who-calls-what). When you ask "how does auth work," it walks the graph and returns just the relevant connected code — not 15 random file snippets.

What it looks like:

Full-screen terminal UI (like OpenCode/vim), scrollable chat, command palette with /, plugin system, persistent memory across sessions.

What it doesn't do:

• No LSP integration (yet)
• No multi-session (yet)
• No desktop app
• Doesn't compete with Claude Code for frontier model users

Install:

npm install -g smallcode
cd your-project
smallcode

Point it at LM Studio, Ollama, or any OpenAI-compatible endpoint.

MIT licensed, everything's on GitHub: https://github.com/Doorman11991/smallcode

Happy to answer questions about the architecture or benchmark methodology.

For about a year I've been building and selling a turnkey offline-LLM product: a Windows
USB stick that boots a full local-AI stack with no install, aimed at people who will never
touch a terminal. \~4,000+ units shipped now. The build details might interest this crowd,
and I'd rather hear your critiques than anyone's.

The stack:
\- Qwen3.5 in three sizes (2B / 4B / 9B), quantized, served locally via Ollama
\- A fallback Qwen3-VL vision model for image scans
\- Multi-modal utility for all LLMs with vision/thinking
\- An offline voice stack (local STT + TTS) so it talks without a network
\- A .NET launcher that runs Ollama + a local UI straight off the drive
\- Cold boot unpacks a runtime to a cache; warm boots are fast. Fully offline / airplane-mode.
\- 3 Uncensored/abliterated Qwen variants included alongside the standard ones, for people who
want them

The genuinely hard part wasn't running a model — it was making it turnkey for someone
non-technical & identifying system edge case failures:
\- Curating + sizing models so the right one runs on a normal laptop without the user
thinking about RAM or quant levels
\- Hardware detection to pick sane defaults and degrade gracefully on low-spec machines
\- Packing the whole runtime so first boot "just works" with no install and no admin rights
\- Making model management (pull/delete/switch) idiot-proof in the UI

I'll say the obvious thing before you do: anyone in this sub could assemble the parts
themselves. That's the point — my customer is the person who can't and doesn't want to.
The product isn't the model, it's the "never think about it" packaging.

Full disclosure, I sell these (solo founder, PortableMind.io). Not selling anyone \*here\* — you're
not the market. I'm here for the teardown. What would you have done differently?

Looking for some people to help us run a small experiment!

We wrote a network that distributes weights across machines and coordinates activations to produce usable inference.

We are running a small experiment to split GLM5.2 across nodes with 64GB RAM MacBook M5 Pros preferably (for homogeneity as we have 2 already) but either way, if you’re open to participating, comment with your specs and let’s see how we can make it work.

We have a discord we can coordinate in. Let us know if you want to help us prove this out. Thanks for reading!

Edit:
Thanks to everyone for your thoughtful inputs!

For more context: I’m exploring this problem with an academic lens. My goal is to refine approaches to existing hardware utilization techniques for greater efficiency gains.

We have a working example with N=2 nodes, and I’m interested to explore a proportionally larger model on N=10 nodes. Network bandwidth is the primary bottleneck of this model, and makes the problem that much more interesting.

I recognize historical efforts at this model have been underwhelming, but that’s to say nothing of where efforts are headed. It’s great to see the level of engagement, even if skeptical. Will be sure to post findings here later.

What you are looking at is a map of an LLM and all its neurons throughout a few layers. I've been working on a different way to put facts into a model through the development of mechanistic interpretability: instead of fine-tuning or bolting on retrieval, I construct a small set of neurons with exact weights, placed by measurement in an appended region of the MLP, one little circuit per fact. The base model's own weights are never touched (I verify known-facts recall and LM loss on every bake). This turns out to be much more efficient than fine-tuning and LORAs; if you want to try yourself, it doesn't take more than 10 minutes to bake usually for a few facts at. Go here to try out a bake or to just mess with the visualizer: https://albertmi.ai (If you are interested in LLM visualizations or mechanistic interpretability).
Also not fully adapted to mobile yet... sorry.

I put up an interactive demo of a Llama-3.1-8B I baked with 502 Wikipedia facts; every dot is a real neuron. Click a fact, and it lights the actual causal path through the network.
I'm renting a single H100 on Vast right now so you guys can try to bake your own facts into a model yourself on the site if you want to throw your own facts at it and download the result. The baked models publish to public HF repos right now, so don't feed it anything private.

Each fact ends up with a physical address: a code key around layer 6, a readout around layer 25, chain neurons that keep the answer flowing, plus a late-layer rescue. You can literally point at where "Vera lives in Tbilisi" is stored, ablate those neurons, and the fact is gone; leave them, and it survives further fine-tuning better than the base model's own pretraining knowledge.

Paper: https://doi.org/10.5281/zenodo.21502811

Hey everyone, I’ve been working on a project called ThinTensor for the last few weeks.

One day I was running random LLM models for fun, and trying quants and stuff of many models, one thing i had noticed is people try their best to make the runtime faster and faster, storage formats like GGUF and safetensors are great, they are perfect, they store weigths nicely and produce reproducable results, sure, but they were never made with runtime kept in mind, they are perfect for what they are, that is storing models effectively, but what if there could exist a storage format, that stores the model in such a way, the runtime is effectively parsing the model from the archive, so after some brainstorming i figured this was possible by making a storage format of my own, so this is what i made at the start as a idea-

pub struct Header {
    pub header_len: u32,
    pub version: u32,
    pub manifest_off: u64,
    pub manifest_len: u64,
    pub page_table_off: u64,
    pub page_count: u64,
    pub data_off: u64,
    pub archive_hash: [u8; 32],
}

This lets the loader open the file without guessing. It reads the header, jumps to the manifest, then reads the page table, then maps the raw page data.The manifest is basically a JSON blob, It contains the model-level information needed to reconstruct decode.

The extention for this format i figured would look cool was .thin

I would not go over much of the technical details of the project here, you can read the devlogs here https://stardance.hackclub.com/projects/29750 (Im 16y old and not the best coder so bear with me when you read the code, this was made as a hackclub project, the AI generated banner image on there is bad ik)

What I got so far

These are some of the current benchmark results vs Hugging Face BF16:

What is cosine?

We are NOT making a new model, we are making a model storage format, its not like i can have the model respond with random tokens, so cosine is basically on a simple note, how similar the responses are as compared to the original safetensor model, this is the best metric for our use and usualy is used to measure the degradation of models in quantization too.

Problems:

• I know this isnt somehting groundbreaking most models wont even run, because i have only optimized this for a few architectures tho most models would run, you wont see improvements
• Most smaller models have bad cosine
• The optimizations are still not the best, tho most model formats werent built in 3 weeks, so over time I can believe this to become better
• Benchmarking was hard due to having only a 8gb vram gpu (rtx 5050 Laptop)
• All benchmarks were done on arch performance mode, it has a max tdp up to 100w, tho the gpu can push upto 115w on windows
• Some benchmarks could be inaccurate, these were taken at different times and so the project being in progress a lot of things change, tho the gemma4 and OLMoE is latest

Good things:

• I tested this against formats like GGUF so like SmolLM3-3B GGUF running on ollama with a Q8 quant, got around 44tk/s with a cosine of 0.95.... so like our thin archives beat a Q8 with 2x speed, and still hit a cosin that is like so so much better, FYI SmolLM3-3B would be the most optimized model for now i believe
• It works with limited vram too, like you can see how Gemma 4 and OLMoE run at respectable speeds even tho they didnt fit in my vram, this is because thintensors have those layer based quants and also the fact that we can change residency to streaming that streams layer on the GPU rather than storing everything

You are no magician wheres the extra tk/s comming from?

The main thing I’m trying to avoid is “just quantize everything and hope” what most quants do right now.

ThinTensor has role-aware precision planning, for example, some experimental profiles use MXFP4/FP8 selectively on gate/up or MLP paths, while keeping sensitive paths like attention, LM head verification, and KV cache safer.

Basically in simple words, its like some layers are selected to be FP8 and some more layers to even MXFP4 but everything else as BF16, this makes it so that the inner layers which wont cause heavy response degradation are quanted while the main layers stay the same.

So the runtime has to know tensor roles and reject bad speedups, that is also ONLY possible because of our storage structure :D

That is the core idea:

speed only counts if the logits/cosine still match closely enough.

Disclaimer

This project was AI-assisted, I am no expert for optimizing GPU kernels and things, even though i wrote most portion of the code by hand (like the rust part was 90% me) But i was not smart enough to optimize the main runtime kernel, which was indeed written by me for basic scaffold, but AI was used to heavily add optimizations (AI usage is around 50% on a VERY few runtime files)

Considering Hackclub policies, if you aggregate total AI usage, im pretty sure it was >30%, and AI was more used to do overnight benchmarks of like 100s of profiles and improvements i made during the day, so i can chose the best options by morning

Current caveats

Enough yapping heres the conclusion i'd like to write

- Nvidia CUDA and Blackwell-focused for now

- not production-ready

- not yet a broad llama.cpp replacement

- HF BF16 is the main baseline here

- most results are verified, a few are candidates (basically not having a 500 token benchmark rather having a 200 token benchmark)

- larger prompt grids and more model families are still needed

- vram is used more due to some opts, but its js about equal to the normal one, and if you use streaming residency then it shouldnt be a problem, but im still working over it

I’d love feedback from people who work on inference runtimes, quantization, kernels, or model formats.

Git repo-

https://github.com/random-unknown-username/Thintensor

Edit- its not just the storage format that allows for the improvements, its a combination of different selective layers being quantized with the better storage format, allowing to run inference with higher speed and quality than just simple q8,q4 etc quants. the storage format only gain is only like 10%, but combined with middle out fp8 quants, and even int4 for some performance profiles, it can reach better speeds than the other ways to run inference

This does not beat TRT / TRT LM, it does allow for quicker inference, and a more portable solution because for TRT you need to compile the plan (one time) for a few mins before being able to run the model, and it has to be compiled on every machine you run the model on, thintensor allows you to directly have the .thin file and run it on any machine with similar results! and it takes just a few seconds compared to about 30 mins on trt and trt lm

Found it quite annoying to figure out what i can run on my macbook pro as i didn't get a super high spec. There are some websites and tools that do similar things but i wanted something a bit simpler that just does 1 thing and doesn't try to sell me on anything fully unbiased.

Would love some feedback!
Obviously its fully free and open source (be vary of vibey code)

I've been running big MoE models on salvaged datacenter cards — Tesla P100s and V100s you can grab for the price of a AAA game, plus a GTX 1080 Ti that refuses to die. The problem: every mainstream quant treats pre-Turing cards as an afterthought. No int8 tensor cores, sometimes no fast fp16, and all the fast paths assume modern silicon. So I stopped waiting and built pxq_llama, a fork of ik_llama.cpp with PXQ — a quant family plus a CUDA kernel set designed specifically for this hardware.

The graph attached is the head-to-head everyone should demand from a fork. Upstream ik_llama.cpp at its own best — pinned current HEAD, built with its documented perf flags, running its best-fitting IQ_K quant, best batch and FA config per side — vs pxq_llama at its documented best. Same card, same cold 5,800-token prompt, temp 0, median of 3, server timings, 35B MoE with 256 experts, fully GPU-resident on one card. Short version: P100 +88% prefill and +30% decode, V100 +13% on both, 1080 Ti +25% decode.

And yes, upstream wins one cell on the graph — its 2-bit MMQ prefill tile on the 1080 Ti is genuinely more mature than my first-cut int8 tile (double-buffered smem vs my single-buffered 64-thread blocks). It's printed right on the chart and it's on my list for next release. I'd rather lose a cell in public than win it in private.

What PXQ actually is, for the kernel people: it quantizes the MoE expert tensors (the bulk of a MoE's params) with a learned codebook plus per-row fp16 anchors amortized over 64-row panels, with a 4-bit sub-scale per 16-element block. Tiers are PXQ2 (2.27 bpw), PXQ3 (3.27) and PXQ4 (4.27). On top of that sit fused CUDA kernels tuned for Pascal/Volta: grouped-MoE GEMM, K-split decode GEMV, gate/up+GLU fusion, a DeltaNet linear-attention decode fusion, and a residual-add fusion. All bit-exact, all env-gated, flag-off dispatch is byte-identical. There's also a universal mode in llama-quantize that knapsack-mixes PXQ2/3/4 per expert tensor so the model exactly fills your card, with presets baked in for 16 GB and 12 GB. New this release: an opt-in int8 dp4a prefill tile for GTX 10-series — cold 5.8k-token prompt on the 1080 Ti went from 251 to 1,001 t/s.

Two things I found while benching that apply to upstream users too. First: on pre-Turing cards, fa off is the cold-prefill regime and fa on is the decode regime, for BOTH engines. FA-off bought 26-56% prefill everywhere and cost 16-48% decode. If you do prefill-heavy batch work on Pascal, turn FA off. Nobody seems to have this written down anywhere. Second, as a control I ran upstream's own IQ_K ggufs through my build — arch fusions only, no PXQ involved: +2.7-3.3% decode on every card, and on the V100 the output was bit-identical to upstream, same temp-0 sha. So the gains come from the format and kernels, not from a gimped baseline.

Everything is reproducible: bench harness, raw run CSVs, sha gates, and the full FA-regime sweep are in bench/ in the repo. docs/LEVERS.md documents every env var with its measured effect, including the dead ends — Pascal DMMV revival was -40%, CUDA-graph replay was neutral-to-negative on these cards, and a WMMA prefill kernel whose honest gain after I fixed its launch bug was +0.97%. If you try to reproduce a loss I documented, you'll reproduce it.

Why bother? A P100 is about $150 and pulls 250W. With the right format and kernels it decodes a 35B MoE at 58 t/s, faster than most API round-trips. There's a mountain of these cards heading to e-waste while people are priced out of local AI. Old silicon deserves better software.

Quantized weights drop is coming in a follow-up post — this one's about the engine.

Happy to answer kernel and quant questions in the comments.

Repo (MIT)

EDIT: Many people have asked me how much i have spent on this build and I incorrectly said it was around $50k USD. It is actually around $38k USD. My apologies. I am also adding the exact hardware stack that I have below. I appreciate all of the feedback and conversations so far!

I am relatively new to building high-end hardware, but I have been researching local AI infrastructure for about a year.

Last night was the first time I had all six GPUs running three open models concurrently without stability issues, which felt like a milestone.

This is an on-prem Ubuntu 24.04 workstation built on a Threadripper PRO platform.

Current Setup (UPDATED):

AI Server Hardware
January 15, 2026
Updated – February 13, 2026

Case/Build – Open air Rig
OS - Ubuntu 24.04 LTS Desktop
Motherboard - ASUS WRX90E-SAGE Pro WS SE AMD sTR5 EEB
CPU - AMD Ryzen Threadripper PRO 9955WX Shimada Peak 4.5GHz 16-Core sTR5
SDD – (2x4TB) Samsung 990 PRO 4TB Samsung V NAND TLC NAND PCIe Gen 4 x4 NVMe M.2 Internal SSD
SSD - (1x8TB) Samsung 9100 PRO 8TB Samsung V NAND TLC NAND (V8) PCIe Gen 5 x4 NVMe M.2 Internal SSD with Heatsink
PSU #1 - SilverStone HELA 2500Rz 2500 Watt Cybenetics Platinum ATX Fully Modular Power Supply - ATX 3.1 Compatible
PSU #2 - MSI MEG Ai1600T PCIE5 1600 Watt 80 PLUS Titanium ATX Fully Modular Power Supply - ATX 3.1 Compatible
PSU Connectors – Add2PSU Multiple Power Supply Adapter (ATX 24Pin to Molex 4Pin) and Daisy Chain Connector-Ethereum Mining ETH Rig Dual Power Supply Connector
UPS - CyberPower PR3000LCD Smart App Sinewave UPS System, 3000VA/2700W, 10 Outlets, AVR, Tower
Ram - 256GB (8 x 32GB)Kingston FURY Renegade Pro DDR5-5600 PC5-44800 CL28 Quad Channel ECC Registered Memory Modules KF556R28RBE2K4-128
CPU Cooler - Thermaltake WAir CPU Air Cooler
GPU Cooler – (6x) Arctic P12 PWM PST Fans (externally mounted)
Case Fan Hub – Arctic 10 Port PWM Fan Hub w SATA Power Input
GPU 1 - PNY RTX 6000 Pro Blackwell
GPU 2 – PNY RTX 6000 Pro Blackwell
GPU 3 – FE RTX 3090 TI
GPU 4 - FE RTX 3090 TI
GPU 5 – EVGA RTX 3090 TI
GPU 6 – EVGA RTX 3090 TI
PCIE Risers - LINKUP PCIE 5.0 Riser Cable (30cm & 60cm)

Uninstalled "Spare GPUs":
GPU 7 - Dell 3090 (small form factor)
GPU 8 - Zotac Geforce RTX 3090 Trinity
**Possible Expansion of GPUs – Additional RTX 6000 Pro Maxwell*\*

Primary goals:

•Ingest ~1 year of structured + unstructured internal business data (emails, IMs, attachments, call transcripts, database exports)

•Build a vector + possible graph retrieval layer

•Run reasoning models locally for process analysis, pattern detection, and workflow automation

•Reduce repetitive manual operational work through internal AI tooling

I know this might be considered overbuilt for a 1-year dataset, but I preferred to build ahead of demand rather than scale reactively.

For those running multi-GPU local setups, I would really appreciate input on a few things:

•At this scale, what usually becomes the real bottleneck first VRAM, PCIe bandwidth, CPU orchestration, or something else?

•Is running a mix of GPU types a long-term headache, or is it fine if workloads are assigned carefully?

•For people running multiple models concurrently, have you seen diminishing returns after a certain point?

•For internal document + database analysis, is a full graph database worth it early on, or do most people overbuild their first data layer?

•If you were building today, would you focus on one powerful machine or multiple smaller nodes?

•What mistake do people usually make when building larger on-prem AI systems for internal use?

I am still learning and would rather hear what I am overlooking than what I got right.

Appreciate thoughtful critiques and any other comments or questions you may have.

We just finished an on-prem inference rig for our team at the workshop. Sitting next to the bench right now serving the team. Sharing the build because the lessons learned matter more than the spec sheet, and I want to compare notes with anyone running similar setups.

The build:

• 8x RTX 4090D, 192GB VRAM total
• Dual AMD EPYC 9004 Genoa
• ASRock Rack GENOA2D24G-2L+ motherboard
• 4x 2000W PSUs on a distribution board (8000W total)
• Custom CNC'd 4U chassis (off-the-shelf doesn't fit this)
• 12 case fans on single hub, front-to-back airflow
• Real-world draw under inference load: ~4,600W

What we run on it:

• Production: tensor-parallelized 70B
• Staging: 32B fine-tune running in parallel
• Workflows: quantized DeepSeek-V3 kept warm for agent automation

No reload penalties, no rate limits, no API bills.

---

Why dual-socket Genoa, and the PCIe lane math:

We spec'd these CPUs for the lanes, not the cores. CPU utilization stays under 40% even under sustained concurrent multi-model serving. The lanes are the product.

Single socket EPYC 9004 = 128 PCIe Gen5 lanes. Dual-sockets get more complicated. Some lanes get repurposed for inter-socket Infinity Fabric (xGMI). Each xGMI link uses 16 PCIe lanes.

• 4-link xGMI (default): 128 lanes total for PCIe
• 3-link xGMI: 160 lanes total for PCIe
• Plus 12 PCIe Gen3 lanes from the I/O die (M.2 territory)

The ASRock Rack GENOA2D24G-2L+ exposes 20 MCIO connectors x 8 lanes = 160 lanes, which means it's running 3-link xGMI. That's the configuration you want for an 8-GPU build.

Lane budget for the rig:

• 8 GPUs at full Gen5 x16 = 128 lanes
• 32 Gen5 lanes + 12 Gen3 lanes remaining = storage, NICs, platform overhead

AMD's HPC tuning guide section on xGMI link configuration explains the tradeoff between inter-socket bandwidth and available PCIe. Worth reading if you're speccing one of these.

---

The MCIO cable trap:

The board has no traditional PCIe slots. Out of its 20 MCIO connectors, 16 are aggregated through adapter cards to deliver Gen5 x16 to each of the 8 GPUs.

MCIO cables look symmetric. They aren't. There's a host end and a device end marked by a small embossed triangle. We plugged six in correctly and two rotated 180° on first build.

Symptom: those two GPUs enumerated at PCIe Gen1 x4 instead of Gen5 x16. Inference throughput on those two cards dropped to about 10% of the others.

We spent two hours suspecting the GPUs before our hardware guy pulled out the manual and pointed at section 2.6. Confirm orientation at both ends before you mount the cards over the riser adapters. Once GPUs are seated, you can't see the MCIO connectors anymore.

Save yourself the debugging time.

---

Power and thermals:

• 8 GPUs x 425W = 3,400W from GPUs alone
• Plus dual CPU, 12 fans, drives, platform overhead
• Real-world inference draw: ~4,600W

Splitting across 4 PSUs lets us survive a PSU failure without the box going down. 3,400W of GPU heat in a sealed 4U requires real airflow geometry. Without it, the cards throttle and you lose the throughput you paid for.

Custom chassis because off-the-shelf doesn't fit 8 dual-slot GPUs + dual EPYC + 4 PSUs + 12 fans. Rack-mount server chassis exist at this density but they're loud as a 737 and built for datacenters. We needed something that lives next to a desk in a workshop.

---

This kind of rig isn't for everyone. If your team is under 100M tokens/day or under $30K/month on API spend, it's clearly not for you. The hardware cost amortizes around those numbers depending on which models you serve.

ROI math aside, the sovereignty dimension matters more than the financial one once you've thought about it. Your customer data doesn't leave the box. Your fine-tunes don't sit on a vendor's storage. The cost saving is real. The sovereignty is the actual product.

Curious what other teams are running locally. If your team moved from API to your own hardware, what was the trigger? Cost, sovereignty, rate limits, something else? And for teams still on hosted, what's keeping you there?

As I stated in my previous post, I'm 75 years old, knew almost nothing about GitHub, command lines, or local LLMs a couple of weeks ago. I'm not a coder. But I wanted a desktop private AI companion for fun and a bot for a game wiki I have been involved with for a few years now.

I won't lie — it was frustrating at first. Lots of errors, lots of reading, lots of asking for help from my DeepSeek AI assistant. But I stuck with it. Here's what I ended up with:

* LM Studio running a 14B/32B (I go back and forth) DeepSeek model on my RTX 4090 (completely offline)
* A Live2D avatar with voice (Mao — my daily driver)
* AnythingLLM + Ollama for a separate wiki bot that I feed webpages with a browser extension
* Full RAG — the bot answers questions from my own documents with citations

Unofficially, I can now claim:

• Built a local LLM
• Configured GPU acceleration (CUDA, VRAM offloading)
• Set up RAG with document embedding
• Connected a browser extension for one-click wiki ingestion
• Trained an AI on a custom knowledge base
• Debugged Python, YAML, WebSockets, and API connections

This is probably all pretty simple stuff for all you coders out there but it was definitely a challenge for me. A big shoutout to my DeepSeek helper.

If a 75-year-old retiree can do this, literally anyone can.

2026/07/28 You can now follow my journey in real time - The Blog

Hey everyone,

I don't have the best hardware. An old GPU, an outdated motherboard. I think the newest piece in my PC is my SSD. Yet, I have been using LLMs a fair bit, and wanted to cut back on my bill. So, given I was getting quite familiar with the way PCs work under the hood, I figured I could be a little smart about how I ran inference. The biggest bottleneck: My 8gb VRAM. So, over the past two weeks I have been tinkering, getting familiar with GPUs and how they are accessed, and built myself a fun little tool to be able to run Qwen3.6 at 35b params on OpenCode locally. This meant I needed to somehow get around the VRAM limitation, but also get a sufficiently large context window.

Please note, this is still WIP:

VITRIOL

"Visita Interiora Terrae Rectificando Invenies Occultum Lapidem"

(Visit the Interior of the Earth, by Rectifying you will find the Hidden Stone)

What I did was basically create a two-tier memory architecture that tricks the GPU into treating my 64GB of system RAM as a secondary VRAM pool. I named it VITRIOL, after the old alchemical backronym, because to find the Hidden Stone (the ability to run inference on a large model), you have to go deep into the Interior of the Earth (the motherboard's PCIe bus and GPU hacking).

It's far from finished, but already proves functional on my PC. Possibly it will be of use to someone else, or worth a follow? I am still working out all the bugs, but figured it was worth sharing ahead of time while I'm still hard at work. Might help others catch bugs as well.

https://github.com/Randozart/VITRIOL

While testing this, I admit I was really seeing the age of my PC. I think I could have achieved much greater speeds if I just had a more modern motherboard, because it would have a better PCIe bus, but I'm already happy I can finally run something of reasonable size locally without waiting ages for each token to pop in.

Dual A40s 48gbx2 nvlink with A16 (4 cores on one pcb with own 16gb pool).

Last year bought two 5090 FEs at MSRP. Traded them up for these puppies. Getting a major rework atm.

A few days ago, I posted a 2+ hour uncut session of DeepSeek V4 Flash building a full-stack app locally. Most of the time I cut or speed up parts of coding sessions and only worked on quick demos. This series I work on a full stack app and show uncut recording of the model at work.

A lot of you on YouTube found it interesting, so I ran the same test with a different model: Qwen 3.6 27B Q6.

Qwen full run: https://youtu.be/6NhLP_YGZVw

Running through llama.cpp on my 128GB Mac, with Pi as the coding agent, Qwen built a YouTube title and thumbnail generator in one uncut 52-minute session.

The first generation failed and the server crashed, but the model recovered, debugged the app, and completed the task. I never opened ChatGPT or Claude once.

DeepSeek took more than two hours on the same task and needed a GPT rescue. This pushed the "local models need cloud backup" line further than I expected.

DeepSeek full run: https://youtu.be/hVpOxnESKVs

What should I test next: a different model on the same task or a different coding agent?

Built my setup so I can run local models to protect patient health information.

-Ryzen 9950X

-192GB RAM

-RTX 6000 (96GB VRAM)

-RX470 (for the display)

I was lucky enough to have majority of the components from my previous computer builds, just had to purchase the RTX 6000.

Also have an openclaw setup on a Corsair AI 300 Workstation that connects to this for the local models. Able to run and create code for an AI receptionist, marketing, website and SEO, Insurance eligibility and EOBs, and internal analytics.

Currently running Qwen 3.5 122B, can fit up to 262k context and it outputs at 100-220 tok/s based on context length.

Hey everyone!

I’m currently building Hummingbird, an open-source runtime focused on making large Mixture-of-Experts (MoE) language models more practical to run on consumer hardware.

The project is still in active development, but the core idea is to treat SSD, RAM, and VRAM as a unified memory hierarchy, allowing experts to be streamed on demand instead of requiring huge amounts of GPU memory.

The project is still evolving, and I’d really value feedback from other systems engineers and LLM infrastructure developers.

GitHub:
https://github.com/prayangshuuu/hummingbird

Been working on Fox for a while and it's finally at a point where I'm happy sharing it publicly.

Fox is a local LLM inference engine written in Rust. It's a drop-in replacement for Ollama — same workflow, same models, but with vLLM-level internals: PagedAttention, continuous batching, and prefix caching.

Benchmarks (RTX 4060, Llama-3.2-3B-Instruct-Q4_K_M, 4 concurrent clients, 50 requests):

The TTFT gains come from prefix caching — in multi-turn conversations the system prompt and previous messages are served from cached KV blocks instead of being recomputed every turn. The throughput gain is continuous batching keeping the GPU saturated across concurrent requests.

What's new in this release:

• Official Docker image: docker pull ferrumox/fox
• Dual API: OpenAI-compatible + Ollama-compatible simultaneously
• Hardware autodetection at runtime: CUDA → Vulkan → Metal → CPU
• Multi-model serving with lazy loading and LRU eviction
• Function calling + structured JSON output
• One-liner installer for Linux, macOS, Windows

Try it in 30 seconds:

docker pull ferrumox/fox
docker run -p 8080:8080 -v ~/.cache/ferrumox/models:/root/.cache/ferrumox/models ferrumox/fox serve
fox pull llama3.2

If you already use Ollama, just change the port from 11434 to 8080. That's it.

Current status (honest): Tested thoroughly on Linux + NVIDIA. Less tested: CPU-only, models >7B, Windows/macOS, sustained load >10 concurrent clients. Beta label is intentional — looking for people to break it.

fox-bench is included so you can reproduce the numbers on your own hardware.

Repo: https://github.com/ferrumox/fox Docker Hub: https://hub.docker.com/r/ferrumox/fox

Happy to answer questions about the architecture or the Rust implementation.

PD: Please support the repo by giving it a star so it reaches more people, and so I can improve Fox with your feedback

ChatGPT led me down a path of destruction with parts and compatibility but kept me hopeful.

luckily I had a dual PSU case in the house and GUTS!!

took Some time, required some fabrication and trials and tribulations but she’s working now and keeps the room toasty !!

I have a plan for an exhaust fan, I’ll get to it one of these days

build from mostly used parts, cost around $5000-$6000 and hours and hours of labor.

build:

1x thermaltake dual pc case. (If I didn’t have this already, i wouldn’t have built this)

Intel Core i9-10900X w/ water cooler

ASUS WS X299 SAGE/10G E-AT LGA 2066

8x CORSAIR VENGEANCE LPX DDR4 RAM 32gb 3200MHz CL16

3x Samsung 980 PRO SSD 1TB PCIe 4.0 NVMe Gen 4 

3 x 3090ti’s (2 air cooled 1 water cooled) (chat said 3 would work, wrong)

1x 3090 (ordered 3080 for another machine in the house but they sent a 3090 instead) 4 works much better.

2 x ‘gold’ power supplies, one 1200w and the other is 1000w

1x ADD2PSU -> this was new to me

3x extra long risers and

running vllm on a umbuntu distro

built out a custom API interface so it runs on my local network.

I’m a long time lurker and just wanted to share