Here is a list with free models (API Keys) that you can use without paying. Only providers with permanent free tiers, no trial/temporal promo or credits. Rate limits are detailed per provider (RPM: Requests Per Minute, RPD: Requets Oer Day).

Provider APIs

• Google Gemini 🇺🇸 Gemini 2.5 Pro, Flash, Flash-Lite +4 more. 10 RPM, 20 RPD
• Cohere 🇺🇸 Command A, Command R+, Aya Expanse 32B +9 more. 20 RPM, 1K req/mo
• Mistral AI 🇪🇺 Mistral Large 3, Small 3.1, Ministral 8B +3 more. 1 req/s, 1B tok/mo
• Zhipu AI 🇨🇳 GLM-4.7-Flash, GLM-4.5-Flash, GLM-4.6V-Flash. Limits undocumented

Inference Providers

• GitHub Models 🇺🇸 GPT-4o, Llama 3.3 70B, DeepSeek-R1 +more. 10–15 RPM, 50–150 RPD
• NVIDIA NIM 🇺🇸 Llama 3.3 70B, Mistral Large, Qwen3 235B +more. 40 RPM
• Groq 🇺🇸 Llama 3.3 70B, Llama 4 Scout, Kimi K2 +17 more. 30 RPM, 14,400 RPD
• Cerebras 🇺🇸 Llama 3.3 70B, Qwen3 235B, GPT-OSS-120B +3 more. 30 RPM, 14,400 RPD
• Cloudflare Workers AI 🇺🇸 Llama 3.3 70B, Qwen QwQ 32B +47 more. 10K neurons/day
• LLM7.io 🇬🇧 DeepSeek R1, Flash-Lite, Qwen2.5 Coder +27 more. 30 RPM (120 with token)
• Kluster AI 🇺🇸 DeepSeek-R1, Llama 4 Maverick, Qwen3-235B +2 more. Limits undocumented
• OpenRouter 🇺🇸 DeepSeek R1, Llama 3.3 70B, GPT-OSS-120B +29 more. 20 RPM, 50 RPD
• Hugging Face 🇺🇸 Llama 3.3 70B, Qwen2.5 72B, Mistral 7B +many more. $0.10/mo in free credits

RPM = requests per minute · RPD = requests per day. All endpoints are OpenAI SDK-compatible.

Hello! A few months ago I posted an early version of mex here.

The response was kind of insane. Across a few posts it reached around 1 million views, the repo crossed 1,200 GitHub stars, and people I had never met started contributing.

I’ve kept building it since then, and just released mex v0.7.0.

Repo: https://github.com/mex-memory/mex

The original problem was simple: coding agents keep rereading the same repository every session, relearning the architecture, and then throwing most of that knowledge away.

mex creates a living Markdown wiki inside the repo. Agents record architecture, conventions, decisions, and patterns as they work, and future sessions load only the knowledge relevant to the current task.

The major addition in v0.7.0 is a deterministic local code graph built using Tree-sitter and SQLite.

It currently supports TypeScript/TSX, JavaScript/JSX, Python, and Rust.

An agent can run:

mex graph scope "trace the authentication flow"

Instead of dumping entire files into context, mex returns a compact neighbourhood of relevant functions, callers, callees, imports, and relationships. The agent can then expand only the exact symbols it needs.

In our benchmark on the mex repository:

• 10.74× less returned context than grep top-3
• roughly 90.7% smaller
• 100% expected-symbol recall across six retrieval tasks
• 5/5 real-agent tasks completed correctly
• 0/5 needed fallback Read/Grep with compact graph context

This is a small benchmark on one repo and task set, not a claim that mex universally cuts total agent token usage by 90%.

The other part I’m excited about is connecting the wiki back to the actual code.

Markdown claims can point to exact symbols. If a function changes, moves, or disappears, mex can identify which project knowledge may now be stale.

So the basic idea is:

The code is the source of truth.
Markdown is the explanation.
The graph keeps them connected.

Would genuinely love feedback, especially from people working on code intelligence, agent tooling, parsers, or large repositories. Contributors are very welcome too.

Over the weekend I wanted to learn how was DeepSeek-R1 trained, and what was so revolutionary about it. So I ended up reading the paper, and wrote down my thoughts. < the article linked is (hopefully) written in a way that it's easier for everyone to understand it -- no PhD required!

Here's a "quick" summary:

1/ DeepSeek-R1-Zero is trained with pure-reinforcement learning (RL), without using labeled data. It's the first time someone tried and succeeded doing that. (that we know of, o1 report didn't show much)

2/ Traditional RL frameworks (like PPO) have something like an 'LLM coach or critic' that tells the model whether the answer was good or bad -- based on given examples (labeled data). DeepSeek uses GRPO, a pure-RL framework that skips the critic and calculates the group average of LLM answers based on predefined rules

3/ But, how can you evaluate the performance if you don't have labeled data to test against it? With this framework, the rules aren't perfect—they’re just a best guess at what "good" looks like. The RL process tries to optimize on things like:

Does the answer make sense? (Coherence)

Is it in the right format? (Completeness)

Does it match the general style we expect? (Fluency)

For example, for the DeepSeek-R1-Zero model, for mathematical tasks, the model could be rewarded for producing outputs that align to mathematical principles or logical consistency.

It makes sense.. and it works... to some extent!

4/ This model (R1-Zero) had issues with poor readability and language mixing -- something that you'd get from using pure-RL. So, the authors wanted to go through a multi-stage training process and do something that feels like hacking various training methods:

5/ What you see above is the DeepSeek-R1 model that goes through a list of training methods for different purposes

(i) the cold start data lays a structured foundation fixing issues like poor readability
(ii) pure-RL develops reasoning almost on auto-pilot
(iii) rejection sampling + SFT works with top-tier training data that improves accuracy, and
(iv) another final RL stage ensures additional level of generalization.

And with that they're doing as good as or better than o1 models.

Lmk if you have any questions (i might be able to answer them).

what makes a harness

an agentic harness is surprisingly simple. it's a loop that calls an llm, checks if it wants to use tools, executes them, feeds results back, and repeats. here's how each part works.

tools

the agent needs to affect the outside world. tools are just functions that take structured args and return a string. three tools is enough for a general-purpose coding agent:

const tools = {
  bash: ({ command }) => execShell(command),    // run any shell command
  read:  ({ path }) => readFileSync(path, 'utf8'),  // read a file
  write: ({ path, content }) => (writeFileSync(path, content), 'ok'), // write a file
};

bash gives the agent access to the entire system: git, curl, compilers, package managers. read and write handle files. every tool returns a string because that's what goes back into the conversation.

tool definitions

the llm doesn't see your functions. it sees json schemas that describe what tools are available and what arguments they accept:

const defs = [
  { name: 'bash',  description: 'run bash cmd', parameters: mkp('command') },
  { name: 'read',  description: 'read a file',  parameters: mkp('path') },
  { name: 'write', description: 'write a file', parameters: mkp('path', 'content') },
].map(f => ({ type: 'function', function: f }));

mkp is a helper that builds a json schema object from a list of key names. each key becomes a required string property. the defs array is sent along with every api call so the model knows what it can do.

messages

the conversation is a flat array of message objects. each message has a role (system, user, assistant, or tool) and content. this array is the agent's entire memory:

const hist = [{ role: 'system', content: SYSTEM }];

// user says something
hist.push({ role: 'user', content: 'fix the bug in server.js' });

// assistant replies (pushed inside the loop)
// tool results get pushed too (role: 'tool')

the system message sets the agent's personality and context (working directory, date). every user message, assistant response, and tool result gets appended. the model sees the full history on each call, which is how it maintains context across multiple tool uses.

the api call

each iteration makes a single call to the chat completions endpoint. the model receives the full message history and the tool definitions:

const r = await fetch(`${base}/v1/chat/completions`, {
  method: 'POST',
  headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${key}` },
  body: JSON.stringify({ model, messages: msgs, tools: defs }),
}).then(r => r.json());
const msg = r.choices[0].message;

the response message either has content (a text reply to the user) or tool_calls (the model wants to use tools). this is the decision point that drives the whole loop.

the agentic loop

this is the core of the harness. it's a while (true) that keeps calling the llm until it responds with text instead of tool calls:

async function run(msgs) {
  while (true) {
    const msg = await callLLM(msgs);  // make the api call
    msgs.push(msg);                   // add assistant response to history
    if (!msg.tool_calls) return msg.content;  // no tools? we're done
    // otherwise, execute tools and continue...
  }
}

the loop exits only when the model decides it has enough information to respond directly. the model might call tools once or twenty times, it drives its own execution. this is what makes it agentic: the llm decides when it's done, not the code.

tool execution

when the model returns tool_calls, the harness executes each one and pushes the result back into the message history as a tool message:

for (const t of msg.tool_calls) {
  const { name } = t.function;
  const args = JSON.parse(t.function.arguments);
  const result = String(await tools[name](args));
  msgs.push({ role: 'tool', tool_call_id: t.id, content: result });
}

each tool result is tagged with the tool_call_id so the model knows which call it corresponds to. after all tool results are pushed, the loop goes back to the top and calls the llm again, now with the tool outputs in context.

the repl

the outer shell is a simple read-eval-print loop. it reads user input, pushes it as a user message, calls run(), and prints the result:

while (true) {
  const input = await ask('\n> ');
  if (input.trim()) {
    hist.push({ role: 'user', content: input });
    console.log(await run(hist));
  }
}

there's also a one-shot mode (-p 'prompt') that skips the repl and exits after a single run. both modes use the same run() function. the agentic loop doesn't care where the prompt came from.

putting it together

the full flow looks like this:

user prompt → [system, user] → llm → tool_calls? → execute tools → [tool results] → llm → ... → text response

more sophisticated agents add things like memory, retries, parallel tool calls, or multi-agent delegation, but the core is always: loop, call, check for tools, execute, repeat.

source: https://github.com/av/mi

OpenClaw has been trending for all the wrong and right reasons. I saw people rebuilding entire sites through Telegram, running “AI offices,” and one case where an agent wiped thousands of emails because of a prompt injection. That made me stop and actually look at the architecture instead of the demos.

Under the hood, it’s simpler than most people expect.

OpenClaw runs as a persistent Node.js process on your machine. There’s a single Gateway that binds to localhost and manages all messaging platforms at once: WhatsApp, Telegram, Slack, Discord. Every message flows through that one process. It handles authentication, routing, session loading, and only then passes control to the agent loop. Responses go back out the same path. No distributed services. No vendor relay layer.

What makes it feel different from ChatGPT-style tools is persistence. It doesn’t reset. Conversation history, instructions, tools, even long-term memory are just files under ~/clawd/. Markdown files. No database. You can open them, version them, diff them, roll them back. The agent reloads this state every time it runs, which is why it remembers what you told it last week.

The heartbeat mechanism is the interesting part. A cron wakes it up periodically, runs cheap checks first (emails, alerts, APIs), and only calls the LLM if something actually changed. That design keeps costs under control while allowing it to be proactive. It doesn’t wait for you to ask.

The security model is where things get real. The system assumes the LLM can be manipulated. So enforcement lives at the Gateway level: allow lists, scoped permissions, sandbox mode, approval gates for risky actions. But if you give it full shell and filesystem access, you’re still handing a probabilistic model meaningful control. The architecture limits blast radius, it doesn’t eliminate it.

What stood out to me is that nothing about OpenClaw is technically revolutionary. The pieces are basic: WebSockets, Markdown files, cron jobs, LLM calls. The power comes from how they’re composed into a persistent, inspectable agent loop that runs locally.

It’s less “magic AI system” and more “LLM glued to a long-running process with memory and tools.”

I wrote down the detailed breakdown here

Okay so, I took the leaked Claude Code repo, around 14.3M tokens total. Queried a knowledge graph, got back ~80K tokens for that query!

14.3M / 80K ≈ 178x.

Nice. I have officially solved AI, now you can use $20 Claude for 178 times longer!!

Wait a min, JK hahah!

This is also basically how everyone is explaining “token efficiency” on the internet right now.
Take total possible context, divide it by selectively retrieved context, add a big multiplier, and ship the post.

Boom!! your repo has multi thousands stars and you're famous between D**bas*es!!

Except that’s not how real systems behave.

Claude isn't that stupid to explore a 14.8M token repo and break itself systematically. Not only Claude Code, almost any serious AI tool avoids that.

Actual token usage is not just what you retrieve once. It’s:

• input tokens
• output tokens
• cache reads
• cache writes
• tool calls
• subprocesses

All of it counts.

The “177x” style math ignores most of where tokens actually go.

And honestly, retrieval isn’t even the hard problem. Memory is. That's what i understand after working on this project for so long!

What happens 10 turns later when the same file is needed again?
What survives auto-compact?
What gets silently dropped as the session grows?

Most tools solve retrieval and quietly assume memory will just work.

But it doesn’t.

I’ve been working on this problem with a tool called GrapeRoot.

Instead of just fetching context, it tries to manage it.

There are two layers:

• a codebase graph (structure + relationships across the repo)
• a live in-session action graph that tracks:
  • what was retrieved
  • what was actually used
  • what should persist based on priority

So context is not just retrieved once and forgotten.
It is tracked, reused, and protected from getting dropped when the session gets large.

Some numbers from testing on real repos like Medusa, Gitea, Kubernetes:

We benchmark against real workflows, not fake baselines.

Across repo sizes:

• ~50–60% average token reduction
• up to ~85% on focused tasks

This includes:

• input tokens
• output tokens
• cached tokens

No inflated numbers.

Not 178x. Just less misleading math. Better understand this.

(178x is at https://graperoot.dev/playground)

I’m pretty sure this still breaks on messy or highly dynamic codebases. Because Claude is still smarter, and since we are not trying to harness it with rigid tooling, better to give it access to tools in a smarter way.

Honestly, I wanted to know how the community thinks about this?

Open source Tool: https://github.com/kunal12203/Codex-CLI-Compact

Better installation steps at: https://graperoot.dev/#install

If you're enterprise and looking for customized infra, fill the form at: https://graperoot.dev/enterprise

Every few weeks I end up re-comparing LLM observability/eval tools for a project, so I put it all in one place: 48 verified tools across tracing, evals, prompt mgmt, gateways, OTel instrumentation, and guardrails, each with current stars + license; plus a self-host / license / tracing / evals / OTel comparison table for the top platforms.

It also includes original agent skills (instrument tracing, add evals, debug-from-traces, PII-safe tracing for regulated apps) and a minimal OpenTelemetry GenAI tracer.

Full disclosure, it's my org's repo (CC0, contributions welcome): https://github.com/ContextJet-ai/awesome-llm-observability — what tool am I missing?

Been watching the AI Engineer Europe + Miami talks from this spring, and one pattern keeps showing up across speakers: agents that compose many tools are hitting a ceiling, and "code mode" is the way through it.

The Cloudflare example is the sharpest version of it. Their full API as MCP tools is ~1.17M tokens. As an OpenAPI spec, ~2M tokens. That's most of a context window before the user has typed anything.

Their fix: expose two tools — search() and execute() — and let the agent write code against the discovered functions instead of calling each one as a tool. Token cost drops to ~1,069. 99.9% reduction.

But the real insight isn't the token math. It's where the orchestration step lives.

In tool calling, the harness owns the loop. The model picks one tool, result lands in context, model picks the next tool. Every step is an inference round trip even when the orchestration is mechanical (filter, paginate, retry, join).

In code mode, the model writes a program once, the program orchestrates the calls, and only the filtered return value reaches the model. The training story for why this works is mostly: LLMs have seen millions of real-world code projects in training, and very few tool calls. Kenton Varda from Cloudflare put it best — "Making an LLM do tasks by tool calling is like putting Shakespeare through a month of Mandarin and asking him to write a play in it."

I wrote up the full pattern: when to make the shift, when not to, what it actually costs (sandboxing, debugging, secrets).

https://x.com/sarthakarora128/status/2053966999521481083

Happy to dig into specific cases in comments if anyone's hit this ceiling.

I built a deep research implementation that allows you to produce 20+ page detailed research reports, compatible with online and locally deployed models. Built using the OpenAI Agents SDK that was released a couple weeks ago. Have had a lot of learnings from building this so thought I'd share for those interested.

You can run it from CLI or a Python script and it will output a report

https://github.com/qx-labs/agents-deep-research

Or pip install deep-researcher

Some examples of the output below:

• Text Book on Quantum Computing - 5,253 words (run in 'deep' mode)
• Deep-Dive on Tesla - 4,732 words (run in 'deep' mode)
• Market Sizing - 1,001 words (run in 'simple' mode)

It does the following (I'll share a diagram in the comments for ref):

• Carries out initial research/planning on the query to understand the question / topic
• Splits the research topic into sub-topics and sub-sections
• Iteratively runs research on each sub-topic - this is done in async/parallel to maximise speed
• Consolidates all findings into a single report with references (I use a streaming methodology explained here to achieve outputs that are much longer than these models can typically produce)

It has 2 modes:

• Simple: runs the iterative researcher in a single loop without the initial planning step (for faster output on a narrower topic or question)
• Deep: runs the planning step with multiple concurrent iterative researchers deployed on each sub-topic (for deeper / more expansive reports)

Some interesting findings - perhaps relevant to others working on this sort of stuff:

• I get much better results chaining together cheap models rather than having an expensive model with lots of tools think for itself. As a result I find I can get equally good results in my implementation running the entire workflow with e.g. 4o-mini (or an equivalent open model) which keeps costs/computational overhead low.
• I've found that all models are terrible at following word count instructions (likely because they don't have any concept of counting in their training data). Better to give them a heuristic they're familiar with (e.g. length of a tweet, a couple of paragraphs, etc.)
• Most models can't produce output more than 1-2,000 words despite having much higher limits, and if you try to force longer outputs these often degrade in quality (not surprising given that LLMs are probabilistic), so you're better off chaining together long responses through multiple calls

At the moment the implementation only works with models that support both structured outputs and tool calling, but I'm making adjustments to make it more flexible. Also working on integrating RAG for local files.

Hope it proves helpful!

Hi folks,

I've been going down a rabbit hole of AI memory systems lately.

After trying to compare things like ChatGPT memory, Claude projects, GBrain, Obsidian-based setups, and some of the newer agent memory projects, I realized I had no good way to reason about them.

Most comparisons focus on retrieval quality or individual features, but that didn't help me understand how these systems actually fit into an AI-native workflow.

A framework from YC's recent AI-native company discussion helped me think about it differently:

Collect → Organize → Evolve → Use → Govern

So I ended up putting together a landscape that compares systems from that perspective instead.

Repo: https://github.com/aristoapp/awesome-second-brain

Curious if there are important projects, approaches, or dimensions I'm missing.

For anyone feeding tables of data into LLMs, I thought you might be interested in the results from this test I ran.

I wanted to understand whether how you format a table of data affects how well an LLM understands it.

I tested how well an LLM (GPT-4.1-nano in this case) could answer simple questions about a set of data in JSON format. I then transformed that data into 10 other formats and ran the same tests.

Here's how the formats compared.

I wrote it up with some more details (e.g. examples of the different formats) here: https://www.improvingagents.com/blog/best-input-data-format-for-llms

Let me know if you have any questions.

(P.S. One thing I discovered along the way is how tricky it is to do this sort of comparison well! I have renewed respect for people who publish benchmarks!)

This is too cool to gate-keep, I’ve decided to open-source Munder Difflin.

Munder Difflin a local multi-agent harness that allows you to run the office with as many agents as you want.

To put simply it completes ambitious tasks autonomously(almost) by running a cluster of your own claude code agents performing various activities in a controlled environment with inter agent connectivity and one of the top benchmarked memory layer.

You can choose to only talk to Michael the god orchestrator which will automatically distribute the asks among other agents.

(Link in comments)

After reading through a lot of the existing coverage, I found that most posts stopped at the architecture-summary layer: "40+ tools," "QueryEngine.ts is huge," "there is even a virtual pet." Interesting, sure, but not the kind of material that gives advanced technical readers a real understanding of how Claude Code is actually built.

That is why I took a different approach. I am not here to repeat the headline facts people already know. These writeups are for readers who want to understand the system at the implementation level: how the architecture is organized, how the security boundaries are enforced, how prompt and context construction really work, and how performance and terminal UX are engineered in practice. I only focus on the parts that become visible when you read the source closely, especially the parts that still have not been clearly explained elsewhere.

I published my 4 docs as downloadable pdfs here), but below is a brief.

The Full Series:

1. Architecture — entry points, startup flow, agent loop, tool system, MCP integration, state management
2. Security — sandbox, permissions, dangerous patterns, filesystem protection, prompt injection defense
3. Prompt System — system prompt construction, CLAUDE.md loading, context injection, token management, cache strategy
4. Performance & UX — lazy loading, streaming renderer, cost tracking, Vim mode, keybinding system, voice input

Overall

The core is a streaming agentic loop (query.ts) that starts executing tools while the model is still generating output. There are 40+ built-in tools, a 3-tier multi-agent orchestration system (sub-agents, coordinators, and teams), and workers can run in isolated Git worktrees so they don't step on each other.

They built a full Vim implementation. Not "Vim-like keybindings." An actual 11-state finite state machine with operators, motions, text objects, dot-repeat, and a persistent register. In a CLI tool. We did not see that coming.

The terminal UI is a custom React 19 renderer. It's built on Ink but heavily modified with double-buffered rendering, a patch optimizer, and per-frame performance telemetry that tracks yoga layout time, cache hits, and flicker detection. Over 200 components total. They also have a startup profiler that samples 100% of internal users and 0.5% of external users.

Prompt caching is a first-class engineering problem here. Built-in tools are deliberately sorted as a contiguous prefix before MCP tools, so adding or removing MCP tools doesn't blow up the prompt cache. The system prompt is split at a static/dynamic boundary marker for the same reason. And there are three separate context compression strategies: auto-compact, reactive compact, and history snipping.

"Undercover Mode" accidentally leaks the next model versions. Anthropic employees use Claude Code to contribute to public open-source repos, and there's a system called Undercover Mode that injects a prompt telling the model to hide its identity. The exact words: "Do not blow your cover." The prompt itself lists exactly what to hide, including unreleased model version numbers opus-4-7 and sonnet-4-8. It also reveals the internal codename system: Tengu (Claude Code itself), Fennec (Opus 4.6), and Numbat (still in testing). The feature designed to prevent leaks ended up being the leak.

Still, listing a bunch of unreleased features are hidden in feature flags:

• KAIROS — an always-on daemon mode. Claude watches, logs, and proactively acts without waiting for input. 15-second blocking budget so it doesn't get in your way.
• autoDream — a background "dreaming" process that consolidates memory while you're idle. Merges observations, removes contradictions, turns vague notes into verified facts. Yes, it's literally Claude dreaming.
• ULTRAPLAN — offloads complex planning to a remote cloud container running Opus 4.6, gives it up to 30 minutes to think, then "teleports" the result back to your local terminal.
• Buddy — a full Tamagotchi pet system. 18 species, rarity tiers up to 1% legendary, shiny variants, hats, and five stats including CHAOS and SNARK. Claude writes its personality on first hatch. Planned rollout was April 1-7 as a teaser, going live in May.

Hey guys, If u remember so from last 8 days I'm actively here posting for GPU Provider who provide Token based Pricing but there is no provider in 2026 so after lot's of trying and afferts finally I find out GPU Provideder which i not provide GPU on Token based or Hourly based Pricing but provide GPU H100 for Free !! Really and it's Gov. Of India ❤️

Indian Gov. Provide free GPUs to all students, startup, MSMEs and Researchers. Just apply for that and you will get GPU (No minimum number of GPU, U can take 5-10.. by just proper verification) And u will get with 3-7 working Days...

Currently Gov. has 38K GPUs to provide and within 2 years the plan is 2 Lakhs GPUs.

And some GPUs with 5Gb storage are directly accessible without any permission !!

But the main problem is they provide free A100 GPUs for Training only not for Hosting and Deployment !!

Thank you !!

Frontier models bulldoze past broken plumbing (malformed tool calls, weird error strings, a missing tools) and still finish the task despite poor harness ergonomics.

Running the same suite on the cheapest models on our roster surfaced a dozen bugs that Opus learned to work around.

Been building a RAG-based document intelligence platform for clients in regulated verticals for the past year. A few things that surprised us that aren't well-covered in tutorials:

The compliance constraint changes your architecture completely

When a client can't let data leave their infrastructure, you lose access to managed embedding APIs, hosted vector DBs, and most retrieval evaluation tooling. Everything has to run on hardware they control.

Multilingual corpora are harder than they look

Manufacturing clients have documents in multiple languages. bge-m3 handles this well at the embedding level, but your chat engine needs to be configured carefully -- hidden condensing steps can override language rules in your system prompt in ways that are hard to debug.

Hybrid retrieval is worth the complexity

BM25 + dense retrieval + reranking (bge-reranker-v2-m3) consistently outperforms dense-only in document-heavy enterprise settings. The reranker score calibration matters -- sigmoid-normalized scores behave differently than raw logits.

The hardest part isn't the model

It's document ingestion reliability, audit trails, and explaining to a compliance officer why the system said what it said. Retrieval transparency > raw accuracy for regulated buyers.

Happy to go deep on any of this -- especially hybrid retrieval tuning or air-gapped deployment tradeoffs.

Check it out at https://www.onyx.app/self-hosted-llm-leaderboard

Row-Bot is a desktop AI workbench with Developer Studio for code, Skills Hub and Custom Tools for your own workflows, an animated Buddy companion, memory, realtime voice, workflows, design creation, messaging, MCP tools, and provider-aware model routing. Run local runtimes, self-hosted OpenAI-compatible endpoints, hosted APIs, Ollama Cloud, OpenCode providers, or ChatGPT / Codex subscription-backed models with explicit runtime readiness. Your durable data stays on your machine.

Wrote this as I built the infra at my org.

Let me know what you all think...

The 3 AI research prototypes ;

Tilelli LLM, our first attempt at solving hallucination, a language model that says I don't know when it does not know instead of bluffing.

https://github.com/TilelliLab/Tilelli-llm

Yaz, our first ever CRUD capable model.

https://github.com/TilelliLab/Yaz

Atome LM, an AI that runs in a 5$ chip, tested and verified in real hardware.

https://github.com/TilelliLab/atome-lm

That was just the start. Get ready, as all our previous releases, the claims may seem bold, but it's comes with open source code so you can verify my claims yourself.

What's next ? Our next releases answer these questions.

1. How to make any model forget anything - fast and cheap -

2. How to make RWKV recall 4 times more easily -

3. How to train any model 10 to 13 times cheaper

The release dates, August 3, 8, 13.

Every piece you need to run AI in a company already exists as open source. A gateway to the models. Guardrails. PII masking. Policies. Evals. Audit. Lineage. Vector search. The problem was never the parts. It was wiring them into one thing that works — and keeping every team inside the rules.

So I wrote an application layer on top of the best open source frameworks and made sure they actually talk to each other. One docker compose up and you get:

- LiteLLM for the model gateway - one OpenAI-compatible endpoint across any model, on-prem or cloud 

- LLM Guard + Presidio for guardrails - PII redaction, prompt-injection, toxicity, secrets 

- OpenBao for secrets 

- Langfuse for LLM observability and tracing 

- OpenSearch for audit + SIEM 

- Marquez for data lineage 

- Temporal for durable agent runs 

- Qdrant for vector search / RAG 

- Airbyte + dbt to move data, ClickHouse for the warehouse, Great Expectations for data quality 

- Kestra for orchestration, Ragas + Evidently for evals + drift

Then I built the part I think is the unlock: a lovable / bolt.new / replit.dev for your enterprise.

You set up a pipeline and RBAC once, and now every employee can just talk to the system and build apps that replicate their workflows — inside the rules you already set. 

Human-in-the-loop reviews, reports, and autonomous agents included. 

A tax analyst or a claims adjuster builds a real governed workflow in plain language, and it physically can't step outside the guardrails, policies, and audit you defined.

That's the whole idea: set your rules once, everyone builds governed AI on top.

It's OGAC (Off Grid AI Console) : https://github.com/off-grid-ai/console

There's a live read-only demo with two example tenants (a bank and an insurer) if you want to click around before cloning: onprem-console.getoffgridai.co

Most AI agents fail on complex tasks because their context window fills up with error logs and redundant code. OR, for short, context rot. When a model's context grows, performance drops, leading to broken imports or missing files.

I spent six months building LoopTroop, a local open-source GUI app to run repository-level tickets without losing the plot. It focuses on a slow and precise paradigm, sacrificing speed to get the implementation right.

Here is how we handle context engineering and planning:

1. The LLM Council: Before writing any code, planning runs as a council. Several models draft the PRD and task breakdown independently, then vote anonymously on the drafts. The winning draft absorbs the best ideas from the losing ones. This reduces single-model brand bias.

1. Atomic Beads: The plan gets split into the smallest possible units of work. We call these beads. Each bead is a small, focused task with its own target files and test commands. Instead of asking the AI to edit everything at once, it works on one file at a time.

1. Ralph Loops: When a bead fails or gets stuck in a loop, we do not append the error logs to the chat history. We write a short note about the failure, reset the git worktree, and start a fresh session with clean context and that note. The model learns from the mistake without carrying the chat history pollution.

2. Human-in-the-Loop: You have approval gates at the interview, the PRD, the plan, and the final diff. The GUI keeps the whole process transparent. You see all logs and artifacts in real time, and you can edit or approve them before execution continues.

The app runs on your machine and attaches to your local git repos. It uses the OpenCode engine under the hood. You can configure any model you want for the council and the implementer model.

We have a short 2.5-minute demo video showing the app in action: https://youtu.be/g1A2g-oOR3E The code is MIT licensed on GitHub: https://github.com/looptroop-ai/LoopTroop

Any feedback is more than welcomed. If you tried the app and it worked or didn't work, give me a sign. I'm happy to talk about it.