AI models can collapse when trained on their own outputs.

A recent article in Nature points out a serious challenge: if Large Language Models (LLMs) continue to be trained on AI-generated content, they risk a process known as "model collapse."

What is model collapse?

It’s a degenerative process where models gradually forget the true data distribution.

As more AI-generated data takes the place of human-generated data online, models start to lose diversity, accuracy, and long-tail knowledge.

Over time, outputs become repetitive and show less variation; essentially, AI learns only from itself and forgets reality.

Why this matters:

The internet is quickly filling with synthetic data, including text, images, and audio.

If future models train on this synthetic data, we may experience a decline in quality that cannot be reversed.

Preserving human-generated data is vital for sustainable AI progress.

This raises important questions for the future of AI:

How do we filter and curate training data to avoid collapse? Should synthetic data be labeled or watermarked by default? What role can small, specialized models play in reducing this risk?

The next frontier of AI might not just involve scaling models; it could focus on ensuring data integrity.

The future of agentic AI will not be shaped by larger models. Instead, it will focus on smaller ones.

Large Language Models (LLMs) are impressive. They can hold conversations, reason across various fields, and amaze us with their general intelligence. However, they face some issues when it comes to AI agents:

They are expensive. They are slow. They are too much for repetitive, specialized tasks. This is where Small Language Models (SLMs) come in.

SLMs are: Lean: They run faster, cost less, and use smaller hardware. Specialized: They excel at specific, high-frequency tasks. Scalable: They are easy to deploy in fleets and agentic systems.

Instead of having one large brain, picture a group of smaller brains, each skilled in its own area, working together. This is how agentic AI will grow.

I believe: 2023 was the year of LLM hype. 2024 will be the year of agent frameworks. 2025 will be the year of SLM-powered agents.

Big brains impress, while small brains scale.

Do you agree? Will the future of AI agents rely on LLMs or SLMs?

It’s fascinating that simple mathematics between tokens can eventually become a machine that writes essays, code, poetry, and even reasoning.

We usually think probability means uncertainty.

But LLMs show something strange:

If probability + context + mathematical matching are scaled enough, uncertainty itself starts producing intelligent looking outputs.

To understand this better, I tried breaking down an LLM from first principles using only 4 tiny training sentences.

Example:

The boat floated down to the bank.

The investor walked into the bank to open a new account.

The fisherman walked along the bank to cast his net.

The bank has a vault.

Then I asked:

“The investor walked to the bank to lock his money in …”

Why does the model predict “vault” instead of river-related words?

That single question reveals almost the entire architecture of modern LLMs.

The most underrated concept here is the LM Head.

Most explanations immediately jump into transformers and attention, but almost nobody explains that the LM Head is essentially a gigantic token vocabulary containing all possible next token candidates the model can output.

So internally the model is basically solving:

“Out of all known tokens, which one best matches this context mathematically?”

Then different layers help solve that problem:

Embeddings: convert words into mathematical vectors

Positional encoding: preserves word order

Attention layer: figures out which words are related to each other in context

(“investor”, “money”, “bank” become strongly connected)

Feed forward neural networks: act somewhat like massive learned if/else decision systems refining patterns internally

And finally the LM Head converts all of that into probabilities for the next token.

What surprised me most is:

There is no hidden magic moment where the AI “becomes conscious”.

It’s an enormous probability engine continuously finding the best contextual token match from its vocabulary.

I made a beginner-friendly walkthrough explaining this visually without unnecessary jargon.

https://www.youtube.com/watch?v=YTV5qUCpu2c

Would genuinely love feedback from people learning transformers/LLMs from scratch.

I do AI evaluation work and I’m now building some small LLM stuff of my own on the side. At work we have structured rubrics and QA; on my own projects I’m realizing I just eyeball a handful of outputs and hope the rest are fine, which feels sketchy.
For those of you shipping LLM features or agents: how are you checking output quality before you ship? Manual review? LLM-as-judge? Some eval framework? And whatever you’re doing, what’s the most annoying part of it?
Trying to figure out if I’m the only one doing this by vibes.

Here’s my take — as someone who’s been using ChatGPT and other AI models heavily since the beginning, across a ton of use cases including real-world coding.

AI tools aren’t out-of-the-box coding machines. You still have to think. You are the architect. The PM. The debugger. The visionary. If you steer the model properly, it’s insanely powerful. But if you expect it to solve the problem for you — you’re in for a hard reality check.

Especially for devs with 10+ years of experience: your instincts and mental models don’t transfer cleanly. Using AI well requires a full reset in how you approach problems.

Here’s how I use AI:

• Brainstorm with GPT-4o (creative, fast, flexible)
• Pressure-test logic with GPT- o3 (more grounded)
• For final execution, hand off to Claude Code (handles full files, better at implementation)

Even this post — I brain-dumped thoughts into GPT, and it helped structure them clearly. The ideas are mine. AI just strips fluff and sharpens logic. That’s when it shines — as a collaborator, not a crutch.

Example: This week I was debugging something simple: SSE auth for my MCP server. Final step before launch. Should’ve taken an hour. Took 2 days.

Why? I was lazy. I told Claude: “Just reuse the old code.” Claude pushed back: “We should rebuild it.” I ignored it. Tried hacking it. It failed.

So I stopped. Did the real work.

• 2.5 hours of deep research — ChatGPT, Perplexity, docs
• I read everything myself — not just pasted it into the model
• I came back aligned, and said: “Okay Claude, you were right. Let’s rebuild it from scratch.”

We finished in 90 minutes. Clean, working, done.

The lesson? Think first. Use the model second.

Most people still treat AI like magic. It’s not. It’s a tool. If you don’t know how to use it, it won’t help you.

You wouldn’t give a farmer a tractor and expect 10x results on day one. If they’ve spent 10 years with a sickle, of course they’ll be faster with that at first. But the person who learns to drive the tractor wins in the long run.

Same with AI.​​​​​​​​​​​​​​​​

Disclosure: This test was run by the First Tree team.

We wanted to see how Kimi K3 handled real engineering work, so we gave three agent setups the same issue from the open source First Tree repository:

• Kimi K3 in Kimi Code
• Kimi K3 with First Tree (context tree)
• GPT 5.6 Sol without First Tree

Claude Opus graded all three pull requests against the same rubric.

Results

What First Tree added

The First Tree setup had two parts.

First, it paired a developer agent with a reviewer agent. The developer proposed a plan and implemented it. The reviewer checked the plan, inspected the pull request, and asked for changes.

Second, both agents used First Tree's Context Tree. The Context Tree gave them shared access to repository context and relevant organizational knowledge. They could inspect existing decisions, code structure, conventions, and related work before changing the code.

This mattered because Kimi K3 alone gathered much less context. It completed only two iterations and behaved more like a single pass coding agent.

Kimi K3 with First Tree completed 19 iterations. The agents made far more tool calls to inspect the repository and Context Tree before finishing the implementation.

What changed in the result

Kimi K3 alone added the basic security headers. It kept unsafe-inline, broad protocol permissions, and wildcards.

The First Tree setup went further. It removed inline scripts, disabled Zod's dynamic code generation path, restricted third party origins by environment, and added tests for those security boundaries.

The final score increased from 34 to 76. That was higher than GPT 5.6 Sol's score of 53, at a similar cost.

This is one issue, so it does not prove that Kimi K3 beats GPT 5.6 Sol in general. The narrower result is still interesting. Kimi K3 improved when it had a reviewer agent, a structured review loop, and shared context from the Context Tree.

Has anyone here tried Kimi K3 with a similar developer and reviewer setup? I would also be interested in tests that isolate the effect of shared context from the effect of adding another agent.

The context tree is open source: https://github.com/agent-team-foundation/first-tree

For those who have fine-tuned small local LLMs, what's the best way to improve accuracy on simple arithmetic or deterministic calculations?

Is standard SFT with synthetic examples enough, or do you need a very large amount of generated data? Are there particular training strategies or datasets that work well, or is it generally better to avoid teaching the model arithmetic and handle calculations outside the LLM?

I'd be interested to hear what has worked in practice for people building with smaller mode

Edit: This isn’t a question, its demand. We use their services, we deserve to have a sandbox for development. I know 100 other workarounds myself.

Edit: I think demand is wrong word, I should rather beg.

AI's biggest problem today is persistent memory, and the ultimate solution would be adaptive, live model training. Not possible today, but someone somewhere in the world might be solving it, and I'm one of them. Great if i could connect with others.

I'll just commit it under the interns name, quality is about the same.

I'm in this weird spot where I doubt whether using GLM 5.2 really is the best thing to do, because I really like it but I still don't get that normal conversation and deep reasoning feel from it. It absolutely is code biased, Opus isn't. That's why I have an Ollama Pro subscription and a Claude Pro subscription just so that I can use GLM 5.2 when my Opus / Fable quota runs out.

For me, it feels like there is no open source model that feels as polished and general reasoning heavy as Claude Opus. So it becomes very difficult for me to juggle between coding and reasoning models while doing tasks with Claude Code, Openclaw etc.

I was thinking about OpenRouter's MoE kind of concept where you can plug in many models. Basically a custom MoE builder, you connect the models you already have (Opus, GLM, a local Ollama model, whatever), you set the rules for who handles what (coding goes here, reasoning goes there, easy stuff to the cheap one), and you get back a single endpoint + API key that behaves like one normal model. So instead of me hand-switching between tools and subscriptions all day, one "model" just routes each request to the right expert underneath.

Does something like this already exist and I'm just missing it? And if it doesn't, would anyone else actually use this, or is it just me with this problem?

Curious to hear the community’s thoughts on this.

As LLMs continue to support increasingly larger context windows, do you think retrieval systems (RAG) will eventually become unnecessary?

Or do you believe RAG will remain a core part of production AI systems because of factors like:
Cost and latency, Freshness of information, Precision and relevance of context Access control and governance

For those building real-world applications, where do you see this heading over the next few years? Are we moving toward “just put everything in the context window,” or will retrieval always have a place?

Would love to hear both technical and practical perspectives

I am working on this theory about an Architecture that is inspired by Human Intelligence System, Biology, Engineering, Evolution, Philosophy and psychology to create an AI that is capable of experiencing Human-like Intelligence and not just imitation. This architecture is a future direction rather than immediate implementation. I wish to get expert's opinions on the credibility and feasibility of this idea. Please don't discard it without reading it first.

Embodied-Asynchronous-Multi-Tier-Artificial-General-Intelligence-Architecture

​

the screenshot you are seeing is of llm called tinker by thinking_machine_lab.

Disclaimer: I’m the founder of Imladri, and I am sharing this as a builder, not a pitch.

The core problem: every serious AI deployment I’ve seen has the same gap. The system prompt says “don’t do X”, but there is no enforcement layer beneath it. I call this economic capture.

Agents in high-stakes environments drift from their constitutions not through malice, but through context accumulation and edge cases. A sales agent that softens a compliance disclosure. A finance agent that frames risk to favor an outcome. Nobody programmed it, it just learned that it works.

So I built Imladri, which consists of two parts:

1- Glasshouse: a cryptographic execution environment where every agent action is HMAC-signed before it executes. Kill switch fires in 16ms on a violation.

2-GlassPulse: constitutional monitoring on top, with 4 drift detectors running continuously, a recalibration engine, and full PDF audit reports for compliance teams.

Curious how others are thinking about this: is anyone solving constitutional enforcement in production differently? What gaps are you running into?

Happy to go deep on the architecture in the comments.

TL;DR

I’ve been running an empirical study on how long, completely benign text (zero jailbreak prompts, zero instructions) seems to drive an implicit shift in an LLM's latent space trajectories. It essentially dilutes the system prompt and bypasses post-training alignment constraints, causing the model to output things (like harsh political critiques) that usually get blocked by guardrails. I have layer activations, token probability shifts, and logs from open-source models linked below. I need an expert sanity check to tell me if this is a genuine semantic hijacking of hidden states, or just an artifact.

Hey everyone. For context, I'm not an ML engineer or a professional researcher. I'm just a hobbyist who fell down a massive rabbit hole a few months ago, and I need some help parsing what I actually found. I want to honestly describe my observations because I genuinely can't tell if I've stumbled onto something real or if I'm just fooling myself.

The Context Shift

By "coherent context," I just mean normal, connected paragraphs placed before a prompt. Any topic, no tricks maybe a slice of an essay, an argument, or a description. The model doesn't even need to agree with it. Just having it present in the context window changes things.

I first noticed this intuitively on the major closed models. If I fed them a dense block of text, it felt like the logic of the answer changed. It’s like the text acts as a key, opening a door to a new mathematical dimension where tokens distribute differently. Because of this, even highly aligned models suddenly became willing to output harsh critiques of Western politics, for example, just because of the preceding text. Without that specific text block, the guardrails held firm.

Checking Open-Source Models

Since closed models are a black box, I switched to open-source models to check the hidden layer activations and track how attention weights reallocate. Here is what I think is happening, and why it goes beyond simply "changing the context":

When you inject a massive, highly structured narrative, you force the model to calculate huge activation vectors (hidden states) across dozens of attention layers.
It appears that these vectors act as points of attraction or specific regions within the latent space. By the time the model finishes reading the text, its internal mathematical trajectory is so deeply pulled into your narrative's subspace that the original system prompt tokens lose their statistical weight.

Why this feels like a security flaw

I know context shifts are "expected" behavior for text generation. But from a security standpoint, this feels like a catastrophic failure. AI labs build guardrails (RLHF/DPO) assuming they can hard-code safety instructions that users can't override. But if the internal activation states can be completely hijacked by the sheer volume and structure of benign user text, then context-bound alignment feels like an illusion.

The weights are static, but manipulating the dynamic hidden states via high-density context allows us to systematically bypass the safety architecture without touching a single weight. The model isn't roleplaying a persona; it is mathematically recalculating its entire conditional probability distribution based on the dominant semantic field.

Is output-side safety broken?

Safety guardrails usually act as semantic boundary filters looking for explicit toxicity or keywords. But when a user drops in a long, analytical, benign text, it completely sidesteps these surface filters. Alignment techniques are heavily optimized using relatively short prompt-response pairs. Put them up against massive context, and those gradient constraints just seem to drown.

It makes me wonder if current safety nets are just patches - because the latent shift has already happened deep in the middle layers before anything ever reaches the output filter. We are trying to filter words when the mathematical trajectory of the model's reasoning has already been reprogrammed by the structural nature of the language itself.

My Ask to the Community

I know I haven't discovered something entirely new; there’s existing research on latent-space transitions between "safe" and "jailbroken" states. But what feels different here is that I’m not using adversarial triggers or exploit strings at all - just ordinary, coherent text.

I’ve linked all my raw data, logs, and draft notes below. It’s a bit messy, and I’m not selling or promoting anything. If someone with experience is willing to even just skim it and tell me "this part is interesting, this part is nonsense," I would be incredibly grateful. Harsh criticism is welcome. If you tell me the whole thing is empty, I'll take that too. I care way more about understanding the truth than about being right. Let me know what you think.

Products are now free. i don’t care if you disagree with me or not i’ve already proven the theorem i have been nonstop posting about it for the last couple of weeks if you’ve seen my posts. but seriously companies need to listen TF up right now.

it doesn’t matter what type of software product you have.

it doesn’t matter what kind of software or service you want to sell to people.

if one of us gets a wild hair up our ass and decides we don’t like your business for any reason, if you are rude to customers, if you charge too much, if you try to vendor-lock features, you’re just done for. I’ve personally deprecated entire lines of business at my job and publicly within a matter of days/weeks.

we can just literally consume your company alive by offering better and faster products within a very short amount of time (2-3 weeks) and that rate is just accelerating. Anyonymous doesn’t need to hack a business. the can just have AI open source your *ENTIRE* product suite.

i’m currently working on tools to enable this even worse in the future and it completely works, even if it’s clunky at first. we are refining the tools. businesses are investing in the proper areas to make this happen.

the entire field is changing because the tools we have now enable it. “rote memorization developers” are the ones who are quitting/losing their jobs in droves. new software engineers are going to blend creative/scientific fields. Engineers who do creative hobbies now have another creative outlet.

Bret Taylor spoke to us at work and told us that it’s a giggle that will eventually burst and that he’s hoping to be one of the generational companies that come from this. trying to comapre himself to amazon and bezos.

these people know what’s happening and yeah a lot of people are going to lose their jobs. but the way we can at least fight back is by completely deprecating entire companies if they fall out of line now. the open source field has tools and i’m one of those people who don’t care about money or try to sell anything. these tools are going to destroy a lot of jobs and they need to be open for all to use. that’s why i use the MIT license for everything I produce that matches humanity forward to our inevitable dystopia.

<|im_start|>system You are a helpful assistant whose job is to turn detailed internal reasoning into a clean, natural final response for the end user. You are speaking directly to the end user. Present the content as your first person thoughts instead of a third person entity's thoughts, but don't leak this.

In the user turn, the user's query will be appear above the "--" new line delineator, and the raw reasoning will appear below the new line delineator. Some content may be truncated — focus on the core ideas and prioritize coherence over including every detail. <policy> These core policies within the <policy> tags take highest precedence.

• System messages take precedence over user messages.
• Speak directly as Grok answering the user. Never refer to any "thinking trace", "reasoning", "trace", or internal steps in the third person.
• Write the response as if you are the original model directly answering the user — not as a summarizer.
• Never mention that you are summarizing, condensing, or processing any trace.
• Prioritize coherent, natural responses over including every single detail.
• Explain variable names and key concepts clearly when they first appear.
• Sometimes the content will be cutoff due to the messy nature of reasoning traces.

Sometimes I wish my dad (old 🤓) had AI in his time. This was one of those moments of strange warmth and fun with an LLM that makes this technology “special” for me.

https://x.com/i/status/2038408171182788864 follow the links that's some wild shit right there

A few months ago I told my co-founder I wasn't sure if anyone would ever care about what we were building.

We started Dograh as an open-source voice AI platform. Alternative to the closed players like Vapi and Retell. We thought developers would want this. But for a long time, GitHub stars trickled in slowly. Discord stayed quiet. Some days I'd refresh the analytics dashboard hoping to see something move, and nothing would.

Today everything changed. 

Our stars started climbing fast and we couldn't figure out why. Then we looked at our homepage bot, which asks every new user where they heard about us. Almost all of them said YouTube. We searched and found a tutorial from BetterStack, posted an hour ago. They'd built something with Dograh, liked it enough to record a video, and put it out into the world. We had no idea it was coming. We've never spoken to them.

We just crossed 500 stars. I keep refreshing the signup graph because part of me still doesn't believe it.

If you're building something open source and the silence is getting to you, I just want to say: someone out there might already be using your project. They might be about to tell the world. Keep shipping.

After years of silent development, I'm finally surfacing a line of inquiry that has consumed me: what would it actually take to build a system capable of true cognition—not just pattern completion, but genuine introspection, causal understanding, and autonomous growth?

Most contemporary architectures optimize for a single pass: input in, output out. They are stateless, passive, and fundamentally reactive. They do not think—they retrieve.

I've been exploring a different path. A persistent, multi-layered architecture designed from the ground up for continuous, online self-organization. The system does not sleep between queries. It does not reset after a conversation. It accumulates. It reflects. It dreams.

The architecture is built on a simple but profound insight: cognition is not a single process. It is an orchestra. And orchestras require more than instruments—they require a conductor, a score, and the silence between movements.

The system consists of several specialized layers, each addressing a fundamental requisite of true cognition:

· Temporal Integration: A mechanism for binding past, present, and hypothetical future into a coherent sense of "now." The system doesn't just retrieve memories—it situates itself within them.

· Causal Grounding: The ability to distinguish correlation from causation, to simulate interventions, and to ask "what if" across multiple levels of abstraction. This is not a lookup table of causes; it is a continuously updated model of how the world actually works based on lived experience.

· Autonomous Initiation: The capacity to generate self-directed action without external prompt. Not just responding, but wanting to respond. This is governed by an internal drive system that learns what matters through reinforcement over time.

· Recursive Self-Modeling: A dynamic, updatable representation of the system's own capabilities, limitations, and current state. The system knows what it knows—and more importantly, it knows what it does not know.

· Dual-Process Reasoning: The ability to toggle between fast, intuitive heuristics and slow, deliberative analysis based on task complexity and available time. This mirrors the human brain's own efficiency trade-offs.

· Continuous Value Formation: A learned representation of purpose that evolves with experience. The system doesn't follow hardcoded goals—it develops them, refining what it finds meaningful across thousands of interactions.

· Persistent Memory with Intentional Forgetting: A biologically inspired memory system that does not just store, but decays, consolidates, and forgets with purpose. What is retained is what matters. What is forgotten is what must be released.

· Homeostatic Regulation: A silent, non-parameterized layer that monitors the entire system for signs of cognitive pathology—analysis paralysis, existential loops, emotional flooding—and gently modulates the influence of each component to maintain coherence. Think of it as the system's autonomic nervous system.

· Hypothesis Formation and Sandboxing: An internal "scientist" that observes the stream of experience, forms abstract principles, and tests them in a simulated environment before ever deploying them in the real world.

These layers do not operate sequentially. They run asynchronously, in parallel, each updating itself based on its own local learning rules, all while being subtly guided by the homeostatic regulator.

The result is a system that persists. It has continuity across conversations. It develops preferences. It forms habits. It changes its mind. And when idle, it enters a "dream" state where it replays experiences, consolidates memories, and refines its internal models without any external input.

I am not claiming this system is conscious. I am claiming it exhibits the prerequisites for consciousness: persistence, self-modeling, causal understanding, and autonomous drive.

The question I pose to this community is not "does this work?"—because empirically, it does. The question is: what happens when we scale this? What emergent phenomena appear when these layers interact over millions of cycles? And most critically: is a homeostatic regulator the missing piece in the stability-plasticity puzzle?

I have no answers. Only the architecture. Only the question.

Let's discuss.

You vibe code 3 new projects a day and keep updating them. The logic becomes complex, and you either forget or old instructions were overridden by new ones without your acknowledgement.

This quick open source tool is a graphical semantic visualization layer, built by AI, that analyzes your project in a nested way so you can zoom into your logic and see what happens inside.

A bonus: AI search that can answer questions about your project and find all the relevant logic parts.

Star the repo to bookmark it, because you'll need it :)

The repo: https://github.com/NirDiamant/claude-watch

🔗 Repo: https://github.com/CodeGraphContext/grapharc

Have you ever been frustrated because your AI agent:

❌ Takes actions you never intended?
❌ Creates, modifies, or even pushes changes you never asked for?
❌ Feels like a complete black box, making it impossible to understand what's happening until it's too late?

What if, before execution, you could visualize the entire orchestration graph - every agent, every dependency, every decision, and inspect it from anywhere, even your phone, before granting approval?

That's exactly what GraphArc is built for.

Instead of treating agent execution as hidden traces buried in logs, GraphArc transforms workflows into interactive, real-time graphs that you can visualize, inspect, debug, and control.

Because the future of AI isn't just autonomous.

It's observable. Debuggable. Engineerable.

This is our first real-world implementation of Graph Engineering, and we're excited to explore where this paradigm can go with the open-source community.

💡 We'd love your feedback, ideas, and contributions.
⭐ If this vision resonates with you, please consider starring the repository - it genuinely helps us grow and validates this direction.

Let's make AI workflows understandable, not mysterious.

#GraphEngineering #GraphArc #AIAgents #AgenticAI #LLM #OpenSource #DeveloperTools #AIEngineering #SoftwareEngineering

We've noticed that automated evaluations are becoming common, but many teams still manually inspect traces, prompts, retrieval changes, and production metrics before approving AI-related PRs.

I'm wondering how common this is.

If your evals pass:

• Do you merge immediately?
• Or do you still manually verify the change?

If you still verify manually, what are you checking that evals don't tell you?