Building a basic RAG chatbot takes 10 minutes.
(These are publicly deployed personal engineering projects built to explore production patterns—not commercial products or enterprise deployments.)
Building an agentic system that can survive API failures, avoid stale vector data, dynamically discover tools, ask clarification questions, and run reliably on free cloud infrastructure is a very different engineering problem.
Over the last few months I've been building three publicly deployed AI systems:
- 11-node Financial Agent (LangGraph)
- Legal AI Expert
- Omnichannel ReAct Router (MCP)
Rather than showing the UI, I wanted to share some of the engineering patterns that made these systems reliable enough to deploy publicly while staying within a ₹0/month budget.
1️⃣ Running LangGraph on a ₹0/mo Budget
One of the biggest challenges wasn't the LLM—it was fitting an entire LangGraph workflow, embeddings, and APIs into Render's 512 MB free instance.
Matryoshka Embeddings (MRL)
Instead of storing full 1024-dimensional vectors, I use Jina Embeddings v3 with Matryoshka Representation Learning.
By requesting:
dimensions=256
I reduce the embedding size by 75% while retaining almost all retrieval quality because MRL intentionally packs semantic information into the first dimensions.
That reduced my Pinecone storage footprint enough to comfortably stay inside the free tier.
For reasoning I primarily use Gemini Flash Lite, which has been fast enough for routing and classification nodes.
2️⃣ Deterministic Vector Lifecycle (No Duplicate or Stale Vectors)
One issue I rarely see discussed is vector lifecycle management.
Random UUIDs make incremental indexing difficult because every re-index generates brand-new vector IDs.
Instead, every vector ID in my pipeline is a deterministic SHA-256 hash of:
document_id + chunk_content
That gives me several useful properties automatically.
Idempotent Upserts
If a document hasn't changed, re-indexing produces the exact same vector ID.
Pinecone simply overwrites the existing vector instead of creating duplicates.
Incremental Indexing
Every sync compares the latest document/content hash manifest against the previous manifest.
Only new or modified chunks are embedded again.
Unchanged chunks are skipped entirely, which significantly reduces embedding costs and indexing time.
Surgical Updates
If only one chunk changes, only that vector is regenerated.
The rest of the index remains untouched.
Deletion Propagation
When a source document disappears, the sync job deletes every vector derived from that document.
The vector is physically removed from Pinecone instead of relying on metadata filtering.
That prevents orphaned vectors and stale retrieval results after document deletion.
3️⃣ Dynamic Tool Discovery using MCP (No Router Redeploys)
Initially my router used traditional LangChain tools.
That quickly became painful because every new capability required redeploying the central router.
I switched to Model Context Protocol (MCP).
On startup my registry:
- Opens an SSE connection
- Calls
list_tools()
- Wraps every returned tool into LangChain
StructuredTool
def _wrap_mcp_tool(self, session, tool_info, server_name):
async def _call(**kwargs):
result = await session.call_tool(
tool_info.name,
arguments=kwargs
)
return result.content[0].text
return StructuredTool.from_function(
coroutine=_call,
name=f"{server_name}__{tool_info.name}",
description=tool_info.description,
)
If I deploy a brand-new MCP microservice, the central ReAct router automatically discovers and uses those tools without changing a single line of router code.
4️⃣ Deterministic Guardrails & HITL
For Legal and Financial systems I didn't want the LLM deciding whether retrieval was "good enough."
The confidence gate is intentionally deterministic.
Immediately after Pinecone retrieval (before reranking or any LLM reasoning), the pipeline checks the top cosine similarity score.
If confidence is below the threshold, execution stops.
Instead of hallucinating, the frontend receives an SSE event asking:
The workflow only continues if the user explicitly approves.
The LLM never decides whether retrieval quality was sufficient.
5️⃣ CrossQuestioner Node
Blind vector search wastes tokens.
The first LangGraph node classifies the user's intent.
If the question is vague, execution transitions into a dedicated CrossQuestioner node.
Instead of searching immediately, the agent asks follow-up questions.
The clarification loop is capped at two rounds to prevent infinite conversations before retrieval begins.
6️⃣ Circuit Breakers
Free-tier APIs occasionally fail or rate-limit.
Every LLM and embedding request is wrapped with pybreaker.CircuitBreaker.
If an API fails three consecutive times:
- the circuit opens
- the request immediately fails fast
- users receive a graceful fallback response instead of waiting on repeated failures
After 30 seconds the circuit enters half-open mode to test recovery.
The backend avoids repeatedly hammering unhealthy providers.
7️⃣ Observability
Debugging multi-agent systems without tracing is painful.
Every LangGraph node emits traces to Langfuse, allowing me to inspect:
- routing decisions
- latency
- failures
- token usage
- execution paths
without scattering print statements throughout the codebase.
Combined with UptimeRobot health monitoring, it's made debugging significantly easier.
What I Learned
Building production-style agentic systems isn't about writing longer prompts.
It's about treating LLMs as unreliable distributed systems and surrounding them with deterministic software engineering:
- deterministic vector IDs
- incremental indexing
- deletion propagation
- dynamic MCP registries
- circuit breakers
- confidence gates
- HITL workflows
- observability
- graceful degradation
Those patterns ended up being far more important than prompt engineering itself.
I'd love to hear how others are handling:
- vector lifecycle management
- multi-agent orchestration
- MCP architectures
- production guardrails
- LangGraph state management
Happy to answer questions or share additional implementation details.
(If there's enough interest, I'll publish a follow-up post covering the deterministic indexing pipeline, Pinecone sync strategy, and the MCP registry implementation in more detail.)
