The Self-Healing Chatbot: From Widget to Production LLMOps

The Genesis#

3 days after the first commit, someone tried to hack the chatbot. No defense. No logs. No tests. Just 80 lines of code and an exposed system prompt. That changed everything.

I'd spent 16 years building systems that run themselves. First in a repair shop. Now in AI. The idea was simple: a portfolio that demonstrates, not describes. The first commit was January 26, 2026: 50 lines of React and 30 of edge function. Claude Sonnet, SSE streaming, no state.

The original chat.js — the entire "architecture" fit in one function

// api/chat.js — Day 1 (Jan 26, 2026)
export default async function handler(req, res) {
  const { messages } = await req.json()
  const response = await anthropic.messages.create({
    model: 'claude-sonnet-4-5-20250929',
    max_tokens: 500,
    system: 'You are Santiago, an AI PM...',
    messages,
    stream: true,
  })
  // Stream SSE to client
  for await (const event of response) {
    res.write(`data: ${JSON.stringify(event)}\n\n`)
  }
}

The Evolution#

Day 1 vs Today

Architecture#

The system has 5 layers. Each was added when the previous one revealed a problem it couldn't solve alone.

Request lifecycle

Tech Stack

Agentic Observability#

Agentic observability means tracing every autonomous decision in an AI pipeline, not just what went in and what came out. Standard LLM observability tracks what went in and what came out. I track every decision the system makes on its own. When a user asks about Jacobo, Langfuse captures 6 generation observations: Claude choosing to search (Sonnet, 200ms), the embedding (OpenAI, 200 tokens), retrieval (pgvector, 10 chunks), Haiku reranking the top 5 (50 tokens out), the final response (Sonnet, 800ms), and quality scoring (Haiku, 0ms added). Each observation carries model ID, real token counts, and calculated cost. A custom ops dashboard aggregates all of this: conversations, costs per span, RAG accuracy, security funnel, eval pass rates, voice analytics, prompt versions, and system health.

How It Was Built: The MMA Loop#

Think of the chatbot as an employee. Cost tracking tells you how much each conversation costs. Online scoring tells you how well it's performing in real-time. CI gate prevents bad changes from reaching production. Trace-to-eval turns today's errors into tomorrow's tests.

The progression was deliberate — the MMA Loop: Measure, Manage, Automate. First you measure, then you manage what you measure, then you automate what you manage. It's the same pattern I used to systematize a physical business, applied to LLMOps.

Agentic RAG#

Why Agentic

In classic RAG, every message goes through the search pipeline. In agentic RAG, Claude decides when to search using tool_use (documented in Anthropic's API as tool_use). "What's your name?" doesn't need to search 56 chunks. "What stack did you use for programmatic SEO?" does. Result: ~60% of conversations don't trigger RAG (measured in Langfuse), saving latency and cost.

Hybrid Search

70% semantic (pgvector with OpenAI embeddings) + 30% keyword (Supabase full-text search, BM25-equivalent), following the hybrid retrieval pattern from RAG research. Embeddings capture meaning; keywords capture proper nouns and technical terms that embeddings sometimes miss.

Re-ranking + Diversification

Haiku selects the top-5 most relevant chunks from the top-10 by ranking. Then diversifyByArticle ensures each distinct article has at least one representative in the final context, preventing any single article from dominating.

Graceful Degradation

The chatbot can answer about Jacobo, Business OS, Programmatic SEO, and n8n for PMs — just ask.

6-Layer Defense#

The 71 Tests#

Examples by category

Assertion Types

70% deterministic (contains, regex, word count) — fast, reproducible, zero LLM cost. 30% LLM-judge (Haiku evaluates quality, tone, relevance) — for responses where there's no "correct" answer but a quality spectrum.

The Closed Loop#

Most LLM applications send a prompt and pray. This chatbot closes the loop.

The 6 Stages

The arrows returning to CI demonstrate that the system feeds itself.

Prompt Versioning + Regression

The system prompt lives in Langfuse as a prompt registry. Each change syncs with hash-based detection (only uploads if changed). Before promoting a new version to production, prompt:regression compares v1 vs v2 responses on the same inputs — human decision, not automatic.

The Developer Feedback Loop

A developer feedback loop is when the AI coding tool that built a system can also diagnose and fix it using production data.

The closed loop extends to the development process itself. Claude Code queries production traces in Langfuse, diagnoses issues in the RAG pipeline, and generates the fix.

In one session, it found that a RAG query had confirmation bias. The search used "n8n for product managers" instead of just "n8n", missing relevant chunks. It proposed the fix and generated an eval to prevent regression.

AI maintaining AI. The chatbot runs in production, Langfuse captures every decision, Claude Code reads the traces and adds a test. The system improves without me touching it.

Real Cost#

Breakdown by span

From Text to Voice#

Everything you just read — RAG, defense, closed-loop — works the same when you speak. Voice is a wrapper around the intelligence that already exists.

Voice Architecture

Shared Intelligence

Voice mode uses the same agentic RAG, the same 6 defense layers, the same closed-loop. The difference is format: no markdown, short sentences, Castilian accent.

The experience is omnichannel. Conversation history persists across modes: ask something via text, switch to voice to go deeper, switch back without losing context. Source badges appear in both modes, deep-linking to the articles mentioned.

Constraints

Lessons#

Frequently Asked Questions#

Did you hear that?