How I Built a Self-Healing Code Pipeline with NATS, Claude AI, and A/B Testing

I run about 30 projects from a home lab. Two servers, a handful of cron jobs, a few web apps, a couple of bots. The standard solo-founder setup. The problem is not building these things. The problem is that something breaks every day and I do not find out until I check manually, which means some things stay broken for days.

So I built PrimeBus: a system that watches all of my projects for changes, detects when something breaks, generates two competing AI-powered fixes, scores them against each other, and either merges the winner or escalates to a human when it is not confident. The entire pipeline runs autonomously on NATS JetStream with Claude as the AI backbone.

This is not a theoretical architecture. It has been running in production since late March 2026, processing hundreds of events per day across real codebases.

The Problem: 30 Projects, One Engineer

Here is a typical week before PrimeBus existed. My Twitter bot stops posting because a dependency updated and broke an import. I do not notice for two days. My YouTubeShorts pipeline silently fails because a thumbnail API changed its response format. My trading scanner logs an error at 3 AM and nobody sees it until the next morning ops check.

The standard solution is monitoring and alerting. I already had that. Prometheus, Grafana, Uptime Kuma. But monitoring tells you something is broken. It does not fix it. I still had to context-switch from whatever I was building, read the logs, understand the failure, write a fix, test it, and commit it. That cycle eats hours, and for a solo operator those hours come directly out of building new things.

What I wanted was a system that could do the full loop: observe, diagnose, fix, validate, and either commit the fix or ask me for help when it was not sure.

The Architecture: Signal, Detect, Resolve

PrimeBus has three stages. Every piece of telemetry flows through NATS JetStream as a structured event.

Signal. Watcher agents poll git repos every 30 seconds, check PyPI for dependency updates daily, and listen for application lifecycle events published by external programs. When a new commit lands, the GitWatcher publishes a change.repo.{project}.push event. When a dependency has a new version, the DepWatcher publishes change.dep.pypi.{package}. External applications like my Twitter bot publish their own events on the app.{name}.* namespace.

Detect. A TestRunner subscribes to push events, checks out the commit in an isolated git worktree, and runs the project's test suite. If tests fail, it publishes a digest.alert event with the failure details. A PerformanceAnalyzer watches application telemetry and uses Claude to detect anomalies in metric patterns.

Resolve. This is where it gets interesting. When a test failure is detected, the FixGenerator fires two parallel Claude API calls with different strategies.

The A/B Code Tournament

This is the core idea that makes PrimeBus different from a simple "AI writes code" pipeline. Every fix attempt produces two competing variants:

• Variant A uses a minimal strategy. Low temperature (0.2), prompt tuned for the smallest possible change that makes tests pass. Think: change one line, fix the import, update the version string.
• Variant B uses a robust strategy. Higher temperature (0.7), prompt tuned for a thorough fix that addresses root cause. Think: refactor the fragile code, add error handling, fix the underlying design issue.

Both variants run concurrently via asyncio.gather. Each one gets applied to its own git worktree, and the test suite runs against it. The ValidationHarness scores each variant on three dimensions:

• Test pass rate (60% weight) — Did the tests actually pass?
• Diff size (25% weight) — Smaller diffs are preferred. Less risk, easier to review.
• Clean apply (15% weight) — Did the diff apply without conflicts?

The ScoringEngine collects both scores, picks the winner, and makes a decision. If the winner scores 75 or above with confidence at or above 0.7, it creates a git branch with the fix committed. If not, it escalates.

The Escalation Path: HumanRail

This is the part that most "autonomous AI" systems skip. What happens when the AI is not confident?

PrimeBus integrates with HumanRail, a human-in-the-loop task routing API I built separately. When a fix does not meet the confidence threshold, PrimeBus creates a task in HumanRail with the failure context, both variant diffs, and the scoring breakdown. A human reviewer (usually me) can approve, reject, or provide guidance.

This is not a fallback bolted on as an afterthought. The escalation path was designed from Sprint 2. The system knows its own limits and routes around them. Every escalation outcome feeds back into the prompt tuning for the next generation of fixes, so the system gets better over time.

Why NATS JetStream

I evaluated Kafka, RabbitMQ, and Redis Streams before settling on NATS JetStream. The decision came down to three things:

• Operational simplicity. NATS is a single binary. My entire event bus runs in a Docker container with 128MB of RAM. Kafka would need Zookeeper and a JVM. For a solo-operator system processing hundreds of events per day, NATS is the right tool.
• Durable consumers. JetStream gives you at-least-once delivery with durable consumer groups. If PrimeBus crashes at 3 AM and restarts at 3:01 AM, it picks up exactly where it left off. No events lost.
• Subject-based routing. NATS subjects like change.repo.*.push let agents subscribe to exactly the events they care about using wildcards. Adding a new telemetry source is just publishing to a new subject. No queue configuration, no topic creation, no admin work.

The architecture uses two isolated NATS instances. HumanRail runs its own NATS on port 4222 for task routing. PrimeBus runs a separate instance on port 4223 for telemetry. A lightweight bridge service forwards relevant events between them.

What Actually Runs

Here is the real agent lineup running in production right now:

• GitWatcher — Polls 4 repos every 30 seconds for new commits
• TestRunner — Runs test suites in isolated worktrees on every push
• DepWatcher — Checks PyPI daily for dependency updates across all projects
• DepUpdater — Auto-bumps dependencies, validates in worktree, creates branch
• FixGenerator — Two-variant Claude API calls when tests fail
• ValidationHarness — Applies diffs to worktrees, runs tests, scores results
• ScoringEngine — Compares A vs B, picks winner or escalates
• PRCreator — Creates git branches from winning fixes, or escalates to HumanRail
• FeedbackAgent — Tracks outcomes (merged, reverted, overridden) and feeds back into prompts
• DigestAgent — Daily summary to Discord with trend analysis
• DiscordAlerter — Real-time alerts on failures and escalations
• PerformanceAnalyzer — Claude-powered anomaly detection on application metrics
• ReactorAgent — Event-driven Claude CLI dispatcher with risk-tiered gates

All agents run in a single Python process managed by systemd. Total memory footprint: about 50MB. Prometheus metrics are exposed on port 9200, scraped every 15 seconds, and visualized in Grafana.

External Publishers: Opening the Bus

PrimeBus is not just an internal system. Any application on my network can publish events to the bus. The pattern is simple: a thin adapter module (about 80 lines of Python) that connects to NATS and publishes structured events. The adapter is self-contained, no import from PrimeBus code, feature-flagged via a PRIMEBUS_ENABLED environment variable, and has a 3-second timeout with all exceptions caught so it never breaks the host application.

My Twitter bot was the first external publisher. It emits 7 event types: content posted, content failed, reply-guy run complete, own-replies complete, metrics generated, and run skipped. YouTubeShorts publishes upload success and failure events. These events flow into the PerformanceAnalyzer, which watches for patterns like "this bot failed 3 times in a row" or "upload latency doubled compared to last week."

Current throughput: about 280 events per day. 180 from external publishers, 100 from internal agents. Well within the system's capacity.

The Feedback Loop

The part that makes this a learning system instead of a static pipeline is the feedback loop. Every fix outcome is tracked in SQLite: was the fix merged, reverted, or overridden by a human? What strategy won (minimal vs robust)? What was the confidence score?

This history gets injected into the fix generation prompts. When PrimeBus generates a fix for a project, it includes context like "the last 3 fixes for this project that succeeded used the minimal strategy" or "robust fixes in this codebase tend to score higher." The system does not just try to fix code. It remembers what worked before and biases toward those patterns.

What I Learned Building This

A/B beats single-shot every time. Before I added the tournament pattern, AI-generated fixes had about a 40% success rate. With two competing strategies, the pipeline finds a good fix about 65% of the time. The minimal strategy wins most often for simple failures (import errors, version bumps), but the robust strategy wins decisively for logic bugs and structural issues.

Telemetry-first design matters. I originally built PrimeBus as a test runner that happened to use NATS. When I refactored it around telemetry collection as the primary job, everything got simpler. The bus exists to collect signals. Acting on those signals is just one of many possible subscribers. This made it trivial to add the PerformanceAnalyzer, the ReactorAgent, and the DigestAgent without touching the core pipeline.

Human escalation is not a weakness. The fact that PrimeBus escalates to a human when it is not confident is a feature, not a limitation. Every competitor in this space either runs autonomously with no escape hatch (dangerous) or requires human approval for everything (slow). The tiered confidence approach means the system handles the easy stuff automatically and only asks for help on genuinely hard problems.

NATS JetStream is underrated. For systems that process hundreds to thousands of events per day, NATS JetStream is dramatically simpler to operate than Kafka while providing the same durability guarantees. The entire event infrastructure runs in a single container with no external dependencies.

The Competitive Landscape

I researched what exists in this space before building PrimeBus. The short answer: no single product builds the full pipeline.

• Renovate and Dependabot handle dependency updates but nothing else
• Devin and SWE-agent generate code but require human triggering and do not do A/B validation
• Datadog and Grafana Cloud observe but do not act
• AlphaCode proved the A/B variant pattern works but it was never productized for general code changes

The gap is the orchestration layer connecting telemetry collection to change detection to AI code generation to A/B validation to feedback loops. That is what PrimeBus fills.

What Is Next

The immediate roadmap is expanding telemetry sources. Infrastructure monitoring via node_exporter anomaly detection. API changelog watching for services like Stripe and GitHub. Cron health monitoring across the entire network. Log aggregation from all projects.

The longer-term vision is multi-repo orchestration. When a change in a shared library triggers downstream test failures, PrimeBus should be able to trace the dependency chain, generate fixes in the affected repos, and validate them together. That is a hard problem, but the event bus architecture makes it possible.

Try It Yourself

PrimeBus is currently an internal tool, but the architecture is reproducible. You need NATS JetStream (free, open source), a Claude API key, and a few hundred lines of Python. The A/B tournament pattern alone is worth implementing even if you do not build the full pipeline.

If you are running multiple projects and want help building autonomous monitoring and self-healing infrastructure, get in touch. This is exactly the kind of system Prime Automation Solutions builds for clients.