The Problem

Code-generating agents confidently produce output that does not compile, references packages that do not exist, and breaks its own files on the next pass. An LLM reviewing an LLM is not a check.

The Solution

A 10-stage multi-agent pipeline (requirements interview, architecture, code generation, security review, QA, deploy) that turns a plain-English brief into a deployed full-stack app. Around 15.7k lines of async Python. Each stage routes to a different model across OpenAI, Anthropic, and Gemini to control cost.

Engineering Decisions

Built a Code Integrity Engine of deterministic, LLM-free checks: AST symbol resolution, dangling foreign keys, hallucinated packages verified with real pip installs, esbuild parse checks, duplicate routes. Each check sends a targeted repair back to the agent and retries. Every check came from a real failed run and is locked in with a regression test.

Key Results

• 55+ documented defects diagnosed and closed across 20+ paid end-to-end runs
• Caught a security reviewer whose own auto-fixes rewrote certified files and quietly put bugs back in. Fixed with a gate that re-runs the original build checks on any post-certification change
• Per-stage model routing across three providers to control cost

Stack

The Problem

Teams use LLM-as-judge evaluators to sign off on agent output. Nobody checks whether the judge actually catches real mistakes or just flags surface patterns.

The Solution

Built an issue-triage workflow on self-hosted Sim, deployed it as an API, and scored it on 536 real issues from Sim and Kubernetes against maintainers' own labels. Dev/test split and an always-guess-the-majority baseline for every field.

Engineering Decisions

Mutation-tested code checks against Sim's LLM-judge Evaluator across 12 injected error types. Code checks caught 100% of invented numbers and broken structure but 0% of summaries with flipped meaning. The judge caught 88 to 100% of injected errors yet only 41 to 54% of the model's real mistakes. Blind hand review showed most judge flags on correct outputs were false alarms (0 of 8 flagged summaries were wrong).

Key Results

• 536 real issues scored across Sim and Kubernetes repos
• Reported 2 bugs upstream (#8353, #8354), both fixed and merged by Sim's co-founder
• Docs PR #8448 correcting the Evaluator block's documented settings (open)
• Calibrated prompt raised Kubernetes priority accuracy from 24% to 33% and within-one-level accuracy from 70% to 87%, stopping there to avoid fitting the test set

Stack

The Problem

When an LLM extracts audit claims from SEC filings, every number and quote it cites has to be verifiable against the source. A fabricated quote hidden inside a correct-looking claim ships silent errors downstream.

The Solution

A harness that checks every citation an LLM attaches to claims extracted from 30 real SEC Item 9A filings. Deterministic checks that never call a model. All 30 filings hand-labeled so the answer key was not written by the system under test.

Engineering Decisions

Mutation-tested the checker: 92% of 539 injected defects caught, 100% of fabricated quotes and flipped values. Gated in CI at --fail-under 0.85. Found 9 defects in the checker itself against 3 model errors, including one that produced a false 23% claim-error rate when the real figure was 10%.

Key Results

• 0 fabricated quotes across 354 cited fields
• 98.9% citation integrity measured end-to-end
• 9 checker bugs caught against 3 real model errors
• 92% of 539 injected defects caught by mutation tests
• CI gate at --fail-under 0.85

Stack

The Problem

In emergency triage, an under-rated urgency level can be fatal. Generic LLM-based triage tools give confident answers with no structural guarantee against incorrect urgency assignments.

The Solution

An ESI triage system where a deterministic Python rules engine (68 YAML-encoded clinical rules with handbook citations) makes every urgency decision. The LLM is barred from choosing the level and only extracts typed facts (a 95-field Pydantic v2 contract) and explains the result.

Engineering Decisions

Reduced dangerous under-triage from 20% to 10% by tuning extraction prompts against a 50-case suite built from ESI handbook practice cases, tracking exact and tolerant accuracy, under- and over-triage, and latency p50/p95. Published medeval-harness v0.1.0 on PyPI, a CLI eval tool with --fail-under gating and standard exit codes, plus a GitHub Actions workflow that fails the build when accuracy drops below 60%.

Key Results

• Under-triage rate reduced from 20% to 10%
• 68 YAML-encoded clinical rules with handbook citations
• 95-field Pydantic v2 contract for extracted facts
• medeval-harness v0.1.0 published on PyPI
• CI gate fails the build below 60% accuracy

Stack