When You Can't Use RL: Optimizing Prompts with GEPA

TL;DR

• An agentic system doesn't run on weights you can train — it runs on prompts you wrote by hand.
• The textbook way to make a policy better is reinforcement learning, but in a production agent built on a hosted model, RL is mostly off the table: no weight access, thousands of paid rollouts, and a reward signal you'd have to invent.
• So we used GEPA — DeepEval's implementation of a reflective, evolutionary prompt optimizer originally proposed by Agrawal et al., 2025 — which treats each prompt as the thing to improve and scores it against a golden dataset.
• We ran it on five live engines. Two big wins — disambiguation 43% → 85%, intent classification 88% → 99%.
• Just as valuable: two honest non-results that told us exactly what prompt optimization can and cannot fix.

The Thing Nobody Tells You About Agents

When people say "AI agent," they picture one clever model making decisions. In practice, our shopping assistant is not one model — it's five separate LLM engines chained together, and each one is driven by a prompt someone on the team wrote by hand:

• an intent classifier that routes a message into one of ~15 intents,
• a disambiguation engine that decides whether to ask a clarifying question,
• an escalation engine that decides whether to hand off to a human,
• a RAG grounding step that decides if a question is in-scope and answerable from real facts,
• a product curation prompt that turns retrieved candidates into recommendations.

Every one of those is a policy. And every one of those policies is a wall of English that grew organically — a rule added here after a bad demo, a caveat bolted on there after a complaint. Nobody could tell you whether the last edit made the engine better or worse. We tuned prompts by vibes and shipped on faith.

That's the real problem with agentic systems. Not that they're hard to build — that they're hard to improve on purpose. You can't improve what you can't measure, and you can't measure a prompt by staring at it.

The Obvious Answer Is RL — and Why We Couldn't Use It

The instinctive answer for "make this policy better against a reward" is reinforcement learning — it's how base models get aligned (RLHF). We considered it seriously, and the deciding factor was cost:

• Rollouts are expensive. RL is sample-hungry — it learns from thousands to millions of trial episodes. Here, every episode is a paid model call, and scoring it often means a second paid LLM-judge call. Burning that many rollouts to nudge one prompt isn't economics that works.
• The reward doesn't exist yet. "Was this recommendation good?" isn't a number the environment hands you — we'd have to build and maintain a separate reward model first, its own labeling project.
• Risk and opacity. RL fine-tuning can quietly degrade what a model was already good at (catastrophic forgetting), with no explanation of what changed — not shippable for a system talking to real customers.

RL isn't the only way to improve a prompt, either — it's just the first instinct that comes to mind. We picked GEPA specifically for its reflection step: it reads a failure in plain English and rewrites from that diagnosis directly, which is what gives it fast convergence.

The reframe: in an agentic system, the policy is natural language. Each engine's behavior comes from a prompt we fully own and can change instantly — so instead of optimizing the model's weights, we optimize its words. That's a search problem we can run cheaply, with no gradients and full interpretability.

That's GEPA.

What GEPA Actually Is

GEPA — Genetic-Pareto prompt evolution — treats a prompt like any candidate solution: score it, find where it's wrong, propose a better one, keep it only if it truly wins, repeat. We used DeepEval's GEPA optimizer as the engine for that loop — including its Pareto-frontier candidate selection, which keeps a diverse pool of non-dominated candidates (higher accuracy, shorter prompt) each generation instead of collapsing to one "best" guess too early — wired into our own production harness so it optimizes against the model actually running in production.

How it works, per generation:

1. Measure — score the current prompt against the golden dataset.
2. Find failures — collect every case it gets wrong.
3. Reflect & rewrite — an LLM reads those failures and proposes a targeted rewrite.
4. Re-score — run the candidate against the full dataset.
5. Keep the winner — only if it measurably beats the current best.
6. Repeat for several generations.

Why this beats RL here: RL's signal is a scalar reward — one number per episode, leaving the algorithm to infer what to change through many noisy steps. GEPA's signal is the model reading its own mistakes in plain language — "you asked a clarifying question on 'gym bag ideas?' — that was already specific enough to search" carries the diagnosis and the fix in one shot. That's why GEPA converges in tens of rollouts where RL needs thousands.

Run setup: 5 generations, 3–5 candidates per generation, using the same model that runs in production as the generator — so we're optimizing for the model that will actually execute the prompt.

The Hard Part: Making "Better" Mean Something

A sloppy fitness function just teaches the optimizer to memorize test cases. Most of our engineering went into the harness that decides whether a rewrite is genuinely better or just lucky.

Golden datasets as the fitness function. Hand-authored, not sampled raw from logs — every case is tagged by category (e.g. should_ask, offtopic, gift) and carries an explicit correctness criterion, so the dataset doubles as a spec. Since the golden set is the fitness function, a rewrite can only get credit for failure modes the dataset actually contains. "Correct" differs per engine:

A judge that isn't a pushover. Curation needs a judge model, and its identity matters: same-family judging (a model judging its own family) is a documented way to inflate scores — a 2025 self-bias study (Play Favorites) found ~14pp average inflation, with some family pairs showing 75–84% self-preferred win rates. So the curation judge is deliberately a different model family from the generator — scored as a majority vote over 3 runs per case. A lenient judge is worse than no judge: it certifies garbage as gold.

Guardrails scaled to where the noise actually is. Intent, disambiguation, and escalation are deterministic label-match tasks at temperature 0 — a rewrite is promoted the moment it beats the incumbent on the full re-scored dataset, since there's no judge noise to filter. Product curation is open-ended, judge-scored, and noisy, so it gets a heavier gate:

• Mean-of-three scoring — every candidate is the average of 3 judge passes, not one lucky run.
• A +2pp promotion margin — a rewrite is only kept if its mean beats the incumbent by at least 2 percentage points.
• A train / held-out split — optimizes on ~2/3 of cases, verified on a held-out ~1/3 it never saw. Wins on training but loses on held-out → overfitting, thrown away.

That held-out guard is why product curation reports no improvement — the harness working, not failing.

The gap this creates: the three deterministic engines get no held-out split at all — promoted and reported on the same golden set they were tuned against. Temperature-0 removes scoring noise, not overfitting risk — two different failure modes, and only curation's harness checks the second one. So treat disambiguation's 84.8% and intent's 98.9% as optimization-set accuracy, not a proven production number yet — that's exactly what the pending live A/B is for.

Results: Before vs After

All numbers below were re-measured on one consistent harness (same generation model, temperature 0, current golden datasets), so before and after are directly comparable. These are offline golden-dataset scores, not live traffic — the validated winners haven't been A/B tested against the current production prompts yet.

Worth asking directly: would an engineer, staring at the same 93 failing disambiguation cases, have hand-written most of this gain in an hour? Probably some of it — the over-asking pattern is visible once you read ten transcripts. What GEPA adds isn't the ability to spot that pattern, it's doing the spotting, rewriting, and re-scoring against all 164 cases every generation without a human in the loop, and catching regressions a manual edit would've shipped blind. The honest comparison isn't GEPA vs. nothing — it's GEPA vs. one engineer with the same failure logs and a lot more time.

Disambiguation: 43% → 85%, the Biggest Win

The disambiguation prompt had one dominant failure mode: over-asking. It fired a clarifying question on queries that were already specific enough to search — "backpack", "luggage please", "I need a bag for a weekend trip", "gym bag ideas?", "bags for college students." Most of the 93 original failures were this single mistake, and every one of them is a friction question stapled to a request that didn't need one.

What GEPA learned, by reading those failures, was a set of explicit don't-ask rules — clear examples of vague/browse/greeting/non-shopping queries that should pass straight through — and a tighter definition of when to actually ask: only when the query is genuinely ambiguous and a different answer would lead to fundamentally different products. Result: 84.8%. The ~25 remaining failures are the genuinely hard ambiguity calls, which is exactly where the difficulty should concentrate.

Intent Classifier: 88% → 99%

The intent classifier was strong on average but failed at the confusable boundaries — the pairs that trip up any classifier. Worst buckets before: similar 64%, unknown 70%, offtopic 75%, details 79%. GPU and celebrity questions were tagged unknown instead of offtopic; "how many liters is this?" (with products on screen) was treated as a fresh search instead of a details lookup; "can I buy 65 of these" was tagged purchase instead of routed to bulk handling.

GEPA added explicit decision rules for each confusable pair, each anchored to a concrete example drawn from the actual failures. Four misses left out of 361 after.

Escalation: 97.5% → 99%

Already near the ceiling. The few failures were over-eager escalations — handing off on near-empty conversations and a too-loose "user seems stuck" trigger. GEPA tightened the conditions so escalation only fires when explicitly warranted. Small absolute gain, but it removed the most annoying false hand-offs.

Where GEPA Did Not Help — and Why That's the Point

Two engines didn't move — and both point straight back to the RL discussion.

• RAG grounding/scope: flat at 60.6%. Most remaining headroom of any engine, but the failures are missing facts — support hours, a discount code, engraving cost, product-specific care instructions. No rewording invents facts that aren't in the knowledge base. Fix: catalog/KB coverage, then re-optimize.
• Product curation: flat at 88.2%. Every rewrite improved training cases and then failed held-out — textbook overfitting, rejected generation after generation. Diagnosis: the ~240-line prompt is already detailed; the bottleneck is the generation model's instruction-following, not a shortage of instructions. Falsifiable next step: score the same, unchanged prompt with a stronger generation model — if the score jumps without touching a word, the model was confirmed as the bottleneck.

The connection: GEPA fixes how a policy is expressed. It can't fix missing knowledge (a data problem) or model capability (the one thing that would need actual RL/fine-tuning to move). Three engines were bottlenecked on expression and moved hard; two were bottlenecked on knowledge/capability, and GEPA correctly refused to fake a win. The most useful output of this project wasn't a number — it was knowing, per engine, which of those buckets the failure lives in.

Lessons Learned

When you can't train the weights, optimize the words. In an agentic system the policy is a prompt, and a prompt is text you own. That turns "improve the model" — impossible on a hosted API — into "search over prompts," which is cheap, gradient-free, and fully interpretable. GEPA is what RL wishes it could be when you don't have the model.

Reflection beats reward when feedback is cheap to read. RL squeezes a scalar out of each episode and takes thousands of steps to infer meaning from it. Letting an LLM read its own failures in English delivers the diagnosis and the fix together, which is why GEPA converges in tens of rollouts, not thousands.

Your fitness function is the whole ballgame. A same-family judge will certify a prompt that a cross-family judge would reject — which is why curation is scored by a different model family than the one generating outputs, with a majority vote over repeated passes. If your score is generous, your optimizer optimizes for looking good, not being good.

Match the guardrails to where the noise lives — but know what you didn't guard against. Mean-of-three scoring, a promotion margin, and a held-out split protect curation from measuring judge-noise luck instead of real improvement. The deterministic engines don't need that margin, since temperature-0 scoring has no run-to-run noise to average out — but skipping the margin isn't the same as skipping the held-out split, and we skipped both. Their headline deltas are optimization-set accuracy, not a generalization guarantee; the live A/B is the actual test of that.

A flat result is a diagnosis, not a failure. The two engines that didn't move told us precisely where the real bottlenecks were — knowledge-base coverage and model capability. Prompt optimization fixes expression. Knowing what it can't fix is how you avoid throwing prompt edits at problems that need data or a better model.

We started this expecting to want RL and finding we couldn't have it. We ended with something better suited to the shape of the problem: an optimizer that improves the part of the system we actually control — the words — and is honest enough to tell us when the words were never the problem.