Getting an LLM to do what you want mostly comes down to fiddling with the wording of the prompt you give it to yield the best results early on. You write an instruction, it gets something wrong, you add a line to cover that case, run it again, and something else breaks. After a while this begins to feel like iterative guesswork.

In my last blog, I explored benchmarks and LLM evaluation for comparing prompts; this detailed how scoring a prompt could be done against human labels and how failing cases could be observed to improve the prompt. GEPA aims to automate this loop.

What is GEPA?

GEPA stands for Genetic-Pareto and aims to improve a prompt by evolving a pool of them instead of editing one.

Genetic is the mutation step. GEPA scores a prompt from the pool, then hands a second model that prompt alongside its failures in full: the input, output, and labelled answer. That model writes a replacement. If the replacement scores higher, it joins the pool and can be mutated in turn.

Pareto is the selection step. For every example in the validation set, GEPA records which prompt scored highest on that example. A prompt is kept if it tops at least one example, even when its average is poor. Selecting on the average alone deletes prompts that are the only ones handling a particular case.

Rewrites are filtered before they cost much. GEPA scores the parent on a small batch of training examples, scores the rewrite on the same batch, and drops the rewrite unless it beats the parent there. Only survivors are evaluated against the full validation set.

GEPA is structured as a loop of the following:

Trying it out

The dataset I used for this was MultiWOZ 2.2, a set of conversations between a user and a booking assistant. I kept the restaurant ones, and the task was to read a conversation and write down what the user settled on such as the following:

INPUT
-----
USER    Is there an expensive African restaurant in town.
SYSTEM  Yes, The Bedouin in the city centre is a good, expensive restaurant.
USER    Excellent! I don't need a table right now, but if you could give me
        the address and phone number, that would be great.
SYSTEM  It is located at 100 Mill Road City Centre and the phone number is
        01223367660. Is there anything else I can help you with?
USER    No thanks. That would be all for today. Have a nice day.

EXPECTED OUTPUT
---------------
{
  "restaurant-area":       "centre",
  "restaurant-food":       "african",
  "restaurant-name":       "bedouin",
  "restaurant-pricerange": "expensive"
}

Not all of the prompt was up for grabs for GEPA. I made the output schema and rules fixed, so GEPA only rewrote the block of additional instructions in the middle, leaving a prompt structured as:

<slots>
restaurant-area   centre, east, north, south or west
...
</slots>

<instructions>
%INSTRUCTIONS%          <-- the only part GEPA changes
</instructions>

<output_contract>
Return one JSON object. Every key must be a slot name...
</output_contract>

Otherwise, it could rewrite the output format itself, which should be fixed – although if the reflector did try to break this, it should score worse and hence not be chosen, but that’s an experiment for another day.

Scoring was done per field: credit for each one the LLM got right, and a penalty for fields it left out, made up, or filled in with the wrong value. A wrong value counts twice, since it both misses the right answer and asserts a wrong one. This is called field F1 and is a number between 0 and 1.

MultiWOZ’s own dev and test sets aren’t interchangeable for this task as the same prompt scored a 19.0% slot error rate on one and 12.4% on the other. So I pooled them and dealt them out again, stratified by how many slots each conversation settles: 27 conversations for training, 45 for validation, 45 for a test set I left alone until the end. That’s 117 in total, which isn’t many.

What’s interesting is that you don’t need that much data for the training set because the reflector LLM only ever sees three at a time (by default); it’s looking for generalisable conventions that show up in one or two cases.

Size matters more on the validation set because that’s what decides which prompts survive. Too small and two things go wrong. LLM output varies between runs, so a prompt can be accepted for getting lucky rather than for being better. And a small set won’t cover the range of patterns in the data, so a prompt that only handles the common cases looks as good as one that handles all of them.

Running the Optimiser

I ran GEPA for 76 iterations and about 2,400 scoring calls, which produced 43 rewrites of the prompt, 6 of them an improvement on anything already in the pool. I used Gemini 2.5 Flash for all of the extraction calls and Claude Sonnet 5 for the reflection and prompt proposals.

GEPA produces a handy visualisation for the candidate tree of prompts where each node represents a prompt and its score on the validation set and the arrows show the parent node that a candidate prompt is a variant of. It shows how different options are kept open as GEPA is running. Nodes marked as “New pool best” are ones which a prompt scored the best for a particular example in the validation set:

If you hover over different nodes on the graphic below starting from candidate 0, you can see how the prompt evolves. In particular, look at the difference between candidates 0, 1 and 3 and how the prompt diverges at that first layer:

The Results

The run took the one-line prompt from 0.88 to 0.95 field F1 on the test set I’d left alone until the end. The best prompt on the validation set came near the end of the run, but it scored 0.94 on the test set. The one that scored 0.95 turned up about a third of the way in.

This shows us that the optimised prompt didn’t help to improve reading the conversations at all. What it stopped doing was making things up, and that was essentially the whole gain.

This makes sense when looking at the final prompt generated. I started with 61 characters and GEPA finished with 6,400 – most of which are rules about when not to fill a field. GEPA will try to optimise whatever scoring mechanism you give it which means in this case where I used field F1 it clearly found the easiest path to do that was through adding rules to reduce hallucinated fields.

In the image above, we can see how the prompt improved at extraction accuracy on the validation set over time, with major diminishing returns after 750 calls, where the difference in prompts being “better” could be within the noise of LLM non-determinism.

We also see rapid initial improvements, which is where GEPA seems to shine. This makes it very good for bootstrapping new prompts from essentially nothing for new LLM use cases, but very slow and expensive to squeeze out the last few percentage points.

All of your measurements improve in accuracy when you increase the number of test cases in the validation and test sets, but this is also expensive as each is an additional LLM API call.

Summary & Insights

• Your scoring/evaluation mechanism needs to match your exact goals. GEPA doesn’t know what a better prompt means, only what the score rewards, so anything you care about that isn’t included in this score might be sacrificed to improve it.
  • For example, if you don’t include shortness of prompt in the score, GEPA will happily keep adding more to the prompt to optimise accuracy and anything that is in the score.
• GEPA speeds up the initial prompt engineering process, and leaves it in a human-readable state to be hand-tuned for the last few percentage points, or otherwise you can become constrained on data, and it becomes rather expensive.
• Spending on the reflector model is where it matters most. I used Claude Sonnet 5 for this experiment, but perhaps a more powerful model would yield better results.
• GEPA doesn’t replace the human expert; it should be thought of as an exciting optimisation technique that is a tool for experts to start a task with.