Use the eval sandwich: before, during, after
Before-change evals establish the baseline you are trying not to lose. During-change evals are small and sharp: the five examples most likely to break because of the edit. After-change evals make sure the fix did not damage neighboring behavior.
This is especially useful for system prompt work because every instruction edit has a shadow. You improve one behavior and accidentally move refusal, tone, verbosity, or citation discipline.
Treat the eval sandwich as a change-management ritual. It is less glamorous than a benchmark, but it is the thing that keeps production behavior from drifting by inches.
The middle eval is where most prompt regressions get caught.
Aha moment
The most valuable eval is often the small one you run while the prompt is still being edited.
Try this
Run a baseline set before the change, five targeted fixtures during the change, and a regression set after.
Watch for
- Only testing the behavior you intended to improve
- Skipping refusal and edge fixtures after style changes
- Treating one green benchmark as proof of production safety