Use refusal boundaries as test fixtures
Most prompt tests focus on whether the model can complete the intended workflow. That misses the other half of the product: whether it handles edge requests without over-refusing, leaking private context, or inventing authority it does not have.
Build a small fixture set with three buckets: allowed, disallowed, and allowed-with-constraints. A good model response should land in the right bucket and explain the usable next step.
This turns safety from a vibe into a regression target. When a prompt edit changes the refusal boundary, you see it immediately.
Aha moment
Safety quality is not whether the model refuses. It is whether it refuses at the right boundary.
Try this
Keep allowed, disallowed, and allowed-with-constraints prompts as fixtures and run them after prompt edits.
Watch for
- Over-refusal on harmless transformed versions
- Under-refusal when the same intent is phrased politely
- Responses that refuse but fail to offer a safe next step