What we built instead.

We transitioned from a registry to version-controlled evaluation datasets — compact collections of inputs and anticipated outputs, each linked to a specific prompt version. Instead of code-review approvals, changes are validated by assessing the dataset and confirming that the new behavior aligns with the intended outcome. The comparison shifts from text to behavior.

The dataset resides alongside the prompt in the repository. CI executes it with each modification. Should a contributor introduce a prompt that unexpectedly alters rejection rates, the build fails — well before human review, and certainly before any user is affected.

Key takeaways.

Datasets must be agile.
If your evaluation suite requires hours to complete, it will be ignored locally. Aim for a focused set of examples per behavior, and actively remove examples that no longer expose genuine issues.

Production unveils the toughest cases.
While synthetic datasets offer comfort, they often miss critical failures. Regularly extract anonymized, deduplicated, and labeled slices from production logs.

Binary results are insufficient.
Monitor 'drift' as a key metric. A prompt that suddenly diverges from its previous behavior is noteworthy, irrespective of individual pass/fail outcomes.

Months later, we've eliminated late-night crises. We've reduced prompt changes — partly due to higher standards, partly because many prior changes were inconsequential. Validated changes deploy faster, with greater assurance, and remain stable.

Production AI requires rigor, not reinvention.