Prompt optimization should not begin with a clever instruction. It should begin with an agreement about what a good result looks like. The prompt is only one part of the system; the evaluation process is what tells us whether a change is actually an improvement.

The first evaluation dimensions should come from good cases supplied by Product and QA. Product usually knows the user outcome: what decision the answer should enable, what information matters, what tone fits the experience, and which trade-offs are acceptable. QA knows the behavioral contract: which details must be present, which edge cases matter, which formats are valid, and which failures have appeared before. Their approved examples turn an abstract idea of quality into observable evidence.

PHASE 1 — EXTRACT THE INITIAL DIMENSIONS. Place the Product and QA good cases side by side and annotate why each one is considered good. Do not start by inventing a generic rubric. Look for repeated signals. If every approved answer identifies the user’s real intent, intent resolution may become a dimension. If the cases consistently separate facts from assumptions, groundedness may become another. If QA expects a specific schema, format compliance should be explicit. If Product values an answer that leads to an action, usefulness or task completion may deserve more weight than elegance.

The initial rubric should be small. Five well-defined dimensions are more useful than fifteen vague ones. A sensible starting set might include task success, factual correctness, completeness, instruction adherence, and usability, but the actual set must be derived from the product. Each dimension should answer four questions: What product risk or value does it represent? What visible evidence earns a good score? What visible failure earns a bad score? Who owns the final judgment when the score is disputed?

Good cases also help define scoring anchors. A number without an anchor is only disguised intuition. On a 0–4 scale, a score of 4 might mean the output matches the important properties of the approved cases with no material correction required. A 2 might mean the main direction is useful but a user or reviewer must repair an important omission. A 0 might mean the result is misleading, unusable, or violates the task. QA should score a sample together until different reviewers apply these anchors consistently.

PHASE 2 — SET THE INITIAL THRESHOLDS. Thresholds should not be chosen because 80 or 90 sounds rigorous. First score the approved good cases with the initial rubric. Their score distribution reveals the realistic quality range for known acceptable work. Then add borderline and clearly bad variants, reviewed by Product and QA, to locate the boundary between acceptable and unacceptable. The initial threshold should separate those groups with enough safety margin for the product’s risk level.

Use both aggregate thresholds and per-dimension floors. A weighted total is useful for trade-offs: S = Σ(wᵢ × sᵢ). But a high average must not hide a critical failure. An answer that is polished and complete but factually wrong should not pass. Correctness, safety, privacy, policy compliance, and required output formats are often gates. Other dimensions, such as brevity and tone, can usually remain weighted preferences. The rubric therefore needs to say not only how much a dimension contributes, but whether it is a score, a minimum floor, or a non-negotiable constraint.

An initial threshold is a hypothesis, not a permanent law. At the beginning, the dataset is small and the product team has seen only a fraction of real behavior. Record the rubric version, threshold version, examples, evaluator configuration, model version, and prompt version together. Without versioning, later score changes cannot be separated from changes in the prompt, model, traffic, or evaluator.

PHASE 3 — EVALUATE CONTINUOUSLY DURING OPERATION. Once the prompt is running, production outputs should continuously feed the evaluation system. The evaluation record should contain the input, relevant context, output, model and prompt versions, latency, token usage, rule-based checks, dimension scores, and reviewer feedback where available. Sensitive data should be removed or handled under the product’s privacy rules before it enters an evaluation dataset.

Not every output needs an expensive human review. Deterministic checks can validate schema, required fields, citations, length limits, or forbidden content. Model-based evaluators can score larger samples and surface suspicious cases. Human reviewers should focus on high-risk outputs, disagreements, novel patterns, and cases close to a threshold. These layers serve different purposes; none should pretend to be a complete judge of quality.

Sampling must reflect actual traffic. Include common requests, difficult cases, low-frequency but high-impact tasks, new user behaviors, and outputs that triggered complaints or manual corrections. Track performance by slice rather than only by global average. A prompt can improve overall while becoming worse for ambiguous questions, long contexts, a particular language, or a critical customer workflow.

The loop is straightforward: run the prompt, collect representative results, submit them to evaluation, inspect failures by dimension and slice, form a specific hypothesis, change one mechanism, and compare the candidate against the baseline. A useful change should improve the target dimension without creating unacceptable regressions in the others. This is closer to product experimentation than to rewriting prose.

PHASE 4 — LET THE RUBRIC EVOLVE. Production evaluation will reveal failures that the initial good cases did not predict. Add a dimension when a recurring and important failure is not represented by the current rubric, when reviewers repeatedly mention the same missing quality, and when that quality can be described with observable anchors. For example, a system may initially score correctness and completeness but later discover that it confidently answers questions that should have been clarified. “Uncertainty handling” may then become a distinct dimension.

Do not keep adding dimensions forever. Merge or remove a dimension when it substantially overlaps another, when reviewers cannot score it consistently, when it never changes a release decision, or when the product no longer values the behavior it represents. A rubric is an operational model of quality, not an archive of every concern the team has ever discussed. Complexity that does not improve decisions should be deleted.

Thresholds should evolve for similarly concrete reasons. Raise a threshold when outputs that technically pass still create user harm or significant manual correction. Lower or reshape it when the evaluator rejects outputs that Product and QA consistently judge acceptable. Introduce a separate threshold for a high-risk traffic slice when a global threshold hides its failures. Whenever a threshold changes, replay historical examples through the new rule and document which decisions would change.

Changes to dimensions and thresholds require governance. Product should own the user and business outcome, QA should own repeatability and coverage, and engineering should own instrumentation, versioning, and runtime constraints. A change should include the evidence that motivated it, examples affected, expected benefit, possible regression, and a rollback path. Otherwise the evaluation system itself becomes an uncontrolled prompt.

PHASE 5 — RECOGNIZE THE LIMITS OF QUANTIFICATION. Not everything valuable can be reduced to a stable number. Originality, taste, empathy, strategic insight, and whether an answer feels appropriate in a delicate situation may resist precise measurement. Forcing them into a score can create false confidence and encourage the system to optimize a proxy instead of the real outcome.

Keep a qualitative channel alongside the metrics. Reviewers should be able to leave explanations, identify surprising strengths or failures, compare two outputs directly, and mark cases that require Product judgment. Periodic editorial or expert review can detect changes that dashboards miss. A metric should act as a sensor, not as a replacement for responsibility.

Constraints are also necessary because optimization always finds shortcuts. Define boundaries for privacy, safety, legal requirements, unsupported claims, latency, cost, tool usage, and data retention. Some are hard runtime limits, some are release gates, and some require human escalation. The evaluation score only applies after these constraints are satisfied. A candidate with a higher quality score is not an improvement if it violates a boundary the product cannot accept.

This creates a living evaluation system. Product and QA good cases establish the first definition of quality. Initial dimensions and thresholds make that definition testable. Runtime outputs continually challenge it. Evaluation results improve the prompt, but also improve the rubric itself by adding missing dimensions, removing weak ones, and recalibrating thresholds. Constraints and human judgment cover the territory that numbers cannot.

The objective is not to quantify everything. It is to quantify what can be observed reliably, preserve judgment where it cannot, and make every prompt change answer a clear question: which product outcome improved, according to what evidence, under which constraints, and at what cost?