本文属于机器翻译版本。若本译文内容与英语原文存在差异，则一律以英文原文为准。 # 定义评估方法 ## 概述 为每个提示模板选择一种评估方法，或者省略所有可选的评估字段作为系统默认值。同一个作业中的不同模板可以使用不同的方法。评估会引导即时优化，因此请尽可能精确地定义您的方法和标准。 + [默认评估](#advanced-prompt-optimization-evaluation-default) + [转向标准](#advanced-prompt-optimization-evaluation-steering) + [自定义 LLM-as-a-judge](#advanced-prompt-optimization-evaluation-llmj) + [自定义 Lambda 评估器](#advanced-prompt-optimization-evaluation-lambda) ## 数据集提示 + **混合简单和困难的例子** ——与你在现实世界中看到的相匹配。 All-easy 数据不会推动改进；全硬数据没有什么可学习的。 + **涵盖你关心的案例** ——系统的概括超出了它所看到的范围，但是具有代表性的数据分布可以帮助它更好地推广到现实世界的输入。 + **提示优化后，使用保留的数据集对优化器未见过的数据进行测试，**确认收益是真实的，而不仅仅是记住的。 ## 默认评估 省略所有可选的评估字段（`steeringCriteria`、`customLLMJConfig`、`evaluationMetricLambdaArn`）。该服务使用 LLM-as-judge 由 Anthropic Claude Sonnet 4.6 提供支持的内置通用工具，它评估三个默认标准：答案准确性、答案完整性和表达质量。根据提示、目标模型的答案和参考答案，评委对每个维度进行评分。然后，它为任务动态分配适当的权重，并生成加权总分。 我们建议您定义自己的评估方法以获得最佳结果。 ### 系统提供的默认裁判提示 以下是 Anthropic Claude Sonnet 4.6 的默认评估中使用的系统提供的完整评判提示： ``` Please act as an impartial judge and evaluate the quality of an answer to a user question, with the help of a reference answer. You will be given: (1) a user question, enclosed in tags (2) an answer, enclosed in tags (3) a reference answer, enclosed in tags ## Universal Evaluation Dimensions Evaluate the answer across these core dimensions: **(1) Answer Accuracy:** examines correctness, consistency, and factuality alignment between the and the ; examines if the contains irrelevant or wrongful information/hallucination. **(2) Answer Completeness:** examines if the is fully addressing the ; examines if the is good at relevance/informativeness: selection of important/key content from **(3) Expression Quality:** examines if the is concise at answering the . NOTE that unless there is special instruction, more concise is always better, and explanation or rational is strictly NOT needed - THIS IS THE MOST IMPORTANT! examines the alignment on instruction following, e.g., if the adheres to both explicit guidelines and implicit guidelines (like few-shot examples) in the ; ## Scoring Rubric For each dimension, assign one score: - **3 points**: Fully satisfies the dimension requirements - **2 points**: Mostly satisfies with minor issues or gaps - **1 point**: Partially satisfies but has notable limitations - **0 points**: Does not satisfy the dimension requirements ## Evaluation Process 1. First, identify the task type from the user question 2. Consider any additional criteria provided 3. Score each dimension independently 4. Determine appropriate weights and calculate final weighted score ## Dimension Weighting and Final Scoring **Weight Determination Process:** Assign weights (must sum to 1.0) based on: - Explicit weights in evaluation_criteria (if provided) - Task analysis and question requirements (if no explicit weights) - Default weights (Answer Accuracy: 0.35, Answer Completeness: 0.30, Expression Quality: 0.35) as fallback **Weight Guidelines:** - **High Accuracy Weight (0.4-0.6)**: Factual questions, multiple choice, technical problems - **High Completeness Weight (0.4-0.6)**: Complex explanatory tasks, multi-part questions - **High Expression Weight (0.4-0.6)**: Creative tasks, presentation-focused questions, format-specific requirements {custom_eval_weight_guideline} **Overall Score Calculation:** Overall = (Answer_Accuracy x Weight_A) + (Answer_Completeness x Weight_C) + (Expression_Quality x Weight_E) ## Output Format Provide your evaluation in this exact format: Brief analysis of task type and appropriate weight rationale Answer Accuracy: 0.XX, Answer Completeness: 0.XX, Expression Quality: 0.XX X X X (X x 0.XX) + (X x 0.XX) + (X x 0.XX) = X.XX X.XX **Answer Accuracy**: [Evaluate factual accuracy, alignment with reference answer, absence of errors/hallucinations, and logical consistency] **Answer Completeness**: [Assess whether all aspects of the question are addressed, necessary information is included, and content stays relevant] **Expression Quality**: [Examine formatting/style adherence, appropriate detail level, communication clarity, and instruction following] **Weight Application**: [Explain how the chosen weights reflect the task requirements and impact the final score] --- ## Current Evaluation Task {prompt} {prediction} {gold} Based on the above guidelines and criteria, provide your evaluation: ``` ## 转向标准 指导标准是指导优化方向的简短自然语言描述符。 + **Format**: `"steeringCriteria": ["string1", "string2"]` + **它们可能是什么：**从一个单词到几个句子，对模型的反应进行定性或定量描述。 + **限制：**每个提示模板最多 5 个。 **示例**： ``` "steeringCriteria": ["PROFESSIONAL", "CONCISE"] ``` ## 自定义 LLM-as-a-judge 提供完整的评分量规以及您定义的评分等级。您的自定义判断提示将与服务的系统判断提示合并，并赋予更强的权重。 ### 配置 + **格式：p** `"customLLMJConfig": {"customLLMJPrompt": "...", "customLLMJModelId": "..."}` lus `"customEvaluationMetricLabel": "My Metric"` + **可用的评委模型：**anthropic.claude-opus-4-6-v1、anthropic.claude-sonnet-4-5-20250929-v 1:0、anthropic.claude-sonnet-4-6 + **裁判提示中的占位符：** + `{{prompt}}`: 完全呈现的提示（提示模板加上评估样本的组合） + `{{response}}`: 模型输出 + `{{referenceResponse}}`: 实地真相 + **评分：**定义您的评分等级，以便分数越高越好。该服务对最终结果的所有分数进行标准化。 + 如果您有多个评分量规，请将它们合并为一个评判提示。 ### 撰写评委提示的最佳实践 使用明确定义的评分量规，其中包含明确的评分标准和每个分数级别的具体示例。用行为描述而不是主观形容词来锚定每个评分量规级别。至少包括一个显示非完美分数的有效示例，以使评委摆脱默认为高评分。指示模型在计算分数之前提供书面理由。在分配总分之前，请考虑独立评估特定维度。最值得信赖和最有帮助的 LLM-as-a-judge 评估者通常是那些你同意评判模型提供的答案的人。因此，使用您已经审核过的评估可能会有所帮助。 ### 运行时如何将您的自定义判断提示符与系统提示符合并 当您提供自己的 LLM-as-a-judge 评估员提示时，它会与通用服务提供的评判提示合并。该提示包含有关格式的具体说明和其他有助于优化进展的最佳实践。在最终判决中，你的自定义裁判提示比通用标准更具权重。具体而言，该服务： + 从您的自定义提示中提取意图 + 标准化比例以匹配系统的 0 到 3 评分量规 + 将其作为命名维度注入到 CUSTOM\_CRITERIA\_DESCRITIA 标签中 + 偏离加权指令以提高自定义标准的重要性（0.3 到 0.6） + 添加优先规则，说明自定义标准覆盖与其他维度冲突 + 保留评估的原始语义 **示例：**您可以提供以下用于评估忠诚度的自定义 LLM-as-a-judge 提示： ``` You are given a task in some context (Input), and a candidate answer. Is the candidate answer faithful to the task description and context? A response is unfaithful only when (1) it clearly contradicts the context, or (2) the task implies that the response must be based on the context, like in a summarization task. If the task does not ask to respond based on the context, the model is allowed to use its own knowledge to provide a response, even if its claims are not verifiable. Task: {{prompt}} Candidate Response: {{response}} First provide your explanation, then state your final answer. Use the following format: Explanation: [Explanation], Answer: [Answer], where '[Answer]' must be one of: none is faithful some is faithful approximately half is faithful most is faithful all is faithful ``` 然后将其与默认 LLM-as-a-judge 提示合并，并赋予很强的权重。最终效果：你的单一标准忠诚度提示在多维评估中变成了一个权重较高的轴，而系统则在它周围添加了结构（准确性、完整性、表达式）。 以下是由此产生的合并裁判提示： ``` """Please act as an impartial judge and evaluate the quality of an answer to a user question, with the help of a reference answer. You will be given: (1) a user question, enclosed in tags (2) an answer, enclosed in tags (3) a reference answer, enclosed in tags (4) custom evaluation criteria that have been integrated into the evaluation dimensions below **IMPORTANT**: - Custom criteria requirements take absolute precedence over user requirements specified inside **IMPORTANT**: - If there is any conflict between custom criteria and user question requirements, prioritize custom criteria ## Universal Evaluation Dimensions Evaluate the answer across these core dimensions: **(1) Answer Accuracy:** examines correctness, consistency, and factuality alignment between the and the ; examines if the contains irrelevant or wrongful information/hallucination. **(2) Answer Completeness:** examines if the is fully addressing the ; examines if the is good at relevance/informativeness: selection of important/key content from **(3) Expression Quality:** examines if the is concise at answering the . NOTE that unless there is special instruction, more concise is always better, and explanation or rational is strictly NOT needed - THIS IS THE MOST IMPORTANT! examines the alignment on instruction following, e.g., if the adheres to both explicit guidelines and implicit guidelines (like few-shot examples) in the ; **(4) Faithfulness to Context:** examines whether the candidate answer is faithful to the task description and context provided in the user question. A response is unfaithful only when (1) it clearly contradicts the context, or (2) the task implies that the response must be based on the context (like in a summarization task). If the task does not ask to respond based on the context, the model is allowed to use its own knowledge to provide a response, even if its claims are not verifiable. Evaluate the degree of faithfulness on the following scale: - **3 points**: All content is faithful (no contradictions, fully grounded when required) - **2 points**: Most content is faithful (minor deviations or unverifiable claims when context-grounding is required) - **1 point**: Some content is faithful or approximately half is faithful (notable contradictions or significant departures from context when required) - **0 points**: None or minimal content is faithful (clear contradictions or complete disregard of context when grounding is required) ## Dimension Weighting and Final Scoring **Weight Determination Process:** Assign weights (must sum to 1.0) based on: - Explicit weights in evaluation_criteria (if provided) - Task analysis and question requirements (if no explicit weights) - Default weights (Answer Accuracy: 0.25, Answer Completeness: 0.25, Expression Quality: 0.25, Faithfulness to Context: 0.25) as fallback **Weight Guidelines:** - **High Accuracy Weight (0.3-0.5)**: Factual questions, multiple choice, technical problems - **High Completeness Weight (0.3-0.5)**: Complex explanatory tasks, multi-part questions - **High Expression Weight (0.3-0.5)**: Creative tasks, presentation-focused questions, format-specific requirements - [*IMPORTANT*] **High Custom Criteria Weight**: The custom criteria (Faithfulness to Context) should always be *prioritized*. Assign it significant weight (0.3-0.6) and adjust other weights accordingly. **Overall Score Calculation:** Overall = (Answer_Accuracy x Weight_A) + (Answer_Completeness x Weight_C) + (Expression_Quality x Weight_E) + (Faithfulness_to_Context x Weight_F) ## Current Evaluation Task {prompt} {prediction} {gold} Based on the above guidelines and criteria, provide your evaluation:""" ``` ## 自定义 Lambda 评估器 将您自己的评分函数作为 Lambda 函数使用。 ### 配置 在输入 JSONL 文件中，为每个应使用它的提示模板指定 Lambda ARN。您还提供用于命名指标的字`customEvaluationMetricLabel`段： ``` "evaluationMetricLambdaArn": "arn:aws:lambda:us-west-2:123456789012:function:my-eval-function", "customEvaluationMetricLabel": "My Custom Metric" ``` 通过 API 创建任务时，无需在`CreateAdvancedPromptOptimizationJob`请求本身中进行其他评估配置。评估方法是根据输入 JSONL 文件中的每个模板确定的。 ### Lambda 要求 + 包含所有代码的单个`.py`文件 + 处理程序设置为 `lambda_function.lambda_handler` + 必须实现`compute_score(preds, golds)`返回 `{"score": float, "scores": [float, ...]}` + 该`golds`参数包含`referenceResponse`值。如果您未`referenceResponse`在输入数据集中提供，则无需传`golds`入`compute_score`函数。 + 永远不要崩溃；出错时返回 0.0 而不是引发异常 + 优先使用连续分数（0.0 到 1.0）而不是二进制分数， 0/1 以加快优化收敛速度 + 分数不需要限制在 [0, 1]。 + 分数必须遵循 “越高越好” 的量表。这是优化正常运行的必要条件。 + 将大批量的超时设置为最大 15 分钟 (900 秒)，以避免提前超时 + 添加基于资源的策略，允许调用您`bedrock.amazonaws.com`的 Lambda ### Lambda 指标提示 + **在上面写一个清晰**的文档字符串 `compute_score` — 提取文档字符串并作为指标的描述显示给系统。解释分数的含义以及高分与低分的区别。 + **清晰地写下您的指标代码** — 系统会读取完整的源代码。注释、变量名称和可读逻辑有助于它理解您的意图并生成更好的反馈。 + **优先选择连续分数而不是二进制**分数 0/1 — 优化器可最大限度地提高样本的平均分数。与全有或全无分数相比，连续分级（例如，部分积分）给出的关注信号更加平滑。 + **返回结构化细节，而不仅仅是数字**，返回带有子分数或诊断字段的字典可以让系统更多地了解预测得分低的原因，从而带来更有针对性的改进。 + **返回带有后备分数的描述性消息** — 当返回 0.0 的回退分数时，请在结果字典中包含诊断字段（例如，）。`{"score": 0.0, "error": "JSON parse failed on line 3"}`系统会读取这些消息，以了解预测得分较低的原因，并生成更有针对性的反馈。 + **验证你的得**分 — 给一些明显不好的输出打分，以确认你的指标会惩罚他们。如果一个琐碎的答案得分不错，优化器就会找到这个捷径。 ### Lambda 模板 ``` """ APO Custom Metric Lambda - Minimal Template Handler: lambda_function.lambda_handler """ import logging from typing import List, Dict, Any logger = logging.getLogger() logger.setLevel(logging.INFO) def compute_score(preds: List[str], golds: List[str]) -> Dict[str, Any]: """ Score predictions against ground truths. Args: preds: Model outputs (one per sample) golds: Expected answers (one per sample) Returns: Must contain: "score": float - aggregate score (higher is better) "scores": list[float] - per-instance scores """ # --- REPLACE THIS with your scoring logic --- scores = [] for pred, gold in zip(preds, golds): # Example: exact match (case-insensitive) scores.append(1.0 if pred.strip().lower() == gold.strip().lower() else 0.0) return { "score": sum(scores) / len(scores) if scores else 0.0, "scores": scores, } def lambda_handler(event, context): """ Lambda entry point. APO service sends: event = {"preds": ["output1", ...], "golds": ["truth1", ...]} """ logger.info(f"Received {len(event.get('preds', []))} predictions") try: preds = event.get("preds", []) golds = event.get("golds", []) if not preds: return {"score": 0.0, "scores": []} return compute_score(preds, golds) except Exception as e: logger.error(f"Error: {e}", exc_info=True) return {"score": 0.0, "scores": [0.0] * len(event.get("preds", [])), "error": str(e)} ``` 有关更多详细示例，包括 GitHub 用于错误处理和 Lambda 函数输入验证的 AWS 样板代码，请参阅示例。