01
BookTracker tagging problem:
Focuses on BookTracker, ModelJudgeEvaluator.
AI & Machine Learning / 26:41
Improve your prompts by hill-climbing with Evaluations
AI & Machine Learningiosipadosmacosvisionos
Key Points
- Covers comparative evaluation techniques to guide prompt engineering and select the right model for an app.
- Explains how to baseline performance, expand the evaluation strategy, and convert results to JSON for integration with other tools.
- Covers when to apply different prompting strategies and how to iteratively refine prompts for best results.
- The session moves through BookTracker tagging problem, Analyzing the evaluation results, Drift between judge and human, Measuring drift with Cohen kappa, Building a judge alignment evaluation, Analyzing alignment failures, and related wrap-up guidance.
- Key concepts include JSON, Evaluations framework, ModelJudgeEvaluator, Swift Testing, model judge, BookTaggingEvaluation.
- Platform coverage: ios, ipados, macos, visionos.
Condensed Flow
02
Analyzing the evaluation results:
Focuses on Swift Testing.
03
Analyzing alignment failures:
Focuses on model judge.
04
Adding few-shot examples to the judge:
Focuses on model judge.
More Detail
Additional details
- Detailed flow: BookTracker tagging problem -> Analyzing the evaluation results -> Drift between judge and human -> Measuring drift with Cohen kappa -> Building a judge alignment evaluation -> Analyzing alignment failures -> Comparative evaluation: control vs experimental -> Refining the scoring dimensions -> ...
- APIs and concepts to recognize: JSON, Evaluations framework, ModelJudgeEvaluator, Swift Testing, model judge, BookTaggingEvaluation.
- Implementation focus: ModelJudgeEvaluator, Evaluations framework.