01
Building the first evaluation:
Focuses on Swift Testing, BookTaggingEvaluation.
AI & Machine Learning / 25:46
Meet the Evaluations framework
AI & Machine Learningiosipadosmacosvisionos
Key Points
- Explains how to evaluate model-driven experiences using the Evaluations framework.
- In a probabilistic world, unit tests alone won't suffice.
- Shows how to define metrics, automatically grade outputs, and aggregate statistics to ensure the AI-powered features perform reliably across Apple platforms.
- The session moves through Demo app Book Tracker: a manual evaluation, Building the first evaluation, Running the evaluation and reading the report, Building robust datasets, Refining metrics and evaluators, Evaluation-driven development and hill-climbing, and related wrap-up guidance.
- Key concepts include Evaluations framework, Foundation Models, Swift Testing, BookTaggingEvaluation, ModelSample, SampleGenerator, Private Cloud Compute, model judge.
- Platform coverage: ios, ipados, macos, visionos.
Condensed Flow
02
Running the evaluation and reading the report:
Focuses on ModelSample.
03
Building robust datasets:
Focuses on SampleGenerator, Evaluations framework.
04
Model judges: qualitative metrics:
Focuses on Private Cloud Compute.
05
Building a model judge:
Focuses on model judge, Private Cloud Compute, Evaluations framework.
06
Refining with score dimensions:
Focuses on ModelJudgePrompt, model judge.
More Detail
Additional details
- Detailed flow: Demo app Book Tracker: a manual evaluation -> Building the first evaluation -> Running the evaluation and reading the report -> Building robust datasets -> Refining metrics and evaluators -> Evaluation-driven development and hill-climbing -> Model judges: qualitative metrics -> Building a model judge -> ...
- APIs and concepts to recognize: Evaluations framework, Foundation Models, Swift Testing, BookTaggingEvaluation, ModelSample, SampleGenerator, Private Cloud Compute, model judge, ModelJudgePrompt, ModelJudgeEvaluator, ModelSamples, TagCount.
- Implementation focus: Foundation Models, Evaluations framework, ModelSample, BookTaggingEvaluation, ModelSamples, ModelJudgeEvaluator, TagCount, TagTotal, model judge, TagQuality.