Create robust evaluations for agentic apps

Key Points

• Explains how to leverage advanced features of the Evaluations framework to build robust evaluations for an app.
• Covers evaluating flows with tool calling and dynamic conditions, and how to define what correct behavior means for the use case.
• Shows how to generate synthetic data, use judges effectively, and validate the datasets for reliable results.
• The session moves through The dataset problem in BookTracker, Generating synthetic data with makeSamples, Customizing generation with SampleGenerator, Sampling strategies, Validating synthetic samples, Comparing evaluation results, and related wrap-up guidance.
• Key concepts include Evaluations framework, ModelSamples, SampleGenerator, ToolCallEvaluator, ModelSample, TrajectoryExpectation, LanguageModelSession, Foundation Models.
• Platform coverage: ios, ipados, macos, visionos.