evals

Write, run, and analyze evaluations for AI agents and LLM applications. Assume we will use EZVals as the eval framework unless you are in a non-python project or the user specifies otherwise.

Traditional ML evals measure model performance on fixed benchmarks with clear accuracy metrics. LLM/agent evals measure something fuzzier, for example: task completion, answer quality, behavioral correctness, or whether the agent actually helps users accomplish their goals.

| Target | The function or agent being evaluated. Takes input, produces output. | | Grader | Function that scores the output. Returns 0-1 or pass/fail. | | Dataset | Collection of test cases (inputs + optional expected outputs). | | Task | Single test case: one input to evaluate. | | Trial | One execution of a task. Multiple trials handle non-determinism. |