Braintrust: artificial intelligence features and setup

Overview

Braintrust is designed for situations where you need to do more than view a run log and want to check whether a model or prompt change caused a regression. Run evaluations, track production, and compare results. It suits a team with a representative set of examples rather than just a gut feeling. Before relying on a score, make sure the examples cover edge cases as well as the happy path.

Features

• Evaluations for agents and prompts
• Tracking production runs
• Detecting regressions after a model change

Business uses

• Checking a support agent against a known set of inquiries before release
• Comparing a new model with the old one on the same examples

Getting started

• Collect a few dozen real examples after removing identifiers.
• Run an initial evaluation and save it as a baseline.
• Make a small prompt change and check whether the score drops on a sensitive example.

What to check

• A good score on a small set does not prove behavior across all customers.
• Update the test set when new real-world failures appear.