Evaluate models offline

Create an offline evaluator

Prerequisites

1. You need to have access to the beta preview of evaluations.
2. You also need to have a project created - if not, please first follow our project creation guides.
3. Finally, you need at least a few datapoints in your project. Use the Editor to generate some datapoints if you don't have any yet.

📘

You need datapoints in your project because we will use these a source of testcases for the testset we will create. If you want to create arbitrary testcases from scratch, see our guide to doing this from the API. We will soon be updating the app to enable arbitrary testcase creation from your browser.

For this example, we're going to evaluate a model who's responsibility is to extract key information from a customer service request and return this information in JSON. In the image below, you can see the model config we've drafted on the left, and an example of it running against a customer query on the right.

Set up a testset

We will create a testset based on existing datapoint that have already been logged in the project.

1. Navigate to the data tab
2. Select the datapoints you would like to convert into testcases
3. From the dropdown menu in the top right (see below), choose Add to testset

4. In the dialog box, give the new testset a name and provide an optional description. Click Create testset.

📘

You can add more testcases to the same testset later by clicking the 'add to existing testset' button at the top.

5. Go to the evaluations tab and click Manage testsets
6. Click into the newly created testset. One testcase will be present for each datapoint you selected in step 3

7. Click on a testcase to inspect its parameters.

📘

A testcase contains inputs (the variables passed into your model config template), an optional sequence of messages (if used for a chat model) and a target representing the desired output.

As we converted existing datapoints into testcases, the target defaults to the output of the source datapoint.

In our example, we created testcases from existing datapoints. The default behaviour is that the original datapoint's output becomes an output field in the target JSON.

In order to access the feature field more easily in our evaluator, we'll modify the testcase targets to be a raw JSON with a feature key.

8. Modify the testcase if you need to make refinements. You can provide an arbitrary JSON object as the target.

Create an offline evaluator

Having set up a testset, we'll now create the evaluator. As with online evaluators, it's a Python function but for offline mode, it also takes a testcase parameter alongside the generated datapoint.

1. Navigate to the evaluations tab
2. Select + New Evaluator and choose Offline
3. Choose Start from scratch

For this example, we'll use the code below to compare the LLM generated output with what we expected for that testcase.

import json
from json import JSONDecodeError

def it_extracts_correct_feature(datapoint, testcase):
    expected_feature = testcase["target"]["feature"]

    try:
        # The model is expected to produce valid JSON output
        # but it could fail to do so.
        output = json.loads(datapoint["output"])
        actual_feature = output.get("feature", None)
        return expected_feature == actual_feature

    except JSONDecodeError:
        # If the model didn't even produce valid JSON, then
        # we evaluate the output as bad.
        return False

4. In the debug console at the bottom of the dialog, click Load from testset and select the testset you created in the previous section. The console will be populated with its testcases.

5. Choose a model config from the dropdown menu.
6. Click the run button at the far right of one of the testcases.

A new debug run will be triggered, which causes an LLM generation using that testcase's inputs and messages parameters. The generated datapoint and the testcase will be passed into the evaluator and the resulting evaluation displays in the Result column.

7. Click Create when you are happy with the evaluator.

Trigger an offline evaluation

Now that you have an offline evaluator and a testset, you can use them to evaluate the performance of any model config in your project.

1. Go to the Evaluations tab.
2. To the right of the Evaluation Runs section, click Run Evaluation
3. In the dialog box, choose a model config to evaluate, and select your newly created testset and evaluator.

4. Click Launch
5. A new evaluation is launched. Click on the card to inspect the results.

It may take some time for the evaluation to complete, depending on how many testcases are in your testset and what model config you are using.

6. Inspect the results of the evaluation.