Setup Evaluation Model

If you choose to run evaluations on Confident AI, you'll need to specify which evaluation model to use when running LLM-as-a-judge metrics. You'll need to configure an evaluation model if you're using any of these features:

• Online evaluations
• Experiments

To setup an evaluation model, you can either:

1. Use Confident AI's default models (no additional setup required).
2. Use OpenAI by supplying your own OpenAI api key.
3. Use a custom endpoint as your evaluation model.

Using Custom LLMs

To use a custom LLM, go to Project Settings > Evaluation Model, and select the "Custom" option under the model provider dropdown. You'll have the opportunity to supply a model "Name", as well as an "Endpoint. The name is straightforward, since it is merely for identification purposes, but for the model endpoint, you'll have to ensure it follows the following rules:

1. It MUST accept a POST request over HTTPS, and must be reachable over the internet.
2. It MUST return a 200 status where the response body is a simple string (not Json, not nothing else, but string only) that represents the generated text of your evaluation model based on the given request body (which contains the input).
3. It MUST correctly parse the request body to extract the input and use your evaluation model to generate a response based on this input.

Here is what the structure of the request body looks like:

{
  "input": "Given the metric score, tell me why..."
}

Please test your endpoint that it meets all the given requirements before continuing. This might be harder to debug than other steps, so you should aim to log any errors and reach out to support@confident-ai.com for any errors during setup.

tip

If you find a model provider that is not support, most of the time it is because no one has asked for it. Simply email support@confident-ai.com as well if that is the case.