Contextual Precision

The contextual precision metric uses LLM-as-a-judge to measure your RAG pipeline's retriever by evaluating whether nodes in your retrieval_context that are relevant to the given input are ranked higher than irrelevant ones. deepeval's contextual precision metric is a self-explaining LLM-Eval, meaning it outputs a reason for its metric score.

info

The ContextualPrecisionMetric focuses on evaluating the re-ranker of your RAG pipeline's retriever by assessing the ranking order of the text chunks in the retrieval_context.

Required Arguments

To use the ContextualPrecisionMetric, you'll have to provide the following arguments when creating an LLMTestCase:

• input
• actual_output
• expected_output
• retrieval_context

The input and actual_output are required to create an LLMTestCase (and hence required by all metrics) even though they might not be used for metric calculation. Read the How Is It Calculated section below to learn more.

Example

from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import ContextualPrecisionMetric

# Replace this with the actual output from your LLM application
actual_output = "We offer a 30-day full refund at no extra cost."

# Replace this with the expected output from your RAG generator
expected_output = "You are eligible for a 30 day full refund at no extra cost."

# Replace this with the actual retrieved context from your RAG pipeline
retrieval_context = ["All customers are eligible for a 30 day full refund at no extra cost."]

metric = ContextualPrecisionMetric(
    threshold=0.7,
    model="gpt-4",
    include_reason=True
)
test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output=actual_output,
    expected_output=expected_output,
    retrieval_context=retrieval_context
)

# To run metric as a standalone
# metric.measure(test_case)
# print(metric.score, metric.reason)

evaluate(test_cases=[test_case], metrics=[metric])

There are SEVEN optional parameters when creating a ContextualPrecisionMetric:

• [Optional] threshold: a float representing the minimum passing threshold, defaulted to 0.5.
• [Optional] model: a string specifying which of OpenAI's GPT models to use, OR any custom LLM model of type DeepEvalBaseLLM. Defaulted to 'gpt-4o'.
• [Optional] include_reason: a boolean which when set to True, will include a reason for its evaluation score. Defaulted to True.
• [Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.
• [Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.
• [Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
• [Optional] evaluation_template: a class of type ContextualPrecisionTemplate, which allows you to override the default prompts used to compute the ContextualPrecisionMetric score. Defaulted to deepeval's ContextualPrecisionTemplate.

As a standalone

You can also run the ContextualPrecisionMetric on a single test case as a standalone, one-off execution.

...

metric.measure(test_case)
print(metric.score, metric.reason)

caution

This is great for debugging or if you wish to build your own evaluation pipeline, but you will NOT get the benefits (testing reports, Confident AI platform) and all the optimizations (speed, caching, computation) the evaluate() function or deepeval test run offers.

How Is It Calculated?

The ContextualPrecisionMetric score is calculated according to the following equation:

$$\text{Contextual Precision} = \frac{1}{\text{Number of Relevant Nodes}} \sum_{k=1}^{n} \left( \frac{\text{Number of Relevant Nodes Up to Position } k}{k} \times r_{k} \right)$$

info

• k is the (i+1)^th node in the retrieval_context
• n is the length of the retrieval_context
• r_k is the binary relevance for the k^th node in the retrieval_context. r_k = 1 for nodes that are relevant, 0 if not.

The ContextualPrecisionMetric first uses an LLM to determine for each node in the retrieval_context whether it is relevant to the input based on information in the expected_output, before calculating the weighted cumulative precision as the contextual precision score. The weighted cumulative precision (WCP) is used because it:

• Emphasizes on Top Results: WCP places a stronger emphasis on the relevance of top-ranked results. This emphasis is important because LLMs tend to give more attention to earlier nodes in the retrieval_context (which may cause downstream hallucination if nodes are ranked incorrectly).
• Rewards Relevant Ordering: WCP can handle varying degrees of relevance (e.g., "highly relevant", "somewhat relevant", "not relevant"). This is in contrast to metrics like precision, which treats all retrieved nodes as equally important.

A higher contextual precision score represents a greater ability of the retrieval system to correctly rank relevant nodes higher in the retrieval_context.

Customize Your Template

Since deepeval's ContextualPrecisionMetric is evaluated by LLM-as-a-judge, you can likely improve your metric accuracy by overriding deepeval's default prompt templates. This is especially helpful if:

• You're using a custom evaluation LLM, especially for smaller models that have weaker instruction following capabilities.
• You want to customize the examples used in the default ContextualPrecisionTemplate to better align with your expectations.

tip

You can learn what the default ContextualPrecisionTemplate looks like here on GitHub, and should read the How Is It Calculated section above to understand how you can tailor it to your needs.

Here's a quick example of how you can override the statement generation step of the ContextualPrecisionMetric algorithm:

from deepeval.metrics import ContextualPrecisionTemplate
from deepeval.metrics.contextual_precision import ContextualPrecisionTemplate

# Define custom template
class CustomTemplate(ContextualPrecisionTemplate):
    @staticmethod
    def generate_verdicts(
        input: str, expected_output: str, retrieval_context: List[str]
    ):
        return f"""Given the input, expected output, and retrieval context, please generate a list of JSON objects to determine whether each node in the retrieval context was remotely useful in arriving at the expected output.

Example JSON:
{{
    "verdicts": [
        {{
            "verdict": "yes",
            "reason": "..."
        }}
    ]
}}
Tthe number of 'verdicts' SHOULD BE STRICTLY EQUAL to that of the contexts.
**

Input:
{input}

Expected output:
{expected_output}

Retrieval Context:
{retrieval_context}

JSON:
"""

# Inject custom template to metric
metric = ContextualPrecisionMetric(evaluation_template=CustomTemplate)
metric.measure(...)