Skip to main content

Image Helpfulness

The Image Helpfulness metric assesses how effectively images contribute to a user's comprehension of the text, including providing additional insights, clarifying complex ideas, or supporting textual details. deepeval's Image Helpfulness metric is a self-explaining MLLM-Eval, meaning it outputs a reason for its metric score.

info

Image Helpfulness evaluates MLLM responses containing text accompanied by retrieved or generated images.

Required Arguments​

To use the ImageHelpfulness, you'll have to provide the following arguments when creating an MLLMTestCase:

  • input
  • actual_output
note

Remember that the actual_output of an MLLMTestCase is a list of strings and Image objects. If multiple images are provided in the actual output, The final score will be the average of each image's helpfulness score.

Example​

from deepeval import evaluate
from deepeval.metrics import ImageHelpfulnessMetric
from deepeval.test_case import MLLMTestCase, MLLMImage

# Replace this with your actual MLLM application output
actual_output=[
    "1. Take the sheet of paper and fold it lengthwise",
    MLLMImage(url="./paper_plane_1", local=True),
    "2. Unfold the paper. Fold the top left and right corners towards the center.",
    MLLMImage(url="./paper_plane_2", local=True),
    ...
]

metric = ImageHelpfulnessMetric(
    threshold=0.7,
    include_reason=True,
)
test_case = MLLMTestCase(
    input=["Provide step-by-step instructions on how to fold a paper airplane."],
    actual_output=actual_output,
)

metric.measure(test_case)
print(metric.score)
print(metric.reason)

# or evaluate test cases in bulk
evaluate([test_case], [metric])

There are five optional parameters when creating a ImageHelpfulnessMetric:

  • [Optional] threshold: a float representing the minimum passing threshold, defaulted to 0.5.
  • [Optional] strict_mode: a boolean which when set to True, enforces a binary metric score: 1 for perfection, 0 otherwise. It also overrides the current threshold and sets it to 1. Defaulted to False.
  • [Optional] async_mode: a boolean which when set to True, enables concurrent execution within the measure() method. Defaulted to True.
  • [Optional] verbose_mode: a boolean which when set to True, prints the intermediate steps used to calculate said metric to the console, as outlined in the How Is It Calculated section. Defaulted to False.
  • [Optional] max_context_size: a number representing the maximum number of characters in each context, as outlined in the How Is It Calculated section. Defaulted to None.

How Is It Calculated?​

The ImageHelpfulness score is calculated as follows:

  1. Individual Image Helpfulness: Each image's helpfulness score is based on the text directly above and below the image, limited by a max_context_size in characters. If max_context_size is not supplied, all available text is used. The equation can be expressed as:
Hi=f(Contextabove,Contextbelow,Imagei)H_i = f(\text{Context}_{\text{above}}, \text{Context}_{\text{below}}, \text{Image}_i)
  1. Final Score: The overall ImageHelpfulness score is the average of all individual image helpfulness scores for each image:
O=∑i=1nHinO = \frac{\sum_{i=1}^n H_i}{n}