Cognee

Quick Summary

Cognee is an open-source framework for anyone to easily implement graph RAG into their LLM application. You can learn more by visiting their website here.

info

With Cognee, you should see an increase in your ContextualRelevancyMetric, ContextualRecallMetric, and ContextualPrecisionMetric scores.

Unlike traditional vector databases that relies on simple embedding retrieval and re-rankings to retrieve retrieval_contexts, Cognee stores and creates a "semantic graph" out of your data, which allows for more accurate retrievals.

Setup Cognee

Simply add your LLM API key to the environment variables:

import os
os.environ["LLM_API_KEY"] = "YOUR_OPENAI_API_KEY"

For those on Networkx, you can also create an account on Graphistry to visualize results:

import cognee

cognee.config.set_graphistry_config({
    "username": "YOUR_USERNAME",
    "password": "YOUR_PASSWORD"
})

Finally, ingest your data into Cognee and run some retrievals:

from cognee.api.v1.search import SearchType

...
text = "Cognee is the Graph RAG Framework"
await cognee.add(text) # add a new piece of information
await cognee.cognify() # create a semantic graph using cognee

retrieval_context = await cognee.search(SearchType.INSIGHTS, query_text="What is Cognee?")
for context in retrieval_context:
    print(context)

Evaluating Cognee RAG Pipelines

Unit testing RAG pipelines powered by Cognee is as simple as defining an EvaluationDataset and generating actual_outputs and retrieval_contexts at evaluation time. Building upon the previous example, first generate all the necessarily parameters required to test RAG:

...

input = "What is Cognee?"
retrieval_context = await cognee.search(SearchType.INSIGHTS, query_text="What is Cognee?")

prompt = """
Answer the user question based on the supporting context

User Question:
{input}

Supporting Context:
{retrieval_context}
"""

actual_output = generate(prompt) # hypothetical function, replace with your own LLM

Then, simply run evaluate():

from deepeval.metrics import (
    ContextualRecallMetric,
    ContextualPrecisionMetric,
    ContextualRelevancyMetric,
)
from deepeval.test_case import LLMTestCase
from deepeval import evaluate

...
test_case = LLMTestCase(
    input=input,
    actual_output=actual_output,
    retrieval_context=retrieval_context,
    expected_output="Cognee is the Graph RAG Framework.",
)
evaluate(
    [test_case],
    metrics=[
        ContextualRecallMetric(),
        ContextualPrecisionMetric(),
        ContextualRelevancyMetric(),
    ],
)

That's it! Do you notice an increase in the contextual metric scores?