Context Quality

Hallucinations and even other quality issues can be often traced back to the quality of the context data that is used in your RAG indexing and retrieval pipelines. AIMon helps you incrementally "fix" the causes of such issues by identifying common problems with your data and RAG pipelines. Currently, AIMon detects the following issues with your context data:

Conflicting Information: This issue is detected when the context contains conflicting information. For example, if the context states that "Paul Graham is an English-born computer scientist" and later states that "He was originally from New York", this would be marked as an inconsistency because being "English-born" implies that he was born in England, not New York.
Missing Punctuation: This issue is detected when the context contains missing punctuation. For example, if the context contains a sentence like "Paul Graham is an English-born computer scientist entrepreneur venture capitalist author and essayist", this would be marked as a missing punctuation issue because it lacks commas between the different roles that Paul Graham has.
Excessive Noise: This issue is detected when the context contains excessive noise. For example, if the context contains a sentence like "Paul Graham is an English-born computer scientist entrepreneur venture capitalist author and essayist. He is best known for his work on Lisp his @@%former startup Viaweb later renamed Yahoo! Store co-founding the influential startup accelerator and seed capital firm Y Combinator his blog and Hacker News", this would be marked as an excessive noise issue because it contains unnecessary characters like "@@%", etc.
Incomplete Sentences: This issue is detected when the context contains incomplete sentences. For example, if the context contains a sentence like "Paul Graham is an English-born. He is best known for his work on Lisp his former startup Viaweb later renamed Yahoo! Store co-founding the influential startup accelerator and seed capital firm Y Combinator his blog and Hacker News", this would be marked as an incomplete sentence issue because the sentence "Paul Graham is an English-born" is incomplete.

The context quality detector outputs a score between 0.0 and 1.0, where 0.0 indicates that the context is of good quality and 1.0 indicates that the context is of poor quality. The detector also provides a list of issues detected in the context and a reasoning field that explains the score and highlights the issues detected.

Request
Response

[
  {
    "context": "Paul Graham is an English-born computer scientist, entrepreneur, venture capitalist, author, and essayist. He is best known for his work on Lisp, his former startup Viaweb (later renamed Yahoo! Store), co-founding the influential startup accelerator and seed capital firm Y Combinator, his blog, and Hacker News. He was originally from New York.",
    "config": {
      "context_classification": {
        "detector_name": "default"
      }
    }
  }
]

[
  {
    "context_classification": {
      "results": [
        {
          "classification": "conflicting_information",
          "reasoning": "The snippet 'He was originally from New York.' is marked as an inconsistency because it conflicts with the earlier statement that Paul Graham is an 'English-born' computer scientist. Being 'English-born' implies that he was born in England, not New York.",
          "snippet": "He was originally from New York."
        }
      ],
      "score": 0.4
    }
  }
]

Code Example

The below example demonstrates how to use the context quality detector in a synchronous manner.

Python
Typescript

from aimon import Detect
import os

# This is a synchronous example
# Use async=True to use it asynchronously
# Use publish=True to publish to the AIMon UI

detect = Detect(
    values_returned=['context', 'generated_text'],
    config={"context_classification": {"detector_name": "default"}},
    publish=True,
    api_key=os.getenv("AIMON_API_KEY"),
    application_name="my_awesome_llm_app",
    model_name="my_awesome_llm_model"
)

@detect
def my_llm_app():
    return "Paul Graham is an English-born computer scientist, entrepreneur, venture capitalist, author, and essayist. He is best known for his work on Lisp, his former startup Viaweb (later renamed Yahoo! Store), co-founding the influential startup accelerator and seed capital firm Y Combinator, his blog, and Hacker News. He was originally from New York.", "This is a generated text"

context, gen_text, aimon_res = my_llm_app()

print(aimon_res)

import Client from "aimon";

// Create the AIMon client using an API Key (retrievable from the UI in your user profile).
const aimon = new Client({ authHeader: "Bearer API_KEY" });

const runDetect = async () => {
  const generatedText = "your_generated_text";
  const context = ["your_context"];
  const userQuery = "your_user_query";
  const config = { context_classification: { detector_name: "default" } };

  // Analyze the quality of the generated output using AIMon
  const response = await aimon.detect(
    generatedText,
    context,
    userQuery,
    config,
  );

  console.log("Response from detect:", response);
}

runDetect();

Code Example​

Code Example