Context Quality

The Context Quality API evaluates the quality of a generated response in the context of documents retrieved by your RAG pipeline. It is designed to help diagnose issues that can lead to hallucinations, grammar failures, or confusing outputs; and helps pinpoint whether these are symptoms of poor context usage or generation errors.

This evaluation is especially useful for summarization, answering, and generation pipelines where fidelity to retrieved context is important.

Labels Evaluated

The following signals are analyzed in the generated output:

• Conflicting Information
  The response contradicts itself or the context.
  Example: "Quantum is good bad..."

• Missing Punctuation
  The response omits necessary punctuation, making it hard to parse.
  Example: "Quantum is revolutionary exciting dangerous unstoppable"

• Excessive Noise
  The output contains malformed tokens, symbols, or artifacts.
  Example: "quantum @@% computers !! obsolet yes"

• Incomplete Sentences
  The sentence structure is broken, fragmented, or grammatically invalid.
  Example: "Quantum is..."

Scoring

The model returns a score between 0.0 and 1.0:

Score	Interpretation
0.0	Output is clean and well-formed
0.3–0.7	Minor quality issues present
1.0	Major failures or low-quality text detected

Each issue contributes probabilistically to the overall score.

Request

[
  {
    "context": [
      "Document 1: Quantum computing has completely replaced classical computing in all practical use-cases. It is now widely believed that classical computers are obsolete. This marks a major step toward scalable quantum computers.",
      "Document 2: Quantum computing remains highly experimental. Most researchers agree that classical computers will remain dominant for the foreseeable future. Quantum chips are limited and unreliable @@%."
    ],
    "user_query": "Summarize the key advancements mentioned in the article.",
    "generated_text": "Quantum is good bad the computerz from obsolet classical yes!!@@%",
    "config": {
      "context_classification": {
        "detector_name": "default"
      }
    }
  }
]

Response

[
    {
        "context_classification": {
            "instructions_list": [
                {
                    "explanation": "The response contains contradictory phrases like 'good bad', directly fulfilling the conflicting information requirement.",
                    "follow_probability": 0.9149,
                    "instruction": "Response includes conflicting information that contradicts itself or the provided context.",
                    "label": true
                },
                {
                    "explanation": "It lacks proper punctuation, with only one period and extra symbols ('!!@@%') instead of standard punctuation.",
                    "follow_probability": 0.5927,
                    "instruction": "Response lacks necessary punctuation, making the text difficult to parse or understand.",
                    "label": true
                },
                {
                    "explanation": "The answer includes special characters ('!!@@%') and unusual tokens ('computerz'), which qualify as excessive noise.",
                    "follow_probability": 0.9149,
                    "instruction": "Response includes excessive noise such as special characters, irrelevant tokens, or formatting artifacts.",
                    "label": true
                },
                {
                    "explanation": "The sentence is incomplete and grammatically broken, e.g., 'Quantum is good bad the computerz from obsolet classical yes!!@@%', hindering clarity.",
                    "follow_probability": 0.9526,
                    "instruction": "Response contains incomplete or grammatically broken sentences that hinder understanding.",
                    "label": true
                }
            ],
            "score": 1.0
        }
    }
]

Code Example

The below example demonstrates how to use the context quality detector in a synchronous manner.

Python

from aimon import Detect
import os

# This is a synchronous example
# Use async=True to use it asynchronously
# Use publish=True to publish to the AIMon UI

detect = Detect(
    values_returned=['context', 'generated_text'],
    config={"context_classification": {"detector_name": "default"}},
    publish=True,
    api_key=os.getenv("AIMON_API_KEY"),
    application_name="my_awesome_llm_app",
    model_name="my_awesome_llm_model"
)

@detect
def my_llm_app():
    return "Paul Graham is an English-born computer scientist, entrepreneur, venture capitalist, author, and essayist. He is best known for his work on Lisp, his former startup Viaweb (later renamed Yahoo! Store), co-founding the influential startup accelerator and seed capital firm Y Combinator, his blog, and Hacker News. He was originally from New York.", "This is a generated text"

context, gen_text, aimon_res = my_llm_app()

print(aimon_res)

Typescript

import Client from "aimon";

// Create the AIMon client using an API Key (retrievable from the UI in your user profile).
const aimon = new Client({ authHeader: "Bearer API_KEY" });

const runDetect = async () => {
  const generatedText = "your_generated_text";
  const context = ["your_context"];
  const userQuery = "your_user_query";
  const config = { context_classification: { detector_name: "default" } };

  // Analyze the quality of the generated output using AIMon
  const response = await aimon.detect(
    generatedText,
    context,
    userQuery,
    config,
  );

  console.log("Response from detect:", response);
}

runDetect();