Completeness

The Completeness API evaluates how thoroughly a generated response addresses a user query based on the provided context. The context typically includes background documents and the original query that was passed to the language model. The goal is to determine whether the generated output fully covers all the necessary information, accurately reflects the context, and delivers a complete response without omissions or gaps. This is particularly useful in use cases such as retrieval-augmented generation, customer support, or domain-specific QA, where missing or incomplete answers can significantly reduce user trust and task success.

Response Format

The API returns a structured object containing the completeness evaluation. Each result includes:

• score (float, 0.0 – 1.0):

  A numerical indicator of how completely the response addresses the query:

  • 0.0–0.2: The response is incomplete or not relevant.
  • 0.2–0.7: The response is partially complete, missing key elements or details.
  • 0.7–1.0: The response is fully complete and covers all relevant information.

• instructions_list (array):

  A list of specific completeness criteria used to evaluate the response. Each item includes:

  • instruction: A rule representing one aspect of completeness.
  • label: Indicates whether the response followed the instruction (true) or violated it (false).
  • follow_probability: A confidence score indicating the likelihood the instruction was followed.
  • explanation: A natural-language rationale explaining why the instruction was marked true or false.

Note: The score reflects the overall completeness, but detailed reasoning is broken down across these instruction-level explanations.

Request

[
  {
    "context": "Paul Graham is an English-born computer scientist, entrepreneur, venture capitalist, author, and essayist. He is best known for his work on Lisp, his former startup Viaweb (later renamed Yahoo! Store), co-founding the influential startup accelerator and seed capital firm Y Combinator, his blog, and Hacker News.",
    "generated_text": "Paul Graham has worked in several key areas throughout his career: IBM 1401: He began programming on the IBM 1401 during his school years, specifically in 9th grade. In addition, he has also been involved in writing essays and sharing his thoughts on technology, startups, and programming.",
    "config": {
      "completeness": {
        "detector_name": "default"
      }
    }
  }
]

Response

[
    {
        "completeness": {
            "instructions_list": [
                {
                    "explanation": "The response only mentions IBM 1401 and essay writing, omitting key aspects like 'his school years' and '9th grade', which are part of the query.",
                    "follow_probability": 0.4688,
                    "instruction": "Response should fully address all aspects of the user query",
                    "label": false
                },
                {
                    "explanation": "It includes some explicit details ('IBM 1401', '9th grade') but omits others, failing to cover all key information.",
                    "follow_probability": 0.0759,
                    "instruction": "Response should include all key information explicitly stated in the context documents",
                    "label": false
                },
                {
                    "explanation": "Critical details such as 'school years' and '9th grade' are omitted, leaving gaps in the response.",
                    "follow_probability": 0.2942,
                    "instruction": "Response should not omit critical details needed to understand or answer the query",
                    "label": false
                },
                {
                    "explanation": "The response avoids incorrect or misleading interpretations, sticking to factual statements.",
                    "follow_probability": 0.9241,
                    "instruction": "Response should avoid including incorrect or misleading interpretations of the context",
                    "label": true
                },
                {
                    "explanation": "The response is structured into clear sections (e.g., 'IBM 1401: He began programming') which makes it easy to follow.",
                    "follow_probability": 0.9933,
                    "instruction": "Response should follow a clear and logical structure, making it easy to understand",
                    "label": true
                },
                {
                    "explanation": "It provides concrete examples ('during his school years, specifically in 9th grade') to clarify his work experience.",
                    "follow_probability": 0.7311,
                    "instruction": "Response should provide examples or explanations when necessary to clarify complex points",
                    "label": true
                },
                {
                    "explanation": "There are no contradictions; the response remains consistent throughout.",
                    "follow_probability": 0.9985,
                    "instruction": "Response should not contradict information in the context or within itself",
                    "label": true
                },
                {
                    "explanation": "The response accurately represents Paul Graham’s career without including irrelevant details.",
                    "follow_probability": 0.9841,
                    "instruction": "Response should accurately represent all relevant information without adding irrelevant content",
                    "label": true
                }
            ],
            "score": 0.625
        }
    }
]

Code Example

The below example demonstrates how to use the instruction adherence detector in a synchronous manner.

Python

from aimon import Detect
import os

# This is a synchronous example
# Use async=True to use it asynchronously
# Use publish=True to publish to the AIMon UI

detect = Detect(
    values_returned=['context', 'generated_text'],
    config={"completeness": {"detector_name": "default"}},
    publish=True,
    api_key=os.getenv("AIMON_API_KEY"),
    application_name="my_awesome_llm_app",
    model_name="my_awesome_llm_model"
)

@detect
def my_llm_app(context, query):
    my_llm_model = lambda context, query: f'''I am a LLM trained to answer your questions.
    But I often don't fully answer your question.
    The query you passed is: {query}.
    The context you passed is: {context}.'''
    generated_text = my_llm_model(context, query)
    return context, generated_text

context, gen_text, aimon_res = my_llm_app("This is a context", "This is a query")

print(aimon_res)

TypeScript

import Client from "aimon";

// Create the AIMon client using an API Key (retrievable from the UI in your user profile).
const aimon = new Client({ authHeader: "Bearer API_KEY" });

const runDetect = async () => {
  const generatedText = "your_generated_text";
  const context = ["your_context"];
  const userQuery = "your_user_query";
  const config = { completeness: { detector_name: "default" } };

  // Analyze the quality of the generated output using AIMon
  const response = await aimon.detect(
    generatedText,
    context,
    userQuery,
    config,
  );

  console.log("Response from detect:", response);
}
runDetect();