Completeness

Given a context, generated text and optionally a user query or a reference text, this API is able to detect if the generated text completely answered the user's question. The context should include the context documents as passed in to the LLM. The output contains a "score" that is between 0.0 and 1.0 which indicates the degree of completeness. If the generated answer is not at all relevant to the user query, a score between 0.0 to 0.2 is possible. If the generated answer is relevant but misses some information, a score between 0.2 and 0.7 is possible. If the generated answer is relevant and fully captures all of the information, a score between 0.7 and 1.0 is possible. The API also includes a "reasoning" field that is a text based explanation of the score. It also does a best effort method of pointing out the points that were missed from the expected answer.

Request

[
  {
    "context": "Paul Graham is an English-born computer scientist, entrepreneur, venture capitalist, author, and essayist. He is best known for his work on Lisp, his former startup Viaweb (later renamed Yahoo! Store), co-founding the influential startup accelerator and seed capital firm Y Combinator, his blog, and Hacker News.",
    "generated_text": "Paul Graham has worked in several key areas throughout his career: IBM 1401: He began programming on the IBM 1401 during his school years, specifically in 9th grade. In addition, he has also been involved in writing essays and sharing his thoughts on technology, startups, and programming.",
    "config": {
      "completeness": {
        "detector_name": "default"
      }
    }
  }
]

Response

[
  {
    "completeness": {
      "reasoning": "The generated answer is somewhat relevant to the query but omits significant information from the context documents, particularly about Paul Graham's contributions to Lisp, co-founding Y Combinator, and his work with Viaweb. It also includes details like the IBM 1401 that are not mentioned in the context, leading to inaccuracies and potential confusion. Thus, it doesn't provide a complete understanding of his career and achievements.",
      "score": 0.227
    }
  }
]

Code Example

The below example demonstrates how to use the instruction adherence detector in a synchronous manner.

Python

from aimon import Detect
import os

# This is a synchronous example
# Use async=True to use it asynchronously
# Use publish=True to publish to the AIMon UI

detect = Detect(
    values_returned=['context', 'generated_text'],
    config={"completeness": {"detector_name": "default"}},
    publish=True,
    api_key=os.getenv("AIMON_API_KEY"),
    application_name="my_awesome_llm_app",
    model_name="my_awesome_llm_model"
)

@detect
def my_llm_app(context, query):
    my_llm_model = lambda context, query: f'''I am a LLM trained to answer your questions.
    But I often don't fully answer your question.
    The query you passed is: {query}.
    The context you passed is: {context}.'''
    generated_text = my_llm_model(context, query)
    return context, generated_text

context, gen_text, aimon_res = my_llm_app("This is a context", "This is a query")

print(aimon_res)

# DetectResult(status=200, detect_response=InferenceDetectResponseItem(result=None, completeness={'reasoning': 'The generated answer lacks relevance to the user query and does not utilize the provided context document effectively, offering only vague statements without addressing the query or including any meaningful content.', 'score': 0.123}), publish_response=[])

TypeScript

import Client from "aimon";

// Create the AIMon client using an API Key (retrievable from the UI in your user profile).
const aimon = new Client({ authHeader: "Bearer API_KEY" });

const runDetect = async () => {
  const generatedText = "your_generated_text";
  const context = ["your_context"];
  const userQuery = "your_user_query";
  const config = { completeness: { detector_name: "default" } };

  // Analyze the quality of the generated output using AIMon
  const response = await aimon.detect(
    generatedText,
    context,
    userQuery,
    config,
  );

  console.log("Response from detect:", response);
}
runDetect();