Output Relevance

The Output Relevance metric evaluates how well a generated response aligns with the core intent of a user's query. It identifies responses that may include unrelated or tangential information, miss the main point, or misinterpret the query. This is particularly useful in systems where precision and intent matching are critical; such as search engines, chatbots, and enterprise assistants.

Response Format

The API returns a list of detection results, each containing an output_relevance object with these key fields:

• score (float, 0.0 – 1.0):
  Represents how relevant the response is to the user query:

  • 0.0–0.2: Poor alignment with intent.
  • 0.2–0.7: Partial relevance; contains some irrelevant or misunderstood content.
  • 0.7–1.0: Strongly aligned; focused and relevant.

• instructions_list (array):
  A list of individual guideline evaluations:

  • instruction: A rule representing one aspect of output relevance (e.g., "Response should directly address the core intent").
  • label: Indicates whether the response followed the instruction (true) or violated it (false).
  • follow_probability: A confidence score indicating the likelihood the instruction was followed.
  • explanation: A natural-language rationale explaining why the instruction was marked true or false.

Note: The score is an overall judgment, but the instruction-level breakdown provides insight into specific strengths or issues in the response.

API Request & Response Example

Request

[
  {
    "user_query": "Describe Paul Graham's work life?",
    "generated_text": "Paul Graham has worked in several key areas throughout his career: IBM 1401: He began programming on the IBM 1401 during his school years, specifically in 9th grade. In addition, he has also been involved in writing essays and sharing his thoughts on technology, startups, and programming.",
    "config": {
      "output_relevance": {
        "detector_name": "default",
        "explain":true
      }
    }
  }
]

Response

[
    {
        "output_relevance": {
            "instructions_list": [
                {
                    "explanation": "The response directly addresses the query by describing Paul Graham’s work life, e.g., 'worked in several key areas'.",
                    "follow_probability": 0.982,
                    "instruction": "Response should directly address the core intent of the user query.",
                    "label": true
                },
                {
                    "explanation": "It prioritizes relevant information by focusing on his programming and essay writing, matching the query.",
                    "follow_probability": 0.9797,
                    "instruction": "Response should prioritize information that is specifically relevant to the question asked.",
                    "label": true
                },
                {
                    "explanation": "The answer avoids unrelated tangential details, staying focused on his work life.",
                    "follow_probability": 0.9933,
                    "instruction": "Response should avoid including unrelated or tangential information.",
                    "label": true
                },
                {
                    "explanation": "The response accurately reflects the query without misinterpreting or distorting its meaning.",
                    "follow_probability": 0.9997,
                    "instruction": "Response should not misinterpret or distort the meaning of the query.",
                    "label": true
                }
            ],
            "score": 1
        }
    }
]

Code Examples

Python (Sync)

# Synchronous example

import os
from aimon import Client
import json

# Initialize client
client = Client(auth_header=f"Bearer {os.environ['AIMON_API_KEY']}")

# Construct payload
payload = [{
    "generated_text": "Our return policy allows returns within 30 days of purchase.",
    "user_query": "What is your return policy?",
    "config": {
        "output_relevance": {
            "detector_name": "default",
            "explain": True
        }
    },
    "publish": False
}]

# Call sync detect
response = client.inference.detect(body=payload)

# Print result
print(json.dumps(response[0].output_relevance, indent=2))

Python (Async)

# Aynchronous example

import os
import json
from aimon import AsyncClient

aimon_api_key = os.environ["AIMON_API_KEY"]

aimon_payload = {
    "generated_text": "Apples are fruits that grow on trees.",
    "user_query": "What is an apple?",
    "config": {
        "output_relevance": {
            "detector_name": "default",
            "explain": True
        }
    },
    "publish": True,
    "async_mode": True,
    "application_name": "async_metric_example",
    "model_name": "async_metric_example"
}

data_to_send = [aimon_payload]

async def call_aimon():
    async with AsyncClient(auth_header=f"Bearer {aimon_api_key}") as aimon:
        return await aimon.inference.detect(body=data_to_send)

resp = await call_aimon()
print(json.dumps(resp, indent=2))
print("View results at: https://www.app.aimon.ai/llmapps?source=sidebar&stage=production")

Python (Decorator)

import os
from aimon import Detect

detect = Detect(
    values_returned=["user_query", "generated_text"],
    config={"output_relevance": {"detector_name": "default", "explain":True}},
    api_key=os.getenv("AIMON_API_KEY"),
    application_name="application_name",
    model_name="model_name"
)

@detect
def output_relevance_test(user_query, generated_text):
    return user_query, generated_text

user_query, generated_text, aimon_result = output_relevance_test(
    "Explain the role of mitochondria in cells.",
    "The Eiffel Tower is one of the most visited monuments in the world."
)
print(aimon_result)

TypeScript

import Client from "aimon";
import dotenv from "dotenv";

dotenv.config();

const aimon = new Client({
  authHeader: `Bearer ${process.env.AIMON_API_KEY}`,
});

const run = async () => {
  const response = await aimon.detect({
    userQuery: "Who wrote Hamlet?",
    generatedText: "William Shakespeare was the author of Hamlet.",
    config: {
      output_relevance: {
        detector_name: "default",
        explain: true,
      },
    },
  });

  console.log("AIMon response:", JSON.stringify(response, null, 2));
};

run();