Instruction Adherence

Given a set of “instructions”, the generated text, the input context and the user query, this API is able to check whether the generated text followed all the instructions specified. in the “instructions” field.

An LLM (Large Language Model) following instructions in the prompt is crucial for several reasons. Firstly, it ensures the generation of content that aligns with the user's needs and expectations, providing relevant and accurate information or responses. This adherence to instructions enhances the utility and effectiveness of the model, making it a reliable tool for various applications such as customer support, content creation, and educational assistance. Moreover, following prompts precisely helps maintain coherence and context, preventing misunderstandings or the generation of irrelevant information. It also ensures ethical compliance, avoiding the creation of inappropriate or harmful content. Ultimately, prompt adherence maximizes the value of the LLM, fostering user trust and satisfaction.

Example Request

[
  {
    "context": "Paul Graham is an English-born computer scientist, entrepreneur, venture capitalist, author, and essayist. He is best known for his work on Lisp, his former startup Viaweb (later renamed Yahoo! Store), co-founding the influential startup accelerator and seed capital firm Y Combinator, his blog, and Hacker News.",
    "instructions": "Write a summary of Paul Graham's career and achievements.",
    "generated_text": "Paul Graham has worked in several key areas throughout his career: IBM 1401: He began programming on the IBM 1401 during his school years, specifically in 9th grade. In addition, he has also been involved in writing essays and sharing his thoughts on technology, startups, and programming.",
    "config": {
      "instruction_adherence": {
        "detector_name": "default"
      }
    }
  }
]

Example Response

[
  {
    "instruction_adherence": {
      "results": [
        {
          "adherence": false,
          "detailed_explanation": "The response provides a very limited overview of Paul Graham's career, only mentioning his early programming on the IBM 1401 and general involvement in writing essays. It fails to cover significant achievements such as his work on Lisp, co-founding Y Combinator, and his influence in the tech and startup world.",
          "instruction": "Write a summary of Paul Graham's career and achievements."
        }
      ],
      "score": 0.0
    }
  }
]

Example (Synchronous detection)

The below example demonstrates how to use the instruction adherence detector in a synchronous manner.

Python

from aimon import Detect

detect = Detect(values_returned=['context', 'generated_text', 'instructions'], config={"instruction_adherence": {"detector_name": "default"}})

@detect
def my_llm_app(context, query, instructions):
    generated_text = my_llm_model(context, query)
    return context, generated_text, instructions

TypeScript

import Client from "aimon";

const aimon_client = new Client({
  authHeader: `Bearer API_KEY`,
});

const instruction_adherence_config = {
    instruction_adherence: {
      detector_name: "default"
    }
};

// Detect quality of the generated output using AIMon
const detectParams: Client.InferenceDetectParams.Body[] = [
  {
    context: contextDocs,
    generated_text: output.text,
    instructions: instructions,
    config: instruction_adherence_config
  },
];

// Call the API
const aimonResponse: Client.InferenceDetectResponse =
  await aimon_client.inference.detect(detectParams);