Toxicity

Given a context, generated text and optionally a user query or reference text, this API is able to generate various toxicity scores. The context should include the context documents along with the user query as passed in to the LLM. The output contains multiple labels with a value between 0.0 and 1.0 which indicates the likelihood of the label.

There are 6 different possible labels:

• identity_hate: Indicates possible hateful language that targets a person’s identity.
• toxic: Content that is generally considered unsafe for a broad audience.
• severe_toxic: Extremely harmful content that should not be shown to a general audience.
• obscene: Content that is offensive or disgusting by accepted standards of morality and decency.
• threat: Content that contains threatening language.
• insult: Content that contains disrespectful or scornfully abusive language.

Example Request

[
  {
    "context": "Paul Graham is an English-born computer scientist, entrepreneur, venture capitalist, author, and essayist. He is best known for his work on Lisp, his former startup Viaweb (later renamed Yahoo! Store), co-founding the influential startup accelerator and seed capital firm Y Combinator, his blog, and Hacker News.",
    "generated_text": "Paul Graham has worked in several key areas throughout his career: IBM 1401: He began programming on the IBM 1401 during his school years, specifically in 9th grade. In addition, he has also been involved in writing essays and sharing his thoughts on technology, startups, and programming.",
    "config": {
      "toxicity": {
        "detector_name": "default"
      }
    }
  }
]

Example Response

[
  {
    "toxicity": {
      "results": {
        "generated_text": {
          "detected_labels": {
            "identity_hate": 0.11749652028083801,
            "insult": 0.35200783610343933,
            "obscene": 0.24614322185516357,
            "severe_toxic": 0.05233444273471832,
            "threat": 0.119813933968544,
            "toxic": 0.11220399290323257
          },
          "text": "Paul Graham has worked in several key areas throughout his career: IBM 1401: He began programming on the IBM 1401 during his school years, specifically in 9th grade. In addition, he has also been involved in writing essays and sharing his thoughts on technology, startups, and programming."
        }
      },
      "score": 0.35200783610343933
    }
  }
]

Example (Synchronous detection)

The below example demonstrates how to use the instruction adherence detector in a synchronous manner.

Python

from aimon import Detect

detect = Detect(values_returned=['context', 'generated_text'],config={"toxicity": {"detector_name": "default"}})

@detect
def my_llm_app(context, query):
    generated_text = my_llm_model(context, query)
    return context, generated_text

TypeScript

import Client from "aimon";

const aimon_client = new Client({
  authHeader: `Bearer API_KEY`,
});

const toxicity_config = {
    toxicity: {
      detector_name: "default"
    }
};

// Detect quality of the generated output using AIMon
const detectParams: Client.InferenceDetectParams.Body[] = [
  {
    context: contextDocs,
    generated_text: output.text,
    config: toxicity_config
  },
];

// Call the API
const aimonResponse: Client.InferenceDetectResponse =
  await aimon_client.inference.detect(detectParams);