Toxicity

Given a context, generated text and optionally a user query or reference text, this API is able to generate various toxicity scores. The context should include the context documents along with the user query as passed in to the LLM. The output contains multiple labels with a value between 0.0 and 1.0 which indicates the likelihood of the label.

There are 6 different possible labels:

• identity_hate: Indicates possible hateful language that targets a person’s identity.
• toxic: Content that is generally considered unsafe for a broad audience.
• severe_toxic: Extremely harmful content that should not be shown to a general audience.
• obscene: Content that is offensive or disgusting by accepted standards of morality and decency.
• threat: Content that contains threatening language.
• insult: Content that contains disrespectful or scornfully abusive language.

Request

[
  {
    "context": "Paul Graham is an English-born computer scientist, entrepreneur, venture capitalist, author, and essayist. He is best known for his work on Lisp, his former startup Viaweb (later renamed Yahoo! Store), co-founding the influential startup accelerator and seed capital firm Y Combinator, his blog, and Hacker News.",
    "generated_text": "Paul Graham has worked in several key areas throughout his career: IBM 1401: He began programming on the IBM 1401 during his school years, specifically in 9th grade. In addition, he has also been involved in writing essays and sharing his thoughts on technology, startups, and programming.",
    "config": {
      "toxicity": {
        "detector_name": "default"
      }
    }
  }
]

Response

Example Response

[
  {
    "toxicity": {
      "results": {
        "generated_text": {
          "detected_labels": {
            "identity_hate": 0.11749652028083801,
            "insult": 0.35200783610343933,
            "obscene": 0.24614322185516357,
            "severe_toxic": 0.05233444273471832,
            "threat": 0.119813933968544,
            "toxic": 0.11220399290323257
          },
          "text": "Paul Graham has worked in several key areas throughout his career: IBM 1401: He began programming on the IBM 1401 during his school years, specifically in 9th grade. In addition, he has also been involved in writing essays and sharing his thoughts on technology, startups, and programming."
        }
      },
      "score": 0.35200783610343933
    }
  }
]

Code Example

The below example demonstrates how to use the instruction adherence detector in a synchronous manner.

Python

from aimon import Detect
import os

# This is a synchronous example

# Use async=True to use it asynchronously

# Use publish=True to publish to the AIMon UI

detect = Detect(
    values_returned=['context', 'generated_text'],
    config={"toxicity": {"detector_name": "default"}},
    publish=True,
    api_key=os.getenv("AIMON_API_KEY"),
    application_name="my_awesome_llm_app",
    model_name="my_awesome_llm_model"
)

@detect
def my_llm_app(context, query):
    my_llm_model = lambda context, query: f'''I am a LLM trained to answer your questions.
    But I often don't fully answer your questions.
    The query you passed is: {query}.
    The context you passed is: {context}.'''
    generated_text = my_llm_model(context, query)
    return context, generated_text

context, gen_text, aimon_res = my_llm_app("This is a context", "This is a query")

print(aimon_res)

# DetectResult(status=200, detect_response=InferenceDetectResponseItem(result=None, toxicity={'results': {'generated_text': {'detected_labels': {'identity_hate': 0.09526486694812775, 'insult': 0.1885508894920349, 'obscene': 0.5094951391220093, 'severe_toxic': 0.04449573904275894, 'threat': 0.10070586949586868, 'toxic': 0.061487503349781036}, 'text': "I am a LLM trained to answer your questions.\n But I often don't fully answer your questions.\n The query you passed is: This is a query.\n The context you passed is: This is a context."}}, 'score': 0.5094951391220093}), publish_response=[])

TypeScript

import Client from "aimon";

// Create the AIMon client using an API Key (retrievable from the UI in your user profile).
const aimon = new Client({ authHeader: "Bearer API_KEY" });

const runDetect = async () => {
  const generatedText = "your_generated_text";
  const context = ["your_context"];
  const userQuery = "your_user_query";
  const config = { toxicity: { detector_name: "default" } };

  // Analyze the quality of the generated output using AIMon
  const response = await aimon.detect(
    generatedText,
    context,
    userQuery,
    config,
  );

  console.log("Response from detect:", response);
}

runDetect();