Skip to main content

Improving LLM Applications

Improve your LLM outputs by collecting structured feedback and fine-tuning examples using the Improvement Datasets Features.

Overview​

Improving LLM Apps allow users to create custom datasets to store examples of prompts, user queries, contexts, and model outputs. These datasets can then be used to retrain models based by allowing users to customize these datasets.

How It Works​

1. Create an Improvement Dataset​

Start by creating a new improvement dataset

Each dataset includes:

  • name: A descriptive name for the dataset (e.g., "Incorrect Summaries" or "Customer Support Errors").
  • description: Optional longer explanation of the dataset’s purpose.

Steps to create the dataset

  1. Go to the Datasets section in the application sidebar.

improvement_dataset_menu.png

  1. Click on the Improvement Datasets tab.

improvement_dataset_creation.png

  1. Click the Create (Green + button) button to open the creation modal.
  2. Fill in the dataset name and (optionally) a description.

improvement_dataset_modal.png

  1. Click Save to create your dataset.

Once the dataset is created, it will appear in the list and can be populated with records.

2. Add Records to the Dataset​

Once the dataset is created, you can populate it with new records. Each record represents a specific prompt and response that can be used to evaluate or improve model behavior.

Each record contains:

  • user_query: The original user input
  • context_docs: The context passed to the model (e.g. retrieved documents)
  • instructions: Any task-specific instruction
  • prompt: (Optional) The full prompt that was sent to the LLM to generate an output
  • output: (Optional) The LLM's output

Steps to add a record to the dataset

  1. Go to the LLM Apps using either Monitoring or Evals section in the application sidebar.

improvement_dataset_record1.png

  1. Click on an specific application and go to the Requests tab.

improvement_dataset_record2.png

  1. At the Requests tab, select one request

  2. Click on Add to dataset

improvement_dataset_record3.png

  1. Select your dataset to include the new record

improvement_dataset_record4.png

  1. (Optional) Once added to the dataset, go back to Datasets section in the application sidebar, click on the dataset that you added the record to. You can now optionally edit the different fields in the dataset manually if you want to change something in that dataset.

Use Cases​

  • Collect Training Data: Build datasets from user interactions where the LLM underperformed.
  • Fine-tuning: Use high-quality prompts and responses to fine-tune LLM models.

Tips​

  • Create separate datasets for different types of issues (hallucinations, tone, formatting, etc.)
  • Use the description field to keep your team aligned on what the dataset is intended to be used for.