Dataset Serialization

Learn how to save and load datasets in different formats, with support for custom evaluators and IDE integration.

Overview

Pydantic Evals supports serializing datasets to files in two formats:

• YAML (.yaml, .yml) - Human-readable, great for version control
• JSON (.json) - Structured, machine-readable

Both formats support: - Automatic JSON schema generation for IDE autocomplete and validation - Custom evaluator serialization/deserialization - Type-safe loading with generic parameters

YAML Format

YAML is the recommended format for most use cases due to its readability and compact syntax.

Basic Example

from typing import Any

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected, IsInstance

# Create a dataset with typed parameters
dataset = Dataset[str, str, Any](
    name='my_tests',
    cases=[
        Case(
            name='test_1',
            inputs='hello',
            expected_output='HELLO',
        ),
    ],
    evaluators=[
        IsInstance(type_name='str'),
        EqualsExpected(),
    ],
)

# Save to YAML
dataset.to_file('my_tests.yaml')

This creates two files:

1. my_tests.yaml - The dataset
2. my_tests_schema.json - JSON schema for IDE support

YAML Output

# yaml-language-server: $schema=my_tests_schema.json
name: my_tests
cases:
- name: test_1
  inputs: hello
  expected_output: HELLO
evaluators:
- IsInstance: str
- EqualsExpected

JSON Schema for IDEs

The first line references the schema file:

# yaml-language-server: $schema=my_tests_schema.json

This enables: - ✅ Autocomplete in VS Code, PyCharm, and other editors - ✅ Inline validation while editing - ✅ Documentation tooltips for fields - ✅ Error highlighting for invalid data

Editor Support

The yaml-language-server comment is supported by:

• VS Code (with YAML extension)
• JetBrains IDEs (PyCharm, IntelliJ, etc.)
• Most editors with YAML language server support

See the YAML Language Server docs for more details.

Loading from YAML

from pathlib import Path
from typing import Any

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected, IsInstance

# First create and save the dataset
Path('my_tests.yaml').parent.mkdir(exist_ok=True)
dataset = Dataset[str, str, Any](
    name='my_tests',
    cases=[Case(name='test_1', inputs='hello', expected_output='HELLO')],
    evaluators=[IsInstance(type_name='str'), EqualsExpected()],
)
dataset.to_file('my_tests.yaml')

# Load the dataset with type parameters
dataset = Dataset[str, str, Any].from_file('my_tests.yaml')


def my_task(text: str) -> str:
    return text.upper()


# Run evaluation
report = dataset.evaluate_sync(my_task)

JSON Format

JSON format is useful for programmatic generation or when strict structure is required.

Basic Example

from typing import Any

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected

dataset = Dataset[str, str, Any](
    name='my_tests',
    cases=[
        Case(name='test_1', inputs='hello', expected_output='HELLO'),
    ],
    evaluators=[EqualsExpected()],
)

# Save to JSON
dataset.to_file('my_tests.json')

JSON Output

{
  "$schema": "my_tests_schema.json",
  "name": "my_tests",
  "cases": [
    {
      "name": "test_1",
      "inputs": "hello",
      "expected_output": "HELLO"
    }
  ],
  "evaluators": [
    "EqualsExpected"
  ]
}

The $schema key at the top enables IDE support similar to YAML.

Loading from JSON

from typing import Any

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected

# First create and save the dataset
dataset = Dataset[str, str, Any](
    name='my_tests',
    cases=[Case(name='test_1', inputs='hello', expected_output='HELLO')],
    evaluators=[EqualsExpected()],
)
dataset.to_file('my_tests.json')

# Load from JSON
dataset = Dataset[str, str, Any].from_file('my_tests.json')

Schema Generation

Automatic Schema Creation

By default, to_file() creates a JSON schema file alongside your dataset:

from typing import Any

from pydantic_evals import Case, Dataset

dataset = Dataset[str, str, Any](cases=[Case(inputs='test')])

# Creates both my_tests.yaml AND my_tests_schema.json
dataset.to_file('my_tests.yaml')

Custom Schema Location

from pathlib import Path
from typing import Any

from pydantic_evals import Case, Dataset

dataset = Dataset[str, str, Any](cases=[Case(inputs='test')])

# Create directories
Path('data').mkdir(exist_ok=True)

# Custom schema filename (relative to dataset file location)
dataset.to_file(
    'data/my_tests.yaml',
    schema_path='my_schema.json',
)

# No schema file
dataset.to_file('my_tests.yaml', schema_path=None)

Schema Path Templates

Use {stem} to reference the dataset filename:

from typing import Any

from pydantic_evals import Case, Dataset

dataset = Dataset[str, str, Any](cases=[Case(inputs='test')])

# Creates: my_tests.yaml and my_tests.schema.json
dataset.to_file(
    'my_tests.yaml',
    schema_path='{stem}.schema.json',
)

Manual Schema Generation

Generate a schema without saving the dataset:

import json
from typing import Any

from pydantic_evals import Dataset

# Get schema as dictionary for a specific dataset type
schema = Dataset[str, str, Any].model_json_schema_with_evaluators()

# Save manually
with open('custom_schema.json', 'w') as f:
    json.dump(schema, f, indent=2)

Custom Evaluators

Custom evaluators require special handling during serialization and deserialization.

Requirements

Custom evaluators must:

1. Be decorated with @dataclass
2. Inherit from Evaluator
3. Be passed to both to_file() and from_file()

Complete Example

from dataclasses import dataclass
from typing import Any

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Evaluator, EvaluatorContext


@dataclass
class CustomThreshold(Evaluator):
    """Check if output length exceeds a threshold."""

    min_length: int
    max_length: int = 100

    def evaluate(self, ctx: EvaluatorContext) -> bool:
        length = len(str(ctx.output))
        return self.min_length <= length <= self.max_length


# Create dataset with custom evaluator
dataset = Dataset[str, str, Any](
    cases=[
        Case(
            name='test_length',
            inputs='example',
            expected_output='long result',
            evaluators=[
                CustomThreshold(min_length=5, max_length=20),
            ],
        ),
    ],
)

# Save with custom evaluator types
dataset.to_file(
    'dataset.yaml',
    custom_evaluator_types=[CustomThreshold],
)

Saved YAML

# yaml-language-server: $schema=dataset_schema.json
cases:
- name: test_length
  inputs: example
  expected_output: long result
  evaluators:
  - CustomThreshold:
      min_length: 5
      max_length: 20

Loading with Custom Evaluators

from dataclasses import dataclass
from typing import Any

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Evaluator, EvaluatorContext


@dataclass
class CustomThreshold(Evaluator):
    """Check if output length exceeds a threshold."""

    min_length: int
    max_length: int = 100

    def evaluate(self, ctx: EvaluatorContext) -> bool:
        length = len(str(ctx.output))
        return self.min_length <= length <= self.max_length


# First create and save the dataset
dataset = Dataset[str, str, Any](
    cases=[
        Case(
            name='test_length',
            inputs='example',
            expected_output='long result',
            evaluators=[CustomThreshold(min_length=5, max_length=20)],
        ),
    ],
)
dataset.to_file('dataset.yaml', custom_evaluator_types=[CustomThreshold])

# Load with custom evaluator registry
dataset = Dataset[str, str, Any].from_file(
    'dataset.yaml',
    custom_evaluator_types=[CustomThreshold],
)

Important

You must pass custom_evaluator_types to both to_file() and from_file().

• to_file(): Includes the evaluator in the JSON schema
• from_file(): Registers the evaluator for deserialization

Evaluator Serialization Formats

Evaluators can be serialized in three forms:

1. Name Only (No Parameters)

evaluators:
- EqualsExpected
- IsInstance: str  # Using default parameter

2. Single Parameter (Short Form)

evaluators:
- IsInstance: str
- Contains: "required text"
- MaxDuration: 2.0

3. Multiple Parameters (Dict Form)

evaluators:
- CustomThreshold:
    min_length: 5
    max_length: 20
- LLMJudge:
    rubric: "Response is accurate"
    model: "openai:gpt-4o"
    include_input: true

Format Comparison

Feature	YAML	JSON
Human readable	✅ Excellent	⚠️ Good
Comments	✅ Yes	❌ No
Compact	✅ Yes	⚠️ Verbose
Machine parsing	✅ Good	✅ Excellent
IDE support	✅ Yes	✅ Yes
Version control	✅ Clean diffs	⚠️ Noisy diffs

Recommendation: Use YAML for most cases, JSON for programmatic generation.

Advanced: Evaluator Serialization Name

Customize how your evaluator appears in serialized files:

from dataclasses import dataclass

from pydantic_evals.evaluators import Evaluator, EvaluatorContext


@dataclass
class VeryLongDescriptiveEvaluatorName(Evaluator):
    @classmethod
    def get_serialization_name(cls) -> str:
        return 'ShortName'

    def evaluate(self, ctx: EvaluatorContext) -> bool:
        return True

In YAML:

evaluators:
- ShortName  # Instead of VeryLongDescriptiveEvaluatorName

Troubleshooting

Schema Not Found in IDE

Problem: YAML file doesn't show autocomplete

Solutions:

1. Check the schema path in the first line of YAML:

   # yaml-language-server: $schema=correct_schema_name.json
   

2. Verify schema file exists in the same directory

3. Restart the language server in your IDE

4. Install YAML extension (VS Code: "YAML" by Red Hat)

Custom Evaluator Not Found

Problem: ValueError: Unknown evaluator name: 'CustomEvaluator'

Solution: Pass custom_evaluator_types when loading:

from dataclasses import dataclass
from typing import Any

from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Evaluator, EvaluatorContext


@dataclass
class CustomEvaluator(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> bool:
        return True


# First create and save with custom evaluator
dataset = Dataset[str, str, Any](
    cases=[Case(inputs='test', evaluators=[CustomEvaluator()])],
)
dataset.to_file('tests.yaml', custom_evaluator_types=[CustomEvaluator])

# Load with custom evaluator types
dataset = Dataset[str, str, Any].from_file(
    'tests.yaml',
    custom_evaluator_types=[CustomEvaluator],  # Required!
)

Format Inference Failed

Problem: ValueError: Cannot infer format from extension

Solution: Specify format explicitly:

from typing import Any

from pydantic_evals import Case, Dataset

dataset = Dataset[str, str, Any](cases=[Case(inputs='test')])

# Explicit format for unusual extensions
dataset.to_file('data.txt', fmt='yaml')
dataset_loaded = Dataset[str, str, Any].from_file('data.txt', fmt='yaml')

Schema Generation Error

Problem: Custom evaluator causes schema generation to fail

Solution: Ensure evaluator is a proper dataclass:

from dataclasses import dataclass

from pydantic_evals.evaluators import Evaluator, EvaluatorContext


# ✅ Correct
@dataclass
class MyEvaluator(Evaluator):
    value: int

    def evaluate(self, ctx: EvaluatorContext) -> bool:
        return True


# ❌ Wrong: Missing @dataclass
class BadEvaluator(Evaluator):
    def __init__(self, value: int):
        self.value = value

    def evaluate(self, ctx: EvaluatorContext) -> bool:
        return True

Next Steps

• Dataset Management - Creating and organizing datasets
• Custom Evaluators - Write custom evaluation logic
• Core Concepts - Understand the data model