Models

PydanticAI is Model-agnostic and has built in support for the following model providers:

OpenAI
Anthropic
Gemini via two different APIs: Generative Language API and VertexAI API
Ollama
Groq
Mistral
Cohere
Bedrock

See OpenAI-compatible models for more examples on how to use models such as OpenRouter, and Grok (xAI) that support the OpenAI SDK.

You can also add support for other models.

PydanticAI also comes with TestModel and FunctionModel for testing and development.

To use each model provider, you need to configure your local environment and make sure you have the right packages installed.

Models, Interfaces, and Providers

PydanticAI uses a few key terms to describe how it interacts with different LLMs:

Model: This refers to the specific LLM model you want to handle your requests (e.g., gpt-4o, claude-3-5-sonnet-latest, gemini-1.5-flash). It's the "brain" that processes your prompts and generates responses. You specify the Model as a parameter to the Interface.
Interface: This refers to a PydanticAI class used to make requests following a specific LLM API (generally by wrapping a vendor-provided SDK, like the openai python SDK). These classes implement a vendor-SDK-agnostic API, ensuring a single PydanticAI agent is portable to different LLM vendors without any other code changes just by swapping out the Interface it uses. Currently, interface classes are named roughly in the format <VendorSdk>Model, for example, we have OpenAIModel, AnthropicModel, GeminiModel, etc. These Model classes will soon be renamed to <VendorSdk>Interface to reflect this terminology better.
Provider: This refers to Interface-specific classes which handle the authentication and connections to an LLM vendor. Passing a non-default Provider as a parameter to an Interface is how you can ensure that your agent will make requests to a specific endpoint, or make use of a specific approach to authentication (e.g., you can use Vertex-specific auth with the GeminiModel by way of the VertexProvider). In particular, this is how you can make use of an AI gateway, or an LLM vendor that offers API compatibility with the vendor SDK used by an existing interface (such as OpenAIModel).

In short, you select a model, PydanticAI uses the appropriate interface class, and the provider handles the connection and authentication to the underlying service.

OpenAI

Install

To use OpenAI models, you need to either install pydantic-ai, or install pydantic-ai-slim with the openai optional group:

pipuv

pip install "pydantic-ai-slim[openai]"

uv add "pydantic-ai-slim[openai]"

Configuration

To use OpenAIModel through their main API, go to platform.openai.com and follow your nose until you find the place to generate an API key.

Environment variable

Once you have the API key, you can set it as an environment variable:

export OPENAI_API_KEY='your-api-key'

You can then use OpenAIModel by name:

openai_model_by_name.py

from pydantic_ai import Agent

agent = Agent('openai:gpt-4o')
...

Or initialise the model directly with just the model name:

openai_model_init.py

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel

model = OpenAIModel('gpt-4o')
agent = Agent(model)
...

By default, the OpenAIModel uses the OpenAIProvider with the base_url set to https://api.openai.com/v1.

`provider` argument

You can provide a custom Provider via the provider argument:

openai_model_provider.py

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.openai import OpenAIProvider

model = OpenAIModel('gpt-4o', provider=OpenAIProvider(api_key='your-api-key'))
agent = Agent(model)
...

Custom OpenAI Client

OpenAIProvider also accepts a custom AsyncOpenAI client via the openai_client parameter, so you can customise the organization, project, base_url etc. as defined in the OpenAI API docs.

You could also use the AsyncAzureOpenAI client to use the Azure OpenAI API.

openai_azure.py

from openai import AsyncAzureOpenAI

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.openai import OpenAIProvider

client = AsyncAzureOpenAI(
    azure_endpoint='...',
    api_version='2024-07-01-preview',
    api_key='your-api-key',
)

model = OpenAIModel(
    'gpt-4o',
    provider=OpenAIProvider(openai_client=client),
)
agent = Agent(model)
...

Anthropic

Install

To use AnthropicModel models, you need to either install pydantic-ai, or install pydantic-ai-slim with the anthropic optional group:

pipuv

pip install "pydantic-ai-slim[anthropic]"

uv add "pydantic-ai-slim[anthropic]"

Configuration

To use Anthropic through their API, go to console.anthropic.com/settings/keys to generate an API key.

AnthropicModelName contains a list of available Anthropic models.

Environment variable

Once you have the API key, you can set it as an environment variable:

export ANTHROPIC_API_KEY='your-api-key'

You can then use AnthropicModel by name:

anthropic_model_by_name.py

from pydantic_ai import Agent

agent = Agent('anthropic:claude-3-5-sonnet-latest')
...

Or initialise the model directly with just the model name:

anthropic_model_init.py

from pydantic_ai import Agent
from pydantic_ai.models.anthropic import AnthropicModel

model = AnthropicModel('claude-3-5-sonnet-latest')
agent = Agent(model)
...

`provider` argument

You can provide a custom Provider via the provider argument:

anthropic_model_provider.py

from pydantic_ai import Agent
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.providers.anthropic import AnthropicProvider

model = AnthropicModel(
    'claude-3-5-sonnet-latest', provider=AnthropicProvider(api_key='your-api-key')
)
agent = Agent(model)
...

Custom HTTP Client

You can customize the AnthropicProvider with a custom httpx.AsyncClient:

anthropic_model_custom_provider.py

from httpx import AsyncClient

from pydantic_ai import Agent
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.providers.anthropic import AnthropicProvider

custom_http_client = AsyncClient(timeout=30)
model = AnthropicModel(
    'claude-3-5-sonnet-latest',
    provider=AnthropicProvider(api_key='your-api-key', http_client=custom_http_client),
)
agent = Agent(model)
...

Gemini

Install

To use GeminiModel models, you just need to install pydantic-ai or pydantic-ai-slim, no extra dependencies are required.

Configuration

GeminiModel let's you use the Google's Gemini models through their Generative Language API, generativelanguage.googleapis.com.

GeminiModelName contains a list of available Gemini models that can be used through this interface.

To use GeminiModel, go to aistudio.google.com and select "Create API key".

Environment variable

Once you have the API key, you can set it as an environment variable:

export GEMINI_API_KEY=your-api-key

You can then use GeminiModel by name:

gemini_model_by_name.py

from pydantic_ai import Agent

agent = Agent('google-gla:gemini-2.0-flash')
...

Note

The google-gla provider prefix represents the Google Generative Language API for GeminiModels. google-vertex is used with Vertex AI.

Or initialise the model directly with just the model name and provider:

gemini_model_init.py

from pydantic_ai import Agent
from pydantic_ai.models.gemini import GeminiModel

model = GeminiModel('gemini-2.0-flash', provider='google-gla')
agent = Agent(model)
...

`provider` argument

You can provide a custom Provider via the provider argument:

gemini_model_provider.py

from pydantic_ai import Agent
from pydantic_ai.models.gemini import GeminiModel
from pydantic_ai.providers.google_gla import GoogleGLAProvider

model = GeminiModel(
    'gemini-2.0-flash', provider=GoogleGLAProvider(api_key='your-api-key')
)
agent = Agent(model)
...

You can also customize the GoogleGLAProvider with a custom http_client:

gemini_model_custom_provider.py

from httpx import AsyncClient

from pydantic_ai import Agent
from pydantic_ai.models.gemini import GeminiModel
from pydantic_ai.providers.google_gla import GoogleGLAProvider

custom_http_client = AsyncClient(timeout=30)
model = GeminiModel(
    'gemini-2.0-flash',
    provider=GoogleGLAProvider(api_key='your-api-key', http_client=custom_http_client),
)
agent = Agent(model)
...

Gemini via VertexAI

If you are an enterprise user, you should use the google-vertex provider with GeminiModel which uses the *-aiplatform.googleapis.com API.

GeminiModelName contains a list of available Gemini models that can be used through this interface.

Install

To use the google-vertex provider with GeminiModel, you need to either install pydantic-ai, or install pydantic-ai-slim with the vertexai optional group:

pipuv

pip install "pydantic-ai-slim[vertexai]"

uv add "pydantic-ai-slim[vertexai]"

Configuration

This interface has a number of advantages over generativelanguage.googleapis.com documented above:

The VertexAI API comes with more enterprise readiness guarantees.
You can purchase provisioned throughput with VertexAI to guarantee capacity.
If you're running PydanticAI inside GCP, you don't need to set up authentication, it should "just work".
You can decide which region to use, which might be important from a regulatory perspective, and might improve latency.

The big disadvantage is that for local development you may need to create and configure a "service account", which I've found extremely painful to get right in the past.

Whichever way you authenticate, you'll need to have VertexAI enabled in your GCP account.

Application default credentials

Luckily if you're running PydanticAI inside GCP, or you have the gcloud CLI installed and configured, you should be able to use VertexAIModel without any additional setup.

To use VertexAIModel, with application default credentials configured (e.g. with gcloud), you can simply use:

vertexai_application_default_credentials.py

from pydantic_ai import Agent
from pydantic_ai.models.gemini import GeminiModel

model = GeminiModel('gemini-2.0-flash', provider='google-vertex')
agent = Agent(model)
...

Internally this uses google.auth.default() from the google-auth package to obtain credentials.

Won't fail until agent.run()

Because google.auth.default() requires network requests and can be slow, it's not run until you call agent.run().

You may also need to pass the project_id argument to GoogleVertexProvider if application default credentials don't set a project, if you pass project_id and it conflicts with the project set by application default credentials, an error is raised.

Service account

If instead of application default credentials, you want to authenticate with a service account, you'll need to create a service account, add it to your GCP project (note: AFAIK this step is necessary even if you created the service account within the project), give that service account the "Vertex AI Service Agent" role, and download the service account JSON file.

Once you have the JSON file, you can use it thus:

vertexai_service_account.py

from pydantic_ai import Agent
from pydantic_ai.models.gemini import GeminiModel
from pydantic_ai.providers.google_vertex import GoogleVertexProvider

model = GeminiModel(
    'gemini-2.0-flash',
    provider=GoogleVertexProvider(service_account_file='path/to/service-account.json'),
)
agent = Agent(model)
...

Alternatively, if you already have the service account information in memory, you can pass it as a dictionary:

vertexai_service_account.py

import json

from pydantic_ai import Agent
from pydantic_ai.models.gemini import GeminiModel
from pydantic_ai.providers.google_vertex import GoogleVertexProvider

service_account_info = json.loads(
    '{"type": "service_account", "project_id": "my-project-id"}'
)
model = GeminiModel(
    'gemini-2.0-flash',
    provider=GoogleVertexProvider(service_account_info=service_account_info),
)
agent = Agent(model)
...

Customising region

Whichever way you authenticate, you can specify which region requests will be sent to via the region argument.

Using a region close to your application can improve latency and might be important from a regulatory perspective.

vertexai_region.py

from pydantic_ai import Agent
from pydantic_ai.models.gemini import GeminiModel
from pydantic_ai.providers.google_vertex import GoogleVertexProvider

model = GeminiModel(
    'gemini-2.0-flash', provider=GoogleVertexProvider(region='asia-east1')
)
agent = Agent(model)
...

You can also customize the GoogleVertexProvider with a custom http_client:

vertexai_custom_provider.py

from httpx import AsyncClient

from pydantic_ai import Agent
from pydantic_ai.models.gemini import GeminiModel
from pydantic_ai.providers.google_vertex import GoogleVertexProvider

custom_http_client = AsyncClient(timeout=30)
model = GeminiModel(
    'gemini-2.0-flash',
    provider=GoogleVertexProvider(region='asia-east1', http_client=custom_http_client),
)
agent = Agent(model)
...

Groq

Install

To use GroqModel, you need to either install pydantic-ai, or install pydantic-ai-slim with the groq optional group:

pipuv

pip install "pydantic-ai-slim[groq]"

uv add "pydantic-ai-slim[groq]"

Configuration

To use Groq through their API, go to console.groq.com/keys and follow your nose until you find the place to generate an API key.

GroqModelName contains a list of available Groq models.

Environment variable

Once you have the API key, you can set it as an environment variable:

export GROQ_API_KEY='your-api-key'

You can then use GroqModel by name:

groq_model_by_name.py

from pydantic_ai import Agent

agent = Agent('groq:llama-3.3-70b-versatile')
...

Or initialise the model directly with just the model name:

groq_model_init.py

from pydantic_ai import Agent
from pydantic_ai.models.groq import GroqModel

model = GroqModel('llama-3.3-70b-versatile')
agent = Agent(model)
...

`provider` argument

You can provide a custom Provider via the provider argument:

groq_model_provider.py

from pydantic_ai import Agent
from pydantic_ai.models.groq import GroqModel
from pydantic_ai.providers.groq import GroqProvider

model = GroqModel(
    'llama-3.3-70b-versatile', provider=GroqProvider(api_key='your-api-key')
)
agent = Agent(model)
...

You can also customize the GroqProvider with a custom httpx.AsyncHTTPClient:

groq_model_custom_provider.py

from httpx import AsyncClient

from pydantic_ai import Agent
from pydantic_ai.models.groq import GroqModel
from pydantic_ai.providers.groq import GroqProvider

custom_http_client = AsyncClient(timeout=30)
model = GroqModel(
    'llama-3.3-70b-versatile',
    provider=GroqProvider(api_key='your-api-key', http_client=custom_http_client),
)
agent = Agent(model)
...

Mistral

Install

To use MistralModel, you need to either install pydantic-ai, or install pydantic-ai-slim with the mistral optional group:

pipuv

pip install "pydantic-ai-slim[mistral]"

uv add "pydantic-ai-slim[mistral]"

Configuration

To use Mistral through their API, go to console.mistral.ai/api-keys/ and follow your nose until you find the place to generate an API key.

LatestMistralModelNames contains a list of the most popular Mistral models.

Environment variable

Once you have the API key, you can set it as an environment variable:

export MISTRAL_API_KEY='your-api-key'

You can then use MistralModel by name:

mistral_model_by_name.py

from pydantic_ai import Agent

agent = Agent('mistral:mistral-large-latest')
...

Or initialise the model directly with just the model name:

mistral_model_init.py

from pydantic_ai import Agent
from pydantic_ai.models.mistral import MistralModel

model = MistralModel('mistral-small-latest')
agent = Agent(model)
...

`provider` argument

You can provide a custom Provider via the provider argument:

groq_model_provider.py

from pydantic_ai import Agent
from pydantic_ai.models.mistral import MistralModel
from pydantic_ai.providers.mistral import MistralProvider

model = MistralModel(
    'mistral-large-latest', provider=MistralProvider(api_key='your-api-key')
)
agent = Agent(model)
...

You can also customize the provider with a custom httpx.AsyncHTTPClient:

groq_model_custom_provider.py

from httpx import AsyncClient

from pydantic_ai import Agent
from pydantic_ai.models.mistral import MistralModel
from pydantic_ai.providers.mistral import MistralProvider

custom_http_client = AsyncClient(timeout=30)
model = MistralModel(
    'mistral-large-latest',
    provider=MistralProvider(api_key='your-api-key', http_client=custom_http_client),
)
agent = Agent(model)
...

Cohere

Install

To use CohereModel, you need to either install pydantic-ai, or install pydantic-ai-slim with the cohere optional group:

pipuv

pip install "pydantic-ai-slim[cohere]"

uv add "pydantic-ai-slim[cohere]"

Configuration

To use Cohere through their API, go to dashboard.cohere.com/api-keys and follow your nose until you find the place to generate an API key.

CohereModelName contains a list of the most popular Cohere models.

Environment variable

Once you have the API key, you can set it as an environment variable:

export CO_API_KEY='your-api-key'

You can then use CohereModel by name:

cohere_model_by_name.py

from pydantic_ai import Agent

agent = Agent('cohere:command')
...

Or initialise the model directly with just the model name:

cohere_model_init.py

from pydantic_ai import Agent
from pydantic_ai.models.cohere import CohereModel

model = CohereModel('command')
agent = Agent(model)
...

`provider` argument

You can provide a custom Provider via the provider argument:

cohere_model_provider.py

from pydantic_ai import Agent
from pydantic_ai.models.cohere import CohereModel
from pydantic_ai.providers.cohere import CohereProvider

model = CohereModel('command', provider=CohereProvider(api_key='your-api-key'))
agent = Agent(model)
...

You can also customize the CohereProvider with a custom http_client:

cohere_model_custom_provider.py

from httpx import AsyncClient

from pydantic_ai import Agent
from pydantic_ai.models.cohere import CohereModel
from pydantic_ai.providers.cohere import CohereProvider

custom_http_client = AsyncClient(timeout=30)
model = CohereModel(
    'command',
    provider=CohereProvider(api_key='your-api-key', http_client=custom_http_client),
)
agent = Agent(model)
...

Bedrock

Install

To use BedrockConverseModel, you need to either install pydantic-ai, or install pydantic-ai-slim with the bedrock optional group:

pipuv

pip install "pydantic-ai-slim[bedrock]"

uv add "pydantic-ai-slim[bedrock]"

Configuration

To use AWS Bedrock, you'll need an AWS account with Bedrock enabled and appropriate credentials. You can use either AWS credentials directly or a pre-configured boto3 client.

BedrockModelName contains a list of available Bedrock models, including models from Anthropic, Amazon, Cohere, Meta, and Mistral.

Environment variables

You can set your AWS credentials as environment variables (among other options:

export AWS_ACCESS_KEY_ID='your-access-key'
export AWS_SECRET_ACCESS_KEY='your-secret-key'
export AWS_DEFAULT_REGION='us-east-1'  # or your preferred region

You can then use BedrockConverseModel by name:

bedrock_model_by_name.py

from pydantic_ai import Agent

agent = Agent('bedrock:anthropic.claude-3-sonnet-20240229-v1:0')
...

Or initialize the model directly with just the model name:

bedrock_model_init.py

from pydantic_ai import Agent
from pydantic_ai.models.bedrock import BedrockConverseModel

model = BedrockConverseModel('anthropic.claude-3-sonnet-20240229-v1:0')
agent = Agent(model)
...

`provider` argument

You can provide a custom BedrockProvider via the provider argument. This is useful when you want to specify credentials directly or use a custom boto3 client:

bedrock_model_provider.py

from pydantic_ai import Agent
from pydantic_ai.models.bedrock import BedrockConverseModel
from pydantic_ai.providers.bedrock import BedrockProvider

# Using AWS credentials directly
model = BedrockConverseModel(
    'anthropic.claude-3-sonnet-20240229-v1:0',
    provider=BedrockProvider(
        region_name='us-east-1',
        aws_access_key_id='your-access-key',
        aws_secret_access_key='your-secret-key',
    ),
)
agent = Agent(model)
...

You can also pass a pre-configured boto3 client:

bedrock_model_boto3.py

import boto3

from pydantic_ai import Agent
from pydantic_ai.models.bedrock import BedrockConverseModel
from pydantic_ai.providers.bedrock import BedrockProvider

# Using a pre-configured boto3 client
bedrock_client = boto3.client('bedrock-runtime', region_name='us-east-1')
model = BedrockConverseModel(
    'anthropic.claude-3-sonnet-20240229-v1:0',
    provider=BedrockProvider(bedrock_client=bedrock_client),
)
agent = Agent(model)
...

OpenAI-compatible Models

Many of the models are compatible with OpenAI API, and thus can be used with OpenAIModel in PydanticAI. Before getting started, check the OpenAI section for installation and configuration instructions.

To use another OpenAI-compatible API, you can make use of the base_url and api_key arguments from OpenAIProvider:

deepseek_model_init.py

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.openai import OpenAIProvider

model = OpenAIModel(
    'model_name',
    provider=OpenAIProvider(
        base_url='https://<openai-compatible-api-endpoint>.com', api_key='your-api-key'
    ),
)
agent = Agent(model)
...

You can also use the provider argument with a custom provider class like the DeepSeekProvider:

deepseek_model_init_provider_class.py

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.deepseek import DeepSeekProvider

model = OpenAIModel(
    'deepseek-chat',
    provider=DeepSeekProvider(api_key='your-deepseek-api-key'),
)
agent = Agent(model)
...

You can also customize any provider with a custom http_client:

deepseek_model_init_provider_custom.py

from httpx import AsyncClient

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.deepseek import DeepSeekProvider

custom_http_client = AsyncClient(timeout=30)
model = OpenAIModel(
    'deepseek-chat',
    provider=DeepSeekProvider(
        api_key='your-deepseek-api-key', http_client=custom_http_client
    ),
)
agent = Agent(model)
...

Ollama

To use Ollama, you must first download the Ollama client, and then download a model using the Ollama model library.

You must also ensure the Ollama server is running when trying to make requests to it. For more information, please see the Ollama documentation.

Example local usage

With ollama installed, you can run the server with the model you want to use:

terminal-run-ollama

ollama run llama3.2

(this will pull the llama3.2 model if you don't already have it downloaded)

Then run your code, here's a minimal example:

ollama_example.py

from pydantic import BaseModel

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.openai import OpenAIProvider


class CityLocation(BaseModel):
    city: str
    country: str


ollama_model = OpenAIModel(
    model_name='llama3.2', provider=OpenAIProvider(base_url='http://localhost:11434/v1')
)
agent = Agent(ollama_model, result_type=CityLocation)

result = agent.run_sync('Where were the olympics held in 2012?')
print(result.data)
#> city='London' country='United Kingdom'
print(result.usage())
"""
Usage(requests=1, request_tokens=57, response_tokens=8, total_tokens=65, details=None)
"""

Example using a remote server

ollama_example_with_remote_server.py

from pydantic import BaseModel

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.openai import OpenAIProvider

ollama_model = OpenAIModel(
    model_name='qwen2.5-coder:7b',  # (1)!
    provider=OpenAIProvider(base_url='http://192.168.1.74:11434/v1'),  # (2)!
)


class CityLocation(BaseModel):
    city: str
    country: str


agent = Agent(model=ollama_model, result_type=CityLocation)

result = agent.run_sync('Where were the olympics held in 2012?')
print(result.data)
#> city='London' country='United Kingdom'
print(result.usage())
"""
Usage(requests=1, request_tokens=57, response_tokens=8, total_tokens=65, details=None)
"""

The name of the model running on the remote server
The url of the remote server

Azure AI Foundry

If you want to use Azure AI Foundry as your provider, you can do so by using the AzureProvider class.

azure_provider_example.py

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.azure import AzureProvider

model = OpenAIModel(
    'gpt-4o',
    provider=AzureProvider(
        azure_endpoint='your-azure-endpoint',
        api_version='your-api-version',
        api_key='your-api-key',
    ),
)
agent = Agent(model)
...

OpenRouter

To use OpenRouter, first create an API key at openrouter.ai/keys.

Once you have the API key, you can use it with the OpenAIProvider:

openrouter_model_init.py

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.openai import OpenAIProvider

model = OpenAIModel(
    'anthropic/claude-3.5-sonnet',
    provider=OpenAIProvider(
        base_url='https://openrouter.ai/api/v1',
        api_key='your-openrouter-api-key',
    ),
)
agent = Agent(model)
...

Grok (xAI)

Go to xAI API Console and create an API key. Once you have the API key, you can use it with the OpenAIProvider:

grok_model_init.py

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.openai import OpenAIProvider

model = OpenAIModel(
    'grok-2-1212',
    provider=OpenAIProvider(base_url='https://api.x.ai/v1', api_key='your-xai-api-key'),
)
agent = Agent(model)
...

Perplexity

Follow the Perplexity getting started guide to create an API key. Then, you can query the Perplexity API with the following:

perplexity_model_init.py

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.openai import OpenAIProvider

model = OpenAIModel(
    'sonar-pro',
    provider=OpenAIProvider(
        base_url='https://api.perplexity.ai',
        api_key='your-perplexity-api-key',
    ),
)
agent = Agent(model)
...

Fireworks AI

Go to Fireworks.AI and create an API key in your account settings. Once you have the API key, you can use it with the OpenAIProvider:

fireworks_model_init.py

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.openai import OpenAIProvider

model = OpenAIModel(
    'accounts/fireworks/models/qwq-32b',  # model library available at https://fireworks.ai/models
    provider=OpenAIProvider(
        base_url='https://api.fireworks.ai/inference/v1',
        api_key='your-fireworks-api-key',
    ),
)
agent = Agent(model)
...

Together AI

Go to Together.ai and create an API key in your account settings. Once you have the API key, you can use it with the OpenAIProvider:

together_model_init.py

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.openai import OpenAIProvider

model = OpenAIModel(
    'meta-llama/Llama-3.3-70B-Instruct-Turbo-Free',  # model library available at https://www.together.ai/models
    provider=OpenAIProvider(
        base_url='https://api.together.xyz/v1',
        api_key='your-together-api-key',
    ),
)
agent = Agent(model)
...

Implementing Custom Models

To implement support for models not already supported, you will need to subclass the Model abstract base class.

For streaming, you'll also need to implement the following abstract base class:

StreamedResponse

The best place to start is to review the source code for existing implementations, e.g. OpenAIModel.

For details on when we'll accept contributions adding new models to PydanticAI, see the contributing guidelines.

Fallback

You can use FallbackModel to attempt multiple models in sequence until one returns a successful result. Under the hood, PydanticAI automatically switches from one model to the next if the current model returns a 4xx or 5xx status code.

In the following example, the agent first makes a request to the OpenAI model (which fails due to an invalid API key), and then falls back to the Anthropic model.

fallback_model.py

from pydantic_ai import Agent
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.models.fallback import FallbackModel
from pydantic_ai.models.openai import OpenAIModel

openai_model = OpenAIModel('gpt-4o')
anthropic_model = AnthropicModel('claude-3-5-sonnet-latest')
fallback_model = FallbackModel(openai_model, anthropic_model)

agent = Agent(fallback_model)
response = agent.run_sync('What is the capital of France?')
print(response.data)
#> Paris

print(response.all_messages())
"""
[
    ModelRequest(
        parts=[
            UserPromptPart(
                content='What is the capital of France?',
                timestamp=datetime.datetime(...),
                part_kind='user-prompt',
            )
        ],
        kind='request',
    ),
    ModelResponse(
        parts=[TextPart(content='Paris', part_kind='text')],
        model_name='claude-3-5-sonnet-latest',
        timestamp=datetime.datetime(...),
        kind='response',
    ),
]
"""

The ModelResponse message above indicates in the model_name field that the result was returned by the Anthropic model, which is the second model specified in the FallbackModel.

Note

Each model's options should be configured individually. For example, base_url, api_key, and custom clients should be set on each model itself, not on the FallbackModel.

In this next example, we demonstrate the exception-handling capabilities of FallbackModel. If all models fail, a FallbackExceptionGroup is raised, which contains all the exceptions encountered during the run execution.

Python >=3.11Python <3.11

fallback_model_failure.py

from pydantic_ai import Agent
from pydantic_ai.exceptions import ModelHTTPError
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.models.fallback import FallbackModel
from pydantic_ai.models.openai import OpenAIModel

openai_model = OpenAIModel('gpt-4o')
anthropic_model = AnthropicModel('claude-3-5-sonnet-latest')
fallback_model = FallbackModel(openai_model, anthropic_model)

agent = Agent(fallback_model)
try:
    response = agent.run_sync('What is the capital of France?')
except* ModelHTTPError as exc_group:
    for exc in exc_group.exceptions:
        print(exc)

Since except* is only supported in Python 3.11+, we use the exceptiongroup backport package for earlier Python versions:

fallback_model_failure.py

from exceptiongroup import catch

from pydantic_ai import Agent
from pydantic_ai.exceptions import ModelHTTPError
from pydantic_ai.models.anthropic import AnthropicModel
from pydantic_ai.models.fallback import FallbackModel
from pydantic_ai.models.openai import OpenAIModel


def model_status_error_handler(exc_group: BaseExceptionGroup) -> None:
    for exc in exc_group.exceptions:
        print(exc)


openai_model = OpenAIModel('gpt-4o')
anthropic_model = AnthropicModel('claude-3-5-sonnet-latest')
fallback_model = FallbackModel(openai_model, anthropic_model)

agent = Agent(fallback_model)
with catch({ModelHTTPError: model_status_error_handler}):
    response = agent.run_sync('What is the capital of France?')

By default, the FallbackModel only moves on to the next model if the current model raises a ModelHTTPError. You can customize this behavior by passing a custom fallback_on argument to the FallbackModel constructor.

Models

Models, Interfaces, and Providers

OpenAI

Install

Configuration

Environment variable

provider argument

Custom OpenAI Client

Anthropic

Install

Configuration

Environment variable

provider argument

Custom HTTP Client

Gemini

Install

Configuration

Environment variable

provider argument

Gemini via VertexAI

Install

Configuration

Application default credentials

Service account

Customising region

Groq

Install

Configuration

Environment variable

provider argument

Mistral

Install

Configuration

Environment variable

provider argument

Cohere

Install

Configuration

Environment variable

provider argument

Bedrock

Install

Configuration

Environment variables

provider argument

OpenAI-compatible Models

Ollama

Example local usage

Example using a remote server

Azure AI Foundry

OpenRouter

Grok (xAI)

Perplexity

Fireworks AI

Together AI

Implementing Custom Models

Fallback

`provider` argument

`provider` argument

`provider` argument

`provider` argument

`provider` argument

`provider` argument

`provider` argument