Calling Multiple AI Providers from .NET: OpenAI, Gemini, Groq and Fallbacks

Multiple AI Providers in .NET with IChatClient

Q: What is IChatClient?

IChatClient is a common chat abstraction from Microsoft.Extensions.AI that allows application code to interact with chat-capable AI services without directly coupling every business service to a provider SDK.

Q: Is fallback the same as retry?

No. Retry repeats an operation against the same provider, while fallback moves to another provider according to an application policy.

In a previous article, I looked at the first step in building an AI-enabled .NET application: connecting an application to a large language model through IChatClient.

That approach works well when an application has one AI provider and one primary model.

Production applications are usually less predictable.

A provider can become temporarily unavailable. A model can hit a rate limit. A particular workload may be too expensive for a premium model. Another model may be faster for a simple classification task. An organization may also need a second provider because of availability, compliance, data-governance, or vendor-dependency requirements.

This raises a practical architectural question:

How can a .NET application use multiple AI providers without spreading provider-specific code throughout the application?

In this article, we'll build that architecture around Microsoft.Extensions.AI and IChatClient. We'll look at provider registration, keyed dependency injection, routing, fallback, resilience, observability, cost tracking, RAG, testing, and production concerns.

The goal is not to use multiple providers simply because we can.

The goal is to make the AI provider a replaceable infrastructure dependency while keeping application logic stable.

Important: A multi-provider design adds operational complexity. If a single provider already satisfies your availability, cost, latency, capability, and governance requirements, using one provider may be the better architecture.

What we'll build

Understand why multiple AI providers can be useful
Register several IChatClient implementations
Use keyed dependency injection
Build a provider fallback router
Separate retry, fallback, routing, and load balancing
Route requests by workload rather than vendor
Track provider, model, tokens, latency, and fallback events
Integrate multi-provider routing with RAG
Handle security and data-governance concerns
Test provider failure scenarios
Design a practical production architecture

Why use multiple AI providers?

Multi-provider architecture is not automatically better architecture.

It introduces additional configuration, testing, monitoring, failure handling, provider-specific capability checks, and operational overhead.

There should be a concrete reason for introducing it.

Common reasons include:

Availability: a secondary provider can provide another path when the primary provider is temporarily unavailable.
Rate limits: different providers may have different quotas and throughput constraints.
Cost: simple workloads may not require the most expensive model available.
Latency: some models may be better suited to high-throughput workloads.
Model capabilities: providers and models can differ in reasoning, structured output, tool calling, context size, and multimodal support.
Vendor independence: application services can remain less coupled to a single provider.
Data governance: certain workloads may be permitted only with particular providers.

For a small application, however, none of these requirements may matter.

If one provider is reliable, affordable, and meets the application's requirements, a single IChatClient is often the simpler choice.

Multi-provider architecture becomes valuable when the operational requirement justifies the additional complexity.

What IChatClient gives us

IChatClient is an abstraction in Microsoft.Extensions.AI for interacting with chat-capable AI services.

The important architectural benefit is that application code can depend on a common interface rather than directly depending on a specific provider SDK.

public sealed class QuestionService(IChatClient chatClient)
{
    public async Task<string> GenerateAsync(
        string topic,
        CancellationToken cancellationToken)
    {
        var response = await chatClient.GetResponseAsync(
            $"Create five questions about {topic}.",
            cancellationToken: cancellationToken);

        return response.Text;
    }
}

The business service does not need to know whether the underlying client represents OpenAI, Gemini, Groq, or another compatible implementation.

That distinction becomes particularly useful when provider selection is moved into infrastructure.

The architecture: application versus provider infrastructure

Provider selection should generally stay out of controllers and core business services.

A simplified architecture looks like this:

API/controller layer: receives the request.
Application layer: understands the business operation.
AI service layer: prepares prompts, options, validation, and AI requests.
Routing layer: determines which provider or model should handle the workload.
Resilience layer: handles appropriate timeouts, retries, and failure policies.
Provider clients: communicate with the external AI services.
Observability layer: records provider, model, latency, usage, failures, and fallback information.

The application should not contain code like this:

if (provider == "openai")
{
    // OpenAI-specific logic
}
else if (provider == "gemini")
{
    // Gemini-specific logic
}

Once provider checks appear throughout application services, provider replacement becomes expensive.

A better design is to make provider selection an infrastructure or policy concern.

Registering multiple AI clients

One approach is to register separate IChatClient instances and distinguish them using dependency-injection keys.

For providers that expose an OpenAI-compatible endpoint, the OpenAI .NET client can be adapted to IChatClient through AsIChatClient(). Microsoft currently documents this integration through the Microsoft.Extensions.AI.OpenAI package.

Conceptually, the registrations can look like this:

var builder = WebApplication.CreateBuilder(args);

var configuration = builder.Configuration;

builder.Services.AddKeyedChatClient(
    "openai",
    new OpenAIClient(
        new ApiKeyCredential(
            configuration["OpenAI:ApiKey"]!))
        .GetChatClient(
            configuration["OpenAI:Model"]!)
        .AsIChatClient());

builder.Services.AddKeyedChatClient(
    "gemini",
    new OpenAIClient(
        new ApiKeyCredential(
            configuration["Gemini:ApiKey"]!),
        new OpenAIClientOptions
        {
            Endpoint = new Uri(
                "https://generativelanguage.googleapis.com/v1beta/openai/")
        })
        .GetChatClient(
            configuration["Gemini:Model"]!)
        .AsIChatClient());

builder.Services.AddKeyedChatClient(
    "groq",
    new OpenAIClient(
        new ApiKeyCredential(
            configuration["Groq:ApiKey"]!),
        new OpenAIClientOptions
        {
            Endpoint = new Uri(
                "https://api.groq.com/openai/v1")
        })
        .GetChatClient(
            configuration["Groq:Model"]!)
        .AsIChatClient());

Version note: Provider endpoints, models, package APIs, and supported capabilities can change. Treat the provider names and model identifiers in this example as configuration examples, and verify the current provider documentation before using them in production.

The corresponding configuration can remain outside the application code:

{
  "OpenAI": {
    "ApiKey": "",
    "Model": "your-openai-model"
  },
  "Gemini": {
    "ApiKey": "",
    "Model": "your-gemini-model"
  },
  "Groq": {
    "ApiKey": "",
    "Model": "your-groq-model"
  }
}

Never commit real API keys to source control.

Use environment variables, .NET user secrets, a secrets manager, or your deployment platform's secret configuration.

OpenAI-compatible does not mean identical

This is one of the most important details in a multi-provider design.

An OpenAI-compatible endpoint gives an application a common API shape. It does not guarantee identical behavior between providers.

Providers can differ in:

Supported models
Context limits
Structured-output support
Tool calling
Streaming behavior
Supported request parameters
Rate limits
Token accounting
Error formats
Model quality

Therefore, compatibility should be treated as an integration convenience rather than a guarantee that providers are interchangeable for every workload.

Using keyed dependency injection

If a particular feature should always use a particular provider, keyed dependency injection may be enough.

For example, a lightweight summarization service could request a specific keyed client:

public sealed class SummarizationService(
    [FromKeyedServices("groq")] IChatClient chatClient)
{
    public async Task<string> SummarizeAsync(
        string text,
        CancellationToken cancellationToken)
    {
        var response = await chatClient.GetResponseAsync(
            $"Summarize this in one sentence:\n{text}",
            cancellationToken: cancellationToken);

        return response.Text;
    }
}

This approach is simple and explicit.

But it also couples the service to the provider choice.

If the requirement becomes:

"Use OpenAI normally, but use Gemini when OpenAI is unavailable."

then the provider decision belongs in a routing component rather than inside the summarization service.

Building a simple fallback router

A fallback router can have a deliberately narrow responsibility:

Read the configured provider order.
Resolve the corresponding IChatClient.
Call the provider.
Determine whether the failure is eligible for fallback.
Try the next provider when policy allows it.
Return the successful response or the final failure.

A simple implementation for demonstrating the concept could look like this:

public sealed class RouterPolicy
{
    public IReadOnlyList<string> ProviderChain { get; init; }
        = [];
}

public sealed class FallbackChatClientRouter(
    IServiceProvider services,
    IOptions<RouterPolicy> policy,
    ILogger<FallbackChatClientRouter> logger)
{
    public async Task<ChatResponse> RouteAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        CancellationToken cancellationToken = default)
    {
        Exception? lastException = null;

        foreach (var provider in policy.Value.ProviderChain)
        {
            var client =
                services.GetRequiredKeyedService<IChatClient>(
                    provider);

            try
            {
                return await client.GetResponseAsync(
                    messages,
                    options,
                    cancellationToken);
            }
            catch (Exception ex)
                when (ex is not OperationCanceledException)
            {
                logger.LogWarning(
                    ex,
                    "AI provider {Provider} failed. " +
                    "Trying the next provider.",
                    provider);

                lastException = ex;
            }
        }

        throw new InvalidOperationException(
            "All configured AI providers failed.",
            lastException);
    }
}

The provider chain can then be configured externally:

{
  "RouterPolicy": {
    "ProviderChain": [
      "openai",
      "gemini",
      "groq"
    ]
  }
}

Changing the preferred provider therefore does not require changing business logic.

Why the simple router is not enough for production

The example above intentionally catches most exceptions so that the routing concept is easy to understand.

That is not a good production policy by itself.

Not every exception means:

"Try another provider."

Failure	Possible response
Temporary provider outage	Fallback may be appropriate
Rate limit	Retry, delay, or fallback depending on policy
Timeout	Potential fallback if the operation is safe to repeat
Invalid API key	Fix configuration rather than repeatedly retrying
Invalid request	Fix the request; another provider may produce the same failure
Unsupported capability	Select a provider known to support the required capability
Safety or policy rejection	Follow the application's safety and governance policy

This distinction is critical because a fallback system is also a system that can create additional requests.

A poor fallback policy can increase latency and cost while hiding the actual root cause of an outage.

Retry, fallback, routing, and load balancing are different

These concepts are often mixed together, but they solve different problems.

Retry: try the same provider again.
Fallback: move to another provider after an eligible failure.
Routing: select a provider according to an application policy.
Load balancing: distribute requests across providers or models.

A simplified flow could look like this:

I would avoid implementing all of these concerns inside one large provider service.

Keeping routing and resilience as separate concerns makes the system easier to test and change.

Routing by workload instead of vendor

This is where multi-provider architecture becomes considerably more useful.

Instead of asking:

"Which provider should I use?"

ask:

"What does this workload require?"

Workload	Important characteristics
Classification	Low latency, high throughput, predictable output
Summarization	Cost and latency may matter more than advanced reasoning
Document extraction	Structured output and accuracy may be important
Complex reasoning	Model capability may matter more than raw latency
Fallback	Availability and compatible capabilities become important

There is no universal "best" provider.

A model that is appropriate for complex reasoning may be unnecessary for checking whether a support message belongs to one of ten categories.

Measure your actual workloads rather than choosing models solely from benchmark charts or marketing claims.

Provider selection as a policy

Once workload-based routing becomes important, provider selection can become an explicit application policy.

public sealed class AiWorkloadPolicy
{
    public string Workload { get; init; } = string.Empty;

    public IReadOnlyList<string> Providers { get; init; }
        = [];

    public bool AllowFallback { get; init; } = true;
}

Configuration can then describe different requirements:

{
  "AiRouting": {
    "Classification": {
      "Providers": [ "groq", "gemini" ],
      "AllowFallback": true
    },
    "Reasoning": {
      "Providers": [ "openai", "gemini" ],
      "AllowFallback": true
    },
    "Extraction": {
      "Providers": [ "openai" ],
      "AllowFallback": false
    }
  }
}

The advantage is that routing becomes an explicit policy rather than a collection of hard-coded conditional statements.

Observability becomes essential

With a single provider, a log entry such as:

AI call completed

does not tell you much.

With multiple providers, you need to know what actually happened.

Useful fields include:

Provider
Model
Workload type
Input token count
Output token count
Total token count
Request duration
Success or failure
Error category
Whether fallback occurred
Original provider
Final provider
Estimated cost

For example:

logger.LogInformation(
    "AI request completed. " +
    "Provider={Provider}, " +
    "Model={Model}, " +
    "InputTokens={InputTokens}, " +
    "OutputTokens={OutputTokens}, " +
    "DurationMs={DurationMs}, " +
    "Fallback={Fallback}",
    provider,
    model,
    inputTokens,
    outputTokens,
    durationMs,
    fallbackUsed);

These fields become particularly useful when investigating unexpected increases in AI spending or latency.

Privacy reminder: Avoid logging complete prompts and responses by default. AI requests can contain personal information, customer data, source code, internal documents, or other sensitive content.

Token usage and cost tracking

Multi-provider systems make cost visibility more important because the same application may use different models with different pricing.

Where usage information is exposed by the provider integration, capture it alongside request metadata.

var response = await router.RouteAsync(
    messages,
    cancellationToken: cancellationToken);

var usage = response.Usage;

logger.LogInformation(
    "AI usage. Input={Input}, Output={Output}, Total={Total}",
    usage?.InputTokenCount,
    usage?.OutputTokenCount,
    usage?.TotalTokenCount);

Keep cost calculation separate from the provider execution layer.

For example, the execution layer can record:

Provider = "openai"
Model = "model-name"
InputTokens = 1200
OutputTokens = 350

A separate pricing component can calculate estimated cost from the current pricing configuration.

This separation matters because provider pricing and model availability change over time.

Do not bury pricing assumptions inside the routing algorithm.

What happens when the primary provider fails?

Imagine that the application normally uses OpenAI and a request encounters a transient provider failure.

A controlled flow could be:

Receive the request.
Validate the request.
Select OpenAI as the primary provider.
Apply the configured timeout.
Retry only if the failure is classified as transient and retrying is appropriate.
If the provider remains unavailable, record the failure.
Move to Gemini if the fallback policy allows it.
Record that fallback occurred.
Return the successful response or a controlled error.

The final step in that sequence is important.

Do not silently hide fallback events from telemetry.

A system can appear healthy from the user's perspective while actually running on its secondary provider for several hours.

Resilience should be designed deliberately

.NET provides dedicated resilience infrastructure for transient failures, including retry, timeout, circuit-breaker, rate-limiting, fallback, and related strategies. Microsoft currently recommends the Microsoft.Extensions.Resilience and Microsoft.Extensions.Http.Resilience packages for these scenarios.

The important architectural point is that resilience should not simply be a giant try/catch around every AI request.

A resilience policy should define:

Which failures are transient
How many retries are allowed
How long the caller can wait
When a provider should be considered unhealthy
Whether another provider can safely receive the request
How failures are reported through telemetry

Microsoft's current HTTP resilience guidance includes strategies such as timeout, retry, circuit breaker, rate limiting, and hedging, while also warning that retry behavior should be chosen carefully for operations where repeating a request can have undesirable effects.

RAG and multi-provider architecture solve different problems

Multi-provider routing and Retrieval-Augmented Generation are sometimes discussed together, but they solve different architectural problems.

RAG asks: What information should the model receive?

Provider routing asks: Which model or provider should generate the response?

A simplified RAG flow looks like this:

This means you do not have to rebuild the retrieval system every time you change the generation provider.

The vector store, chunking strategy, document metadata, and retrieval logic can remain independent from the chat provider.

Where embeddings fit

Embeddings are another separate part of the AI architecture.

An embedding model converts text into a numerical representation that can be stored and searched using a vector database.

A typical pipeline is:

Document
   |
   v
Chunking
   |
   v
Embedding model
   |
   v
Vector database
   |
   v
Similarity search
   |
   v
Relevant context
   |
   v
Chat model

The embedding provider and chat provider do not necessarily have to be the same vendor.

That is another reason to keep AI capabilities separated by responsibility rather than building one provider-specific abstraction around the entire AI stack.

Where Semantic Kernel fits

Semantic Kernel operates at a different level from IChatClient.

You do not need Semantic Kernel simply to send a prompt to an LLM.

For a straightforward application, IChatClient may be sufficient.

A higher-level orchestration framework becomes more interesting when the application needs capabilities such as:

Plugins
Functions
Prompt orchestration
AI service coordination
Memory-related workflows
Agent-oriented workflows

Layer	Example	Responsibility
Model provider	OpenAI, Gemini, Groq	Provides AI models
AI abstraction	`IChatClient`	Common application-facing chat interface
Retrieval	Embeddings + vector store	Find relevant application data
Routing	Provider router	Select provider or model
Orchestration	Semantic Kernel	Coordinate higher-level AI workflows

Testing provider fallback

Fallback logic should not be considered complete simply because the code compiles.

Failure paths should be tested deliberately.

At minimum, test:

Primary provider succeeds.
Primary provider times out.
Primary provider returns a rate-limit failure.
Primary provider fails and the secondary provider succeeds.
Primary and secondary providers both fail.
Cancellation is requested by the caller.
A non-retryable request error occurs.
Provider configuration is missing.
The provider returns an unsupported response.

For example:

OpenAI   → failure
Gemini   → success

The test should verify that the router returns the Gemini response and records the fallback event.

Also test:

OpenAI   → success
Gemini   → should not be called

This protects against accidentally invoking multiple providers for every request.

Don't accidentally double your AI bill

There is a subtle operational risk with fallback systems.

Suppose the first provider successfully processes a request, but your application fails while receiving, parsing, or validating the result.

If the fallback policy blindly sends the request to another provider, one logical application request may produce multiple billable AI calls.

This is why it is useful to distinguish:

Transport failure
Provider failure
Timeout
Application parsing failure
Invalid structured output
Business validation failure

Fallback should therefore be based on a deliberate policy rather than:

catch (Exception)
{
    TryAnotherProvider();
}

The second provider is not automatically a solution to every problem.

Security and data governance

Every additional AI provider is another external system that may receive application data.

Provider selection should therefore be part of the application's security and governance architecture.

At minimum:

Keep API credentials outside source control.
Do not log sensitive prompts or responses by default.
Define which workloads can use each provider.
Validate user-controlled input before sending it to models.
Consider prompt-injection risks when retrieved content is supplied to the model.
Validate structured model output before using it in business operations.
Document what data each provider is permitted to process.
Do not assume identical privacy, retention, or regional-processing behavior across providers.

This becomes particularly important for applications processing customer documents, internal business information, source code, financial information, or other sensitive data.

Common mistakes when implementing multi-provider AI

1. Putting provider checks everywhere

If controllers and business services contain provider-specific conditions, the abstraction is already breaking down.

Keep provider selection centralized.

2. Treating every error as a fallback condition

An invalid request will not become valid simply because it is sent to another provider.

3. Using the same model for every workload

A simple classification request and a complex reasoning task do not necessarily need the same model.

4. Ignoring observability

Without provider and model telemetry, it becomes difficult to explain cost increases, latency changes, or unexpected fallback activity.

5. Assuming compatibility means identical behavior

OpenAI-compatible APIs can still differ considerably in supported features and model behavior.

6. Building the abstraction before there is a requirement

A router with five providers, complex policies, and multiple fallback paths is not automatically more production-ready.

It may simply be more complicated.

When should you use multiple providers?

I would consider multi-provider architecture when at least one of these requirements is real:

You need provider redundancy.
Your workloads have significantly different model requirements.
There are meaningful cost differences between workloads.
You have provider-specific quotas or throughput constraints.
Different workloads require different data-governance policies.
You want the ability to change providers without rewriting application services.

If none of these apply, start with one provider.

You can still build around IChatClient so that adding another provider later does not require rewriting the application.

This leads to an important architectural principle:

Design for replaceability without implementing every possible provider on day one.

A practical production architecture

Putting the pieces together, a production AI subsystem might look like this:

The important architectural property is that the application owns the policy while provider implementations remain replaceable infrastructure dependencies.

Production checklist

Provider API keys are stored securely.
Provider selection is centralized.
Timeouts are explicitly configured.
Retries are limited and applied only to appropriate failures.
Fallback conditions are explicitly defined.
Cancellation is respected.
Provider and model are included in telemetry.
Token usage is captured where available.
Fallback events are observable.
AI costs can be estimated.
Provider-specific capability differences are documented.
Structured AI output is validated before use.
Sensitive data is handled according to the application's data policy.
Fallback behavior is covered by automated tests.
Provider configuration can be changed without modifying business logic.

Frequently asked questions

Can a .NET application use multiple AI providers?

Yes. Multiple AI clients can be registered and selected through dependency injection or an application-level routing component.

What is IChatClient?

IChatClient is a common chat abstraction from Microsoft.Extensions.AI. It allows application code to interact with chat-capable AI services without directly coupling every business service to a provider SDK.

Can I use OpenAI, Gemini, and Groq in the same .NET application?

Yes. They can be integrated as separate AI clients when their supported APIs and capabilities fit the application's requirements.

Is fallback the same as retry?

No. Retry repeats the operation against the same provider. Fallback moves to another provider according to an application policy.

Should every AI request have a fallback provider?

No. Some workloads have provider-specific requirements, and some failures should not result in another request. Fallback should be based on an explicit policy.

Can RAG and multi-provider routing be used together?

Yes. RAG determines which application data should be supplied to the model, while provider routing determines which model or provider generates the response.

Do I need Semantic Kernel to use IChatClient?

No. IChatClient is sufficient for many straightforward AI integrations. A higher-level orchestration framework becomes useful when the application requires more advanced workflows.

Does OpenAI-compatible mean all providers behave the same?

No. API compatibility does not guarantee identical model capabilities, parameters, limits, structured-output behavior, tool calling, latency, or quality.

Does Microsoft.Extensions.AI already provide routing?

Current Microsoft documentation describes chat routing as an experimental capability. It includes abstractions such as RoutingChatClient and failover-oriented clients including FailoverChatClient and OrderedFailoverChatClient. Because these APIs are experimental, applications should verify the version-specific documentation before depending on them in production.

Final thoughts

Multi-provider AI architecture is not really about putting three API keys into a configuration file.

The more important idea is creating a clean boundary between the application and the AI infrastructure behind it.

IChatClient gives .NET applications a useful abstraction for that boundary. A routing layer can then decide which provider should handle a workload, while resilience policies handle transient failures and observability records what actually happened.

RAG can remain focused on retrieving relevant application data. Embeddings can remain focused on vector representations and search. Higher-level frameworks can provide orchestration when the application actually needs it.

The result is an AI subsystem that can evolve without forcing the rest of the application to change every time a model or provider changes.

But there is one principle I would keep above all the others:

Don't add multi-provider complexity until you have a real reason for it.

Start with the simplest architecture that satisfies the requirement. Use abstractions such as IChatClient to keep the door open for future changes, and introduce routing, fallback, and multiple providers when availability, cost, performance, capability, or governance actually requires them.

Key takeaways

IChatClient provides a useful abstraction between .NET application code and AI providers.
Multiple providers can improve availability, workload-specific routing, cost control, and vendor flexibility.
OpenAI-compatible APIs do not guarantee identical provider capabilities.
Provider selection should live in routing or policy infrastructure rather than throughout business logic.
Retry, fallback, routing, and load balancing are different concerns.
Fallback should happen only for failures that the application considers eligible.
Provider, model, token usage, latency, failures, and fallback events should be observable.
RAG and provider routing solve different problems and can be combined.
Every additional provider introduces security, governance, testing, and operational considerations.
The best architecture is usually the simplest one that satisfies the real requirements.

Transitioning to AI: Engineering the Next Paradigm