Calling LLMs from .NET: IChatClient in Practice

IChatClient in .NET: A Practical Guide for Building AI Applications

If you are building AI features in a .NET application, getting your first response from a large language model is usually the easy part.

The harder problem appears when the application grows. Provider-specific SDKs start appearing inside application services, prompts become coupled to infrastructure code, conversation history needs to be managed, and concerns such as logging, telemetry, caching, rate limiting, security, and cost tracking begin spreading across the codebase.

This is where IChatClient becomes useful.

IChatClient is part of the Microsoft.Extensions.AI ecosystem. It provides a common .NET abstraction for interacting with chat-capable AI services while allowing the underlying implementation to come from a provider SDK or another compatible implementation.

The architectural idea is simple: your application can depend on an AI capability without every application service needing to know which provider executes the request.

Important: IChatClient is an abstraction, not an AI model. The actual model, provider, authentication, capabilities, limits, and pricing still come from the underlying implementation.

What this article covers

• Why an AI abstraction matters in .NET applications
• What IChatClient actually provides
• Basic requests and streaming responses
• Conversation history and token usage
• ChatOptions and request configuration
• Dependency injection in ASP.NET Core
• Client pipelines for caching, telemetry, and custom behavior
• Function calling and AI tools
• Multiple providers and provider portability
• RAG, resilience, security, and testing
• A practical production architecture

Why an AI abstraction matters in a .NET application

A common starting point for an AI feature is to call a provider SDK directly from an application service.

That approach can be perfectly reasonable for a small experiment. The problem is not the first API call. The problem is what happens after the application has several AI-powered features.

Imagine that an application contains services for document summarization, customer support, classification, content generation, and retrieval-augmented generation.

If every service directly understands a provider-specific SDK, changing providers can become a cross-cutting refactoring exercise.

An abstraction creates a cleaner boundary:

This does not mean every application should dynamically switch providers. The main benefit is separation of responsibilities.

Where IChatClient fits in the .NET AI stack

Microsoft.Extensions.AI provides abstractions and supporting components for integrating AI capabilities into .NET applications.

The ecosystem includes the IChatClient abstraction for chat interactions and supports cross-cutting functionality such as caching, telemetry, and function invocation through composable client pipelines.

The important distinction is between the application layer and the provider implementation.

Layer	Responsibility	Example
Application	Business workflow and user experience	Summarize a support ticket
AI service	Prompt construction and AI-specific business logic	Build a summarization request
IChatClient	Common chat interaction abstraction	Send messages and receive responses
Pipeline	Cross-cutting AI concerns	Telemetry, caching, rate limiting
Provider implementation	Actual communication with an AI service	Provider-specific client
Model	Generates the actual AI response	Selected chat model

What is IChatClient?

At its core, IChatClient represents a client capable of sending chat messages to an AI service and receiving a response.

The interface exposes both normal and streaming operations. The current API includes GetResponseAsync for a complete response and GetStreamingResponseAsync for incremental updates.

It also provides access to provider-specific or implementation-specific services when needed, which is useful when an application needs to escape the abstraction for a capability that is not represented directly by the common API.

This is an important design characteristic: the abstraction gives you portability without pretending that all AI providers have exactly the same capabilities.

The simplest IChatClient request

A basic request can be represented using ChatMessage objects.

using Microsoft.Extensions.AI;

var messages = new[]
{
    new ChatMessage(
        ChatRole.User,
        "Explain retrieval-augmented generation in simple terms.")
};

ChatResponse response =
    await client.GetResponseAsync(messages);

Console.WriteLine(response.Text);

The application does not need to know how the underlying provider constructs HTTP requests, serializes the provider-specific payload, or handles its SDK objects.

That separation becomes more valuable as the application grows.

Streaming responses

Streaming is useful when users should see an AI response as it is generated instead of waiting for the complete response.

The streaming API returns an asynchronous sequence of ChatResponseUpdate values. Those updates collectively represent the response.

await foreach (ChatResponseUpdate update in
    client.GetStreamingResponseAsync(messages))
{
    Console.Write(update);
}

In a real web application, you would normally translate those updates into the transport mechanism used by the frontend, such as server-sent events or another streaming protocol.

Streaming is especially useful for:

• Chat interfaces
• Long-form generation
• Interactive assistants
• Applications where perceived response time matters

It also changes application design. You need to think about cancellation, partial responses, connection failures, and how the final response is stored.

Conversation history is your application's responsibility

An LLM request is generally based on the messages you send with that request. Your application therefore needs a strategy for conversation history when building multi-turn experiences.

var messages = new List<ChatMessage>
{
    new(
        ChatRole.System,
        "You are a helpful .NET assistant."),

    new(
        ChatRole.User,
        "What is dependency injection?")
};

var firstResponse =
    await client.GetResponseAsync(messages);

messages.AddRange(firstResponse.Messages);

messages.Add(
    new ChatMessage(
        ChatRole.User,
        "How does it work in ASP.NET Core?"));

var secondResponse =
    await client.GetResponseAsync(messages);

Console.WriteLine(secondResponse.Text);

The important architectural point is that conversation state should not accidentally become hidden global state inside your AI client.

Your application should decide whether history lives in memory, a database, a distributed cache, or another conversation-state store.

Why conversation history becomes expensive

A conversation can become increasingly expensive as more messages are included in subsequent requests.

More history can also make prompts harder to reason about and may eventually exceed model context limits.

A production application may therefore need strategies such as:

• Limiting the number of previous messages
• Summarizing older conversation turns
• Keeping only relevant messages
• Separating long-term memory from the active prompt
• Using retrieval instead of inserting large documents directly into every request

This is one reason why IChatClient should not be confused with a complete conversational-memory solution. It provides the chat interaction abstraction; your application still owns conversation design.

Controlling requests with ChatOptions

ChatOptions lets callers provide additional configuration for an individual request. The current API exposes common settings as strongly typed properties and also provides mechanisms for additional provider-specific options.

var options = new ChatOptions
{
    Temperature = 0.2f,
    MaxOutputTokens = 500
};

ChatResponse response =
    await client.GetResponseAsync(
        messages,
        options);

The exact options that matter depend on the model and provider.

This creates an important engineering rule: do not assume that an option has identical behavior across every provider simply because it exists in the common abstraction.

Provider-specific behavior should be tested rather than assumed.

Dependency injection in ASP.NET Core

Dependency injection is one of the biggest reasons an abstraction like IChatClient fits naturally into modern .NET applications.

Instead of creating an AI client directly inside every service, register the client at the application composition root and inject it where required.

public sealed class AiService(IChatClient chatClient)
{
    public async Task<string> AskAsync(
        string question,
        CancellationToken cancellationToken = default)
    {
        var messages = new[]
        {
            new ChatMessage(
                ChatRole.User,
                question)
        };

        var response =
            await chatClient.GetResponseAsync(
                messages,
                cancellationToken: cancellationToken);

        return response.Text;
    }
}

The service now depends on IChatClient rather than a specific provider SDK.

This also improves testability because the application service can be tested against a fake or controlled implementation.

Building an IChatClient pipeline

One of the most useful ideas around IChatClient is composition.

Instead of placing every cross-cutting concern directly into business services, the client can be wrapped with additional behavior.

Current Microsoft documentation shows client composition for concerns including caching, OpenTelemetry, function invocation, and custom behavior.

Caching

Caching can be useful for deterministic or repeatable requests where returning a previously generated result is acceptable.

However, caching AI responses is not automatically correct.

Before enabling it, consider:

• Whether identical prompts should produce reusable results
• Whether the prompt contains user-specific data
• How long cached data should remain valid
• Whether the underlying model or system prompt can change
• Whether cached responses could expose information between users

Telemetry

AI systems need observability just like ordinary distributed applications.

Useful telemetry can include request duration, failures, model identifiers, token usage, and application-level operation names.

The Microsoft.Extensions.AI ecosystem includes an OpenTelemetry chat client that can add telemetry around an IChatClient pipeline.

Logging

Logging should focus on operational information rather than blindly storing complete prompts and responses.

AI requests may contain confidential business information, personal data, source documents, or user-generated content.

A useful logging strategy records identifiers, timing, model information, success or failure, and usage metadata while applying appropriate redaction rules.

Custom middleware

Custom pipeline behavior is useful when an application needs functionality that is not provided by an existing delegating client.

Examples include application-specific rate limiting, request tagging, policy checks, custom metrics, or cost accounting.

The ChatClientBuilder API supports custom delegating behavior through Use, allowing an application to execute logic before and after the underlying client operation.

Function calling and AI tools

Modern AI applications often need more than text generation.

An AI assistant may need to call a function such as:

• Get the current order status
• Look up a customer record
• Retrieve internal documentation
• Calculate a value
• Create a support ticket

Microsoft.Extensions.AI supports function invocation through the chat client pipeline.

IChatClient client = new ChatClientBuilder(baseClient)
    .UseFunctionInvocation()
    .UseOpenTelemetry()
    .Build();

Function calling introduces an important security boundary.

A function exposed to an AI model should be treated as an application capability, not as harmless generated code.

Authorization, input validation, auditing, and business rules should remain enforced by your application.

Tracking AI usage and tokens

AI cost management becomes much easier when usage information is captured close to the AI request.

When available from the underlying implementation, the response usage information can be inspected through response.Usage.

var usage = response.Usage;

if (usage is not null)
{
    Console.WriteLine(
        $"Input: {usage.InputTokenCount}");

    Console.WriteLine(
        $"Output: {usage.OutputTokenCount}");

    Console.WriteLine(
        $"Total: {usage.TotalTokenCount}");
}

Token counts can then be associated with an application operation, tenant, user, feature, or model.

A practical AI usage record might contain:

• Request identifier
• Application feature
• Model identifier
• Input token count
• Output token count
• Total token count
• Duration
• Success or failure
• Provider

Actual monetary cost should be calculated from the pricing rules applicable to the provider and model being used rather than assuming that every token has the same price.

Using multiple AI providers

One of the strongest reasons to introduce an AI abstraction is to reduce provider coupling.

For example, an application might use one provider for general chat, another for a specific model capability, and a third provider as a fallback.

The important distinction is between provider portability and provider equivalence.

The abstraction can make the calling code more portable, but different providers may still differ in:

• Model capabilities
• Tool calling behavior
• Structured output support
• Context limits
• Streaming behavior
• Safety features
• Pricing
• Latency

Therefore, switching providers should be treated as an engineering decision rather than simply changing one configuration value and assuming identical behavior.

IChatClient and RAG are different layers

It is easy to confuse IChatClient with retrieval-augmented generation because both appear in modern AI applications.

They solve different problems.

Technology	Main responsibility
IChatClient	Common abstraction for chat model interaction
Embedding generation	Convert content into vector representations
Vector database	Store and retrieve vectorized information
RAG pipeline	Retrieve relevant information and include it in model context
Application service	Coordinate the complete business workflow

A typical RAG application might therefore look like:

This separation is useful because you can change the retrieval strategy without changing the fundamental chat-client abstraction.

IChatClient vs Semantic Kernel

IChatClient and Semantic Kernel are not necessarily competing technologies.

They operate at different abstraction levels.

Concern	IChatClient	Semantic Kernel
Chat client abstraction	Core purpose	Can use AI service abstractions within a broader framework
Provider integration	Strong focus	Supported as part of a broader AI framework
Middleware pipeline	Strong focus	Broader orchestration capabilities
Plugins / tools	Supported through AI function concepts	Major framework capability
Agent orchestration	Not its primary purpose	Better suited to broader orchestration scenarios

A simple application that mainly needs a provider-independent chat abstraction may only need IChatClient.

A larger AI application involving orchestration, plugins, planners, agents, or framework-level workflows may benefit from a higher-level framework.

The correct choice depends on the architecture rather than the popularity of either technology.

Resilience belongs around the AI client

AI calls are network calls, so they inherit many of the same reliability problems as other external services.

A production application should think about:

• Timeouts
• Cancellation
• Transient failures
• Rate limits
• Provider outages
• Retry behavior
• Fallback providers
• Concurrency limits

Retries deserve particular attention.

Retrying every failed AI request automatically can increase latency and cost, and careless retries can make rate-limit problems worse.

A better strategy is to classify failures and retry only where the operation and provider semantics make that safe.

Security is still your responsibility

An AI abstraction does not automatically make an application secure.

AI applications can introduce additional security concerns because prompts, retrieved documents, tool definitions, and model responses may contain untrusted or sensitive information.

At minimum, consider:

• Protecting provider API credentials
• Validating user input
• Controlling which tools a model can invoke
• Applying authorization before sensitive operations
• Preventing cross-user conversation leakage
• Redacting sensitive information from logs
• Protecting retrieved content and internal documents
• Applying output validation where business rules require it

A particularly important rule is: never treat an AI-generated instruction as an authorization decision.

The model can suggest an action, but the application must decide whether that action is permitted.

Testing services that depend on IChatClient

One practical advantage of depending on an interface is that application services do not need to construct a real provider client during every unit test.

Conceptually, production and test environments can provide different implementations:

// Production
services.AddChatClient(productionClient);

// Tests
services.AddSingleton<IChatClient>(
    new FakeChatClient());

The exact registration depends on how the application constructs its client pipeline, but the architectural principle remains the same: business services should be testable without requiring a live AI provider.

A useful fake client can return deterministic responses for scenarios such as:

• Successful generation
• Provider failures
• Empty responses
• Cancellation
• Unexpected tool requests
• Malformed or incomplete model output

A production-oriented architecture

A practical .NET AI architecture can separate business logic, AI orchestration, cross-cutting concerns, and provider infrastructure.

The architecture can then evolve independently:

• The API can change without replacing provider infrastructure.
• AI services can change prompts without changing controllers.
• Telemetry can be added without duplicating instrumentation.
• Caching can be introduced at the client pipeline level.
• Provider implementations can be replaced or added independently.
• Tests can substitute controlled AI implementations.

What IChatClient does not solve

It is important not to treat IChatClient as a complete AI application framework.

It does not automatically solve:

• Prompt engineering
• Conversation memory design
• RAG architecture
• Vector database selection
• Authorization
• Content safety policies
• Business validation
• AI evaluation
• Application-specific cost controls
• Provider-specific model capabilities

This distinction is important because good AI architecture is more than selecting a chat abstraction.

When should you use IChatClient?

IChatClient is particularly useful when an application needs a clean boundary between application code and AI provider infrastructure.

It is a strong fit when:

• You are building AI functionality in a .NET application.
• You want application services to depend on an abstraction.
• You expect to evaluate multiple AI providers or models.
• You need composable telemetry or caching.
• You want to centralize AI-related cross-cutting behavior.
• You want easier unit testing of application services.
• You need streaming responses or tool invocation.

It may be unnecessary for a very small throwaway experiment where directly using a provider SDK is simpler and provider portability has no value.

A practical IChatClient production checklist

• Abstraction: Keep application services dependent on IChatClient where appropriate.
• Configuration: Keep provider credentials outside source code.
• History: Define an explicit strategy for conversation state.
• Tokens: Track usage where cost and capacity matter.
• Telemetry: Measure latency, failures, model usage, and important operations.
• Resilience: Define timeout, retry, rate-limit, and cancellation behavior.
• Security: Validate tools and enforce authorization in application code.
• Caching: Cache only where the data and behavior make caching safe.
• Testing: Use controlled clients for deterministic tests.
• Provider differences: Test capabilities instead of assuming complete equivalence.

Common mistakes when adopting IChatClient

Putting provider-specific code everywhere

Introducing IChatClient but continuing to expose provider-specific SDK objects throughout the application defeats much of the architectural benefit.

Keeping conversation state inside the client

Conversation state usually belongs to the application because different features may have different retention, privacy, and persistence requirements.

Logging complete prompts without considering privacy

Prompts can contain sensitive information. Observability should be designed with data minimization and redaction in mind.

Assuming every provider behaves identically

A common interface does not make model capabilities identical. Provider-specific testing remains necessary.

Adding retries without understanding cost

An automatic retry can become another billable AI request. Retry policies should therefore consider both reliability and economics.

Frequently asked questions

What is IChatClient in .NET?

IChatClient is an abstraction in the Microsoft.Extensions.AI ecosystem for interacting with chat-capable AI services. It provides common operations for complete and streaming chat responses while allowing different implementations underneath.

Is IChatClient a replacement for OpenAI or other provider SDKs?

Not exactly. A provider SDK can supply the concrete implementation behind an IChatClient. The abstraction lets application code depend less directly on that provider implementation.

Can IChatClient stream AI responses?

Yes. GetStreamingResponseAsync returns an asynchronous sequence of ChatResponseUpdate values that can be consumed incrementally.

Can I use function calling with IChatClient?

Yes. The Microsoft.Extensions.AI client pipeline supports function invocation, allowing AI applications to expose application functions as tools while keeping the invocation behavior in a composable layer.

Does IChatClient provide conversation memory?

No. Your application should decide how conversation history is collected, stored, summarized, filtered, and supplied to subsequent requests.

Is IChatClient useful for RAG?

Yes, but it represents only one part of a RAG architecture. Retrieval, embeddings, document processing, vector search, prompt construction, and the final generation step are separate concerns.

Does IChatClient eliminate provider differences?

No. It provides a common abstraction, but providers and models can still differ in capabilities, limits, pricing, tool behavior, structured output, and other characteristics.

Further reading

For the most accurate API details, always check the current Microsoft documentation because the Microsoft.Extensions.AI APIs and provider integrations continue to evolve.

• Microsoft documentation: Use the IChatClient interface
• Microsoft documentation: Microsoft.Extensions.AI libraries
• Related article: Retrieval-Augmented Generation in .NET
• Related article: Understanding Embeddings in AI Applications

Key takeaways

IChatClient is valuable because it gives .NET applications a consistent boundary around chat-based AI services.

The biggest benefit is not simply that you can replace one provider with another. The more important benefit is architectural separation.

Your application can keep business logic, conversation management, authorization, RAG workflows, and cost policies separate from the implementation that actually communicates with an AI provider.

The abstraction also fits naturally with dependency injection and composable client pipelines. Caching, telemetry, function invocation, and custom policies can be introduced without duplicating the same logic across every AI-enabled service.

At the same time, IChatClient is not a shortcut around AI engineering. Production applications still need thoughtful decisions about prompts, conversation history, retrieval, security, resilience, observability, testing, and cost.

That is ultimately where the abstraction becomes most useful: it gives those concerns a cleaner place to live.

API note: Microsoft.Extensions.AI is an evolving .NET AI ecosystem. Verify package versions, provider capabilities, and API signatures against the current official documentation before copying production code directly from an older article.