Embeddings in AI: What They Are, How They Work, and Why They Matter in .NET

Understanding Embeddings in RAG with .NET

This article focuses on the practical engineering side of embeddings, semantic retrieval, and RAG applications built with .NET.

If you've started building Retrieval-Augmented Generation (RAG) applications, you've probably heard the word embedding more times than you expected.

Documents become embeddings. Questions become embeddings. A vector database stores those embeddings, and a similarity search retrieves vectors that are close to the vector representing the user's question.

The workflow looks simple when drawn as a diagram. The more interesting engineering question is what happens between the text and the vector:

How can a list of numbers represent the meaning of a sentence?

That is the question this article focuses on.

Rather than treating embeddings as another AI buzzword, we'll build a practical mental model of what happens when text is converted into a vector, why related pieces of text can produce similar representations, how vector similarity works, and where embeddings fit into a real .NET application.

The goal isn't to turn this into a mathematics textbook. It is to give you enough understanding to design, debug, and evaluate an embedding-based application yourself.

What You'll Learn

• What an embedding actually is.
• Why embeddings are useful for AI applications.
• How text is converted into a vector.
• Why semantically related text can produce nearby vectors.
• What vector dimensions represent.
• How cosine similarity works.
• The difference between keyword and semantic search.
• How embedding models affect retrieval quality.
• How embeddings fit into a RAG pipeline.
• How to generate embeddings in .NET.
• How PostgreSQL and pgvector can store and search vectors.
• Why changing an embedding model can require re-indexing.
• Common embedding problems that appear in production.

Start With a Simple Example

Consider these three sentences:

A: Employees receive twenty weeks of parental leave.

B: Staff members can take five months of parental leave.

C: The database server was restarted after maintenance.

A person can immediately recognize that sentences A and B are related. Sentence C is discussing a completely different subject.

A traditional keyword search can identify exact word matches, but A and B don't use exactly the same wording. One says employees while the other says staff members. One says twenty weeks while the other says five months.

Embeddings provide another way to compare the content. Instead of comparing only individual words, an embedding model converts each piece of text into a numerical representation that can be compared mathematically.

Important: Embeddings do not directly answer a user's question. In a RAG system, they help retrieve potentially relevant information that can later be supplied to the language model.

What Is an Embedding?

An embedding is a numerical representation of information.

In the RAG applications most developers build, that information is usually text. Depending on the model, embeddings can also be used with other types of data such as images, audio, documents, or source code.

For example, consider this sentence:

Employees are eligible for twenty weeks of parental leave.

An embedding model transforms it into a vector that might conceptually look like:

[
    0.021,
   -0.183,
    0.441,
    0.072,
    ...
]

The real vector can contain hundreds or thousands of numerical values depending on the embedding model.

One of the most important things to understand is that an individual value normally does not have a simple, human-readable interpretation.

A value such as 0.441 does not mean "parental leave." You generally cannot look at dimension 417 and conclude that it represents "employees."

The useful information is distributed across the vector. What matters to an application is how the complete vector relates to other vectors.

Why Convert Text Into Numbers?

Computers are very good at comparing numbers. They can calculate distances, angles, rankings, and other mathematical relationships efficiently.

Computers do not naturally have a simple operation called:

"Do these two sentences mean approximately the same thing?"

An embedding model creates a representation that makes this type of comparison possible.

This is the basic foundation of semantic search. The model creates a representation that allows the application to compare a query with previously indexed content.

How Does an Embedding Model Capture Relationships?

An embedding model has been trained to produce useful representations of content. During training, the model learns patterns and relationships from its training data.

A useful way to visualize this is to imagine a large map.

We cannot realistically draw a 768-dimensional or 1,536-dimensional embedding space on a normal screen. Instead, imagine a simplified two-dimensional map:

                Security
                   ●

      Parental ●  ● Leave policies
        leave

                       ● Database
                         maintenance

              ● Network configuration

This drawing is only an analogy. Real embedding spaces contain many more dimensions.

The important idea is that related pieces of information can have similar representations, making them easier to retrieve using mathematical similarity measures.

It is also important not to interpret an embedding space too literally. A vector's dimensions are learned numerical features, not a predefined dictionary of concepts where each coordinate corresponds to one human meaning.

Embeddings Are Not Keyword Search

This distinction is particularly important when designing a RAG system.

Imagine that a company policy contains:

Employees can take up to twenty weeks of parental leave.

A user might ask:

How long can I stay away from work after having a baby?

The two sentences do not share many important words.

A keyword search primarily looks for matching terms. Semantic retrieval attempts to identify related meaning through vector representations.

Approach	What it primarily looks for	Useful for
Keyword / lexical search	Matching terms and lexical patterns	Names, identifiers, error codes, exact phrases, product codes
Semantic search	Similarity between learned vector representations	Conceptual matches, natural-language questions, related content
Hybrid search	A combination of lexical and semantic signals	Applications where both exact terms and meaning matter

These approaches do not have to compete with each other. Many production retrieval systems combine lexical and semantic retrieval because exact terms can be extremely valuable for identifiers, product names, error codes, version numbers, and other precise information.

What Does Vector Dimension Mean?

You will often see embedding models described using dimensions such as 384, 768, 1,024, 1,536, or other values.

The dimension tells you how many numerical values are contained in the vector.

For example:

Embedding A = [x1, x2, x3, ..., x1536]

would contain 1,536 values.

It is tempting to think that each position corresponds to a specific concept. That is usually not a useful mental model.

Instead, think of the vector as a single learned representation whose useful meaning comes from the relationship between the complete vector and other vectors.

Don't ask: "What does dimension 417 mean?"
Ask: "How does this complete vector compare with another vector?"

How Do We Compare Embeddings?

Once text has been converted into vectors, the application needs a way to measure how similar those vectors are.

Several distance metrics can be used. One common choice for text embeddings is cosine similarity.

Cosine Similarity in Simple Terms

Imagine two arrows starting at the same point.

             Vector B
                /
               /
              /
             /
            /  Vector A
           /  /
          /  /
         ●----------------

Cosine similarity looks at the angle between the vectors rather than simply measuring the distance between their endpoints.

The formula is:

cosine similarity = (A · B) / (||A|| × ||B||)

You normally do not need to calculate this manually. Vector databases and vector-search libraries can perform the calculation for you.

As an application developer, the important concept is that a vector store can use a mathematical relationship between the query vector and stored vectors to rank candidate chunks.

Metric choice matters. Do not assume that cosine similarity is automatically the correct metric for every embedding model and database configuration. Follow the embedding model's guidance and ensure that the database index and query operator use compatible distance semantics.

Where Embeddings Fit in a RAG Application

Now the role of embeddings in RAG becomes much clearer.

Imagine that your company has 10,000 pages of internal documentation and you want employees to ask questions about those documents.

The first stage is indexing the documents:

Documents
    ↓
Extract text
    ↓
Split into chunks
    ↓
Generate embeddings
    ↓
Store vectors + text + metadata
    ↓
Vector database

When a user asks a question, the application follows a different path:

User question
    ↓
Generate query embedding
    ↓
Search stored vectors
    ↓
Select relevant chunks
    ↓
Build LLM context
    ↓
Generate answer

The embedding does not answer the question. It helps the application find information that can be supplied to the language model as context.

Document Embeddings and Query Embeddings

A RAG application generally creates embeddings in two places: during document ingestion and when processing a user query.

During Document Ingestion

Each document chunk is converted into a vector and stored.

Chunk:
"Employees can take twenty weeks of parental leave."

        ↓

Document embedding

        ↓

Vector database

During a User Query

The user's question is converted into another vector.

Question:
"How much parental leave is available?"

        ↓

Query embedding

        ↓

Vector search

The search compares the query representation with stored document representations.

This also explains why the embedding model is part of the retrieval contract. The query and indexed documents need compatible representations for the similarity search to be meaningful.

A Practical .NET Embedding Example

Modern .NET applications can use Microsoft.Extensions.AI to work with AI services through common abstractions.

A small service can hide the embedding provider from the rest of the application:

using Microsoft.Extensions.AI;

public sealed class EmbeddingService(
    IEmbeddingGenerator<string, Embedding<float>> generator)
{
    public async Task<Embedding<float>> GenerateAsync(
        string text,
        CancellationToken cancellationToken)
    {
        return await generator.GenerateAsync(
            text,
            cancellationToken: cancellationToken);
    }
}

The important architectural benefit is separation of concerns.

Your retrieval service does not need provider-specific embedding code scattered throughout the application. The embedding generator can be injected into the component that needs it.

This becomes useful when you need to evaluate different embedding models or providers without rewriting the retrieval layer.

Production Considerations for .NET

A production embedding service should usually do more than call the provider. Depending on the application, you may also need cancellation support, retry policies appropriate for transient failures, rate limiting, logging, cost tracking, batching, and model/version information.

Be especially careful with automatic retries. A failed embedding request can be retried safely in some ingestion workflows, but aggressive retries can increase latency and cost when a provider is already throttling requests.

What Should You Store With an Embedding?

A vector by itself is rarely enough for a production RAG application.

You normally want the vector associated with the original text and metadata that allows the application to identify, filter, authorize, and display the retrieved content.

{
    "documentId": "policy-123",
    "chunkId": "policy-123-07",
    "content": "Employees can take up to twenty weeks...",
    "page": 14,
    "section": "Parental Leave",
    "tenantId": "company-a",
    "source": "employee-handbook.pdf",
    "embedding": [ ... ]
}

Metadata becomes particularly important when your application needs filtering.

For example, a user should not retrieve documents belonging to another tenant simply because those documents happen to have a high similarity score.

In a multi-tenant system, authorization should be enforced independently of semantic similarity. A high similarity score must never override document permissions.

Storing Embeddings With PostgreSQL and pgvector

If your application already uses PostgreSQL, you may not need to introduce an entirely separate database just to support vector search.

pgvector adds vector data types and similarity-search capabilities to PostgreSQL.

A simplified table might look like this:

CREATE TABLE document_chunks
(
    id UUID PRIMARY KEY,
    document_id UUID NOT NULL,
    content TEXT NOT NULL,
    page_number INT NULL,
    section_name TEXT NULL,
    embedding VECTOR(1536),
    created_at TIMESTAMPTZ NOT NULL
);

The important detail is:

embedding VECTOR(1536)

The vector dimension must match the embedding representation your application stores. If the selected model produces vectors with a different dimension, the schema and indexing strategy need to account for that.

In a real application, the table would commonly contain additional fields for document identity, tenant boundaries, permissions, model/version information, ingestion status, and other application-specific metadata.

Searching the Vector Store

Assume the user asks:

How much parental leave can I take?

Your application generates an embedding for the question and then searches the stored vectors.

A simplified pgvector query might look like this:

SELECT
    id,
    document_id,
    content,
    page_number,
    section_name
FROM document_chunks
ORDER BY embedding <=> @queryEmbedding
LIMIT 5;

The exact operator depends on the distance metric and index strategy you choose. The important part is the overall flow: the query vector is compared against stored vectors and the closest candidates are returned.

In production, the similarity query is usually only one part of retrieval. Metadata filters, authorization boundaries, lexical search, reranking, and result thresholds may also be involved.

Why Metadata Filtering Matters

Similarity alone is not enough for many business applications.

Consider a multi-tenant SaaS application with documents belonging to thousands of customers.

A vector search might mathematically find another customer's document to be extremely similar to the current user's question.

That document still must not be returned.

A production retrieval system therefore often combines:

• Vector similarity.
• Tenant or user filtering.
• Document permissions.
• Document status.
• Language or category filters.
• Optional keyword constraints.

This is one of the important differences between a simple RAG demo and a production retrieval system.

Security principle: Retrieval should operate inside the user's authorization boundary. Do not use similarity search as an authorization mechanism.

Why Changing the Embedding Model Matters

Suppose you initially generate all of your document embeddings using model A. Six months later, you decide that model B is cheaper or produces better retrieval results.

It may be tempting to change the model used for user queries and leave all existing document vectors untouched. That is generally not a safe assumption.

Different embedding models can produce vectors with different dimensions and different learned representations. Vectors from different models should not automatically be treated as if they occupy the same representation space.

A model migration may therefore require:

1. Creating embeddings with the new model.
2. Updating or creating the stored vectors.
3. Rebuilding the relevant vector indexes when required.
4. Testing retrieval quality.
5. Switching query generation to the new model.
6. Removing the old vectors only after the migration has been validated.

This is why the embedding model and vector dimension should be treated as part of the data contract of a RAG system rather than as an insignificant configuration detail.

For larger systems, storing the embedding model or embedding version alongside indexed data can make migrations and troubleshooting considerably easier.

Chunking Affects Embedding Quality

Embedding quality is not determined by the model alone. The text supplied to the model matters too.

Suppose a policy contains:

Employees are eligible for twenty weeks of parental leave
if they meet the following conditions...

If your chunking algorithm separates the main statement from the conditions, retrieval may return a chunk that appears relevant but does not contain enough information to answer the question correctly.

On the other hand, extremely large chunks can combine several unrelated topics into a single embedding. This can make the retrieved context less focused.

A useful chunking strategy therefore needs to preserve enough context while keeping individual chunks focused. The correct strategy depends on the document structure, query patterns, embedding model, and downstream context limits.

Chunk Boundaries Should Preserve Meaning

For structured documents, natural boundaries such as headings, sections, paragraphs, tables, and list groups can often provide better starting points than blindly splitting text at a fixed character count.

Chunking should also preserve useful metadata such as the document name, section heading, page number, and source location so that retrieved content can be traced back to the original document.

Common Embedding Mistakes

1. Treating Embeddings as Magic

Embeddings improve retrieval, but they do not guarantee that the correct document will always be returned. Retrieval remains a system-design problem involving data quality, chunking, queries, filtering, model choice, and evaluation.

2. Choosing a Model Only by Price

Cost matters, especially at scale, but retrieval quality, latency, supported languages, vector dimensions, operational requirements, and evaluation results matter too.

3. Ignoring Metadata

A vector without document identity, permissions, tenant information, source information, or other useful metadata can make production retrieval much harder.

4. Assuming the Highest Score Means "Correct"

Similarity is a retrieval signal, not a correctness guarantee.

A highly similar chunk can still fail to contain the information required to answer the question.

5. Never Evaluating Retrieval Quality

A RAG application can produce convincing answers during a demo while consistently retrieving the wrong source documents.

Create a small evaluation dataset containing realistic questions and expected source documents. Then test whether the expected information appears in the top retrieved results.

6. Mixing Embedding Models

Document embeddings and query embeddings need to be compatible. Accidentally indexing documents with one model and generating queries with another can lead to poor or meaningless retrieval.

7. Ignoring Embedding Failures During Ingestion

An ingestion pipeline should not silently mark a document as successfully indexed if some of its chunks failed to receive embeddings. Track ingestion status and make failed chunks observable so incomplete indexes can be repaired.

How to Troubleshoot Poor Vector Search Results

When a RAG application retrieves the wrong content, don't immediately assume that the LLM is the problem. Debug the retrieval pipeline separately.

1. Inspect the original document. Make sure the information actually exists in the source.
2. Inspect the chunks. Verify that important context was not lost during chunking.
3. Inspect the generated query. Make sure the application is embedding the question you expect.
4. Inspect retrieved chunks. Log document IDs, scores, and source metadata during development.
5. Test different top-K values. Returning only one result may be too restrictive for some questions.
6. Check metadata filters. Incorrect filters can silently remove useful documents.
7. Check the embedding model. Confirm that document and query embeddings are generated using compatible models.
8. Evaluate retrieval independently from generation. First determine whether the correct evidence was retrieved. Then evaluate whether the LLM used that evidence correctly.

A useful debugging rule: If the correct evidence never appears in the retrieved candidates, changing the prompt of the final LLM will not fix the retrieval problem.

Embeddings Are Useful Beyond RAG

RAG is one of the most common applications, but embeddings can be useful for many other problems.

• Semantic document search.
• Finding similar support tickets.
• Duplicate-content detection.
• Recommendation systems.
• Finding related documentation.
• Grouping similar content.
• Semantic code search.
• Question-to-document matching.

The common pattern is straightforward:

Content
   ↓
Embedding
   ↓
Vector representation
   ↓
Compare with other vectors
   ↓
Rank related content

A Practical Mental Model

If you remember only one thing from this article, remember this:

An embedding is not the answer. It is a numerical representation that makes comparison and retrieval possible.

The complete mental model is:

Text
  ↓
Embedding model
  ↓
Vector representation
  ↓
Vector database
  ↓
Similarity search
  ↓
Relevant content
  ↓
LLM
  ↓
Answer

Once this model becomes familiar, vector databases and semantic search become considerably easier to reason about.

What to Monitor in a Real .NET RAG System

Building the first prototype is relatively easy. Operating a RAG system reliably is where additional engineering becomes important.

For a production .NET application, I would track at least:

• Embedding generation latency.
• Embedding API failures.
• Embedding token usage where the provider exposes it.
• Embedding cost.
• Number of chunks indexed.
• Number of chunks retrieved per query.
• Similarity or distance scores.
• Retrieval latency.
• LLM latency after retrieval.
• Failed or low-quality retrievals.
• Document versions.
• Embedding model and version information.

Logging these values makes it much easier to answer questions such as: "Did retrieval get worse after we changed the embedding model?" or "Why did the cost of document ingestion increase this month?"

Avoid logging sensitive document content indiscriminately. Retrieval observability should be designed around the application's privacy, authorization, and data-retention requirements.

Embeddings Are Only One Part of RAG

It is easy to focus heavily on embeddings because they are the part of RAG that introduces vectors and new infrastructure.

In practice, retrieval quality depends on the entire pipeline:

Source documents
      ↓
Text extraction
      ↓
Chunking
      ↓
Embedding model
      ↓
Vector storage
      ↓
Metadata filtering
      ↓
Similarity search
      ↓
Reranking / selection
      ↓
Context construction
      ↓
LLM
      ↓
Answer

A better embedding model cannot compensate for badly extracted text, poor chunking, missing permissions, incorrect metadata, or a retrieval strategy that returns irrelevant context.

This is why RAG quality should be evaluated as a pipeline rather than by looking at the embedding model in isolation.

Final Thoughts

Embeddings are one of those AI concepts that can sound much more complicated than they need to be.

At the application level, the basic workflow is:

Text
  ↓
Embedding model
  ↓
Vector
  ↓
Store
  ↓
Compare with another vector
  ↓
Retrieve related content

The interesting engineering work starts after that.

You need to choose an appropriate embedding model, design useful chunks, preserve metadata, enforce document permissions, select a suitable similarity strategy, measure retrieval quality, and monitor the pipeline once it reaches production.

For .NET developers building RAG applications, embeddings are the bridge between ordinary application data and vector-based retrieval.

Once the vector makes sense, RAG becomes much easier to reason about.

Frequently Asked Questions

Is an embedding the same thing as an LLM response?

No. An embedding is a numerical representation used primarily for comparison, retrieval, clustering, or related tasks. An LLM response is generated text.

Do I need a vector database to use embeddings?

Not necessarily. You can generate embeddings and compare them in application code for small datasets. A vector database becomes useful when you need efficient similarity search across a larger collection of vectors and want indexing and filtering capabilities.

Does a larger embedding dimension always mean better results?

No. Vector dimension is only one property of an embedding model. Retrieval quality depends on the model, the data, chunking strategy, query formulation, similarity method, filtering, and evaluation methodology.

Do I need to re-embed documents when changing models?

In most cases, you should plan for re-embedding when moving to a different embedding model because the new model can produce vectors with a different dimension and representation space.

Are embeddings useful outside RAG?

Yes. They can be used for semantic search, recommendations, similarity detection, clustering, document matching, support-ticket analysis, and semantic code search, among other applications.

Should I use vector search alone for a production RAG system?

Not necessarily. Vector search is useful for semantic retrieval, but production systems may benefit from metadata filtering, lexical or keyword search, reranking, authorization checks, and retrieval evaluation. The right combination depends on the application's data and query patterns.

Key Takeaways

• An embedding is a numerical representation of content, not an answer generated by an LLM.
• The complete vector matters more than the meaning of any individual dimension.
• Semantic search can retrieve conceptually related content even when exact keywords differ.
• Document and query embeddings must use compatible representations.
• Chunking and metadata can have as much practical impact on retrieval as the embedding model itself.
• Similarity scores are retrieval signals, not guarantees that the retrieved content is correct.
• Changing embedding models can require re-embedding data and rebuilding vector indexes.
• Production RAG systems should monitor retrieval quality, latency, failures, cost, permissions, and model versions.

References and Further Reading

• Microsoft documentation for Microsoft.Extensions.AI
• Microsoft documentation covering .NET AI application development
• PostgreSQL documentation for PostgreSQL database features
• pgvector documentation for PostgreSQL vector similarity search