Embedding (dense vector representation)

Definition

An embedding is a dense vector — a list of numbers, typically 384 to 3072 dimensions long — that represents a piece of data in a way that captures its semantic meaning. Text embeddings represent sentences or documents; image embeddings represent images; multimodal embeddings can represent both in a shared space. The key property: items with similar meaning produce similar vectors, regardless of surface differences.

You compute similarity between embeddings using distance metrics — most commonly cosine similarity or dot product for normalized vectors. A cosine similarity of 1.0 means identical direction (same meaning); 0.0 means unrelated; -1.0 means opposite. In practice, most useful pairs sit in the 0.3 to 0.9 range.

Embeddings came into their own with Word2Vec in 2013, which showed that word meaning could be captured in dense vectors trained via a simple prediction objective. The famous demonstration: king - man + woman ≈ queen. Vector arithmetic on semantic space. Modern embedding models (OpenAI text-embedding-3, Cohere embed-v3, BGE, E5) generalize this to sentences and documents, using contrastive training on massive pairs of semantically related text.

Embeddings are the substrate of the entire modern retrieval-augmented AI stack. When you build a RAG system, you embed your documents at indexing time and queries at retrieval time, finding matches by vector similarity. When ChatGPT does 'memory' or when Claude has Projects, embeddings power the retrieval underneath. When you upload PDFs to any AI product and ask questions, embeddings are involved.

Real-world example

Why embeddings work — a concrete demonstration

Consider three sentences: 'The cat sat on the mat.', 'A feline rested on the rug.', and 'Docker containers are lightweight.'. To a keyword search, sentences 1 and 3 share more words with a query like 'What did the cat do?' than sentence 2 does (which shares zero words). But an embedding model produces vectors where sentence 2 is close to sentence 1 (both about a cat resting on fabric) and sentence 3 is far away. Cosine similarity turns semantic meaning into a number — and that number drives everything from RAG retrieval to recommendation systems.

# Computing embeddings with the OpenAI API
from openai import OpenAI

client = OpenAI()

def embed(text: str) -> list[float]:
    """Return the embedding vector for a piece of text."""
    response = client.embeddings.create(
        input=text,
        model="text-embedding-3-large",
    )
    return response.data[0].embedding

# Compute embeddings for three sentences
e1 = embed("The cat sat on the mat.")
e2 = embed("A feline rested on the rug.")
e3 = embed("Docker containers are lightweight.")

# Compute cosine similarity
import numpy as np

def cosine_sim(a, b):
    a, b = np.array(a), np.array(b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

print(cosine_sim(e1, e2))  # ~0.65 — semantically similar
print(cosine_sim(e1, e3))  # ~0.10 — unrelated
print(cosine_sim(e2, e3))  # ~0.11 — unrelated

# For 100K+ documents, you'd store vectors in a
# vector database (Pinecone, Weaviate, Qdrant, pgvector)
# and use ANN indexes (HNSW, IVF) for fast retrieval.

When you'll encounter this

• You're building semantic search — the reference use case for embeddings.
• You're building a RAG system — embeddings are how the retrieval half works.
• You need to cluster or categorize documents — embeddings + clustering algorithms (k-means, HDBSCAN) group similar content.
• You're building a recommendation system — user and item embeddings capture preference and content in the same space.
• You need deduplication or near-duplicate detection — high similarity between embeddings often means duplicates even with surface differences.

How it works

Common misconceptions

Related terms

Where you'll see this in practice

Learn more

• Read the deep-dive: How Embeddings Work
• Read: How RAG Works (End-to-End)
• Take the Master RAG in 10 Lessons path
• Browse Vector Databases & RAG category