Embedding Backends Compared: Hash, Sentence-Transformers, OpenAI

When building Cognix, I needed an embedding backend that worked without API keys, ran on a laptop, and produced vectors good enough for useful semantic search. That meant evaluating three very different approaches before settling on a configurable cascade.

What an Embedding Is

An embedding turns a piece of text into a fixed-length vector of floats. The useful property: similar texts produce similar vectors. Similarity is measured by cosine distance — the angle between two vectors in high-dimensional space.

python

import numpy as np
 
def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

The quality of your retrieval is only as good as your embedding model's ability to place semantically similar texts near each other.

Option 1: SHA-256 Hash Embeddings (Cognix Default)

The simplest possible approach: hash individual words, accumulate into a fixed-length vector, normalize.

python

import hashlib
import numpy as np
 
def hash_embed(text: str, dim: int = 128) -> np.ndarray:
    words = text.lower().split()
    vec = np.zeros(dim, dtype=np.float64)
    for word in words:
        digest = hashlib.sha256(word.encode()).digest()
        indices = [int.from_bytes(digest[i:i+2], 'big') % dim for i in range(0, 16, 2)]
        for idx in indices:
            vec[idx] += 1.0
    norm = np.linalg.norm(vec)
    return vec / norm if norm > 0 else vec

Pros: Deterministic, zero dependencies, zero latency, works offline. Good enough for keyword-dense technical documents where you're mostly matching exact terms.

Cons: Zero semantic understanding. "car" and "automobile" get completely different vectors. Synonym matching fails entirely. Dimensionality is low (128) so the space is crowded.

When to use: Default fallback, offline environments, testing pipelines without model dependencies, very keyword-structured data like code or config files.

Option 2: Sentence-Transformers

all-MiniLM-L6-v2 produces 384-dimensional vectors trained on 1B+ sentence pairs to encode semantic meaning:

python

from sentence_transformers import SentenceTransformer
 
model = SentenceTransformer("all-MiniLM-L6-v2")
 
def st_embed(text: str) -> np.ndarray:
    vec = model.encode(text, normalize_embeddings=True)
    return vec  # shape: (384,)

The model is 80MB and runs on CPU in ~50-100ms per chunk. It understands that "the vehicle broke down" and "my car stopped working" are semantically similar.

Pros: True semantic understanding, runs locally, one-time download, L2-normalized output ready for cosine search.

Cons: First-load model download (~80MB), 50-100ms latency per embed on CPU (fine for batch ingest, slow for real-time if you have thousands of chunks), still limited context window (256 tokens).

When to use: Local setups where you want real semantic search without API costs. Best for prose documents, Q&A, and anything where synonym matching matters.

Option 3: OpenAI text-embedding-3-small

1536-dimensional vectors from a model trained on vastly more data:

python

from openai import OpenAI
 
client = OpenAI()
 
def openai_embed(text: str) -> np.ndarray:
    resp = client.embeddings.create(
        model="text-embedding-3-small",
        input=text,
    )
    return np.array(resp.data[0].embedding)

Pros: Highest quality, handles nuance, 8192-token context window so long chunks don't get truncated, fast (API latency, not CPU-bound).

Cons: Requires API key, costs money ($0.02/million tokens), network dependency, data leaves your machine.

When to use: Production systems where retrieval quality is critical and data sensitivity allows cloud processing.

Comparing Them in Practice

For a corpus of 500 technical documents with queries like "how does the authentication middleware work":

| Backend | P@5 | Latency (per chunk) | Offline | Cost | |---|---|---|---|---| | Hash (128d) | ~0.42 | <1ms | ✓ | Free | | Sentence-Transformers | ~0.71 | ~80ms CPU | ✓ | Free | | OpenAI 3-small | ~0.84 | ~200ms (network) | ✗ | $0.02/1M tokens |

The hash backend holds up better than you'd expect when you pair it with keyword search in a hybrid retrieval pipeline — which is exactly what Cognix does with RRF. The semantic gap is partially covered by the keyword layer.

The Cognix Approach

Rather than picking one, Cognix makes the backend configurable:

bash

COGNIX_LOCAL_EMBEDDING_BACKEND=hash               # default, always works
COGNIX_LOCAL_EMBEDDING_BACKEND=sentence-transformers  # local semantic
COGNIX_CLOUD_EMBEDDINGS_ENABLED=true              # OpenAI fallback

The retrieval pipeline runs hybrid search regardless of backend — keyword match, SQLite vector cosine, and ChromaDB semantic — then merges results with Reciprocal Rank Fusion. A weaker embedding backend loses some signal in the semantic layer but the keyword layer compensates, making the system reasonably robust even at the default hash setting.