Running AI Locally Without API Keys

Cloud AI APIs are convenient until they're not: cost at scale, data privacy constraints, rate limits, network dependency, and the friction of managing API keys in every environment. A surprising amount of useful AI functionality runs perfectly well on a developer laptop with no external dependencies.

The Local AI Stack

Three layers cover most use cases:

1. Embedding: sentence-transformers for semantic vector generation
2. Inference: Ollama for local LLM completions
3. Fallback: deterministic extractive synthesis when no model is available

Embeddings with sentence-transformers

from sentence_transformers import SentenceTransformer
import numpy as np
 
# One-time download (~80MB), then runs fully offline
model = SentenceTransformer("all-MiniLM-L6-v2")
 
def embed(text: str) -> np.ndarray:
    return model.encode(text, normalize_embeddings=True)
 
# Semantic similarity without any API
a = embed("the server crashed with a memory error")
b = embed("the process ran out of RAM and terminated")
c = embed("the weather is nice today")
 
print(np.dot(a, b))  # → ~0.82 (similar)
print(np.dot(a, c))  # → ~0.11 (unrelated)

The model is 80MB on disk. First call downloads it to ~/.cache/huggingface/. Every subsequent call is pure local inference. On CPU, expect ~50-100ms per 256-token chunk.

LLM Inference with Ollama

Ollama runs LLMs locally with a REST API that mirrors OpenAI's interface:

# Install (macOS/Linux)
curl -fsSL https://ollama.ai/install.sh | sh
 
# Pull a model
ollama pull qwen2.5:3b    # 2GB, runs on 8GB RAM
ollama pull llama3.2:3b   # similar, good for Q&A

Call it from Python:

import urllib.request
import json
 
def local_completion(prompt: str, model: str = "qwen2.5:3b") -> str:
    data = json.dumps({
        "model": model,
        "prompt": prompt,
        "stream": False
    }).encode()
 
    req = urllib.request.Request(
        "http://localhost:11434/api/generate",
        data=data,
        headers={"Content-Type": "application/json"}
    )
    with urllib.request.urlopen(req) as resp:
        return json.loads(resp.read())["response"]

No SDK, no dependencies beyond the standard library. Cognix uses this exact pattern — OllamaLLMClient in Neural Nexus's bridge layer is stdlib-only for the same reason.

Deterministic Fallback

When no Ollama instance is running and no API key is configured, you can still produce useful answers extractively:

def extractive_answer(query: str, chunks: list[str]) -> str:
    query_words = set(query.lower().split())
    scored = []
    for i, chunk in enumerate(chunks):
        chunk_words = set(chunk.lower().split())
        score = len(query_words & chunk_words) / max(len(query_words), 1)
        scored.append((score, i, chunk))
 
    scored.sort(reverse=True)
    top_sentences = []
    for _, _, chunk in scored[:3]:
        # Extract sentences containing query keywords
        for sentence in chunk.split('. '):
            if any(w in sentence.lower() for w in query_words):
                top_sentences.append(sentence.strip())
                if len(top_sentences) >= 5:
                    break
 
    return '. '.join(top_sentences) + '.' if top_sentences else chunks[0]

This is always available, always fast, and produces acceptable answers for keyword-dense technical queries. It's the last resort in Cognix's four-tier synthesis cascade.

Trade-offs vs Cloud

| | Local | Cloud | |---|---|---| | Privacy | Data stays on device | Data leaves your machine | | Cost | Free after hardware | Pay per token | | Latency | 500ms-5s (CPU) | 200-800ms (network) | | Quality | Good for 3B-7B models | Best-in-class | | Availability | Works offline | Requires internet | | Setup | One-time model download | API key management |

When Local Is the Right Choice

Local inference is the right default when:

• You're indexing personal documents (health records, finances, private code)
• You're building a tool for air-gapped or regulated environments
• You need to process millions of tokens and cloud costs are prohibitive
• You want CI/CD pipelines that run without secrets

Cloud is better when quality matters more than privacy and latency is acceptable. Cognix makes this explicit: the synthesis cascade tries local options first and only escalates to cloud providers if explicitly enabled.

Practical Size Guide

| Model | Size | RAM needed | Quality | |---|---|---|---| | all-MiniLM-L6-v2 (embeddings) | 80MB | 200MB | Good semantic search | | qwen2.5:3b | 2GB | 4GB | Q&A, summarization | | llama3.2:3b | 2GB | 4GB | Instruction following | | qwen2.5:7b | 4.5GB | 8GB | Better reasoning | | llama3.1:8b | 4.7GB | 8GB | Strong general purpose |

A developer machine with 16GB RAM can run 7B models comfortably alongside a local database and web server.