Cognix Architecture

Data Flow

data/raw/ (immutable)
  └─ parsers.py (v4)         → plain text per file
  └─ chunking.py             → 450-word sliding windows, 70-word overlap
  └─ embeddings.py           → vector per chunk (hash / ST / OpenAI)
  └─ SQLite                  → raw_files, chunks, chunk_embeddings, jobs
  └─ ChromaDB                → vector index (data/chroma/)

Query (web UI → POST /api/ask)
  └─ retrieval.py            → EvidencePack (8 chunks)
  └─ llm.py                  → grounded answer + confidence score
  └─ wiki/outputs/analysis/  → saved draft markdown

Chunking

chunking.py — sliding window, word-based:

• max_words = 450, overlap_words = 70, step = 380 words
• Token estimate: len(text.split()) * 4 // 3
• Returns TextChunk(index, text, token_estimate)

Retrieval Pipeline

retrieve_evidence(question, limit=8) returns an EvidencePack:

Step 1 — File name match Tokenizes the question, scores all raw_files by name/path overlap. If strong hits exist, returns their chunks directly without going further.

Step 2 — Query decomposition For complex queries (≥14 words or comparison/relationship terms), calls the configured model to split into 2–4 sub-queries. Falls back to deterministic regex splitting if no model available.

Step 3 — Parallel search (per sub-query)

• retrieve_from_chroma() — ChromaDB semantic search
• retrieve_by_keyword() — exact keyword group matching across all chunks in SQLite
• retrieve_from_sqlite() — cosine similarity against hash embeddings in chunk_embeddings, filtered by keyword

Step 4 — Reciprocal Rank Fusion (k=60) Merges all ranked lists; best score per chunk_id kept.

Step 5 — Rerank and filter Keyword coverage bonus + source path bonus applied. Chunks below 45% of top score are dropped. Optional trained reranker applied if configured.

Step 6 — Extension filter If the question mentions a file type (e.g. "in my PDFs"), restricts to those extensions.

Answer Synthesis

synthesize_answer(question, chunks, style) — three-tier cascade:

1. Cognix Micro Model — local trained synthesis artifact (data/models/cognix-micro-synthesis.json), used when COGNIX_SYNTHESIS_BACKEND=cognix-micro
2. SFT Adapter — supervised fine-tuned adapter (data/models/cognix-sft-adapter.json), used when COGNIX_SYNTHESIS_BACKEND=cognix-sft-adapter
3. Configured provider — iterates enabled providers (OpenAI → Anthropic → Ollama), skipping any with blocked status (auth_failed, model_unavailable, quota_exhausted, rate_limited, missing_key)
4. Extractive fallback — sentence-level keyword scoring against retrieved chunks; formats as memo/brief with inline citations (source: path, chunk: id) — always available, no key required

Database Schema

V1 Tables (db/schema.sql)

V2 Tables (db/schema_v2.py)

Intelligence Pipeline

Located at services/intelligence/. Triggered by POST /api/intelligence/run or nightly scheduler.

GapDetector — finds concepts with ≤1 evidence source or zero retrieval hits → gap findings

ContradictionDetector — compares claim embeddings for semantic conflict → contradiction / contradiction_candidate findings; optionally uses LLM for verification

StalenessDetector — flags files not updated in >90 days or chunks with no recent query hits

After detection, generate_intelligence_brief() writes a markdown briefing to wiki/_intelligence/ and stores it in briefings.

Concept Graph

SQLite-backed via graph_edges. API endpoints:

• GET /api/graph/concept/{slug} — concept node + 1-hop neighbors
• GET /api/graph/neighbors/{slug} — recursive neighbors up to depth 3 (via recursive CTE)

Edges are created by the intelligence pipeline when claims reference related concepts.

Configuration Reference

All environment variables are prefixed COGNIX_:

API Reference