ADR-006: Vector embeddings and hybrid search for knowledge retrieval
Adopt hybrid retrieval combining lexical BM25 matching with vector embeddings (text-embedding-004) for Ask Alan.
Launch stateThese entries are placeholders while the real case studies and experiment write-ups are prepared for publication.
Context
- Current retrieval relies primarily on lexical keyword token matching across titles, topics, and descriptions.
- Queries with natural conceptual overlap (e.g., 'zero trust admission gates' vs 'secure supply chain signing') fail to retrieve ADRs if exact terms differ.
Decision
- Implement hybrid retrieval: combine lexical BM25 search with dense vector similarity via pgvector.
- Generate chunked document embeddings using Google's text-embedding-004 model via @google/genai.
- Reciprocal Rank Fusion (RRF) to merge and rank lexical and semantic candidates before synthesis.
Alternatives considered
- Pure lexical search. Rejected due to vulnerability to vocabulary mismatch.
- Pure vector search without keyword filters. Rejected because exact identifiers (e.g. CVE codes, technology names) get lost in vector space.
Consequences
- High recall and semantic understanding for conceptual engineering questions.
- Requires embedding generation pipeline during content indexing / write triggers.
- Maintains strict grounding: retrieved chunks must still be explicitly cited in the generated answer.
Roadmap & Marcos de Implementação
- Standardization on text-embedding-004(Phase 1: Embedding Model Selection)Concluído
Selected Google text-embedding-004 (768 dimensions) for superior semantic understanding across cloud native and DevSecOps domains.
- Reciprocal Rank Fusion (RRF)(Phase 2: Hybrid Candidate Fusion Algorithm)Em desenvolvimento
Implemented fusion ranking combining lexical keyword BM25 candidates with semantic cosine similarity scores before LLM generation.
- PostgreSQL pgvector Schema Migration(Phase 3: Vector Storage & Ingestion Pipeline)Planejado
Provisioning pgvector table with automated re-indexing triggers executing whenever content is committed via /write.
- Semantic Recall & Grounding Evaluation(Phase 4: Benchmark & Evaluation Testing)Planejado
Evaluation suite testing conceptual paraphrase queries (e.g. 'zero trust ingress' vs 'Istio mutual TLS') to ensure 100% citation grounding.