ACCEPTEDRoadmap: V1.3Draft2025-04-20 — updated 2025-07-10
ADR-002: Retrieval over fine-tuning for runbook answers
Answer operational questions by retrieving from runbooks rather than fine-tuning a model on internal incidents.
Launch stateThese entries are placeholders while the real case studies and experiment write-ups are prepared for publication.
Context
- On-call engineers needed faster answers to known operational questions without relying on undocumented tribal knowledge.
- Fine-tuning on incident history risked baking outdated or sensitive details into model weights.
Decision
- Use retrieval over indexed runbooks and incident notes to ground answers.
- Require every answer to cite a source or refuse explicitly.
Alternatives considered
- Fine-tune a small model on incident history. Rejected because updates are expensive and citations are unreliable.
- Use a general model without grounding. Rejected because it invents steps for procedural questions.
Consequences
- Answers stay tied to current documents; updating a runbook immediately changes what the system can say.
- Evaluation must include refusal cases as first-class tests.
- Chunking and embedding quality become the main engineering work, not model training.
Roadmap & Marcos de Implementação
V1.3Grounded Operational Knowledge
Progresso da Execução Arquitetural100% CONCLUÍDO
- Hierarchical Document Chunking(Phase 1: Runbook & Incident Ingestion)Concluído
Created automated chunking pipelines separating procedural steps, symptom patterns, and remediation guides.
- Strict Zero-Hallucination Guardrail(Phase 2: Citation Verification & Refusal Policy)Concluído
Enforced explicit citation rules: model responses must cite verified anchors or refuse to answer speculative questions.
- Regression & Drift Testing(Phase 3: Accuracy Evaluation Suite)Concluído
Implemented test harness verifying that doc updates immediately change system output with zero model fine-tuning overhead.