Core RAG Pipeline Stages
- Document Ingestion & Chunking: Text parsing, token-based chunking, overlap strategies, and metadata preservation.
- Vector Storage & Search: Embedding generation, Qdrant / Pinecone / pgvector indexing, HNSW cosine similarity.
- Hybrid Search & Reranking: Combining dense vector search with sparse BM25 keyword search, followed by Cohere/Cross-Encoder reranking.
- Evaluation & Grounding: Measuring context recall, faithfulness, answer relevance, and hallucination rates.