RAG Projects

# Document assistant with citations and retrieval evaluation

> Project scaffold: replace TODO fields with your implementation and measured results before publishing. This template does not contain a completed application.

## Objective
Build a question-answering assistant over a small permitted document collection that cites source passages, refuses unsupported questions and measures retrieval separately from answer quality.

## Architecture
```mermaid
flowchart TD
  N0["Permitted documents and source IDs"]
  N1["Extraction and versioned chunks"]
  N2["Embeddings and document store"]
  N3["Query retrieval and optional reranking"]
  N4["Context-bound answer generation"]
  N5["Citation validation and evaluation report"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
```

Indexing and question answering are separate pipelines. Carry document version, source URI and chunk ID through retrieval into citations. An empty or insufficient evidence set routes to an explicit unsupported-answer response.

## Tools and prerequisites
Python, embeddings, HTTP APIs and basic retrieval concepts.
- Python and Haystack for modular pipelines
- A local document store for the first baseline
- A compatible embedding model and local or hosted generator
- pytest and a versioned question/evidence fixture
- A simple CLI or FastAPI endpoint for the demo

## Repository structure (proposed)
```text
README.md
src/                 # Your implementation
tests/               # Unit, integration and failure-case tests
fixtures/            # Small synthetic or permitted inputs
config/              # Non-secret configuration examples
docs/architecture.md # Decisions and tradeoffs
docs/runbook.md      # Recovery, rerun and cleanup procedures
reports/             # Sanitized evidence and measured results
.env.example         # Variable names and safe placeholders only
```

## Setup and execution
- TODO: Record supported OS, runtime versions, pinned dependencies and hardware requirements.
- TODO: Add exact commands to install, configure, start and run a sample input after implementing them.
- TODO: Document environment variables in .env.example; keep secrets and local .env files out of Git.
- TODO: Include expected sample output and any optional hosted-service costs.

## Implementation checklist
- [ ] Select documents you own or can reuse, record their source and reuse terms, and create a version manifest. Begin with text or Markdown so extraction errors are easy to inspect.
- [ ] Build an indexer that emits stable chunk IDs, source locations and document hashes. Compare two chunk sizes and document the behavior when a source changes or is deleted.
- [ ] Create a held-out question set with relevant passage IDs, unsupported questions and ambiguous requests. Keep threshold tuning questions separate from final evaluation questions.
- [ ] Implement a retrieval baseline and measure recall@k against the labeled passage set. Inspect misses before adding optional reranking, and preserve the baseline for comparison.
- [ ] Generate answers using only retrieved context and return source IDs. Check that each cited ID belongs to the retrieved evidence, then manually score whether the cited passage actually supports the claim.
- [ ] Add abstention for insufficient evidence, bounded input length and a clear separation between document text and instructions. Test a document containing misleading instructions; never give retrieved text authority to run tools.
- [ ] Evaluate retrieval recall, supported-answer rate, unsupported-question refusal rate and p95 latency with fixed model/configuration versions. If using a hosted model, record token usage and actual charges separately.
- [ ] Publish an error analysis with at least one retrieval miss, one unsupported answer and one correct refusal. Include source deletion/reindexing tests and a demo showing citations opening the original passage.

## Tests and acceptance evidence
- [ ] A document/chunk manifest and a labeled question set
- [ ] A baseline versus changed-retrieval evaluation report
- [ ] Citation identity checks plus human evidence-support review
- [ ] Unsupported-question, document-update and deletion tests
- TODO: Add the exact test command and a link to a passing run.
- TODO: Explain at least one failure case and how it is detected or recovered.

## Results
Do not replace missing measurements with estimates presented as observations.

| Metric | Baseline | Result | Dataset / hardware / run link |
| --- | --- | --- | --- |
| TODO: choose a project metric | Not measured | Not measured | TODO |

## Limitations and next steps
- TODO: State what this lab does not establish about production reliability, security or model quality.
- TODO: Document cleanup, retained data and optional infrastructure charges.
- TODO: Link an issue for the next improvement and explain its priority.

## Interview preparation
- How can you tell a retrieval failure from a generation failure?
- Why does a valid citation ID not guarantee the answer is supported?
- How do chunk size and top-k affect latency, context noise and recall?
- How do you remove a deleted document from both the index and cached answers?

## Resume draft (use only after completing the work)
Replace bracketed values with real evidence and remove claims you did not implement.
- Built a citation-first RAG assistant with versioned document ingestion, retrieval evaluation and explicit unsupported-question handling over [N] permitted documents.
- Compared [retrieval configurations] on [N] held-out questions, reporting recall@[k], manually checked answer support and [p95 latency] with reproducible evaluation fixtures.

## References, attribution and your contribution
- [Haystack](https://github.com/deepset-ai/haystack): Python framework reference for modular indexing, retrieval and generation pipelines. Its README links to current tutorials, component documentation and example recipes.

Haystack supplies the pipeline framework. Your contribution is the corpus, chunk lifecycle, labeled evaluation set, citation checks, abstention behavior and failure analysis. Local compute is sufficient for a small baseline; hosted models may cost money.

- TODO: Record the exact upstream commit/tag you consulted and any files reused or modified.
- TODO: Preserve required copyright/license notices and check each repository's reuse terms before copying code. Choose a license only for work you have rights to license.
- TODO: Explain your own implementation and link its commits; do not claim authorship of upstream code.

Project brief: [BonusMantra](https://bonusmantra.com/career-roadmaps/rag-engineer/projects/); reference pages checked 2026-09-06.