# Support ticket classification with reproducible evaluation

> Project scaffold: replace TODO fields with your implementation and measured results before publishing. This template does not contain a completed application.

## Objective
Build a service that routes synthetic support tickets to billing, technical support or account assistance, and sends uncertain predictions to human review. Demonstrate repeatable evaluation and a versioned model release.

## Architecture
```mermaid
flowchart TD
  N0["Synthetic labeled tickets"]
  N1["Versioned train / validation / test split"]
  N2["TF-IDF classifier and MLflow runs"]
  N3["Selected model artifact"]
  N4["FastAPI prediction endpoint"]
  N5["Review queue and aggregate metrics"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
```

Training is an offline path. The API loads a selected artifact at startup; it does not train on incoming requests. Store low-confidence ticket IDs in a review queue and keep ticket text out of request logs.

## Tools and prerequisites
Python, basic classification metrics, HTTP APIs and Git.
- Python and scikit-learn for the baseline classifier
- MLflow for experiment parameters, metrics and artifacts
- FastAPI and Pydantic for validated requests
- SQLite for a local review queue
- pytest and Docker for repeatable testing and packaging

## Repository structure (proposed)
```text
README.md
src/                 # Your implementation
tests/               # Unit, integration and failure-case tests
fixtures/            # Small synthetic or permitted inputs
config/              # Non-secret configuration examples
docs/architecture.md # Decisions and tradeoffs
docs/runbook.md      # Recovery, rerun and cleanup procedures
reports/             # Sanitized evidence and measured results
.env.example         # Variable names and safe placeholders only
```

## Setup and execution
- TODO: Record supported OS, runtime versions, pinned dependencies and hardware requirements.
- TODO: Add exact commands to install, configure, start and run a sample input after implementing them.
- TODO: Document environment variables in .env.example; keep secrets and local .env files out of Git.
- TODO: Include expected sample output and any optional hosted-service costs.

## Implementation checklist
- [ ] Write a label guide and create a small synthetic dataset with ticket ID, text and category. Include ambiguous and unrelated requests. Document that synthetic results do not represent real customer performance.
- [ ] Deduplicate before splitting and keep near-duplicate tickets in the same split. Commit the split manifest and random seed; reserve the test set until model and threshold selection are complete.
- [ ] Fit a majority-class baseline, then a TF-IDF plus logistic regression pipeline using only training data. Record macro-F1, per-class recall, dataset checksum and parameters in MLflow.
- [ ] Use validation data to choose a confidence threshold. Report how coverage and accuracy change as more requests go to review; do not interpret uncalibrated confidence as guaranteed correctness.
- [ ] Package the complete preprocessing and model artifact together. Add POST /predict with bounded text length, a model version in the response and a review decision for uncertain predictions.
- [ ] Test malformed input, empty text, out-of-domain examples and identical predictions after reloading the artifact. Keep a regression fixture for a previously misclassified ticket.
- [ ] Run a local load test with a fixed concurrency and request mix. Save p50/p95 latency, error rate and hardware details alongside the untouched test-set report.
- [ ] Record a short demo that trains a candidate, compares it with the baseline and switches back to a previous artifact. Publish your evaluation and review-queue code with attribution.

## Tests and acceptance evidence
- [ ] A split manifest and evaluation report with per-class results
- [ ] Two tracked experiments and a documented selection decision
- [ ] A passing API regression test and a reproducible latency run
- [ ] A model card describing synthetic-data limitations and review behavior
- TODO: Add the exact test command and a link to a passing run.
- TODO: Explain at least one failure case and how it is detected or recovered.

## Results
Do not replace missing measurements with estimates presented as observations.

| Metric | Baseline | Result | Dataset / hardware / run link |
| --- | --- | --- | --- |
| TODO: choose a project metric | Not measured | Not measured | TODO |

## Limitations and next steps
- TODO: State what this lab does not establish about production reliability, security or model quality.
- TODO: Document cleanup, retained data and optional infrastructure charges.
- TODO: Link an issue for the next improvement and explain its priority.

## Interview preparation
- Why can random row splitting inflate results when tickets contain duplicates?
- Why choose macro-F1 instead of accuracy, and which class errors matter most?
- How did you choose the review threshold and measure the coverage tradeoff?
- What must be versioned together to reproduce a prediction or roll back a release?

## Resume draft (use only after completing the work)
Replace bracketed values with real evidence and remove claims you did not implement.
- Built a versioned support-ticket classification API with MLflow experiment tracking, input validation and a human-review fallback; evaluated macro-F1 on [N] held-out synthetic examples.
- Compared [N] classifier configurations against a majority baseline and measured [p95 latency] at [concurrency] on [hardware], documenting failure cases and deployment tradeoffs.

## References, attribution and your contribution
- [MLflow](https://github.com/mlflow/mlflow): Reference for experiment tracking, evaluation and model lifecycle concepts. Follow its linked model-training documentation for current APIs.

Add your own dataset split checks, threshold analysis, review queue and rollback demonstration. These deliverables are the proposed BonusMantra project, not features copied from an upstream example.

- TODO: Record the exact upstream commit/tag you consulted and any files reused or modified.
- TODO: Preserve required copyright/license notices and check each repository's reuse terms before copying code. Choose a license only for work you have rights to license.
- TODO: Explain your own implementation and link its commits; do not claim authorship of upstream code.

Project brief: [BonusMantra](https://bonusmantra.com/career-roadmaps/ai-engineer/projects/); reference pages checked 2026-09-06.