# Kubernetes release pipeline with failure recovery

> Project scaffold: replace TODO fields with your implementation and measured results before publishing. This template does not contain a completed application.

## Objective
Deliver a small service through a repeatable build and Kubernetes rollout, then prove that a bad release is detected and a previous image can be restored. Use a local cluster before attempting an optional cloud deployment.

## Architecture
```mermaid
flowchart TD
  N0["Pull request and tests"]
  N1["Container build"]
  N2["Image registry with immutable digest"]
  N3["Kubernetes Deployment and Service"]
  N4["Smoke test and rollout observation"]
  N5["Rollback to recorded digest"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
```

CI builds the image; a deployment job applies the recorded digest to an isolated environment. Readiness controls traffic eligibility. Smoke-test failure triggers the recovery procedure, while metrics and logs explain what happened.

## Tools and prerequisites
Containers, basic Kubernetes objects, Git and CI workflows.
- Docker for packaging
- GitHub Actions for CI
- kind or minikube for a local Kubernetes cluster
- kubectl and Kustomize for deployment
- Prometheus and a small HTTP load-test script for measurements

## Repository structure (proposed)
```text
README.md
src/                 # Your implementation
tests/               # Unit, integration and failure-case tests
fixtures/            # Small synthetic or permitted inputs
config/              # Non-secret configuration examples
docs/architecture.md # Decisions and tradeoffs
docs/runbook.md      # Recovery, rerun and cleanup procedures
reports/             # Sanitized evidence and measured results
.env.example         # Variable names and safe placeholders only
```

## Setup and execution
- TODO: Record supported OS, runtime versions, pinned dependencies and hardware requirements.
- TODO: Add exact commands to install, configure, start and run a sample input after implementing them.
- TODO: Document environment variables in .env.example; keep secrets and local .env files out of Git.
- TODO: Include expected sample output and any optional hosted-service costs.

## Implementation checklist
- [ ] Study Online Boutique's service boundaries and Kubernetes manifests. Start with your own small HTTP service or a documented subset with its dependencies; the complete demo needs more resources.
- [ ] Create a health endpoint and an independently testable user endpoint. Add unit tests and a Dockerfile, then record exact tool versions and a repeatable local start procedure.
- [ ] Configure pull-request CI to run tests and build the container. Pin third-party actions to reviewed commits and grant the job only the permissions it needs.
- [ ] Publish an image from an approved branch and record its digest. Create a Deployment and Service with resource requests, limits, readiness and liveness probes; explain why each probe exists.
- [ ] Apply the manifests to a local cluster, wait for rollout completion and run a user-endpoint smoke test. Keep the cluster accessible to a local deployment job; do not assume a hosted runner can reach your laptop.
- [ ] Introduce a controlled faulty version in the lab, such as an endpoint returning an error. Capture detection time and distinguish a failed rollout from an application failure after rollout.
- [ ] Restore the last known good digest and rerun the smoke test. Record recovery time, requests lost during recovery and the exact point from which each measurement starts.
- [ ] Publish sanitized logs, manifests and a recovery runbook. Include a cleanup procedure and a local resource budget; optional cloud execution requires a separate cost estimate.

## Tests and acceptance evidence
- [ ] CI run showing a tested image and its immutable digest
- [ ] Deployment manifests with explained probes and resource limits
- [ ] A failed release followed by successful recovery
- [ ] A runbook with measured detection and recovery timelines
- TODO: Add the exact test command and a link to a passing run.
- TODO: Explain at least one failure case and how it is detected or recovered.

## Results
Do not replace missing measurements with estimates presented as observations.

| Metric | Baseline | Result | Dataset / hardware / run link |
| --- | --- | --- | --- |
| TODO: choose a project metric | Not measured | Not measured | TODO |

## Limitations and next steps
- TODO: State what this lab does not establish about production reliability, security or model quality.
- TODO: Document cleanup, retained data and optional infrastructure charges.
- TODO: Link an issue for the next improvement and explain its priority.

## Interview preparation
- How do readiness and liveness probes affect a deployment differently?
- Why is an image digest more reproducible than a mutable tag?
- What happens when an application passes health checks but fails a user journey?
- When would database changes make a simple image rollback unsafe?

## Resume draft (use only after completing the work)
Replace bracketed values with real evidence and remove claims you did not implement.
- Built a CI-to-Kubernetes delivery lab with immutable images, readiness checks and automated smoke tests; demonstrated recovery from a deliberately faulty release.
- Measured [detection time] and [recovery time] across [N] controlled rollout failures, and documented image rollback limitations and a recovery runbook.

## References, attribution and your contribution
- [Google Cloud Online Boutique](https://github.com/GoogleCloudPlatform/microservices-demo): Reference application with microservice source code and Kubernetes deployment manifests. Its README includes a service architecture and cloud quickstart; this proposed lab uses a smaller local scope.

Use the reference to understand deployment structure, then own the CI pipeline, smoke tests, failure experiment and recovery evidence. Attribute any reused manifests and preserve their notices.

- TODO: Record the exact upstream commit/tag you consulted and any files reused or modified.
- TODO: Preserve required copyright/license notices and check each repository's reuse terms before copying code. Choose a license only for work you have rights to license.
- TODO: Explain your own implementation and link its commits; do not claim authorship of upstream code.

Project brief: [BonusMantra](https://bonusmantra.com/career-roadmaps/devops-engineer/projects/); reference pages checked 2026-09-06.