Apex Coding AcademyExplore the curriculum
TIER 3 · PRODUCTION CAPSTONE

Regional Farm Control Plane

Operate a multi-tenant worker control plane with fair scheduling, signed leases, Kubernetes isolation, authorized route pools, telemetry, autoscaling, failover, and cost controls.

Download starter bundle (.zip)Back to course

System architecture

  1. An admission API validates tenant quotas and records immutable job identities in Postgres.
  2. Redis Streams partitions ready work while a scheduler applies deficit round-robin fairness.
  3. Workers claim expiring signed leases, checkpoint effects, and renew only while healthy.
  4. Kubernetes isolates service accounts, namespaces, network policy, resources, and disruption budgets.
  5. OpenTelemetry traces join admission, queue, worker, route, and authoritative verification spans.
  6. A regional reconciler promotes work only after fencing the failed region and checking operation state.

Acceptance requirements

  • Sustain the supplied load profile while meeting queue-latency and success-rate SLOs.
  • Guarantee tenant fairness and global/provider rate budgets during burst traffic.
  • Use lease fencing so a paused worker cannot commit after another worker takes over.
  • Allow only contract-authorized route classes; health scoring cannot override policy.
  • Scale from queue age and service time with explicit floors, ceilings, and cost budgets.
  • Demonstrate zone disruption and regional failover without duplicate external effects.

Required failure drills

  • Pause a worker beyond its lease, resume it, and prove its stale commit is rejected.
  • Evict one zone under load; the disruption budget must preserve SLO capacity.
  • Fail Redis briefly; producers must shed load and workers must reconcile safely.
  • Mark a route unhealthy; authorized jobs reroute while prohibited route classes remain prohibited.
  • Fence the primary region and promote the secondary using authoritative operation checks.

Submission deliverables

  • Admission, scheduler, worker lease, reconciliation, and telemetry services
  • Docker Compose developer stack plus production Kubernetes manifests
  • Load generator, chaos drills, SLO dashboard, alerts, and cost guardrails
  • Threat model covering tenant isolation, routes, secrets, leases, and supply chain
  • Regional recovery report proving RPO, RTO, and no duplicate effects

Automated verification

$ npm ci$ npm test$ docker compose up --build --wait$ npm run test:load$ kubectl apply --dry-run=server -f k8s/

100-point professional rubric

Concurrency and lease correctnessFencing and duplicate-effect tests pass under load.25 pts
Fairness and backpressureTenant shares and queue bounds meet the supplied profile.20 pts
Kubernetes securityPolicy tests prove isolation and least privilege.20 pts
Reliability and recoveryZone and region drills meet documented RPO/RTO.20 pts
Telemetry and costSLO alerts and budget ceilings trigger in controlled tests.15 pts

Starter repository contents

OPERATIONS.mddocker-compose.ymlk8s/network-policy.yamlk8s/worker.yamlpackage.jsonscripts/load.mjssrc/autoscale.mjssrc/lease.mjssrc/scheduler.mjstest/control-plane.test.mjs