Designing a code deployment system

The problem

Design a code deployment system (CI/CD like Spinnaker, or an internal deployer): take committed code, build it into an artifact, and roll it out to a fleet of thousands of servers across regions safely — catching bad deploys before they take down production, and rolling back fast when they do.

Step 1 — Requirements

Functional: build code → artifact; distribute the artifact to many servers; deploy with a strategy (rolling/blue-green/canary); run health checks; roll back on failure; track deploy status/history; manage configuration per environment.

Non-functional: safety/reliability (a bad deploy shouldn’t cause an outage), speed (deploy thousands of hosts quickly), availability (zero-downtime deploys), auditability, and global reach.

Step 2 — The pipeline stages

commit → CI build → tests → artifact (versioned) → artifact store
       → deploy orchestrator → distribute to fleet → health check → promote/rollback

• Build once, deploy many — produce an immutable, versioned artifact (container image or package) and store it in a registry/object store. Building once guarantees every host runs identical bits.

Step 3 — Artifact distribution at scale

Pushing a large artifact to thousands of hosts from one registry is a bottleneck (and a network hotspot). Distribute efficiently:

• CDN / regional mirrors of the registry so hosts pull from nearby.
• Peer-to-peer distribution (BitTorrent-style, e.g. Dragonfly) — hosts share chunks with each other, so the registry serves the artifact a few times, not N times.
• Content-addressed layers (container image layers) so only changed layers transfer (reuse the dedup/chunking idea).

Step 4 — Rollout strategies (the safety core)

Never flip all servers at once. Progressive strategies:

• Rolling — update servers in batches (e.g. 10% at a time), health-checking each batch before the next. Zero downtime, slow blast radius.
• Blue-green — stand up a full new environment (green) alongside the old (blue); shift traffic over once green is healthy; instant rollback by switching back. Costs double capacity briefly.
• Canary — route a small % of traffic to the new version, watch metrics (errors, latency), and auto-roll-forward or roll-back based on them. The safest; catches bad deploys with minimal user impact.

Recommend canary + rolling and explain the blast-radius control.

Step 5 — Health checks and automatic rollback

Each updated host (or canary cohort) is health-checked (readiness + key metrics: error rate, latency, crashes). If a batch/canary fails its checks, the orchestrator halts and rolls back automatically to the last-known-good version. Fast rollback (re-point to the previous artifact) is what makes aggressive deploys safe.

Step 6 — Orchestration and config

• A deploy orchestrator (state machine per deployment) drives batches, tracks per-host status, and coordinates rollback. Make it idempotent and resumable (a deploy can be retried after the orchestrator restarts).
• Configuration is versioned and environment-scoped (dev/staging/prod), deployed separately from code; secrets via a secrets manager.

Trade-offs to raise

• Canary (safest, slower, needs good metrics) vs blue-green (instant rollback, 2× cost) vs rolling (cheap, larger blast radius).
• Speed vs safety — bigger batches deploy faster but risk more; canary tunes this.
• Immutable artifacts (reproducible, easy rollback) vs in-place updates (faster, riskier).

The interview cue

“Build an immutable versioned artifact once, distribute via regional mirrors / P2P so the registry isn’t a bottleneck, then a deploy orchestrator does a canary → rolling rollout with health checks and automatic rollback to last-known-good. Config is versioned and environment-scoped.” Progressive rollout + health-gated auto-rollback is the safety story this problem tests.