Designing a file sync service (Dropbox)

The problem

Design Dropbox/Google Drive: users store files and have them sync across all their devices and collaborators — upload once, see it everywhere, including after being offline. The crux is efficient sync of large, changing files and keeping distributed copies consistent.

Step 1 — Requirements

Functional: upload/download files; sync changes to all the user’s devices; share files/folders; handle offline edits and reconcile; versioning.

Non-functional: durability (never lose a file), efficient transfer (don’t re-upload a whole file for a one-line change), scalability (huge total storage), and reasonable consistency (a change should propagate quickly).

Step 2 — The key idea: chunking

Files are split into fixed (or content-defined) chunks (e.g. 4 MB). Chunking unlocks everything:

• Efficient sync — change one chunk, upload only that chunk, not the whole file.
• Deduplication — identical chunks (same hash) are stored once across all users (the same installer uploaded by millions = stored once). Big storage + bandwidth win (Chapter 3 dedup).
• Parallel transfer and resumable uploads — chunks transfer independently.
• Integrity — each chunk has a checksum.

Each chunk is named by its content hash (content-addressed), so dedup and verification are automatic.

Step 3 — Split the system: metadata vs blobs

• Block/blob storage (S3-style) holds the chunks — the bulk of the data.
• Metadata service (a database) holds the file tree: files, versions, and the ordered list of chunk hashes that reconstruct each file, plus permissions and device sync state. Small but heavily queried.

file → [chunk hashes] in metadata DB;  chunks → blob store (deduped, content-addressed)

Step 4 — The sync mechanism

A client agent watches the local folder and talks to the server:

• Upload: diff locally, find changed chunks (by hash), upload only new chunks to blob storage, then commit the new file version (chunk list) to metadata.
• Notify: the server pushes a change notification to the user’s other devices (long-poll/WebSocket from a notification service).
• Download: a notified device fetches the new metadata, sees which chunk hashes it lacks, and downloads only those from blob storage.

So only deltas move, both up and down.

Step 5 — Consistency, conflicts, and offline

• Versioning — each change creates a new version (list of chunks); old versions are retained (cheap, since chunks are shared) for history and undo.
• Conflicts — two devices edit the same file offline. Detect via version vectors; resolve by keeping both as conflicted copies (“file (Bob’s conflicted copy)”) rather than silently losing one — the pragmatic, safe choice.
• Offline — the client queues changes locally and syncs on reconnect.

Step 6 — Scale

• Blob store scales horizontally (S3); dedup cuts its size dramatically.
• Metadata DB shards by user/account; it’s the hot path for sync.
• A CDN can serve frequently downloaded public files.

Trade-offs to raise

• Fixed vs content-defined chunking — content-defined survives insertions (boundaries shift gracefully) but is more complex; fixed is simpler.
• Dedup scope — global dedup saves the most but raises privacy/security considerations (encrypt per-user, dedup within trust boundaries).
• Conflict policy — conflicted copies (never lose data) vs last-write-wins.

The interview cue

“Chunk files (content-hash addressed) so we upload/download only deltas and dedup identical chunks globally; metadata service stores the file→chunk-list and versions, blob store holds chunks; a client agent diffs locally and a notification service pushes changes to other devices; conflicts become conflicted copies.” Chunking + metadata/blob split + delta sync is the heart of the design.