Building Instagram

The upload path (direct to blob, async processing)

Clients upload large media directly to the object store via a pre-signed URL, so app servers never proxy bytes; processing happens async:

def request_upload(user_id):
    media_id = uuid()
    key = f"uploads/{user_id}/{media_id}"
    return {"upload_url": blob_store.presign_put(key, ttl="10m"), "media_id": media_id}

def finalize_post(user_id, media_id, caption):
    key = f"uploads/{user_id}/{media_id}"
    post_id = snowflake()
    posts.put(post_id, {"user": user_id, "caption": caption,
                        "media_key": key, "ts": now(), "status": "processing"})
    media_queue.publish({"post_id": post_id, "key": key})   # async variant generation
    return post_id

Image-variant generation (async worker)

Generate the sizes the feed needs once, so reads never resize:

VARIANTS = {"thumb": 150, "feed": 1080, "full": 2048}
def media_worker(msg):
    original = blob_store.get(msg["key"])
    urls = {}
    for name, width in VARIANTS.items():
        img = resize(original, width)                # + transcode video to HLS segments
        out_key = f"media/{msg['post_id']}/{name}.webp"
        blob_store.put(out_key, img)
        urls[name] = cdn_url(out_key)
    posts.update(msg["post_id"], media_urls=urls, status="ready")
    fanout_queue.publish({"post_id": msg["post_id"], "user_id": author_of(msg)})  # then fan out

Fan-out happens after processing, so a post appears in feeds only once its images are ready.

Feed assembly (reuse the fan-out)

Identical to Twitter — push post ids into followers’ feeds (hybrid for celebrities), read the precomputed list, hydrate, and attach CDN image URLs:

def home_feed(user_id, k=20):
    ids = redis.lrange(f"feed:{user_id}", 0, k)      # precomputed (pushed) post ids
    ids += pull_celebrity_posts(user_id, k)          # merge big accounts at read time
    posts_ = hydrate(top_by_time(ids, k))            # batch-fetch metadata
    return [{**p, "image": p.media_urls["feed"]} for p in posts_]   # CDN url, right size

Serving images

The client requests the right variant for its context (thumb in a grid, feed size in the feed, full on tap) directly from the CDN — so the app/DB never touch image bytes, and users get appropriately-sized, edge-cached images.

Storage and counts

• Media in the object store (erasure-coded) + CDN — the dominant cost; dedup identical uploads by content hash.
• Posts/graph/feeds sharded as in Twitter.
• Likes/comments counts → async approximate counters (incrementing a DB row per like at scale is a hotspot); comment lists paginate from a store.

Failure handling and edge cases

• Upload fails mid-way → pre-signed PUT is idempotent by key; a sweeper cleans orphaned uploads with no finalized post.
• Processing fails → retry; the post stays processing (not yet in feeds) until variants exist or it’s dead-lettered.
• Celebrity post → pull-and-merge (no fan-out storm).
• Hot image → CDN absorbs it entirely.
• Feed cache loss → rebuild from followees’ recent posts.

The takeaway

Concrete signals: direct-to-blob pre-signed uploads, async variant generation (resize/transcode once), CDN-served right-sized images, fan-out reused from Twitter, and approximate counters. The media plane (blob + CDN + variants) layered on the feed pattern is exactly how every photo/video social app is built — TikTok and the news feed are variations.