Building a news feed (Facebook)

The read pipeline: generate → rank → hydrate

def news_feed(user_id, k=20):
    candidates = gather_candidates(user_id, n=500)   # fan-out feed + groups/pages
    candidates = cheap_filter(candidates)            # dedup, drop seen/blocked → ~300
    features   = feature_store.batch(user_id, candidates)   # author affinity, recency, popularity...
    scored     = ranking_model.predict(features)     # P(engage) per candidate
    ranked     = sort_desc(scored)
    ranked     = interleave_ads(ranked, ad_auction(user_id))
    return hydrate(ranked[:k])                        # post content + CDN media

The structure is candidate generation (recall) → ML ranking (precision) → hydrate, the same two-stage shape as the search engine.

Candidate generation

Reuse fan-out feeds for friends, and pull recent posts from followed pages/groups:

def gather_candidates(user_id, n):
    feed_ids   = redis.lrange(f"feed:{user_id}", 0, n)        # pushed friend posts
    page_ids   = recent_posts(pages_followed(user_id), n)      # pulled page/group posts
    return dedupe(feed_ids + page_ids)

The ranking model

Score predicted engagement from features; in practice a gradient-boosted tree or neural net, but conceptually:

def score(features):
    # learned weights over: author affinity, post weight (type/eng), recency decay,
    # user-topic match, post popularity, time-of-day, ...
    return model.predict(features)          # P(like|comment|share|dwell)

A feature store serves these features at low latency (precomputed affinities, rolling engagement counts, user embeddings). Keeping feature fetch fast is what makes read-time ranking viable.

Incremental re-ranking and freshness

• Rolling engagement counts (likes/comments in the last hour) update a post’s features in near-real-time (stream processing), so a post gaining traction climbs.
• Negative feedback (hide, “see less”) immediately down-ranks similar content and trains the model.
• Cache a user’s ranked feed briefly (seconds–minutes) to absorb rapid refreshes, but re-rank on meaningful new activity.

Hydration and serving

Ranked post ids are hydrated into full posts (batched cache/DB reads), with media URLs pointing at the CDN at the right size (Instagram pattern). Pagination continues the ranking from where it left off (cursor).

Storage and scale

• Candidate feeds in Redis (fan-out), posts sharded, media on CDN.
• Feature store (Redis/specialized) for low-latency features.
• Ranking service is a stateless, horizontally-scaled fleet behind the read path; the heavy model runs only on the filtered candidate set.

Failure handling

• Ranking service down/slow → fall back to chronological order (degrade gracefully — a feed in time order beats no feed).
• Feature store miss → score with defaults; don’t block the feed.
• Celebrity/page with huge reach → pulled at read (no fan-out storm).
• Stale counts → acceptable; eventual consistency.

The takeaway

Concrete signals: two-stage candidate-generation-then-ML-rank, a low-latency feature store, stream-updated engagement features for freshness, graceful fallback to chronological, and CDN hydration. Fan-out gives you what could appear; the ranking pipeline decides what does — the reusable recipe for any ranked feed (TikTok’s “For You” is the same shape with stronger ML).