Skip to content
System design course
Ch.4 · Designing real systems·how to build it ·7 min read

Building a payment system

Implement idempotent charge handling, atomic double-entry ledger writes, the payment state machine over async webhooks, and reconciliation.


Idempotent charge handling

The first thing every money operation does: dedupe on the idempotency key and return the stored result on retries:

def charge(idempotency_key, amount, source, dest):
    existing = idempotency.get(idempotency_key)
    if existing:
        return existing.result                       # retry → SAME result, no second charge
    with txn():
        # reserve the key atomically so concurrent retries can't both proceed
        idempotency.insert(idempotency_key, status="processing")
    result = process_charge(amount, source, dest)    # do the work exactly once
    idempotency.complete(idempotency_key, result)
    return result

Reserving the key inside a transaction (unique constraint on the key) means two concurrent identical requests can’t both process.

Atomic double-entry ledger write

Money movement appends balanced debit/credit entries in one transaction — the books never go half-updated:

def post_entry(txn_id, debit_acct, credit_acct, amount):
    with db.txn():
        ledger.append(txn_id, account=debit_acct,  direction="debit",  amount=amount)
        ledger.append(txn_id, account=credit_acct, direction="credit", amount=amount)
        # invariant enforced/asserted: sum(debits) == sum(credits) for txn_id

Balances are derived from entries (or kept as a materialized total updated in the same transaction). The ledger is append-only — a refund is a new reversing entry, never an edit. This is what makes it auditable.

The payment state machine over async events

A payment advances through states as async bank/PSP events arrive — idempotently:

STATES = ["created", "authorized", "captured", "settled"]   # or failed/refunded

def on_psp_event(event):                              # webhook from the provider
    with txn():
        p = payments.get(event.payment_id)
        if p.applied(event.id): return                # idempotent: ignore duplicates
        if event.type == "authorized":
            p.state = "authorized"
        elif event.type == "captured":
            p.state = "captured"
            post_entry(p.id, "customer_funds", "merchant_payable", p.amount)  # ledger
        elif event.type == "failed":
            p.state = "failed"
        p.record_applied(event.id); payments.put(p)

Webhooks can duplicate or arrive out of order → dedup by event id and guard transitions.

PSP adapters with retries

Each provider behind an adapter (Strategy), wrapped with idempotency + retries; never trust a single synchronous response — wait for the webhook/settlement as truth:

def call_psp(adapter, request):
    return retry(lambda: adapter.charge(request, idempotency_key=request.key),
                 on=TransientError, backoff="exponential")

Reconciliation

A scheduled job compares the ledger to the provider’s settlement file and flags discrepancies:

def reconcile(date):
    ours   = ledger.entries_for(date)
    theirs = parse_settlement(psp.settlement_file(date))
    for diff in symmetric_difference(ours, theirs):
        alert("reconciliation mismatch", diff)        # missing/dup/amount mismatch
        open_investigation(diff)                        # repair workflow

This catches anything that slipped (a webhook never arrived, a duplicate) — the safety net.

Scale and failure handling

  • Crash mid-charge → idempotency key + transactional ledger mean a retry is safe and produces no duplicate.
  • Duplicate/out-of-order webhooks → event-id dedup + state-machine guards.
  • PSP down → retries + the payment waits in authorized/created; reconcile later.
  • Partial failure → the ledger is transactional; either both entries commit or neither.
  • CP stance → on uncertainty, don’t double-act; mark pending and reconcile.

The takeaway

Concrete signals: idempotency keys (retry-safe, no double-charge), an append-only double-entry ledger with atomic balanced entries (auditable truth), a payment state machine over async webhooks (idempotent), and reconciliation against settlement. The ledger + idempotency pattern is the foundation for the digital wallet and UPI next.