Design a Payment System
DifficultyHard | HelloInterview: problem breakdown
Problem Statement
Design a payment processing system that handles financial transactions between payers and payees. Transactions must be reliable, auditable, and exactly-once โ no double charges.
๐Real-world: Payments run on a 700-year-old idea: double-entry bookkeeping (every transaction is two balanced entries โ debit one account, credit another โ so the books always reconcile and you can prove where money went). Modern systems make the ledger append-only and immutable for exactly the audit reasons a security person cares about. The defining engineering challenge is idempotency: networks retry, users double-click, and "charge $50" must happen exactly once even if the request arrives three times โ solved with a client-supplied idempotency key plus a dedup table, the same pattern Stripe popularized. Multi-step flows (charge card โ update ledger โ notify merchant) can't use a single ACID transaction across services, so you use the Saga pattern with compensating actions to roll back. Security framing: payments are where strong consistency, immutability, idempotency, and auditability all become non-negotiable at once โ "eventually consistent" loses lawsuits, and PCI-DSS scope shapes the whole architecture.
Requirements
Functional
- Initiate a payment (payer โ payee, amount, currency)
- Payment status tracking (pending, processing, completed, failed)
- Refund capability
- Transaction history
- Support for multiple payment methods (card, bank transfer, wallet)
Non-Functional
- Exactly-once execution โ no double charges under any failure scenario
- Consistency: strong โ money must balance (debit = credit)
- Durability: no lost transactions even under node failure
- Latency: < 2s for card payment end-to-end
- 1M transactions/day โ ~12 TPS average; 100 TPS peak
- Financial audit trail: immutable, append-only
Core Design
The Fundamental Problem: Distributed Failure
Money involves at minimum two systems: your ledger and the external payment processor (Stripe, bank). What happens if:
- You debit the user but the external payment API call fails?
- The external payment succeeds but your database write fails?
- The service crashes between the two operations?
Solutionidempotency + distributed transaction patterns.
Idempotency Keys
Every payment request must carry an idempotency key โ a unique ID generated by the client.
POST /payments
{
"idempotency_key": "client-uuid-1234",
"amount": 9999,
"currency": "USD",
"payer_id": "user_a",
"payee_id": "merchant_b"
}Server-side:
- Check idempotency store: has
client-uuid-1234been processed? - If yes: return stored result (don't process again)
- If no: process and store result atomically
This makes retries safe: client can retry a failed request N times without double-charging.
Double-Entry Ledger
Financial systems use double-entry bookkeeping: every transaction has a matching debit and credit. Ledger always balances.
ledger_entries(
id BIGSERIAL PRIMARY KEY,
transaction_id UUID NOT NULL,
account_id UUID NOT NULL,
amount BIGINT NOT NULL, -- positive = credit, negative = debit, in cents
currency CHAR(3) NOT NULL,
entry_type ENUM('debit','credit'),
created_at TIMESTAMP NOT NULL -- immutable, append-only
)
-- Payment: $50 from Alice to Bob
INSERT INTO ledger_entries VALUES
(txn_id, alice_account, -5000, 'USD', 'debit'), -- Alice loses $50
(txn_id, bob_account, +5000, 'USD', 'credit'); -- Bob gains $50Balance of an accountSELECT SUM(amount) FROM ledger_entries WHERE account_id = ?
This is inherently audit-proof: no rows are updated or deleted, only inserted.
Transaction State Machine
PENDING โ PROCESSING โ COMPLETED
โ
FAILED โ (refund creates a new COMPLETED reversal transaction)Persistence guarantees:
- Create payment record in DB (PENDING) before calling external processor
- Call external processor (Stripe/bank)
- Update record to PROCESSING (storing external transaction ID)
- On callback/webhook: update to COMPLETED or FAILED
Saga Pattern for Distributed Transactions
When a payment spans multiple services (e.g., deduct from wallet, charge card, credit merchant), use the Saga pattern instead of distributed 2PC.
Saga: ChargeCard โ UpdateWallet โ CreditMerchant โ NotifyUser
If CreditMerchant fails:
Compensating transactions:
โ RevertWallet (compensate UpdateWallet)
โ RefundCard (compensate ChargeCard)Each step is atomic in its own DB. Compensating transactions undo side effects. This achieves eventual consistency without distributed locking.
Architecture
Client โ API Gateway โ Payment Service
|
โโโโโโโโโโโโโโโโโโโโโโ
โ โ
Idempotency Store Payment DB (Postgres)
(Redis) [payments, ledger_entries]
|
External Processor
(Stripe / ACH)
|
Webhook Handler โ Stripe sends async callbacks
|
Notification Service
Reconciliation Service (daily batch: check DB vs Stripe)Webhook Handling
External payment processors send webhooks (HTTP callbacks) on payment completion. These must be:
HMAC signature from Stripe must match
webhooks may be delivered more than once
return 200 immediately; process async
@app.post("/webhooks/stripe")
def stripe_webhook(request):
# Verify signature
sig = request.headers["Stripe-Signature"]
stripe.Webhook.construct_event(request.body, sig, STRIPE_WEBHOOK_SECRET)
# Store event ID for idempotency
if not idempotency_store.exists(event.id):
idempotency_store.set(event.id, True, ttl=24*3600)
message_queue.publish("payment_events", event)
return {"status": "ok"} # ACK immediatelyKey Design Decisions
| Decision | Choice | Reason |
|---|---|---|
| Idempotency | Client-generated UUID + server-side store | Safe retries |
| Ledger | Append-only double-entry | Audit trail; balance correctness |
| Distributed txn | Saga pattern | Avoids 2PC; compensating transactions |
| Consistency | Strong (synchronous DB write) | Money must not be lost |
| Refunds | New credit transaction | Don't modify history; create reversal |
| Reconciliation | Daily batch vs external processor | Detect discrepancies; catch bugs |
Security Considerations
| Threat | Mitigation |
|---|---|
| Double charge | Idempotency keys + dedup table; distributed lock on payment initiation |
| Payment fraud | 3DS2 authentication; ML fraud scoring; velocity limits |
| Man-in-the-middle | TLS everywhere; certificate pinning for mobile SDK |
| Webhook spoofing | HMAC signature verification on every incoming webhook |
| Unauthorized payment | Explicit payer authorization (re-auth for high-value); signed payment intents |
| SQL injection in amount | Always use parameterized queries; store amounts as integers (cents), never floats |
| Insider threat / ledger manipulation | Append-only ledger; DB user has INSERT-only on ledger_entries, no UPDATE/DELETE |
| PCI-DSS compliance | Never store raw card data; use tokenization (Stripe token); all card data encrypted at rest |
| SSRF / account enumeration | Validate account IDs exist before any operation; no timing differences between valid/invalid |
๐ฐReal incident โ Bangladesh Bank SWIFT heist (2016). Attackers installed malware on Bangladesh Bank's systems connected to the SWIFT interbank network and sent 35 fraudulent transfer requests to the Federal Reserve Bank of New York for nearly $1 billion. Most were blocked โ one partly because a recipient bank name contained the word "Jupiter," which matched a sanctions-list entry โ but $81 million reached accounts in the Philippines and largely disappeared through casinos. The malware also altered the printed confirmation reports to hide the transfers.
๐ฏOn the job. Check that every payment API accepts an idempotency key and that the key is stored atomically with the payment record โ retries after a timeout are the most common cause of double charges. Then check reconciliation: an independent daily comparison of your ledger against the bank's statement is what catches both bugs and fraud that every real-time control missed.
Interview Tips
explain why it's needed, how to implement it (idempotency key + dedup store), and what failures it prevents.
is a differentiator โ most candidates don't know this accounting concept. It immediately signals production financial systems knowledge.
mention distributed two-phase commit and explain why it's avoided (blocking, single coordinator SPOF). Saga with compensating transactions is preferred.
floating point precision errors in financial calculations are a classic bug (0.1 + 0.2 โ 0.3). Store as integer cents/satoshis.
mention a daily batch job that compares your internal ledger against the external processor's records. Discrepancies are bugs or fraud.