Build a Payment API for 1 Million Transactions a Day

Build for a million payments and their failure paths#

A million-payment day is not, by itself, a reason to split your system into dozens of services. It averages about 11.6 business payments per second. What matters is the busiest window, the work each payment creates, and what happens when a provider completes a charge just before your connection drops.

Start by defining a transaction for your capacity model. Here it means one business payment instruction. An authorization, capture, refund, status lookup and webhook are separate operations around that instruction. Counting them all as payments hides the load and makes success rates hard to interpret.

Turn daily volume into a peak capacity model#

These are worked assumptions, not a benchmark or a guarantee. The concurrency calculation uses mean time in flight; use measured latency distributions and a bounded concurrency limit for operations. A spike lasting five minutes is a different queueing problem from a spike lasting an hour. Measure the largest tenant, synchronized renewals and batch imports as well as aggregate traffic.

Model authorization and capture separately if your flow calls the provider twice. Add status recovery, refunds, polling and webhook fetches to the provider budget. Check its account, endpoint and burst limits; additional API replicas cannot make an upstream limit disappear. Keep a separate budget for recovery traffic so a callback outage cannot consume every available provider request.

For example, if 116 jobs arrive each second while workers can complete only 100, backlog grows by about 16 jobs per second. Over ten minutes that adds about 9,600 jobs. When arrivals fall to 60 per second and service capacity stays at 100, the backlog takes roughly four minutes to drain. Those estimates assume constant rates; validate queue age and recovery time with your actual workload.

Define the API contract before the external call#

Return a durable payment ID when you accept a command. Say explicitly whether the response means accepted for processing, authorized, captured or settled. An HTTP success from your API must not make a client believe bank cash has arrived.

Scope the idempotency record to authenticated tenant, operation and client key. Store a canonical payload digest alongside it. Enforce uniqueness in the database, then create the command and any local reservation in the same transaction. A check followed by an unprotected insert lets concurrent requests both proceed.

Keep your internal record long enough to cover client retries, recovery and financial investigation. Provider retention is a separate limit. Stripe’s idempotency documentation says keys can be removed after they are at least 24 hours old; a reused key after pruning can initiate a new request. It also documents parameter checks and saved responses, including failures. Do not assume another provider uses the same rules.

Make crashes recoverable at the provider boundary#

Before sending an external instruction, persist an attempt ID, provider account, operation, amount, currency and the provider idempotency key. Use a conditional state update or transactional claim so two workers do not both start a new attempt. A worker lease helps schedule work, but expiry of the lease does not prove the previous external call failed.

Do not switch providers, change rails or issue a fresh key while the first attempt is unknown. A new destination is not a cancellation of the original request. Escalate unresolved attempts with their provider references and timestamps; create a replacement only after evidence establishes the original did not execute, or after a separately authorized reversal and new payment.

A local database transaction cannot atomically commit a remote provider charge. Keep that uncertainty visible instead of claiming end-to-end exactly-once delivery. Your practical goal is one business obligation, durable attempts and controlled recovery.

Keep payment state and accounting truth distinct#

Races show up when updates from different paths arrive close together or out of order. If multiple paths can write state without a guard, lifecycle integrity breaks.

For a card flow, requested, processing, authorized, captured and failed can describe execution; settlement, refunds and disputes need their own records. A captured payment can later be refunded or disputed. Do not treat a single terminal label as permission to discard every later event.

Validate a proposed transition against stored state in the same transaction that records it. Persist the triggering attempt or event ID and an append-only transition record. Use a unique business-effect identity for journal postings so a second event about the same capture cannot post the same financial effect twice.

A balanced double-entry ledger records your accounting entries; provider statements and bank records establish external settlement evidence. Reconciliation connects those records and explains timing differences, fees and mismatches. It should not silently overwrite the ledger to make a dashboard balance match.

Choose a database by the atomicity, query and recovery requirements of this write path. At this volume, measure a straightforward transactional design before introducing manual sharding. If you add derived stores, designate their data as projections and keep authoritative updates out of uncoordinated dual writes.

Receive webhooks durably, then process them once locally#

Verify the signature against the raw request body before trusting tenant routing or payload fields. Resolve the provider account to your own tenant mapping, validate the event, durably capture it and then acknowledge delivery. If capture fails, return a failure so the provider can retry. Keep expensive business processing out of the receiver.

Stripe’s webhook guidance documents duplicate deliveries, unordered events and asynchronous processing. Use the provider account and event ID to identify delivery duplicates. Distinct events can concern the same payment, so payment ID alone is not an event deduplication key.

In the worker transaction, check or insert the event-processing marker together with the guarded state change, journal effect and outgoing outbox record. If the worker crashes before commit, all those local changes roll back and the event can run again. If it crashes after commit, the marker prevents local effects from running twice. A marker written before the business transaction would instead risk losing an event.

Use the current provider resource where needed to resolve late or incomplete callbacks. Do not sort solely by receipt time or reject a legitimate refund because capture already completed. Separate delivery deduplication from the business-effect guard for two distinct events describing the same effect.

An outbox publisher may deliver a committed message more than once. Its downstream consumer needs its own atomic processing guard. The AWS transactional outbox pattern explains why the state update and outgoing event belong in one local transaction; it does not make the remote payment part of that transaction.

Apply bounded backoff with jitter to transient local processing failures. Send exhausted events to a visible dead-letter queue with event ID, payment ID, provider account, attempt count, last error and a named owner. A replay goes through the same guards as the first processing attempt.

Instrument operations for detection, triage and proof#

Track business acceptance, provider attempts and financially confirmed outcomes separately. Publish oldest unknown-attempt age, oldest unprocessed-event age, reconciliation exception count and amount, and queue drain time. Define the denominator of a duplicate-attempt rate; a safely replayed client request is different from an unintended second charge.

Carry tenant ID, payment ID, attempt ID, provider account and reference through requests, events and reconciliation rows. Store structured evidence with access controls and retention rules. Avoid putting raw payment credentials or unnecessary personal data into logs or idempotency keys.

Your operator view should answer which obligation is waiting, whether an external attempt might have executed, and what evidence permits the next action. API latency alone cannot answer those questions.

Related: How to Build a Deterministic Ledger for a Payment Platform.

Prove recovery before increasing traffic#

• Send simultaneous identical requests and confirm one durable command; send the same key with a changed payload and confirm rejection.
• Kill a worker before the provider call, after the call and after local commit. Confirm recovery preserves the attempt identity and exposes unknown outcomes.
• Deliver duplicate and out-of-order webhooks, including a later refund or dispute. Confirm each legitimate financial effect posts once.
• Stop outbox publishing, resume it and redeliver messages. Confirm committed events are recovered and downstream effects are guarded.
• Run the measured peak workload with realistic database writes, provider latency and rate limits. Include recovery traffic and the largest tenant burst.
• Compare command records, provider references, ledger entries, settlement reports and bank movements for the same sample. Explain every difference rather than relying on a matching total.

Assign thresholds to your own service objectives: maximum processing delay, unknown-attempt age, queue recovery time and unresolved reconciliation amount. Add orchestration when multiple durable steps need coordinated recovery; daily volume alone does not require it.