AhmadRaza365 Logo

AhmadRaza365

Blog Post

Inventory Sync Without Chaos: Real-Time vs Batch, Race Conditions, and Data Integrity

April 3, 2026
Inventory Sync Without Chaos: Real-Time vs Batch, Race Conditions, and Data Integrity

If you’ve ever seen a customer charged twice, two orders created for one checkout, or the same webhook processed multiple times, your system isn’t “buggy”, it’s missing idempotency and a reliable event pipeline.

In this post I’ll show how I design order flows in MERN/Next.js so that:

  • Clicking “Pay” twice doesn’t create two orders
  • Stripe/PayPal webhook retries don’t duplicate fulfillment
  • Network timeouts don’t leave you with “paid but no order”
  • You can safely retry anything (client → API, API → payment provider, provider → webhook)

The real reason duplicates happen (it’s not just “users double-click”)

In ecommerce/fintech, duplicates are usually caused by retries across boundaries:

  1. Client retry: mobile data drops, user taps again
  2. API retry: load balancer times out, client retries
  3. Provider retry: Stripe/PayPal retries webhook delivery
  4. Worker retry: your queue re-processes a job

Retries are good. Your system must be designed to tolerate them.

The goal is simple:

For any “real-world action” (create order, capture payment, mark fulfilled), the system should produce at most one durable effect, even if the request is processed multiple times.


A production-safe mental model: separate Intent, Order, and Payment

I typically split the flow into three layers:

  • Checkout Intent (ephemeral): user session/cart state
  • Order (durable): your internal record of what should be fulfilled
  • Payment (durable): provider objects (PaymentIntent, Charge, Capture, etc.) + your internal payment ledger

Key rule:

  • An Order must have a stable, unique identifier before you do anything irreversible.
  • A Payment confirmation must be linked to exactly one Order.

This reduces the chance of “paid but no order” or “order exists but payment unknown”.


Step 1, Choose your idempotency keys (and where they live)

An idempotency key is a unique token representing one logical operation.

Common operations that need keys:

  • create_order_from_cart
  • create_payment_intent
  • confirm_payment
  • apply_webhook_event
  • mark_order_fulfilled

Practical key strategy I use

  • Client-generated key for “create order” (UUID v4)
  • Provider event ID for webhooks (evt_... in Stripe)
  • Job ID for queue workers (BullMQ jobId)

Store these keys in MongoDB with a unique index.

MongoDB schema sketch

// collection: idempotency_keys
{
  _id: "create_order:9b7b3d...",        // namespaced key
  scope: "create_order",
  key: "9b7b3d...",
  status: "completed",                  // started|completed|failed
  response: { orderId: "ord_123" },     // what you want to return on retry
  createdAt: ISODate("..."),
  expiresAt: ISODate("...")
}

Indexes:

db.idempotency_keys.createIndex({ _id: 1 }, { unique: true });
db.idempotency_keys.createIndex({ expiresAt: 1 }, { expireAfterSeconds: 0 });

TTL keeps the collection from growing forever.


Step 2, Make “Create Order” idempotent (Node/Express + Mongo)

When the user hits Place Order, your API should:

  1. Attempt to reserve the idempotency key
  2. If it already exists and is completed, return the same response
  3. If it exists and is in-progress, return 409 or a “processing” response
  4. If it doesn’t exist, create the order exactly once

Minimal middleware-ish implementation

import crypto from 'crypto';
import { IdempotencyKey } from './models/IdempotencyKey.js';

export async function withIdempotency({ scope, key, handler }) {
  const docId = `${scope}:${key}`;

  // 1) Try to create the key (atomic due to unique _id)
  try {
    await IdempotencyKey.create({
      _id: docId,
      scope,
      key,
      status: 'started',
      createdAt: new Date(),
      expiresAt: new Date(Date.now() + 1000 * 60 * 60 * 24), // 24h
    });
  } catch (e) {
    const existing = await IdempotencyKey.findById(docId).lean();
    if (!existing) throw e;

    if (existing.status === 'completed') {
      return existing.response; // return same result on retry
    }

    // in-progress / failed,  your policy decision
    throw Object.assign(new Error('Request already in progress'), { statusCode: 409 });
  }

  // 2) Execute handler
  try {
    const result = await handler();

    // 3) Persist response for future retries
    await IdempotencyKey.updateOne(
      { _id: docId },
      { $set: { status: 'completed', response: result } }
    );

    return result;
  } catch (err) {
    await IdempotencyKey.updateOne(
      { _id: docId },
      { $set: { status: 'failed', error: String(err) } }
    );
    throw err;
  }
}

In your route:

app.post('/api/orders', async (req, res) => {
  const idemKey = req.header('Idempotency-Key');
  if (!idemKey) return res.status(400).json({ error: 'Missing Idempotency-Key' });

  const out = await withIdempotency({
    scope: 'create_order',
    key: idemKey,
    handler: async () => {
      // IMPORTANT: derive price server-side, never trust client totals
      // - load cart
      // - compute totals (tax/shipping/discount)
      // - create order with status = 'pending_payment'
      // - create provider PaymentIntent linked to orderId
      return { orderId: 'ord_...', paymentIntentClientSecret: '...' };
    },
  });

  res.json(out);
});

Critical detail: don’t create two provider intents

If your handler creates a Stripe PaymentIntent, that creation must be part of the same idempotent block. Stripe supports idempotency too, use the same key when calling Stripe.


Step 3, Webhooks must be processed exactly once (even if delivered 20 times)

Stripe/PayPal webhooks are “at least once delivery”. Treat them like a queue, not like a normal HTTP request.

The webhook endpoint should do only three things

  1. Verify signature (security)
  2. Store the event ID (idempotency)
  3. Enqueue processing (reliability)

That’s it.

Stripe webhook verification (Express)

import express from 'express';
import Stripe from 'stripe';

const stripe = new Stripe(process.env.STRIPE_SECRET_KEY);

// Stripe needs RAW body for signature verification
app.post('/api/webhooks/stripe', express.raw({ type: 'application/json' }), async (req, res) => {
  const sig = req.headers['stripe-signature'];

  let event;
  try {
    event = stripe.webhooks.constructEvent(req.body, sig, process.env.STRIPE_WEBHOOK_SECRET);
  } catch (err) {
    return res.status(400).send(`Webhook Error: ${err.message}`);
  }

  // Now: enqueue event.id for async processing
  await webhookQueue.add(
    'stripe_event',
    { eventId: event.id },
    { jobId: event.id } // BullMQ de-duplicates by jobId
  );

  res.json({ received: true });
});

Worker processing: de-dupe by eventId + make your updates idempotent

In the worker you:

  • Fetch the event from Stripe API (optional but useful for replays)
  • Apply it to your order/payment records
  • Record that this event has been applied (unique index)

Mongo pattern:

// collection: processed_events
{ _id: "stripe:evt_123", processedAt: new Date() }

If insert fails due to duplicate key, you already processed it.


Step 4, Use an order state machine (so your own retries don’t break you)

A clean state machine prevents “random status flips”:

  • pending_payment
  • paid
  • failed
  • fulfilled
  • cancelled
  • refunded / partially_refunded

Key rule:

  • States should only move forward in well-defined transitions.
  • Each transition should be safe to apply multiple times.

Example: “mark as paid” should be an atomic update:

await Orders.updateOne(
  { _id: orderId, status: 'pending_payment' },
  { $set: { status: 'paid', paidAt: new Date() } }
);

If it runs again, it matches zero documents, and that’s fine.


Step 5, Handle the hardest bug: “payment succeeded, API timed out”

This is where teams lose money.

If the client didn’t receive a response, it will retry. Your API must return the same order/payment info.

This is why the idempotency key stores a response payload.

Also:

  • Prefer creating the Order first, then creating provider intent linked to the order.
  • If provider call succeeds but your response fails, the next retry will read the stored response.

If you already created a PaymentIntent but didn’t persist it, you’ll create a second one. That’s how duplicates start.


Step 6, Observability: how you catch duplicates before customers do

Minimum instrumentation I add on revenue paths:

  • A request ID (propagate from client → API → queue)
  • Structured logs with: orderId, idemKey, eventId, paymentIntentId
  • Metrics counters:
    • orders.created
    • orders.duplicate_create_attempts
    • webhooks.received
    • webhooks.duplicate_events
    • payment.confirmation_failures

If you use something like Sentry + OpenTelemetry + a log aggregator, you can follow one checkout across systems.


Quick “Never Duplicates” checklist (what I verify in an audit)

API layer

  • Every POST that creates money-impacting state requires an Idempotency-Key
  • Key is stored in DB with unique constraint and TTL
  • Response is stored and returned for retries
  • Order totals computed server-side (tax/shipping/discount)

Payment layer

  • Payment provider calls use provider idempotency (Stripe supports this)
  • Payment objects link to orderId in metadata
  • Webhooks don’t directly run heavy logic (enqueue instead)

Webhook layer

  • Signature verification uses raw body
  • Each provider event ID is processed once (processed_events unique index)
  • Worker jobs have stable jobId and safe retries

Data layer

  • Order state transitions are atomic and monotonic
  • Unique constraints exist where they matter (orderNumber, providerTransactionId)

Final note (lead-dev perspective)

The teams that win aren’t the ones who “never get retries”, they’re the ones who design for retries.

If you want, I can do a focused stabilization sprint:

  • map your current checkout/payment flow
  • add idempotency + webhook pipeline
  • patch the top duplicate/timeout failure modes
  • add monitoring so you can prove the fix

Not a rewrite, just production hardening where revenue depends on it.

You can find me on different platforms