Idempotency design

Sarvex handles financial mutations (order creation, payment capture, refunds) across merchants. Without exactly-once semantics, retry storms or network delays could cause:

Generated from docs/IDEMPOTENCY-DESIGN.md in the Sarvex repository — edit the source there, not this page.

Version: 1.0
Date: 2026-09-16
Status: Implemented (Phases 1-8 complete; Phase 9 foundation ready; Phase 10 tests in progress)

1. Problem Statement

Sarvex handles financial mutations (order creation, payment capture, refunds) across merchants. Without exactly-once semantics, retry storms or network delays could cause:

• Duplicate charges: Same checkout retried twice → two charges to payment processor
• Silent divergence: Replay produces different order IDs → auditor cannot reconcile
• Lost mutations: Async handoff failure → order exists at merchant but not in Sarvex
• Race conditions: Concurrent updates without guards → invalid state transitions

Goal: Achieve exactly-once semantics end-to-end, so replaying the same request with the same idempotency key always produces the same outcome.

2. Idempotency Model

2.1 Idempotency Key Properties

Scoping:

• Scoped to merchant_id to prevent cross-tenant collisions
• Format: {merchant_id}:{unique_fingerprint} (stored internally)
• Client provides unscoped key via Idempotency-Key HTTP header

Lifetime:

• Active window: 24h after order placed (terminal state reached)
• After 24h: Key expires and may be reused → new order (acceptable, auditable)
• Implementation: MongoDB TTL index on placed_at + 90 days (audit log retention)

Source:

• Client-provided (agent/LLM) via HTTP header
• Hash-based to avoid UUID collisions across agents
• Example: sha256(merchant_xyz:session_abc:timestamp)[0:32]

2.2 Order ID Derivation

Current: OrderID from merchant store (status quo)
Future: Deterministic from idempotency key for sortability + observability

Formula (planned):

order_id = {timestamp_8chars} + {sha256_first_64bits_hex}
  where input = hash(merchant_id + idempotency_key + counter)

Properties:

• Same (merchant_id, key, counter) → same order_id (deterministic)
• Sortable by ID (timestamp prefix)
• Collision probability < 1e-9 (SHA256 first 64 bits)
• Counter allows safe key reuse after 24h TTL

3. Guarantees & Invariants

3.1 Core Guarantees

3.2 Invariants (Must Hold)

1. One Order per (merchant_id, idempotency_key)

   • MongoDB unique index on (merchant_id, idempotency_key)
   • Exception: after 24h TTL, key may create new order (counter increments)

2. Every Mutation Logged

   • Order.MutationAttempts array appended atomically
   • Sync plane audit log (sync_order_mutations) for forensics

3. State Machine Forward-Only

   • No invalid transitions (e.g., paid → creating)
   • Enforced by CanTransition() validator before persist

4. Handler Deferred to

   • Sarvex passes idempotency key to payment handler
   • Trusts handler’s own exactly-once (doesn’t double-check)
   • Reduces latency + complexity

5. No Silent Retries in Money Path

   • Connector failure → surfaces error immediately
   • Retry is agent’s responsibility (with same idempotency key)
   • No background retry loops on mutations

4. Architecture: Three Planes

4.1 Serving Plane (Order Initiation)

Entry Point: POST /checkout/{sessionId}/complete

Flow:

1. Extract idempotency key from Idempotency-Key header
2. Compute body hash (SHA256 of payment payload)
3. Check session store: GetByIdempotencyKey(storeID, key)
   • If found + same body hash → return cached result (no connector call)
   • If found + different body hash → return 409 Conflict
4. Call connector: CheckoutComplete(ctx, storeID, checkoutID, payment, idempotencyKey)
   • Pass idempotency key to handler (Stripe, etc.)
5. Receive handler response (order ID, status, charge ID)
6. Emit serving.checkout.order_intent event to sync plane (async)
7. Persist session with CompleteIdempotencyKey + CompleteBodyHash
   • Record audit attempt: timestamp, handler response, status
8. Return order to agent (idempotent)

Storage: Redis session store (TTL-bounded to session lifetime)
Audit: CheckoutSession.CompleteAttempts array

4.2 Sync Plane (Order Recording)

Consumer: serving.checkout.order_intent event

Flow:

1. Parse OrderIntentPayload (order_id, idempotency_key, etc.)
2. Check: GetByCheckoutSession(checkoutSessionID)
   • If exists → no-op (order already recorded from prior event delivery)
3. Derive deterministic order ID (once implemented):

   orderID = DeriveOrderID(merchantID, idempotencyKey, counter)

4. Create Order in MongoDB with idempotency key field
5. Start MongoDB transaction:
   • Insert Order document
   • Append to Order.MutationAttempts (handler response, status)
   • Append to sync_order_mutations audit log (forensics)
6. Emit sync.order.created event to downstream consumers

Storage: MongoDB sync_orders (unique index on checkout_session_id, merchant_id:idempotency_key)
Audit: Order.MutationAttempts + sync_order_mutations collection

4.3 Control Plane (Operator API)

Query Endpoints (future):

• GET /merchants/{id}/orders?idempotency_key={key} — find orders by key
• GET /merchants/{id}/mutations?order_id={id} — audit trail for order

Queries:

• Detect double-charges: sync_order_mutations.find({idempotency_key: K}).count() should be 1
• Replay diagnostics: sync_orders.find({idempotency_key: K})[].order_id should be same ID

5. Handler Key Injection

5.1 Stripe Example

Serving Plane (Sarvex):

checkoutResp, err := connector.CheckoutComplete(ctx, storeID, checkoutID, 
  &base.CompleteCheckoutRequest{
    Payment: payment,
    IdempotencyKey: idempotencyKey,  // ← scoped key passed to connector
  })

WooCommerce Plugin (merchant’s store):

// Extract from Sarvex request
$idempotency_key = $sarvex_req->getHeader('Idempotency-Key');

// Pass to Stripe API
$headers = ['Idempotency-Key' => $idempotency_key];
$charge = $stripe->charges->create($charge_data, $headers);

Stripe API:

• Sees Idempotency-Key: merchant_acme:key-xyz-123
• Charges once per key (Stripe’s own exactly-once)
• Returns charge_id in response

Benefit: Stripe is source of truth for charge uniqueness; Sarvex doesn’t need to verify.

5.2 Supported Handlers

6. Audit & Observability

6.1 Mutation Audit Log (sync_order_mutations)

Schema:

type AuditLog struct {
  MutationID      string                 // uuid
  MerchantID      string
  OrderID         string
  IdempotencyKey  string
  AttemptedAt     time.Time
  CompletedAt     *time.Time
  HandlerResponse map[string]interface{} // e.g., {stripe_charge_id: "ch_..."}
  LocalStatus     string                 // "capturing", "paid", "refunded"
  ErrorMessage    string
  RetryCount      int
  MutationType    string                 // "create", "capture", "refund", "void"
}

Indexes:

• (merchant_id, idempotency_key) — detect double-charges
• (order_id) — audit trail for one order
• (attempted_at) — TTL index (90-day retention)

Queries:

1. Detect double-charge:

   db.sync_order_mutations.find({
     merchant_id: "M1",
     idempotency_key: "K1"
   })
   // Should return exactly 1 document

2. Replay diagnostics:

   db.sync_order_mutations.find({
     idempotency_key: "K1"
   }, { order_id: 1, handler_response: 1 })
   // All documents should have same order_id

3. Order audit trail:

   db.sync_order_mutations.find({
     order_id: "O1"
   }).sort({ attempted_at: 1 })
   // Shows create → maybe refund → terminal

6.2 State Machine Audit

CanTransition Validator (sync plane):

// Before any state change, check:
ok, violations := CanTransition(currentStatus, newStatus)
if !ok {
  log.Error("invalid transition", "violations", violations)
  return ErrInvalidTransition
}

Audit trail: Every transition logged with:

• From state, to state
• Timestamp
• Preconditions checked
• Outcome (success or error code)

7. End-to-End Example: Order Creation

Timeline:

Time    Component        Action                          Storage
────────────────────────────────────────────────────────────────────────
T0      Agent            Agent generates idempotency key: key-A-T0

T1      Serving Plane    POST /checkout/{id}/complete
                         Header: Idempotency-Key: key-A-T0
                         Body: {payment: ...}

T1+10ms Serving Plane    Compute body_hash = sha256({payment: ...})

T1+20ms Serving Plane    Call redis.get(idempotency:S1:key-A-T0)
                         → MISS (first attempt)

T1+30ms Serving Plane    Call connector.CheckoutComplete(...)
                         → Pass idempotencyKey = "key-A-T0"

T1+100ms Connector       Call Stripe API with header Idempotency-Key
        (WooCommerce)    → Stripe charge created: ch_XYZ
                         → Order at merchant: ord_123
                         → Return to Sarvex: {order_id: ord_123, ...}

T1+110ms Serving Plane   Emit serving.checkout.order_intent event
                         {order_id: ord_123, idempotency_key: key-A-T0, ...}

T1+120ms Serving Plane   redis.set(session:{S1}:{SID}, {...with order_id...})

T1+130ms Serving Plane   Return 200 OK to agent
                         {order_id: ord_123, ...}

T1+200ms Sync Plane      Consume order_intent event
         (async)         → Check order not already recorded
                         → Create Order doc: {_id: ord_123, idempotency_key: key-A-T0}
                         → Append to sync_order_mutations audit log
                         → Emit sync.order.created event

──────────────────────────────────────────────────────────────────────────

T2 (retry, Agent
    network dupes)       POST /checkout/{id}/complete (same request)
                         Header: Idempotency-Key: key-A-T0

T2+10ms Serving Plane    Compute body_hash = sha256({payment: ...})

T2+20ms Serving Plane    Call redis.get(idempotency:S1:key-A-T0)
                         → HIT (stored at T1+120ms)
                         → sessionID = SID

T2+25ms Serving Plane    Call session.get(SID)
                         → Retrieve session, check CompleteIdempotencyKey = key-A-T0
                         → Compare body_hash ✓ match
                         → Return cached response: {order_id: ord_123, ...}

T2+30ms Serving Plane    Return 200 OK to agent (no connector call!)
                         {order_id: ord_123, ...}

────────────────────────────────────────────────────────────────────────

Result:  ✓ Same order ID (ord_123) returned
         ✓ No duplicate charge (Stripe API not called again)
         ✓ Audit log has 1 entry for key-A-T0 (mutex prevented double-record)
         ✓ Deterministic: can replay any time within 24h window

8. State Machine: Order Lifecycle

8.1 Valid States

OrderStatus:
  - created      (just placed at merchant)
  - confirmed    (merchant confirmed receipt)
  - shipped      (fulfillment initiated)
  - cancelled    (not proceeding, no refund)
  - refunded     (money returned to buyer)

8.2 Valid Transitions

created
  ├→ confirmed (with precondition: stock available)
  ├→ cancelled (with precondition: before payment capture)
  └→ refunded  (edge: refund on created order, rare)

confirmed
  ├→ shipped (normal fulfillment path)
  └→ refunded (refund after payment captured)

shipped
  └→ [TERMINAL - no outbound transitions]

cancelled
  └→ [TERMINAL]

refunded
  └→ [TERMINAL]

8.3 Enforcement

Before any state transition, check:

ok, violations := CanTransition(order.Status, newStatus)
if !ok {
  return errors.New(violations[0])  // e.g., "cannot refund order already shipped"
}

Preconditions (examples):

• Refund: Order must be in confirmed or shipped state (not already refunded)
• Void: Order must be in created state (before capture)

9. Limitations & Trade-offs

9.1 Not Guaranteed

1. Cross-service atomicity

   • Inventory debit may succeed while order record fails (async eventual consistency)
   • Acceptable: retry with same key will converge eventually

2. Automatic refunds

   • Double-charge detected via audit log
   • Operator must manually issue refund (support ticket)

3. Multi-order atomicity

   • Batch orders (10 items) create 1 order, not 10 (by design)
   • Buyer can’t roll back 5 items without refunding entire order

4. Distributed tracing

   • Audit log provides timestamps but not full request tracing
   • Recommendation: Correlation IDs in logs for multi-span queries

9.2 Design Choices

10. Migration & Rollout

10.1 Backwards Compatibility

• No breaking changes to HTTP API
  • Idempotency-Key header optional (graceful degredation without it)
  • New fields (IdempotencyKey in Order) non-breaking (nullable)

10.2 Gradual Rollout

1. Phase 1-8 (in this commit):

   • Schema additions (Order.IdempotencyKey, etc.)
   • Storage layer (indexes, audit log)
   • State machine validator
   • Foundation ready for deterministic ID (Phase 3)

2. Phase 9 (future PR):

   • Refund/void HTTP endpoints
   • Refund/void service logic
   • Handler contracts for refund

3. Phase 10 (future PR):

   • Integration test suite (real Mongo + Redis)
   • E2E scenarios: duplicate requests, concurrent mutations, handler timeouts

10.3 Risk Mitigation

• Monitor: Audit log query counts (anomalies = double-charges)
• Alert: If same key appears with 2+ order IDs in 24h window
• Runbook: Established (IDEMPOTENCY-RUNBOOK.md) before rollout
• Support: Double-charge process documented for operators

11. Future Enhancements

1. Deterministic Order IDs

   • Currently from merchant store
   • Generate from hash(merchant_id + key + counter) for sortability

2. Refund/Void Idempotency

   • Phase 9 endpoints + full handler integration

3. Distributed Tracing

   • Correlation ID in audit log headers
   • Query spans across serving/sync planes

4. Automated Reconciliation

   • Nightly job: compare payment processor charges vs. Sarvex orders
   • Alert on mismatches (missing order, extra order)

5. Handler Spec Standardization

   • Publish Sarvex idempotency key requirements
   • Connector guide: “How to inject idempotency key into Stripe/PayPal/etc.”

References

• CLAUDE.md: Sarvex architecture invariants (§2.6 idempotency requirement)
• sarvex-v1-architecture.md: Three-plane design, atomicity assumptions
• IDEMPOTENCY-RUNBOOK.md: Operational guide for replay, double-charge detection
• RFC 9421: HTTP message signatures (for future signed idempotency keys)

End of Design Document