Commerce Service API

Incident Report for Norce

Postmortem

Post-Mortem: Commerce API Service Degradation — March 13, 2026

Duration: 2026-03-13 11:55 – 16:15 CET

Impact: High latency, timeouts, and intermittent failures on the Commerce API and Legacy API for a significant portion of customers in the multi-tenant production environment.

Status page history: https://norce.statuspage.io/incidents/m05jyh3crwzd

Summary

On March 13th, the Commerce API experienced severe service degradation caused by a misconfiguration made by a customer's external solution partner. The solution partner was performing a large-scale product import into the customer's staging environment and had incorrectly configured their webhook subscriptions — subscribing to far more event types than necessary and pointing the webhook target at their own production endpoint. That endpoint called GetProductByPartNo against our production Commerce API for products that only existed in the staging environment, resulting in 100% cache misses and direct database load. Combined with our webhook retry mechanism (up to 15 retries per failed delivery), this produced traffic exceeding 10x the normal level for the entire multi-tenant environment.

The excessive load caused thread pool exhaustion and connection pool saturation, leading to cascading failures across the affected infrastructure shard. Recovery was further complicated by a "thundering herd" effect — each time the environment stabilized, all affected customers' applications reconnected simultaneously, producing up to 5x normal traffic and causing repeated collapses.

Timeline (CET)

~11:55 — Issue first noticed; troubleshooting begins.
12:18 — First update posted to Status Page.
~14:10 — Primary environment recovers.
~14:30 — Environment degrades again (coinciding with the solution partner restarting the product import).
~14:35 — Customer migration to secondary environment begins in batches.
~15:45 — Secondary environment degrades after migrating batch containing affected customer; batch rolled back.
~16:07 — Affected customer identified through customer-by-customer migration.
~16:15 — Affected customer isolated at load balancer; all remaining customers restored on secondary environment.
Evening — Solution partner identifies and corrects the webhook misconfiguration.

Root Cause

The solution partner's webhook configuration had two critical errors: (1) overly broad event subscriptions caused each product update to trigger 3+ webhook calls, and (2) an environment mismatch routed those calls through to our production API for products that didn't exist in production. Every request missed the cache and hit the database directly. Our webhook retry mechanism amplified the problem by retrying each failed delivery up to 15 times.

What We Did

Tuned application thread pool and timeout settings to provide more headroom under load.
Migrated customers in batches to a healthy secondary environment.
Identified and isolated the affected customer at the load balancer level.
Established contact with the solution partner, who corrected the misconfiguration.

What We've Changed

Rate limiting at the load balancer: Per-customer and per-IP rate limiting is now in place on all Commerce API endpoints, ensuring no single customer can overwhelm shared infrastructure.
Nginx-based traffic monitoring: Request-per-second metrics are now measured from load balancer access logs instead of APM data, providing accurate visibility even when backend services are degraded.
Updated alerting: Alerts are now connected to load balancer-level metrics to detect traffic spikes earlier.

What We're Implementing

Webhook circuit breaker: We are implementing automatic disabling of webhook event subscriptions after a threshold of consecutive delivery failures. This would have stopped the retry storm before it escalated.

We apologize for the disruption and are committed to these improvements to prevent similar incidents in the future.

Posted Mar 16, 2026 - 08:02 CET

Resolved

This incident has been resolved. All Commerce API services are operating normally.

Root cause: The disruption was caused by an abnormally high volume of API requests originating from a single client integration. The request volume — orders of magnitude above normal levels — exceeded platform capacity and caused cascading performance degradation across all customers.

Resolution: The affected client has been identified and isolated to a dedicated environment. All other customer traffic is running on a separate, healthy environment with normal performance. The underlying issue in the client's integration has been addressed.

During this incident, no data was lost. All orders, baskets, and customer data remained intact throughout.

We have also deployed infrastructure improvements to increase the platform's resilience to sudden traffic spikes. We will continue to monitor the environment and follow up with the affected customer's team to prevent recurrence.

A formal post-mortem report will be published early next week with a detailed timeline of the incident, root cause analysis, and the steps we are taking to prevent similar issues in the future.

We apologize for the disruption and thank you for your patience. If you continue to experience any issues, please contact support.

Posted Mar 13, 2026 - 18:19 CET

Monitoring

All customer traffic has been successfully migrated and the Commerce API is operating normally. Response times and error rates have returned to expected levels.

We have deployed infrastructure improvements to address the underlying bottleneck that contributed to the extended recovery time. These changes improve how the platform handles concurrent connections during high-load scenarios.

We are continuing to actively monitor the environment to ensure stability holds. At this time, all services are functioning as expected.

Impact: No ongoing impact. Normal service has been restored. No data was lost during this incident.

Next update: We will post a final resolution update once we have confirmed sustained stability over the coming hours.

Posted Mar 13, 2026 - 16:31 CET

Identified

Following our earlier recovery, the Commerce API experienced a return to degraded performance after approximately 20 minutes of stable operation. The platform is once again experiencing intermittent timeouts and slow responses under sustained elevated traffic.

Our engineering team has identified and deployed a fix targeting a key bottleneck in how internal services handle concurrent connections. This fix is being rolled out progressively and we expect it to significantly improve resilience under high load.

In parallel, we are taking additional measures to manage traffic levels and gradually restore normal operations. We are seeing steady improvement as these measures take effect.

Impact: Customers may continue to experience intermittent slow responses or timeout errors on Commerce API calls. No data loss has occurred.

What you can do: If your integration uses retry logic, we continue to recommend using backoff between attempts (e.g., waiting 1–2 seconds before retrying). This helps the platform recover more quickly and benefits all customers.

Next update: We will provide another update within 30 minutes or as conditions change.

Posted Mar 13, 2026 - 15:00 CET

Monitoring

The Commerce API environment has begun to stabilize. The majority of requests are now completing successfully, and we are seeing steady improvement across all services.

The initial cause — a sudden and unexpected doubling of inbound traffic — has been addressed. Recovery was prolonged by retry storms as client applications reconnected simultaneously, but the platform is now handling traffic at expected levels.

We are continuing to monitor closely to ensure stability holds. Some customers may still experience occasional slow responses as the final services fully recover.

Impact: Most API requests are succeeding normally. Intermittent slowness may still occur briefly as full stabilization completes. No data loss has occurred throughout this incident.

Next update: We will post a final resolution update once we have confirmed sustained stability.

Posted Mar 13, 2026 - 14:14 CET

Update

We are continuing to work on restoring full service to the Commerce API. The initial cause — a sudden spike in traffic to approximately double the normal volume — has been identified and addressed.

However, full recovery is taking longer than expected. As services come back online, accumulated retry traffic from client applications is creating repeated load spikes that are slowing stabilization. Our team is actively implementing measures to manage this and restore capacity in a controlled manner.

We understand this is impacting your operations and we are treating this with the highest priority.

Impact: Customers may continue to experience intermittent timeouts or errors on Commerce API calls. No data has been lost.

What you can do: If your integration supports it, please consider temporarily reducing request frequency or adding backoff between retries. This will help the platform recover more quickly and benefit all customers.

Next update: We will provide another update within 30 minutes or sooner if the situation changes significantly.

Posted Mar 13, 2026 - 13:44 CET

Update

We have identified the cause of the ongoing service disruption affecting the Commerce API. An unexpected surge in traffic, approximately double the normal volume, overwhelmed backend services and triggered a cascading effect across the platform.

Our team is actively working to restore full service. Recovery is being complicated by a retry storm — as client applications detect the service returning, they reconnect simultaneously, generating up to five times the normal request volume. This additional load is delaying full stabilization.

We are implementing controlled measures to gradually restore capacity and manage the inbound traffic load. We expect incremental improvement as these measures take effect.

Impact: Customers may continue to experience intermittent timeouts or slow responses on Commerce API calls. No data loss has occurred.

What you can do: If your integration supports it, we recommend implementing exponential backoff on retries to help ease the recovery process. Reducing retry frequency will help the platform stabilize more quickly.

Next update: We will post another update within 30 minutes or sooner if the situation changes.

Posted Mar 13, 2026 - 13:06 CET

Identified

We are currently experiencing intermittent service degradation affecting the Commerce API. Some API requests may take longer than usual or return timeout errors.

Our team has identified the issue and is actively working to restore full service. The root cause is related to increased load on backend infrastructure components, which is causing some requests to respond slower than expected. Automated recovery mechanisms are in progress.

We will provide updates as the situation develops.

Impact: Some customers may experience slow responses or intermittent errors when calling Commerce API endpoints (e.g. product lookups). No data loss has occurred.

Posted Mar 13, 2026 - 12:39 CET

Investigating

We're currently experiencing a degradation in availability for the Norce Commerce Services API. Our engineers are investigating. Next update will be at 12:45 or earlier if the problem is identified.

Posted Mar 13, 2026 - 12:18 CET

This incident affected: Norce Commerce (Norce Commerce).