One data center went down and took ChatGPT, Claude, and Grok with it. What's your fallback?

The short version

On September 3, 2026, ChatGPT and Codex, Claude, and Grok all degraded or went down inside the same window. The first theory was a shared dependency somewhere upstream. SpaceX later confirmed the cause: a failure at a compute center in Memphis that also affected what it described as compute partners, a reference to infrastructure that Anthropic relies on. The Register reported the confirmation, and Cybernews followed with analysis of the wider pattern of outage clusters.

Three separate companies. Three separate brands. One physical failure point. If your product calls one of those APIs on the critical path and has no second route, your product was one building away from an outage you could not fix, explain, or shorten.

Why this is a concentration problem, not an AI problem

It is tempting to file this under AI reliability. It belongs in a much older folder: vendor concentration risk. The same failure mode has taken down payment flows, authentication, search, email delivery, and image processing for years. What changed is how quickly AI features moved onto the critical path, and how little of that movement came with a fallback plan.

Concentration risk hides well, because it does not look like concentration from the inside. You picked 3 vendors. You compared their status pages. You read their uptime commitments. What you could not see from the dashboard was that some of them rent capacity from the same place. Logical diversity is not physical diversity, and a contract with 3 suppliers does not guarantee 3 independent failure paths.

The questions worth answering before the next one

You do not need an architecture review to start. You need honest answers to a small set of questions, written down where your team can find them.

• Which external calls sit on the path your customers cannot work around? Checkout, login, search, and publishing usually qualify. Summaries and suggestions usually do not.
• For each of those, what happens at 30 seconds of failure, 30 minutes, and 3 hours? If the answer is the same for all 3, you have no strategy, only hope.
• Who tells your customers, through which channel, and how long does that take?
• If one tenant sends a flood of retries, whose experience degrades? Just theirs, or everyone on the same infrastructure?

Tenant isolation is the part that pays off quietly

In a multi-tenant platform, the goal is not zero failures. The goal is that a failure stays small. That means limiting the blast radius per tenant, so that one store hammering a provider, hitting a rate limit, or holding open a thousand slow connections does not take down the tenant next door.

The practical pieces are unglamorous. Per-tenant rate limits and concurrency caps. Separate queues, or at least separate queue partitions, so one backlog does not starve another. Circuit breakers that trip per provider and per tenant rather than globally. Timeouts short enough that a slow dependency fails fast instead of consuming your worker pool. Bulkheads around anything that talks to a third party, so a stuck call cannot spread.

None of this shows up in a demo. All of it shows up on the day a provider has a bad hour.

Graceful degradation, decided in advance

Every external dependency needs a defined behaviour for when it is unavailable. Pick one before you need it:

• Serve the last good result. Cache aggressively for anything that does not need to be fresh. A 6 hour old product description is better than an error.
• Fail to a simpler version. If semantic search is down, fall back to keyword search. If an AI summary is unavailable, show the first paragraph. The feature narrows, the page still loads.
• Queue and reconcile. For writes that can wait, accept the request, confirm receipt, and process when the dependency returns. Tell the customer what state their request is in.
• Hide the feature. A control that quietly disappears is better than one that spins and then apologises.

The pattern that makes all 4 possible is a thin abstraction in front of each provider category, with the provider chosen by configuration rather than by code. One interface for text generation, one for search, one for mail. Switching providers then becomes a flag change you can make under pressure, not a deploy you have to write under pressure.

Redundancy you have never tested is a guess

A second provider that has never served real traffic is a plan, not a fallback. Differences in token limits, response format, latency, content filtering, and error semantics all surface at the worst moment.

Two habits close that gap. Send a small share of live traffic to the secondary continuously, so it stays warm and its differences stay visible. Then run scheduled failure exercises, where you turn off the primary in a controlled window and watch what your platform actually does. The findings are usually mundane: a hardcoded model name, a retry loop with no ceiling, an alert that goes to someone who left. Mundane findings are the point.

The runbook and the message

When a large provider goes down, your engineering work is often finished in 10 minutes. The remaining 3 hours are communication. That part deserves preparation too: a named owner for each dependency, a status page you can update without a deploy, a short template that says what is affected, what still works, and when you will post again. Give your support team permission to send it without waiting for a full diagnosis.

Your customers rarely judge you on whether something broke. They judge you on whether you knew, and whether you told them.

How we build this at Nudova

Resilience does not retrofit well. Once a provider call is woven through 40 files, isolating it becomes a project rather than a setting. So the platforms we build start with the boring decisions already made: provider access behind a single interface, per-tenant limits set at the infrastructure layer, defined degraded behaviour for every external call, and a runbook written while the system is still calm.

The outcome is not a promise of perfect uptime. No one can offer that, including the companies that went dark on September 3. The outcome is narrower and more useful: when a giant has a bad day, your store keeps taking orders, your admin panel keeps loading, and the feature that depends on the failed provider degrades in a way you chose rather than one you discover.

If you are not sure what your platform does when a single provider disappears for 3 hours, that is worth finding out on a quiet Tuesday rather than during the next outage cluster.