Zone Outages: Multi-Zone Is Proven by a Drill

The grid feed to the building failed at 05:53 UTC on 20 October 2026, and backup power took over as designed: the battery banks carried the load while the diesel generators were meant to start. The generators never took it. The programmable logic controllers that run the start and transfer sequence failed to run it, the primary and the backup alike. At 06:00 the batteries were spent, and the host servers, switching fabric and local storage arrays of Google Cloud zone europe-west2-b in London lost power together. That is the chain of events in the preliminary incident report Google posted on 22 October; the final report with preventative actions is still to come. By Google's incident clock the disruption ran from 06:00 to 12:10 UTC, six hours and ten minutes, and the impact list names 40 products, among them GKE, Cloud Run, Cloud SQL, Spanner, Pub/Sub and Persistent Disk.

Revolut, HSBC and Lloyds confirmed disruption that morning, the BBC reported in a piece first published at 08:46 UTC, and HM Courts and Tribunals Service said it was aware of an issue with the video platform used for remote hearings. Revolut said, in a statement the BBC carried, that its customers' money and data “remained safe throughout”; Reuters put its UK customer base at about 13 million. Downdetector also listed Royal Mail, Rightmove, DPD, BT and Screwfix, The Register reported, and a listing there counts complaints rather than confirming a cause. Nothing published says how any of these organisations was spread across the region's three zones, and Google says customers in the other two zones were hit as well. Times here are UTC; London was on BST, one hour ahead.

The power was back in an hour, the zone in six

The report says the utility feed was restored within an hour. The other five hours went on resetting and testing the generator controllers, restarting Google's infrastructure in sequence and re-admitting traffic to the zone gradually, in the report's words “allowing accumulated job backlogs to clear smoothly”. Raised error rates, API timeouts and processing delays lasted until 11:29 UTC, when most services had recovered. Compute Engine then had a tail of its own: new VM creation resumed at 11:46, and by 12:10 over 99% of Compute Engine VMs were healthy, which is where Google stops the clock. The closure notice on the public page is stamped 14:03 UTC and speaks of a “very small number of customers” still affected. Press figures of about eight hours run to that notice.

• One piece of advice. From 09:24 UTC the workaround field read: “If feasible, the impacted customer may try to migrate their workload impacted to another available location (zone or region).” The closure notice replaced it with “None as the issue has been mitigated.”
• A facility run by a third party, Google says. The report says only “the datacenter facility hosting availability zone europe-west2-b”. A Google spokesperson told the BBC and Reuters the disruption was “at a third-party facility” in the region.

Zones a and c hurt in two ways of their own

The power outage “was isolated strictly to datacenter facilities in europe-west2-b”, the report says, and still “some customers running workloads in unimpacted zones (europe-west2-a and europe-west2-c) or across regional architectures also experienced unexpected impact.” Google gives two causes: “instantaneous demand surges on unimpacted compute capacity” and “synchronous control-plane validation dependencies across zonal boundaries”. In plain terms, everyone's failover arrived in zones a and c at the same moment and saturated the compute slots there, which produced admission queues, delayed database failovers and intermittent 500 and 503 errors across the region. Separately, regional control planes called “metadata and catalog components in the offline zone”, and those calls timed out until Google's engineers routed around them.

The products named as degraded by these effects were “BigQuery, Looker, Google Compute Engine, Google Kubernetes Engine, Regional Persistent Disk, and Cloud SQL”. Regional Persistent Disk and Cloud SQL are what a customer buys to survive a zone, and the report counts them among the services whose failovers were hit by the capacity exhaustion and by control-plane timeouts into the dead zone. The report also records what worked: “Cloud SQL, AlloyDB, Memorystore for Redis, and Cloud Bigtable failed over into nearby unimpacted zones”.

Google's own reports hold a precedent. In April 2023 zone europe-west9-a in Paris lost more than half of its Compute Engine instances and Persistent Disk volumes to a water leak that “originated in a non-Google portion of the facility”, reached a UPS room and led to a fire. That report explained the regional spill by noting that “regional Spanner's replicas were not correctly distributed across the three buildings in the region”. Two European zone incidents in three and a half years, both starting in the building, both reaching customers who had paid for more than one zone.

List the zonal things from the API

The architecture diagram says regional. The resources decide, and the words to check are in Google's documentation. GKE zonal clusters “have a single control plane in a single zone”, and “Nodes in a regional cluster can run in multiple zones or a single zone depending on the configured node locations”; Autopilot clusters are always regional. Cloud SQL “doesn't recover standalone instances from a zonal outage automatically”, while an HA instance has a primary and a secondary zone and fails over in about sixty seconds. Memorystore for Redis Basic Tier “provides a cache with no replication”. Filestore Basic tiers are zonal; the Regional tier is built to survive a zone. A zonal persistent disk dies with its zone. The script below pulls all of that from the project, one zone at a time.

#!/usr/bin/env bash
set -euo pipefail
Z=europe-west2-b

# ZONAL: gone with the zone
gcloud compute instances list \
  --filter="zone:$Z" \
  --format="table(name,status)"
gcloud compute disks list \
  --filter="zone:$Z" \
  --format="table(name,sizeGb)"

# HA OR NOT: one field decides
F="table(name,gceZone,"
F+="settings.availabilityType)"
gcloud sql instances list \
  --format="$F"
gcloud redis instances list \
  --region=europe-west2 \
  --format="table(name,tier)"

# NODES: regional label, one zone?
F="table(name,location,"
F+="locations.list())"
gcloud container clusters list \
  --format="$F"

# HEADROOM: slots held in a and c
F="table(zone.basename(),"
F+="specificReservation.count)"
gcloud compute reservations list \
  --format="$F"

Read the output as the list of things that stop when zone b stops. Every instance and disk in the first two tables is one. In the Cloud SQL table, settings.availabilityType prints ZONAL or REGIONAL; REGIONAL with a gceZone of europe-west2-b means a failover is coming, with about a minute of unavailability. A Redis tier of BASIC is a cache with no second copy. In the cluster table, a regional location beside a single entry in locations is the trap the GKE page describes: a replicated control plane over nodes that all sit in one zone. Run it for each zone and attach to every row what breaks upstream when it goes dark. Do the same for the vendors in your request path, since a supplier with one zone gives your service one zone.

The drill has to run while the region is busy

The arithmetic of a three-zone region is unforgiving. If each zone runs at 70% and one goes dark, the other two are asked for 105% each, and the autoscaler's request for more joins the same queue as everyone else's. The report describes Google's engineers adjusting “scheduler allocations and capacity limits across unimpacted infrastructure to prioritize active customer reservations”, which is what counted during the surge. A Compute Engine reservation holds capacity in a named zone and is charged whether or not a VM uses it; Google's overview calls it “a high level of assurance of capacity”. For a service that must survive a zone, reserved headroom in the other two is the price of the label, sized to the share of load the lost zone carried.

Then cut a zone deliberately. Cordon every node in it with kubectl cordon -l topology.kubernetes.io/zone=europe-west2-b, drain the same selector, and watch where the pods land and how long they wait for a node. Trigger a Cloud SQL failover with gcloud sql instances failover and time the gap your application sees against the sixty seconds in the documentation. Check that load balancer health checks and DNS do what the runbook claims. Write down the recovery time, the thing that broke that nobody had listed, and the date. Then repeat it at the busiest hour of the quarter, because a drill in an empty region proves only that the empty region works. Google's 09:24 UTC advice to migrate workloads elsewhere was sound; it is usable by a team that has migrated them before.

Watch zone b lose power

Press Pull the breaker below. The panel is a three-zone region with your components placed in it. Set each one zonal or regional, choose whether slots are held in zones a and c, then cut zone b. Zonal placements go dark at once. Regional placements fail over, and whether that failover finishes in a minute or sits in an admission queue depends on the headroom you reserved. The regional control plane's calls into zone b time out until the panel routes around them. The multi-zone label on the architecture card only reads as proven when the run ends with every request answered.

Detection and recovery are yours

Google's own monitoring and probers alerted its engineers at 06:08 UTC, eight minutes after the batteries ran out. The public status page first posted at 08:35 UTC, two hours and twenty-seven minutes later, with a start time written as “2026-10-19 23:08 PDT”. The zone and the power event were first named at 11:12 UTC. StackPulse, which tracks about 9,800 status pages, counted 70 that reported a problem that day, 50 of them before Google posted, and 21 that named Google or GCP in the title. Those are StackPulse's figures, published on 20 October. Both ends of that gap are on Google's own pages, the report and the status history.

A team that waited for the page lost two and a half hours. A synthetic probe per zone on the real critical path (a login, a payment, a hearing joined) fires inside a minute and names the zone; the public page took five hours to do that. Alert on the regional error-rate shift too, since the report says storage writes in the region saw a transient error rate during the initial failover period. The general case for asking live questions of a system is made in the observability guide; the point here is narrower. A probe per zone is how you learn the blast radius at 06:0x rather than at 11:12.

Recovery runs past the provider's clock. Google's incident ended at 12:10 UTC and its closure notice came at 14:03. StackPulse counts 58 downstream pages that resolved the same day and 43 of them after Google's closure, with Redis at 17:24, Sourcegraph at 17:44 and Elastic Cloud at 20:11. Twelve pages were still open at the end of the day, 8 of them Revolut partners. Those are the hours of replaying Pub/Sub backlogs, restarting subscriber fleets that gave up, warming caches that come back empty and waiting on partners doing the same. A recovery runbook starts at the all-clear with a list: queues to drain and in what order, workers to restart, caches to warm, partners to check before the page goes green. The longest part of your outage can be somebody else's.

Regulators are asking the question you should answer first

Reuters reported on 21 October that the European Banking Authority had been in contact with Google Cloud for more information; the EBA said that designating a critical third-party provider “establishes an EU-level oversight framework” and “does not transfer responsibility for incident management from the provider”. The Financial Conduct Authority said it was “engaging with Google over the incident”; the Bank of England declined to comment. On 20 October, the day of the outage, the Treasury Committee published a letter from its chair, Dame Meg Hillier, to the Governor of the Bank of England and the chief executive of the FCA, asking how they will use their powers in relation to the incident. Its page lists the four critical third parties designated in July 2026: Microsoft Ireland Operations Limited, Google Cloud EMEA Limited, Amazon Web Services EMEA SARL and Oracle Corporation UK Limited. Reuters noted that a bank relying on a third party remains responsible for its own services.

A bank or payment firm will be asked, in some form, what it knew about its zones and when. The answers that hold up are artefacts with dates on them: the drill that was run, the inventory it tested and the probe that fired before the provider's page. Whether to spread across providers or regions is a different question, covered in the deployment models guide. The zone question is answered inside one region, by evidence that the design was exercised.