Tuesday 4th December 2018

Compute SysEleven Stack issues in region DBL

Some Instances in DBL were unreachable between 10:55 and 11:03.

Update 11:05: The problem appears to be currently resolved but we are still investigating to understand the root cause.

Update 11:45: All Instances in our DBL region were affected by a loss of network.

The outage was caused by a maintenance command issued on one isolated and empty compute node that unexpectedly affected all hypervisors in the whole cluster. It raised the log level which led to our SDN agents to be overwhelmed by traffic.

We will conduct a post-mortem to find ways to prevent that error from happening again.