Past Incidents

At the moment we are facing issues with control plane services. The problems are affecting a limited number of clusters in dbl and cbk.

We will keep you informed and apologise for any inconveniences.

Update 10:45 PM: The issue has been identified and fixed. All control plane components are operational again.

At the moment we are facing an issue with slow etcds which makes some operations impossible, like updates to the cluster. Running workloads should not be affected unless they use the Kubernetes API Server directly.

We will keep you informed and apologise for any inconveniences.

Update: We have recovered from the incident.

We were facing issues with the SysEleven Stack compute API in the CBK region. The outage was short and only affected spawning new virtual machines or changing existing resources. All services are up and running again.

We apologise for any inconveniences.

We are facing problems at our datacenter location Berlin Lützowstraße. Some systems are currently unreachable, including the Openstack API. We are investigating the problem.

Update 2022-04-19 15:45: Our datacenter provider reported an external power outage at our datacenter Berlin Lützowstraße. While most systems were not affected, some have been shut down unexpectedly. Power supply has been recovered and systems are being ramped up again.

Update 2022-04-19 15:53: Several systems are recovering. We are still working on the recovery of all affected systems.

Update 2022-04-19 16:32: Part of the Openstack nodes are still not operational. This affects the availability of the cluster's network storage. We are still working on the problem with highest priority.

Update 2022-04-19 17:57: Recovery of underlying services such as Quobyte (SDS) and Midonet (SDN) to full functionality is still ongoing. All involved teams are working on the problems with high pressure and we will keep you updated with the progress.

Update 2022-04-19 19:45: All services have been recovered to work, which includes Network, Storage, API, DNS, and Loadbalancers. The consistency checks of existing data are still running.