Past Incidents

Root cause:

We created a oauth service with dex to authenticate our web applications against. Dex writes all of the session information into the etcd cluster. Due to a misconfigured monitoring we created over 500000 sessions in the etcd cluster. The database size grow from about 300MB to 700MB. That slowed down all cluster operations. During the whole incident customer applications were not affected, the MetaKube dashboard including cluster creation were regularly not working correctly when one of the master apiservers ran out of memory.

At the moment we are facing an issue with the Nova API in the region DBL.

Creating new VMs results in an error since 11:30 AM CET.

We are investigating.

We will keep you informed and apologise for any inconveniences.

UPDATE 11:53 CET: We have resolved the issue, and isolated the root cause.

We were facing issues with the SysEleven Stack Neutron API between Dec 29th 2018 and Jan 2nd 2019. Some Neutron API requests were slow, and a few requests might have timed out and failed. The reason was an almost-out-of-memory situation in the software of one of the three SDN controllers. The problem was fixed at 11:26 CET on Jan 2nd 2019.

Network connectivity of VMs were not affected by this issue.

We apologise for any inconveniences.

At the moment we are facing connectivitiy issues in region CBK on one compute node.

We will keep you informed and apologise for any inconveniences.

Update 11:23: The issue is fixed. Reason for outage: The SDN agent on one of our compute nodes crashed because of memory pressure. We are investigating the reason for the memory pressure.