Past Incidents

Currently it will take a long time to create servers with the Local SSD Storage (L1.*) flavors. The wait time increases linearly to Local SSD Storage capacity of the chosen flavor. Eventually, creating the server will succeed.

This is due to a mitigation we put in place that solves sporadic performance problems, when a virtual machine has been created or deleted. We are right now working on a better mitigation, that will not affect creating servers.

The issue only affects spawning new virtual machines with the L1.* flavor (Local SSD Storage).

We apologise for any inconveniences.

UPDATE 2019-02-13: The issues are fully resolved now. Creating and deleting Local SSD Storage instances will only take a few seconds now, and the time for creating and deleting VMs is constant, no matter what disk capacity.

Root cause:

We created a oauth service with dex to authenticate our web applications against. Dex writes all of the session information into the etcd cluster. Due to a misconfigured monitoring we created over 500000 sessions in the etcd cluster. The database size grow from about 300MB to 700MB. That slowed down all cluster operations. During the whole incident customer applications were not affected, the MetaKube dashboard including cluster creation were regularly not working correctly when one of the master apiservers ran out of memory.