Past Incidents

Currently we are facing issues with the SysEleven Stack Cinder API. We are working on a solution and keep you informed. The outage only affects spawning new virtual machines or changing existing resources.

We apologise for any inconveniences.

Update: The Cinder API is operational again

At the moment we are facing issues with the distributed file system, a core component of the SysEleven Stack. At the moment it is possible that access to volumes or disks of some VMs is slow.

We will keep you informed and apologise for any inconveniences.

At the moment we are facing issues with the Cinder service API in the DBL region.

We will keep you informed and apologise for any inconveniences.

Update: The problems with the Cinder API are solved

At the moment we are facing major issues in the region DBL.

We are investigating.

UPDATE 14:27: A major storage issue affects VMs using cinder volumes or the M1 flavor severely. Also S3 is affected. We are working hard on identifying and resolving the problem.

UPDATE 15:30: The block storage (Cinder volumes and VMs with M1 flavor), as well as the S3 object storage, are available again. We are still investigating thoroughly, together with the software vendor of our storage system. If you are still experiencing issues please contact us immediately at cloudsupport@syseleven.de

We will keep you informed and apologise for any inconveniences.

UPDATE 16:10: We isolated the root cause and will implement measures that prevent this from happening again.

We are currently seeing problems in the MetaKube UI. Load projects and clusters takes a lot longer than normal.

At the moment we are facing issues with the storage system (both block storage and object storage) in region DBL.

We will keep you informed and apologise for any inconveniences.

Update 2020-12-04 00:40: The storage system recovered. We are still watching.

Update 2020-12-04 01:35: The problem is solved.

Between December 2nd, 2020 00:00 AM and 07:00 AM (UTC +1) we will perform planned maintenance in Region CBK

We expect two short network interruptions up to one minute and degradation in storage performance.

• network storage performance (m1 flavors and cinder volumes) will be degraded
• SEOS(S3) performance will be degraded
• local storage (L1 flavors) will not be affected, except for cinder volumes mounted on local storage instances
• network availability will be affected
• the API will not be affected
• state of virtual machines will not be affected

Should you have any trouble following the maintenance work please contact us.

Currently we are facing issues with the SysEleven Stack API. We are working on a solution and keep you informed. The outage only affects spawning new virtual machines or changing existing resources.

The problem is identified and we work on a fix.

Update 15:08 : We fixed the issue accordingly

We apologise for any inconveniences.

Currently we are facing issues with the SysEleven Stack API. We are working on a solution and keep you informed. The outage only affects spawning new virtual machines or changing existing resources.

We apologise for any inconveniences.

Update 16:30 : We identified the issue and fixed it accordingly

The MetaKube Cluster creation is currently not working. We are investigating the problem now.

Between November 19th, 2020 00:00 AM and 07:00 AM (UTC +1) we will perform planned maintenance in Region CBK

• Network storage performance (m1 flavors and cinder volumes) will be degraded
• SEOS(S3) performance will be degraded
• Local storage (L1 flavors) will not be affected, except for cinder volumes mounted on local storage instances

We will perform a planned maintenance on our network storage system.

Should you have any trouble following the maintenance work please contact us.

We are currently working on restoring one etcd replica in most of our MetaKube customer clusters. We lost a local storage node in the cloud region DBL last night. We did not think this would have any customer impact as we run 5 replicas in every MetaKube Cluster. We just noticed that it is not possible to add new node deployments to existing Clusters which indeed is a problem. We are working on fixing this problem as soon as possible.

Between 17:22 and 17:27 we experienced storage issues for Object Storage and Block Storage in region DBL.

The problem was identified. We are currently observing.

The issues were fixed.

We will keep you informed and apologise for any inconveniences.