SysEleven Status

SysEleven Stack Neutron API issues in Region DBL

SysEleven Status — Wed, 05 Dec 2018 12:30:00 +0000

We are currently seeing a slightly elevated rate of errors in the Neutron API. Our engineers are currently looking into them and will update this status page post with more information soon. We apologise for any inconveniences. **UPDATE 14:30:** We determined the impact. The errors are impacting a very small number of attempts to launch VMs. So far only one customer is affected. We are now looking for a solution. **UPDATE 15:30:** The issue also affects Neutron API performance. If you are experience any troubles, please contact cloudsupport@syseleven.de **UPDATE 16:15:** The API performance issues are now resolved. **UPDATE 17:00:** The API issues are fully resolved now.

SysEleven Stack issues in region DBL

SysEleven Status — Tue, 04 Dec 2018 09:55:00 +0000

Some Instances in DBL were unreachable between 10:55 and 11:03. **Update 11:05**: The problem appears to be currently resolved but we are still investigating to understand the root cause. **Update 11:45**: All Instances in our DBL region were affected by a loss of network. The outage was caused by a maintenance command issued on one isolated and empty compute node that unexpectedly affected all hypervisors in the whole cluster. It raised the log level which led to our SDN agents to be overwhelmed by traffic. We will conduct a post-mortem to find ways to prevent that error from happening again.

SysEleven Stack issues in region CBK

SysEleven Status — Tue, 27 Nov 2018 10:00:00 +0000

From approximately 11h to 14h we had issues with some instances in region CBK being unavailable for up to ten minutes due to unexpected impact of a hypervisor update. We have stopped those updates and investigate how to do them without such impact.

SysEleven Stack API issues in Region CBK and DBL

SysEleven Status — Mon, 12 Nov 2018 13:45:00 +0000

Currently we are facing issues with the SysEleven Stack API. We are working on a solution and keep you informed. The outage only affects spawning new virtual machines or changing existing resources. We apologise for any inconveniences. --- **Update: 15:00** The problem was found and could be fixed immediately.

SysEleven Stack maintenance announcement

SysEleven Status — Tue, 06 Nov 2018 13:14:07 +0000

--- **Between November 20th, 2018 14:00 and 16:00 (UTC +1)** we will perform planned maintenance in **Region DBL** --- We plan to update components of the software-defined networking layer. During the maintenance, the following degradations in service will occur: - The network connectivity of the virtual machines will have multiple interruptions of up to 45 seconds. This will affect all virtual machines on a compute node at the same time. We will try to avoid concurrent connectivity loss of multiple compute nodes. - access to the API/Dashboard will **not** be affected - SEOS (S3) will **not** be affected - storage availability will **not** be affected --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack Block Storage issues in region CBK

SysEleven Status — Tue, 06 Nov 2018 12:42:00 +0000

At the moment we are facing issues with the distributed file system, a core component of the SysEleven Stack. At the moment it is possible that virtaul machines can't be spawned or have no write access to their volumes(disks). We will keep you informed and apologise for any inconveniences. --- **Update: 14:22** We identified a suspicious hardware node that slowed down the overall cluster performance. We are working to isolate that node and bring back cluster write performance. **Update: 14:23** We isolated the affected hardware and performance is back to normal. If you experience any problems please contact us.

SysEleven Stack issues in region DBL

SysEleven Status — Thu, 25 Oct 2018 01:04:00 +0000

At the moment we are facing connectivitiy issues in region DBL. We will keep you informed and apologise for any inconveniences. **UPDATE 4:14:** Any network connectivity issues are solved now. Reason for outage was a faulty line card.

SysEleven Stack block storage issues in regions CBK and DBL

SysEleven Status — Mon, 15 Oct 2018 10:05:00 +0000

Today we were facing issues with the distributed file system, a core component of the SysEleven Stack. **Affected times for the region CBK** 13:12 – 13:17 (MESZ) 14:32 – 14:37 (MESZ) **Affected times for the region DBL** 12:05 – 12:15 (MESZ) 12:51 – 13:00 (MESZ) 13:50 – 14:00 (MESZ) During these times it was possible that virtual machines did not respond due to IO wait. It might also have affected creating virtual machines and cinder volumes. We will keep you informed and apologise for any inconveniences. **UPDATE 16:20** We identified that regular maintenance work on storage nodes (reboots) have indirectly caused this incident. The maintenance work is stopped until we find and resolve the root cause in close collaboration with the storage software vendor. We will keep you informed on the results.

SysEleven Stack API issues in Region DBL

SysEleven Status — Tue, 02 Oct 2018 11:40:00 +0000

Currently you may experience troubles with the network API . Spawning and deleting instances or other network resources is not possible. If you experience any unexpected behaviour don't hesitate to contact us. We apologise for any inconveniences. **UPDATE 22:45** We have identified and resolved the networking API troubles. We found a bad entry in the network state database of our SDN software, that caused a cascading failure: Internal locks could not be acquired anymore, which caused the API errors, which in turn caused networking troubles during live migrations in a small number of virtual machines. We now understood the root cause and will post more information on actions we will take to prevent this from happening again once we conducted the post-mortem. **UPDATE 15. October** We are currently working the following action points: * We identified and are working to resolve the root cause in our SDN software * We will improve the monitoring of our networking APIs, to identify and correlate future troubles earlier * We will improve monitoring of virtual machine port binding of our SDN software, so we can identify and fix troubled VMs faster in the future

SysEleven Stack API issues in Region DBL

SysEleven Status — Mon, 01 Oct 2018 11:30:00 +0000

Currently we are facing issues with the SysEleven Stack API. We are working on a solution and keep you informed. The outage only affects changing network resources, as well as spawning new virtual machines. Existing virtual machines are not affected. Please contact us, should you have any troubles or need any assistance. We apologise for any inconveniences. **UPDATE 16:15:** We found out that a number of virtual machines lost their network connectivity due to the API issue. We are trying to bring them back online as soon as possible. Impacted customers have been informed by e-mail. **UPDATE 17:25:** We resolved the API issues and connectivity issues for virtual machines. All customers that have been affected by connectivity issues should have been informed by e-mail. Please contact us should you still have issues or need assistance.

SysEleven Managed Hosting Network issues in region BKI1

SysEleven Status — Fri, 21 Sep 2018 12:15:00 +0000

At the moment it is possible that virtual machines face issues with network connectivity. We are working on a solution. We will keep you informed and apologise for any inconveniences. **Update 21/09/2018 14:40** Most systems are accessible and we are working on some remaining problems. **Update 21/09/2018 15:20** Systems experienced another downtime of 10 minutes and are coming online now.

SysEleven Stack emergency maintenance

SysEleven Status — Fri, 14 Sep 2018 08:55:00 +0000

--- **Between September 14th, 2018 11:15 AM and 12:00 noon (UTC +2)** we will perform emergency maintenance in **both Regions** --- We urgently fix the problem that caused the incident yesterday evening to prevent the issue from reoccurring. There should be no impact, however, there is a little risk that similar problems occour one last time. - storage availability **may be** affected - SEOS (S3) availability **may be** affected - virtual machines **may be** affected indirectly by "freezes" accessing their storage - network availability will **not** be affected but services might hang while waiting for storage - the API will **not** be affected --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack issues in both regions

SysEleven Status — Thu, 13 Sep 2018 19:00:00 +0000

Around 21:00 CEST we had an incident which affected storage of SysEleven Stack instances in both regions. The issue disappeared and we are still investigating cause and effect. Affected customers are encouraged to open support tickets with affected resources, obeservations and timestamps, if available. Update 10:40 CEST: The cause has been identified and we are planning an emergency maintenance to fix it to avoid the problem to reoccur. Update 11:15 CEST: We are fixing the root cause in an emergency maintenance. Update 12:00 CEST: The root cause has been fixed. To our knowledge there were some glitches in region DBL, but everything went smooth in region CBK.

SysEleven Stack storage maintenance announcement

SysEleven Status — Wed, 29 Aug 2018 09:29:00 +0000

--- **Between August 29th, 2018 11:30 AM and 17:00 PM (UTC +2)** we will perform planned maintenance in **Region DBL** --- We expect temporary performance impact and marginal impact on some virtual machines due to necessary live-migrations. - virtual machines **may be** affected by short "freezes" (<0.5sec) due to live migration - state of virtual machines will **not** be affected - network availability will **not** be affected - storage availability will **not** be affected - storage performance **can be** temporarily degraded - SEOS (S3) availability will **not** be affected - SEOS (S3) performance **can be** temporarily degraded - the API will **not** be affected --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack network maintainance announcement

SysEleven Status — Fri, 24 Aug 2018 12:35:22 +0000

--- **Between August 27th, 2018 07:00 AM and 08:00 PM (UTC +2)** we will perform planned maintenance in **Region CBK** --- We expect up to three short network interruptions (of less than 15 seconds each) per virtual machine. - network availability **will be** affected (see above) - state of virtual machines will **not** be affected - storage availability will **not** be affected --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack API issues in Region DBL

SysEleven Status — Thu, 19 Jul 2018 08:00:00 +0000

Currently it may not be possible to create Cinder Volumes using the SysEleven Stack API. We are working on a solution and keep you informed. The outage only affects creating new cinder volumes. We apologise for any inconveniences. **UPDATE 16:00:** The Cinder API should be functioning normally again. Please contact cloudsupport@syseleven.de if you are still experiencing issues.

SysEleven Stack Block Storage issues in region DBL

SysEleven Status — Tue, 17 Jul 2018 10:15:00 +0000

At the moment we are facing issues with the distributed file system, a core component of the SysEleven Stack. At the moment it is possible that virtaul machines can't be spawned or have no write access to their volumes(disks). We are working hard to resolving the incident and will keep you informed and apologise for any inconveniences. **Update 12:34:** We fixed the issues and are now investigating the root cause.

SysEleven Stack API issues in all regions

SysEleven Status — Mon, 09 Jul 2018 09:45:00 +0000

Currently we are facing issues with the SysEleven Stack API. We are working on a solution and keep you informed. The outage only affects spawning new virtual machines or changing existing resources. We apologize for any inconveniences. **Update: 12:20** We resolved all issues.

SysEleven Stack API issues in Region DBL

SysEleven Status — Wed, 04 Jul 2018 12:31:00 +0000

Currently we are facing issues with the SysEleven Stack API. We are working on a solution and keep you informed. The outage only affects spawning new virtual machines or changing existing resources. We apologize for any inconveniences. **Update 14:44** The API is available again. The reason for the short unavailability was due to a software bug.

SysEleven Stack Object Storage issues in region CBK

SysEleven Status — Tue, 03 Jul 2018 16:30:00 +0000

At the moment we are facing issues with the Object Storage in Region CBK. At the moment it is possible that objects can't be read or written. We will keep you informed and apologise for any inconveniences. **UPDATE 19:54:** We fixed the issues. Every service is up and running again.

SysEleven Stack Compute issues in Region CBK

SysEleven Status — Tue, 03 Jul 2018 12:50:00 +0000

Currently we are facing issues with the SysEleven Stack Compute service in Region CBK. Thus it is not possible to create new instances at the moment. Also it is not possible to create new Volumes in this region. Running VMs are not affected. We apologise for any inconviniences. **UPDATE 16:00:** We fixed the issues and are now investigating the root cause.

Cinder API issues in the DBL region

SysEleven Status — Tue, 03 Jul 2018 09:45:00 +0000

We are currently facing issues with the cinder volume API. Only creating, updating or attaching cinder volumes in the DBL region is affected. Running VMs with cinder volumes are NOT affacted. **UPDATE 12:15:** We solved the issues and are watching the systems closely. **UPDATE 13:00:** The issues are solved and we are working on a root cause analysis.

Maintenance work in our storage infrastructure in Region CBK

SysEleven Status — Tue, 03 Jul 2018 08:18:00 +0000

**Between July 3rd, 2018 10:00 AM and 18:00 PM (UTC +1)** we will perform planned maintenance in **Region CBK** --- We do **not** expect impact on running virtual machines or the S3 service. --- Should you have any trouble following the maintenance work please contact us.

Release maintenance announcement - SysEleven Managed Kubernetes (Kubermatic)

SysEleven Status — Thu, 07 Jun 2018 15:27:15 +0000

**Between June 11th, 2018 2:00 PM (UTC +1) and 2:30 PM (UTC +1)** we will perform planned maintenance --- we would like to inform you about the change of your login details from your GitHub to SysElevenStack credentials on 11.06.2018 at 2:00 PM UTC+1. After the change the registration will be easier and more secure, because you can log in the [K8s-Dashboard](https://k8s.01.syseleven.de/) using your SysElevenStack account. --- **Please note:** Your K8s cluster will not be visible in the frontend after the change, nevertheless, the clusters will still be available and reachable via API. --- **Action required:** Please send us your GitHub and SysElevenStack e-mail address so that we can migrate your cluster and SSH-keys in order to make them visible again.

SysEleven Stack API issues in region DBL

SysEleven Status — Wed, 06 Jun 2018 13:26:00 +0000

Currently we are facing issues with the SysEleven Stack API. We are working on a solution. - The outage only affects spawning new virtual machines or changing existing ressources. --- **Update: 16:59** The issue is solved, the API is up and responding properly again. The problem was also caused by a network fabric failure. We apologise for any inconveniences.

SysEleven Stack Block and Object Storage issues in region DBL

SysEleven Status — Wed, 06 Jun 2018 12:18:00 +0000

At the moment we are facing issues with the distributed file system, a core component of the SysEleven Stack. * Virtul machines might have trouble spawning or have no access to their volumes (disks) * SEOS/S3 access might be slow or interrupted --- **Update: 14:34** We identified the cause and are working on a fix. **Update: 14:46** The issue is solved, everything is up and running properly again. The problem was caused by a network fabric failure. --- We will keep you informed and apologise for any inconveniences.

SysEleven Stack maintainance announcement

SysEleven Status — Tue, 05 Jun 2018 13:01:05 +0000

--- **Between June 5th, 2018 10:15 (CET) and 11:45 (CET)** we will perform planned maintenance on the SysEleven Stack * Some instances may become unreachable for a few seconds during this timeframe * The short interruption of instances network connectivity will take place randomly * The instances will keep running; the unavailability is only related to the network component --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack API issues

SysEleven Status — Fri, 18 May 2018 08:20:00 +0000

Currently we are facing an outage inside the SysEleven Stack cluster components. We are working on a solution and will keep you informed. - Affected are the regions DBL and CBK. Since the API is non functional also the Dashboard service is affected. - Spawning new virtual machines is not possible - Running instances or other OpenStack resources are not affected. **Update 10:30** - Any issues are resolved now and every component of the SysEleven Stack is working again. We apologise for any inconviniences.

SysEleven Stack maintainance announcement

SysEleven Status — Tue, 15 May 2018 13:28:10 +0000

--- **Between May 23, 2018 14:00 (CET) and May 23, 2018 15:00 (CET)** we will perform planned maintenance on the SysEleven Stack. Our plan is to have no customer-facing impact, but there is a possibility that the API will not be available in both regions - leading to a short loss of control. --- During this period the API and Dashboard will not be available in both regions. Existing resources like virtual machines will not be affected in any way. SEOS (S3) will not be affected in any way.

Wartungsarbeiten an redundanter Stromversorgung

SysEleven Status — Mon, 14 May 2018 12:17:00 +0000

**Update** Gegen 5:00 Uhr wurden die Wartungsarbeiten erfolgreich und ohne Unterbrechung abgeschlossen. **Ankündigung** Am Dienstag, den 15.05.2018 zwischen 00:00 und 06:00 Uhr finden in unserem Rechenzentrum Wartungsarbeiten an der redundanten Stromversorgung eines Serverschranks statt. In diesem Zuge wird die Redundanz der Stromversorgung der in Server in diesem Rack unterbrochen werden müssen, was jedoch keinen Einfluss auf die Verfügbarkeit Ihrer Dienste haben wird. Vorsorglich haben wir alle direkt betroffenen Kunden direkt kontaktiert und informiert.

SysEleven PVC Störung einer Hardware Node

SysEleven Status — Fri, 11 May 2018 07:50:00 +0000

**Update 10:45** Es sind keine weiteren Auffälligkeiten festgestellt worden. Damit beenden wir den Vorfall. **Update 10:27** Alle Systeme arbeiten wieder wie gewohnt. Wir haben die Services geprüft und konnten keine weiteren Auffälligkeiten feststellen. **Update 10:18** Die Betroffene Hardware startete wieder wie gewohnt. Derzeit werden noch alle Gastsysteme gestartet und überprüft. **Update 10:10** Betroffene Hardware wird neugestartet **Ankündigung 09:50** Unsere Überwachung hat eine Hardware Node identifiziert die in einem Zustand ist, der außerhalb der normalen Parameter liegt. Die entsprechenden Kunden wurde per E-Mail kontaktiert und das weitere Vorgehen wird mit ihnen ggf. individuell abgestimmt. Sollten Sie Probleme identifiziert haben, aber nicht per E-Mail kontaktiert sein, wenden Sie sich bitte an support@syseleven.de oder rufen Sie uns an.

SysEleven Datacenter BKI - NFS partial outage

SysEleven Status — Wed, 09 May 2018 17:25:00 +0000

**Update 20:20 Uhr** All systems are working in the expected parameters. We apologize for any inconvenience. **Update 19:55 Uhr** We fixed the problem and monitor all systems. **Announcement 19:25** We are currently facing a service failure in our central storage appliance in our datacenter Berlin BKI. This leads to problems in access to NFS shares. We are working on a solution with high pressure.