Affected times for the region CBK

13:12 – 13:17 (MESZ)

14:32 – 14:37 (MESZ)

Affected times for the region DBL

12:05 – 12:15 (MESZ)

12:51 – 13:00 (MESZ)

13:50 – 14:00 (MESZ)

During these times it was possible that virtual machines did not respond due to IO wait. It might also have affected creating virtual machines and cinder volumes.

We will keep you informed and apologise for any inconveniences.

UPDATE 16:20

We identified that regular maintenance work on storage nodes (reboots) have indirectly caused this incident. The maintenance work is stopped until we find and resolve the root cause in close collaboration with the storage software vendor.

We will keep you informed on the results.

We apologise for any inconveniences.

UPDATE 22:45

We have identified and resolved the networking API troubles.

We found a bad entry in the network state database of our SDN software, that caused a cascading failure: Internal locks could not be acquired anymore, which caused the API errors, which in turn caused networking troubles during live migrations in a small number of virtual machines.

We now understood the root cause and will post more information on actions we will take to prevent this from happening again once we conducted the post-mortem.

UPDATE 15. October

We are currently working the following action points:

• We identified and are working to resolve the root cause in our SDN software
• We will improve the monitoring of our networking APIs, to identify and correlate future troubles earlier
• We will improve monitoring of virtual machine port binding of our SDN software, so we can identify and fix troubled VMs faster in the future

Existing virtual machines are not affected.

Please contact us, should you have any troubles or need any assistance.

We apologise for any inconveniences.

UPDATE 16:15:

We found out that a number of virtual machines lost their network connectivity due to the API issue. We are trying to bring them back online as soon as possible. Impacted customers have been informed by e-mail.

UPDATE 17:25:

We resolved the API issues and connectivity issues for virtual machines.

All customers that have been affected by connectivity issues should have been informed by e-mail.

Please contact us should you still have issues or need assistance.

Update 21/09/2018 14:40
Most systems are accessible and we are working on some remaining problems.

Update 21/09/2018 15:20
Systems experienced another downtime of 10 minutes and are coming online now.

Between September 14th, 2018 11:15 AM and 12:00 noon (UTC +2) we will perform emergency maintenance in both Regions

We urgently fix the problem that caused the incident yesterday evening to prevent the issue from reoccurring. There should be no impact, however, there is a little risk that similar problems occour one last time.

• storage availability may be affected
• SEOS (S3) availability may be affected
• virtual machines may be affected indirectly by "freezes" accessing their storage
• network availability will not be affected but services might hang while waiting for storage
• the API will not be affected

Should you have any trouble following the maintenance work please contact us.

Update 10:40 CEST: The cause has been identified and we are planning an emergency maintenance to fix it to avoid the problem to reoccur.

Update 11:15 CEST: We are fixing the root cause in an emergency maintenance.

Update 12:00 CEST: The root cause has been fixed. To our knowledge there were some glitches in region DBL, but everything went smooth in region CBK.

Between August 29th, 2018 11:30 AM and 17:00 PM (UTC +2) we will perform planned maintenance in Region DBL

We expect temporary performance impact and marginal impact on some virtual machines due to necessary live-migrations.

• virtual machines may be affected by short "freezes" (<0.5sec) due to live migration
• state of virtual machines will not be affected
• network availability will not be affected
• storage availability will not be affected
• storage performance can be temporarily degraded
• SEOS (S3) availability will not be affected
• SEOS (S3) performance can be temporarily degraded
• the API will not be affected

Should you have any trouble following the maintenance work please contact us.

Between August 27th, 2018 07:00 AM and 08:00 PM (UTC +2) we will perform planned maintenance in Region CBK

We expect up to three short network interruptions (of less than 15 seconds each) per virtual machine.

• network availability will be affected (see above)
• state of virtual machines will not be affected
• storage availability will not be affected

Should you have any trouble following the maintenance work please contact us.

We are working on a solution and keep you informed.

The outage only affects creating new cinder volumes.

We apologise for any inconveniences.

UPDATE 16:00: The Cinder API should be functioning normally again. Please contact cloudsupport@syseleven.de if you are still experiencing issues.

We are working hard to resolving the incident and will keep you informed and apologise for any inconveniences.

Update 12:34:
We fixed the issues and are now investigating the root cause.

We apologize for any inconveniences.

Update: 12:20
We resolved all issues.

We apologize for any inconveniences.

Update 14:44
The API is available again. The reason for the short unavailability was due to a software bug.

We will keep you informed and apologise for any inconveniences.

UPDATE 19:54: We fixed the issues. Every service is up and running again.

Thus it is not possible to create new instances at the moment. Also it is not possible to create new Volumes in this region.

Running VMs are not affected.

We apologise for any inconviniences.

UPDATE 16:00:
We fixed the issues and are now investigating the root cause.

Only creating, updating or attaching cinder volumes in the DBL region is affected.

Running VMs with cinder volumes are NOT affacted.

UPDATE 12:15:
We solved the issues and are watching the systems closely.

UPDATE 13:00:
The issues are solved and we are working on a root cause analysis.

We do not expect impact on running virtual machines or the S3 service.

Should you have any trouble following the maintenance work please contact us.

we would like to inform you about the change of your login details from your GitHub to SysElevenStack credentials on 11.06.2018 at 2:00 PM UTC+1. After the change the registration will be easier and more secure, because you can log in the K8s-Dashboard using your SysElevenStack account.

Please note: Your K8s cluster will not be visible in the frontend after the change, nevertheless, the clusters will still be available and reachable via API.

Action required: Please send us your GitHub and SysElevenStack e-mail address so that we can migrate your cluster and SSH-keys in order to make them visible again.­

• The outage only affects spawning new virtual machines or changing existing ressources.

Update: 16:59
The issue is solved, the API is up and responding properly again.
The problem was also caused by a network fabric failure.

We apologise for any inconveniences.

• Virtul machines might have trouble spawning or have no access to their volumes (disks)
• SEOS/S3 access might be slow or interrupted

Update: 14:34
We identified the cause and are working on a fix.

Update: 14:46
The issue is solved, everything is up and running properly again.
The problem was caused by a network fabric failure.

We will keep you informed and apologise for any inconveniences.

Between June 5th, 2018 10:15 (CET) and 11:45 (CET) we will perform planned maintenance on the SysEleven Stack

• Some instances may become unreachable for a few seconds during this timeframe
• The short interruption of instances network connectivity will take place randomly
• The instances will keep running; the unavailability is only related to the network component

Should you have any trouble following the maintenance work please contact us.

• Affected are the regions DBL and CBK. Since the API is non functional also the Dashboard service is affected.
• Spawning new virtual machines is not possible
• Running instances or other OpenStack resources are not affected.

Update 10:30

• Any issues are resolved now and every component of the SysEleven Stack is working again.

We apologise for any inconviniences.

Between May 23, 2018 14:00 (CET) and May 23, 2018 15:00 (CET) we will perform planned maintenance on the SysEleven Stack. Our plan is to have no customer-facing impact, but there is a possibility that the API will not be available in both regions - leading to a short loss of control.

During this period the API and Dashboard will not be available in both regions.

Existing resources like virtual machines will not be affected in any way. SEOS (S3) will not be affected in any way.

Ankündigung
Am Dienstag, den 15.05.2018 zwischen 00:00 und 06:00 Uhr finden in unserem Rechenzentrum Wartungsarbeiten an der redundanten Stromversorgung eines Serverschranks statt.

In diesem Zuge wird die Redundanz der Stromversorgung der in Server in diesem Rack unterbrochen werden müssen, was jedoch keinen Einfluss auf die Verfügbarkeit Ihrer Dienste haben wird.

Vorsorglich haben wir alle direkt betroffenen Kunden direkt kontaktiert und informiert.

Update 10:27
Alle Systeme arbeiten wieder wie gewohnt. Wir haben die Services geprüft und konnten keine weiteren Auffälligkeiten feststellen.

Update 10:18
Die Betroffene Hardware startete wieder wie gewohnt. Derzeit werden noch alle Gastsysteme gestartet und überprüft.

Update 10:10
Betroffene Hardware wird neugestartet

Ankündigung 09:50
Unsere Überwachung hat eine Hardware Node identifiziert die in einem Zustand ist, der außerhalb der normalen Parameter liegt.
Die entsprechenden Kunden wurde per E-Mail kontaktiert und das weitere Vorgehen wird mit ihnen ggf. individuell abgestimmt.
Sollten Sie Probleme identifiziert haben, aber nicht per E-Mail kontaktiert sein, wenden Sie sich bitte an support@syseleven.de oder rufen Sie uns an.

Update 19:55 Uhr
We fixed the problem and monitor all systems.

Announcement 19:25
We are currently facing a service failure in our central storage appliance in our datacenter Berlin BKI. This leads to problems in access to NFS shares. We are working on a solution with high pressure.