At the moment we are facing connectivitiy issues in region DBL.

We will keep you informed and apologise for any inconveniences.

UPDATE 4:14: Any network connectivity issues are solved now. Reason for outage was a faulty line card.

Today we were facing issues with the distributed file system, a core component of the SysEleven Stack.

Affected times for the region CBK

13:12 – 13:17 (MESZ)

14:32 – 14:37 (MESZ)

Affected times for the region DBL

12:05 – 12:15 (MESZ)

12:51 – 13:00 (MESZ)

13:50 – 14:00 (MESZ)

During these times it was possible that virtual machines did not respond due to IO wait. It might also have affected creating virtual machines and cinder volumes.

We will keep you informed and apologise for any inconveniences.

UPDATE 16:20

We identified that regular maintenance work on storage nodes (reboots) have indirectly caused this incident. The maintenance work is stopped until we find and resolve the root cause in close collaboration with the storage software vendor.

We will keep you informed on the results.

Currently you may experience troubles with the network API . Spawning and deleting instances or other network resources is not possible. If you experience any unexpected behaviour don't hesitate to contact us.

We apologise for any inconveniences.

UPDATE 22:45

We have identified and resolved the networking API troubles.

We found a bad entry in the network state database of our SDN software, that caused a cascading failure: Internal locks could not be acquired anymore, which caused the API errors, which in turn caused networking troubles during live migrations in a small number of virtual machines.

We now understood the root cause and will post more information on actions we will take to prevent this from happening again once we conducted the post-mortem.

UPDATE 15. October

We are currently working the following action points:

• We identified and are working to resolve the root cause in our SDN software
• We will improve the monitoring of our networking APIs, to identify and correlate future troubles earlier
• We will improve monitoring of virtual machine port binding of our SDN software, so we can identify and fix troubled VMs faster in the future

Currently we are facing issues with the SysEleven Stack API. We are working on a solution and keep you informed. The outage only affects changing network resources, as well as spawning new virtual machines.

Existing virtual machines are not affected.

Please contact us, should you have any troubles or need any assistance.

We apologise for any inconveniences.

UPDATE 16:15:

We found out that a number of virtual machines lost their network connectivity due to the API issue. We are trying to bring them back online as soon as possible. Impacted customers have been informed by e-mail.

UPDATE 17:25:

We resolved the API issues and connectivity issues for virtual machines.

All customers that have been affected by connectivity issues should have been informed by e-mail.

Please contact us should you still have issues or need assistance.