Past Incidents

Between November 20th, 2018 14:00 and 16:00 (UTC +1) we will perform planned maintenance in Region DBL

We plan to update components of the software-defined networking layer.

During the maintenance, the following degradations in service will occur:

• The network connectivity of the virtual machines will have multiple interruptions of up to 45 seconds. This will affect all virtual machines on a compute node at the same time. We will try to avoid concurrent connectivity loss of multiple compute nodes.

• access to the API/Dashboard will not be affected

• SEOS (S3) will not be affected

• storage availability will not be affected

Should you have any trouble following the maintenance work please contact us.

At the moment we are facing issues with the distributed file system, a core component of the SysEleven Stack. At the moment it is possible that virtaul machines can't be spawned or have no write access to their volumes(disks).

We will keep you informed and apologise for any inconveniences.

Update: 14:22
We identified a suspicious hardware node that slowed down the overall cluster performance. We are working to isolate that node and bring back cluster write performance.

Update: 14:23
We isolated the affected hardware and performance is back to normal. If you experience any problems please contact us.

At the moment we are facing connectivitiy issues in region DBL.

We will keep you informed and apologise for any inconveniences.

UPDATE 4:14: Any network connectivity issues are solved now. Reason for outage was a faulty line card.

Today we were facing issues with the distributed file system, a core component of the SysEleven Stack.

Affected times for the region CBK

13:12 – 13:17 (MESZ)

14:32 – 14:37 (MESZ)

Affected times for the region DBL

12:05 – 12:15 (MESZ)

12:51 – 13:00 (MESZ)

13:50 – 14:00 (MESZ)

During these times it was possible that virtual machines did not respond due to IO wait. It might also have affected creating virtual machines and cinder volumes.

We will keep you informed and apologise for any inconveniences.

UPDATE 16:20

We identified that regular maintenance work on storage nodes (reboots) have indirectly caused this incident. The maintenance work is stopped until we find and resolve the root cause in close collaboration with the storage software vendor.

We will keep you informed on the results.