SysEleven Status

SysEleven Stack issues in region DBL

SysEleven Status — Mon, 14 Oct 2019 13:11:00 +0000

At 15:11 CEST we experienced an unexpected restart of services that are part of our software defined networking (SDN). This led to 30 second downtime for some VMs and VPN services in DBL. **Update 16:00**: We have found the root cause and will make sure that this will not happen again.

DNS Störung/Failure

SysEleven Status — Wed, 09 Oct 2019 12:45:00 +0000

We are currently facing a failure in outgoing DNS resolving. We are in contact with the supplier and are working on a solution with high pressure. Update: The issue has been resolved.

SysEleven Stack network maintenance announcement

SysEleven Status — Tue, 08 Oct 2019 13:48:22 +0000

--- **Between October 18th, 2019 00:00 AM and 01:00 AM (UTC+2)** we will perform planned maintenance in **Region DBL** --- During the maintenance window we will update our network service in region DBL. Availability of the following services will be reduced: - interruption of network connectivity for 30 sec --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack storage and network maintenance announcement

SysEleven Status — Tue, 08 Oct 2019 13:42:22 +0000

--- **Between October 16th, 2019 00:00 AM and 04:00 AM (UTC +2)** we will perform planned maintenance in **Region CBK** --- During the maintenance window we will update our storage system and a network service in region CBK. Availability of the following services will be reduced: - degradation in storage performance - interruption of network connectivity for 30 sec --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack Network and Storage issues in region DBL

SysEleven Status — Thu, 26 Sep 2019 08:23:00 +0000

We received reports for an intermittent partial outage in networking and storage in DBL. Total duration was around minute. The issues are resolved already. We are looking into what caused these issues. **Update 10:36**: We identified an issue in our switching fabric. The issues were caused by the short time the network took to reconverge. **Update 10:42**: We observed slow IO in our distributed storage layer. We are investigating. **Update 11:11**: All systems are operational again. We are watching closely and continue investigating. **Update 11:28**: We observed slow IO in our distributed storage layer. We are investigating. **Update 11:38**: We identified the root cause in our storage layer. All issues are now resolved. We will conduct a post-mortem to make sure this will not happen again.

SysEleven Stack storage maintenance announcement

SysEleven Status — Thu, 12 Sep 2019 13:20:23 +0000

--- **Between September 19th, 2019 00:00 AM and 04:00 AM (UTC +2)** we will perform planned maintenance in **Region DBL** --- During the maintenance window we will update our storage system in region DBL. There may be a degradation in storage performance. --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack API issues in Region DBL

SysEleven Status — Mon, 02 Sep 2019 15:30:00 +0000

Currently we are facing issues with the SysEleven Stack API. We are working on a solution and keep you informed. The outage only affects spawning new virtual machines or changing existing resources. We apologise for any inconveniences. **UPDATE 18:05**: There was a problem with controller hardware. It is solved and we are working on a longterm solution. Please contact us at cloudsupport@syseleven.de if you are still encountering any issues.

SysEleven Stack storage maintenance announcement

SysEleven Status — Thu, 29 Aug 2019 08:20:57 +0000

--- **Between September 6th, 2019 0:00 AM and 04:00 AM (UTC +2)** we will perform planned maintenance in **Region DBL** --- During the maintenance window we will update our storage system in region DBL. There may be a degradation in storage performance. --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack network maintenance announcement

SysEleven Status — Mon, 26 Aug 2019 13:33:48 +0000

--- **Between September 3rd, 2019 00:00 AM and 01:00 AM (UTC +2)** we will perform planned maintenance in **Region DBL** --- We will update our network service and expect network interruptions of less than 30 seconds per virtual machine. - network availability **will be** affected - state of virtual machines will **not** be affected - storage availability will **not** be affected - SEOS (S3) will **not** be affected - the API will **not** be affected --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack DNSaaS API issues in Region DBL/CBK

SysEleven Status — Thu, 22 Aug 2019 10:23:00 +0000

Currently we are facing issues with the SysEleven Stack DNSaaS API. We are working on a solution and keep you informed. The outage only affects creating or changing existing DNS resources. We apologise for any inconveniences. **UPDATE 12:55**: There was a problem with our Designate Infrastructure. It is solved and we are working on a longterm solution. Please contact us at cloudsupport@syseleven.de if you are still encountering any issues.

SysEleven Zabbix maintenance ende

SysEleven Status — Tue, 20 Aug 2019 03:29:08 +0000

Gerne informieren wir Sie darüber, dass die Wartung erfolgreich abgeschlossen wurde. Zabbix wird nun in der Version 4.2 betrieben. Bei Darstellungsproblemen leeren sie bitte den Cache ihres Browsers für die Seite https://zabbix.syseleven.de/.

SysEleven Zabbix maintenance start

SysEleven Status — Mon, 19 Aug 2019 22:33:41 +0000

Wie bereits angekündigt, beginnen wir jetzt mit der Wartung der SysEleven Zabbix Instanz. Vorausgesetzt, alles läuft nach Plan, sollten wir bis 06:00 Uhr MEZ fertig sein. Während der Wartung treten die folgenden Beeinträchtigungen im Betrieb auf: * Erreichbarkeit von https://zabbix.syseleven.de/ eingeschränkt.

SysEleven Stack maintenance announcement

SysEleven Status — Mon, 19 Aug 2019 09:34:14 +0000

--- **Between August 23rd, 2019 00:00 AM and 01:00 AM (CEST)** we will perform planned maintenance in **Region CBK** --- During the maintenance window we will perform a configuration update on our networking components in our region CBK. During the maintenance window the following services will not be accessible: - interruption of network connectivity to virtual machines of less than 30 seconds --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack maintenance announcement

SysEleven Status — Thu, 08 Aug 2019 14:47:15 +0000

--- **Between August 15th, 2019 0:00 AM and 04:00 AM (UTC +2)** we will perform planned maintenance in **Region DBL** --- During the maintenance window we will update our storage system and a network service in region DBL. Availability of the following services will be reduced: - degradation in storage performance - interruption of network connectivity for 30 sec --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack storage maintenance announcement

SysEleven Status — Wed, 31 Jul 2019 22:09:18 +0000

--- **Between August 1st, 2019 0:00 AM and 02:00 PM (CEST)** we perform planned maintenance in **Region DBL** --- We expect temporary performance impact. --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack issues with VPNaaS in region CBK

SysEleven Status — Thu, 25 Jul 2019 14:00:00 +0000

At the moment we are investigating issues with VPN as a Service in CBK. There might be troubles when creating new VPN connections. Existing connections should continue to work. Please contact our support team at cloudsupport@syseleven.de if you are experiencing any issues. --- Update 2019-07-26 14:35: Due to a misbehavior of the SDN software, scheduling of new VPN services was unreliable. Now the service is fully functional again. ---

SysEleven Stack emergency maintenance announcement

SysEleven Status — Wed, 17 Jul 2019 10:00:00 +0000

--- **Between July 17th, 2018 12:00 (noon) and 12:30 PM (CEST, UTC +2)** we will perform emergency maintenance in **Region CBK** and **Region DBL** --- During the maintenance window we will roll back the configuration change of the previous maintenance on our networking components in our region CBK and DBL due to unpredicted side effects for some customers. During the maintenance window the following services will be affected: - interruption of network connectivity to virtual machines of less than 30 seconds --- Update 12:20: The emergency maintenance has been carried out as planned. --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack maintenance announcement

SysEleven Status — Mon, 15 Jul 2019 22:00:00 +0000

--- **Between July 16, 2019 0:00 AM and 06:00 AM (CEST)** we will perform a planned maintenance in **Region CBK** and **Region DBL** --- During the maintenance window we will perform a configuration update on our networking components in our region CBK and DBL, and update our storage system only in the DBL region. During the maintenance window the following services will not be accessible: - interruption of network connectivity to virtual machines of less than 30 seconds - degradation on storage performance --- We stopped updating the storage system. We observed issues with slow I/O during the time of the update. We apologize for any inconvenience that may have caused. --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack network maintenance announcement

SysEleven Status — Tue, 25 Jun 2019 15:00:05 +0000

--- **Between June 27, 2019 1:00 AM and 02:00 AM (CEST)** we will perform a planned maintenance in **Region CBK** and **Region DBL** --- During the maintenance window we will perform a configuration update on our networking components in our region CBK and DBL. During the maintenance window the following services will not be accessible: - interruption of network connectivity to virtual machines of less than 30 seconds --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack maintenance announcement

SysEleven Status — Wed, 15 May 2019 16:22:53 +0000

--- **Between May 24, 2019 0:00 AM and 01:00 AM (UTC +2)** we will perform planned maintenance in **Region CBK** --- During the maintenance window we will update the SDN software in our DBL region in order to fix an issue with the management of LBaaS load balancer health monitors. During the maintenance window the following services will not be accessible: - No access to the SysEleven Stack Network API (Neutron) - interruption of network connectivity to virtual machines of less than a minute Because of that network components cannot be created or re-configured during this time. --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack maintenance announcement

SysEleven Status — Wed, 15 May 2019 16:20:27 +0000

--- **Between May 23, 2019 0:00 AM and 01:00 AM (UTC +2)** we will perform planned maintenance in **Region DBL** --- During the maintenance window we will update the SDN software in our DBL region in order to fix an issue with the management of LBaaS load balancer health monitors. During the maintenance window the following services will not be accessible: - No access to the SysEleven Stack Network API (Neutron) - interruption of network connectivity to virtual machines of less than a minute Because of that network components cannot be created or re-configured during this time. --- Should you have any trouble following the maintenance work please contact us.

MetaKube partial outage

SysEleven Status — Wed, 17 Apr 2019 06:47:00 +0000

**2019-04-17 08:47 am** We are currently experiencing issues with the OpenStack infrastructure, MetaKube and some MetaKube clusters are running on. At the moment we are fixing the underlying issue and analyze which MetaKube clusters are affected and bring them back online again. **2019-04-17 09:22 am** The first effected Clusters are back online. Our MetaKube Engineering team is still working on a solution for every Cluster. If you mention any issues please feel free to contact our [customer Support](https://www.syseleven.de/support/). **2019-04-17 10:04 am** Most of the effected Clusters are up and running in a normal stage. We are still working on this incident and come back with more information asap. **2019-04-17 10:32 am** Our monitoring and our personal checks show, that all customer clusters are up and running in a normale state. We close this incident and work on the root cause. We will send out a RFO Report (Reason For Outage) as soon as we have identified all aspects of this issue. **If you still have problems with your cluster or identify unnormal behaviour in your cluster, please contact out [customer Support](https://www.syseleven.de/support/).**

SysEleven network partial outage

SysEleven Status — Tue, 09 Apr 2019 12:45:00 +0000

A network outage occured on April 9 at approximately 2:45 pm. This caused a temporary interruption in the availability of services from the internet. At the moment all services are available. We are investigating the issue. Update: We have temporarily disabled one of our upstream traffic partners who seemed to cause the issues. We expect no further impact on our reachability. We already have request the RFO (Reason for outage).

Maintenance work on our infrastructure in DC DBL

SysEleven Status — Mon, 08 Apr 2019 09:38:09 +0000

--- **Between April 17th, 2019 0:00 and 6:00 AM (UTC +2)** we will perform planned maintenance in **Region DBL** --- During this period we will update the storage system. During the maintenance window the following services will be affected: - Slower storage access --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack API issues in region CBK

SysEleven Status — Wed, 03 Apr 2019 23:08:00 +0000

At the moment we are facing API issues in region CBK. It is not possible to start, stop or create virtual machines. It is also not possible to create or delete cinder volumes. Running VMs and the object storage (S3) are not affected. We will keep you informed and apologise for any inconveniences. **UPDATE 05:45**: We are still working on resolving these issues together with our technology partners. **UPDATE 06:25**: All issues are now resolved.

SysEleven Stack maintenance announcement

SysEleven Status — Thu, 28 Mar 2019 14:15:55 +0000

--- **Between April 4th, 2019 0:00 and 5:00 AM (UTC +2)** we will perform planned maintenance in **Region CBK** --- During this period we will update the network and storage system. During the maintenance window the following services will be affected: - Network interruption of 60 seconds - Slower storage access --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack issues in region CBK

SysEleven Status — Wed, 06 Mar 2019 15:53:00 +0000

At the moment we are facing timeouts in the Network (Neutron) API in region CBK. Managing network resources may not work at this time. We will keep you informed and apologise for any inconveniences. **Update 2019-03-06 17:12**: The issue has been resolved. We applied a bug fix and now the Neutron Network API in region CBK is fully available again.

SysEleven Stack maintenance announcement in region CBK

SysEleven Status — Fri, 01 Mar 2019 15:25:09 +0000

--- **Between March 7th, 2019 0:00 AM and 01:00 AM (UTC +1)** we will perform planned maintenance in **Region CBK** --- During this period the Cinder (Volume) API in the CBK region is not available - leading to a loss of control. During this period there will be: - no access to Cinder API, causing loss of control over block volumes. Creation, deletion, modification, attachment, and detachment of volumes will not work. - running virtual machines will **not** be affected, attached volumes will not be affected. - data on the volumes will not be corrupted. - SEOS (S3) will **not** be affected. - network availability will **not** be affected. **After the maintenance is over, the storage availability zone name will be changed from "nova" to "cbk1".** No Heat stack or Terraform state changes are required, if the availability zone was not explicitly specified. --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack Cinder API issues in Region CBK

SysEleven Status — Wed, 27 Feb 2019 15:45:00 +0000

Currently we are facing issues with the SysEleven Stack API. We have noticed, that when creating a new Cinder Volume, it will be in "error" state immediately. Cinder volumes that have been created before already are not affected. We are working on a solution and keep you informed. We apologise for any inconveniences. **UPDATE 16:56**: The issues have been resolved. We are now investigating the root cause.

SysEleven Stack API issues

SysEleven Status — Mon, 18 Feb 2019 13:22:00 +0000

We observed an elevated rate of errors, creating local storage virtual machines (L1.* flavor). Engineers are now looking into the problem and we will keep you informed here. **UPDATE 15:45**: Creating L1.* virtual machines currently requires unusual amounts of disk space on the hypervisors. The elevated error rate was because not enough space was left, for short amounts of time. Our engineers have resolved this issue for now, and are still working on a permanent solution. Please contact us at cloudsupport@syseleven.de if you are still encountering any issues.

SysEleven Stack issues in region DBL

SysEleven Status — Wed, 13 Feb 2019 19:42:00 +0000

At the moment we are facing storage issues in region DBL. Some IO requests may take longer than usual. We will keep you informed and apologise for any inconveniences.

SysEleven Stack maintenance announcement

SysEleven Status — Wed, 13 Feb 2019 16:38:43 +0000

--- **Between February 27th 2019 00:00 AM and 05:00 AM (UTC +1)** we will perform planned maintenance in **Region CBK** --- During this period the API in the CBK region is not available - leading to a loss of control. During this period there will be: - no access to the API/Dashboard (loss of control) - running virtual machines will **not** be affected - SEOS (S3) will **not** be affected - network availability will **not** be affected - storage availability will **not** be affected --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack API issues in Region DBL

SysEleven Status — Tue, 12 Feb 2019 23:12:00 +0000

Currently we are facing issues with the SysEleven Stack API in the DBL Region. We are working on a solution and keep you informed. The outage only affects spawning new virtual machines or changing existing resources. We apologise for any inconveniences. **UPDATE 00:30**: We have identified the cause of the API issues in DBL and resolved the issues for most API operations. Due to a bug in OpenStack API components, the DBL API services used CBK identity URLs for validating tokens, which did not work due to the planned maintenance. We changed all internal identity URLs in the endpoint registry to region-agnostic URLs. We are now working on fully resolving this issue for all API operations in DBL. **UPDATE 00:50**: All issues are resolved.

SysEleven Backbone Network Maintenance Announcement

SysEleven Status — Tue, 12 Feb 2019 13:11:16 +0000

SysEleven Network Maintenance Announcement - Location: bki1 - Reason: Software Update - Between February 14th, 2019 00:00 AM and 06:00 AM (local time) we will perform planned maintenance in our bki1 Location - Reason: Software Updates Should you have any trouble following the maintenance work please contact us.

SysEleven Backbone Network Maintenance Announcement

SysEleven Status — Tue, 12 Feb 2019 10:34:44 +0000

SysEleven Network Maintenance Announcement - Location: fix5 - Reason: Software Update - Between February 20th, 2019 00:00 AM and 06:00 AM (local time) we will perform planned maintenance in our fix5 Location - Reason: Software Updates Should you have any trouble following the maintenance work please contact us.

SysEleven Backbone Network Maintenance Announcement

SysEleven Status — Tue, 12 Feb 2019 10:23:10 +0000

SysEleven Network Maintenance Announcement - Location: blu1 - Reason: Software Update - Between February 19th, 2019 00:00 AM and 06:00 AM (local time) we will perform planned maintenance in our blu1 Location - Reason: Software Updates Should you have any trouble following the maintenance work please contact us.

SysEleven Stack: Long wait time when creating Local SSD Storage servers in all regions

SysEleven Status — Mon, 11 Feb 2019 09:21:00 +0000

Currently it will take a long time to create servers with the Local SSD Storage (L1.*) flavors. The wait time increases linearly to Local SSD Storage capacity of the chosen flavor. Eventually, creating the server will succeed. This is due to a mitigation we put in place that solves sporadic performance problems, when a virtual machine has been created or deleted. We are right now working on a better mitigation, that will not affect creating servers. The issue only affects spawning new virtual machines with the L1.* flavor (Local SSD Storage). We apologise for any inconveniences. **UPDATE 2019-02-13:** The issues are fully resolved now. Creating and deleting Local SSD Storage instances will only take a few seconds now, and the time for creating and deleting VMs is constant, no matter what disk capacity.

SysEleven Stack maintenance announcement in CBK

SysEleven Status — Thu, 07 Feb 2019 13:55:00 +0000

--- In order to offer the same features in both regions we will roll out new versions of our networking components in the CBK region. The maintenance window is planned for 2019-02-13 from 00:00-05:00 (CET) During the maintenance window we will update the networking components in the CBK region. There will be the following service degradations within the maintenance window: - API endpoints of the CBK region are not available. - Virtual machines of the CBK region will face several downtimes. They will be unavailable for approximately one minute in total. The DBL region will not be affected. **UPDATE 01:55**: The maintenance has been completed. ---

SysEleven Network Maintenance Announcement - Location: bno1 - Reason: Software Update

SysEleven Status — Thu, 24 Jan 2019 18:05:00 +0000

--- ** Between January 25th, 2019 00:00 AM and 06:00 AM (local time) we will perform planned maintenance in our bno1 Location ** --- ** Reason: Software Updates ** --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack storage maintenance announcement

SysEleven Status — Tue, 22 Jan 2019 23:00:00 +0000

--- **Between January 23rd, 2018 00:00 AM and 06:00 AM (UTC +1)** we will perform planned storage maintenance in **Region DBL** --- During the maintenance window we will update the distributed storage software in our DBL region in order to fix a memory leak. It is possible that IO requests are slow during the maintenance window. Affected services: - block storage - ephemeral storage - object storage Not affected: - local SSD storage instances --- *UPDATE 04:32 AM (CET)*: The maintenance has been completed successfully. Should you have any trouble following the maintenance work please contact us at cloudsupport@syseleven.de

MetaKube partial outage

SysEleven Status — Sat, 12 Jan 2019 11:00:00 +0000

Root cause: We created a oauth service with dex to authenticate our web applications against. Dex writes all of the session information into the etcd cluster. Due to a misconfigured monitoring we created over 500000 sessions in the etcd cluster. The database size grow from about 300MB to 700MB. That slowed down all cluster operations. During the whole incident customer applications were not affected, the MetaKube dashboard including cluster creation were regularly not working correctly when one of the master apiservers ran out of memory.

[completed] SysEleven Stack network maintenance announcement in region DBL

SysEleven Status — Thu, 10 Jan 2019 14:11:00 +0000

--- **Between January 17th, 2019 07:00 AM and 08:00 AM (UTC +1)** we will perform planned maintenance in **Region DBL** --- During the maintenance window we will update the SDN software in our DBL region in order to fix a memory leak. During the maintenance window the following services will not be accessible: - access to the SysEleven Stack Network API (Neutron) **will be interrupted** in region DBL The following services will not be affected: - network availability will **not** be affected - state of virtual machines will **not** be affected - storage availability will **not** be affected - SEOS (S3) will **not** be affected --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack API issue in region DBL

SysEleven Status — Thu, 10 Jan 2019 10:30:00 +0000

At the moment we are facing an issue with the Nova API in the region DBL. Creating new VMs results in an error since 11:30 AM CET. We are investigating. We will keep you informed and apologise for any inconveniences. **UPDATE 11:53 CET**: We have resolved the issue, and isolated the root cause.

SysEleven Stack Neutron API issues in Region DBL

SysEleven Status — Wed, 02 Jan 2019 10:00:00 +0000

We were facing issues with the SysEleven Stack Neutron API between Dec 29th 2018 and Jan 2nd 2019. Some Neutron API requests were slow, and a few requests might have timed out and failed. The reason was an almost-out-of-memory situation in the software of one of the three SDN controllers. The problem was fixed at 11:26 CET on Jan 2nd 2019. Network connectivity of VMs were not affected by this issue. We apologise for any inconveniences.

SysEleven Stack network issues in region CBK

SysEleven Status — Wed, 19 Dec 2018 10:20:00 +0000

At the moment we are facing connectivitiy issues in region CBK on one compute node. We will keep you informed and apologise for any inconveniences. **Update 11:23:** The issue is fixed. Reason for outage: The SDN agent on one of our compute nodes crashed because of memory pressure. We are investigating the reason for the memory pressure.

SysEleven Stack Neutron API issues in Region DBL

SysEleven Status — Wed, 05 Dec 2018 12:30:00 +0000

We are currently seeing a slightly elevated rate of errors in the Neutron API. Our engineers are currently looking into them and will update this status page post with more information soon. We apologise for any inconveniences. **UPDATE 14:30:** We determined the impact. The errors are impacting a very small number of attempts to launch VMs. So far only one customer is affected. We are now looking for a solution. **UPDATE 15:30:** The issue also affects Neutron API performance. If you are experience any troubles, please contact cloudsupport@syseleven.de **UPDATE 16:15:** The API performance issues are now resolved. **UPDATE 17:00:** The API issues are fully resolved now.

SysEleven Stack issues in region DBL

SysEleven Status — Tue, 04 Dec 2018 09:55:00 +0000

Some Instances in DBL were unreachable between 10:55 and 11:03. **Update 11:05**: The problem appears to be currently resolved but we are still investigating to understand the root cause. **Update 11:45**: All Instances in our DBL region were affected by a loss of network. The outage was caused by a maintenance command issued on one isolated and empty compute node that unexpectedly affected all hypervisors in the whole cluster. It raised the log level which led to our SDN agents to be overwhelmed by traffic. We will conduct a post-mortem to find ways to prevent that error from happening again.

SysEleven Stack issues in region CBK

SysEleven Status — Tue, 27 Nov 2018 10:00:00 +0000

From approximately 11h to 14h we had issues with some instances in region CBK being unavailable for up to ten minutes due to unexpected impact of a hypervisor update. We have stopped those updates and investigate how to do them without such impact.

SysEleven Stack API issues in Region CBK and DBL

SysEleven Status — Mon, 12 Nov 2018 13:45:00 +0000

Currently we are facing issues with the SysEleven Stack API. We are working on a solution and keep you informed. The outage only affects spawning new virtual machines or changing existing resources. We apologise for any inconveniences. --- **Update: 15:00** The problem was found and could be fixed immediately.

SysEleven Stack maintenance announcement

SysEleven Status — Tue, 06 Nov 2018 13:14:07 +0000

--- **Between November 20th, 2018 14:00 and 16:00 (UTC +1)** we will perform planned maintenance in **Region DBL** --- We plan to update components of the software-defined networking layer. During the maintenance, the following degradations in service will occur: - The network connectivity of the virtual machines will have multiple interruptions of up to 45 seconds. This will affect all virtual machines on a compute node at the same time. We will try to avoid concurrent connectivity loss of multiple compute nodes. - access to the API/Dashboard will **not** be affected - SEOS (S3) will **not** be affected - storage availability will **not** be affected --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack Block Storage issues in region CBK

SysEleven Status — Tue, 06 Nov 2018 12:42:00 +0000

At the moment we are facing issues with the distributed file system, a core component of the SysEleven Stack. At the moment it is possible that virtaul machines can't be spawned or have no write access to their volumes(disks). We will keep you informed and apologise for any inconveniences. --- **Update: 14:22** We identified a suspicious hardware node that slowed down the overall cluster performance. We are working to isolate that node and bring back cluster write performance. **Update: 14:23** We isolated the affected hardware and performance is back to normal. If you experience any problems please contact us.

SysEleven Stack issues in region DBL

SysEleven Status — Thu, 25 Oct 2018 01:04:00 +0000

At the moment we are facing connectivitiy issues in region DBL. We will keep you informed and apologise for any inconveniences. **UPDATE 4:14:** Any network connectivity issues are solved now. Reason for outage was a faulty line card.

SysEleven Stack block storage issues in regions CBK and DBL

SysEleven Status — Mon, 15 Oct 2018 10:05:00 +0000

Today we were facing issues with the distributed file system, a core component of the SysEleven Stack. **Affected times for the region CBK** 13:12 – 13:17 (MESZ) 14:32 – 14:37 (MESZ) **Affected times for the region DBL** 12:05 – 12:15 (MESZ) 12:51 – 13:00 (MESZ) 13:50 – 14:00 (MESZ) During these times it was possible that virtual machines did not respond due to IO wait. It might also have affected creating virtual machines and cinder volumes. We will keep you informed and apologise for any inconveniences. **UPDATE 16:20** We identified that regular maintenance work on storage nodes (reboots) have indirectly caused this incident. The maintenance work is stopped until we find and resolve the root cause in close collaboration with the storage software vendor. We will keep you informed on the results.

SysEleven Stack API issues in Region DBL

SysEleven Status — Tue, 02 Oct 2018 11:40:00 +0000

Currently you may experience troubles with the network API . Spawning and deleting instances or other network resources is not possible. If you experience any unexpected behaviour don't hesitate to contact us. We apologise for any inconveniences. **UPDATE 22:45** We have identified and resolved the networking API troubles. We found a bad entry in the network state database of our SDN software, that caused a cascading failure: Internal locks could not be acquired anymore, which caused the API errors, which in turn caused networking troubles during live migrations in a small number of virtual machines. We now understood the root cause and will post more information on actions we will take to prevent this from happening again once we conducted the post-mortem. **UPDATE 15. October** We are currently working the following action points: * We identified and are working to resolve the root cause in our SDN software * We will improve the monitoring of our networking APIs, to identify and correlate future troubles earlier * We will improve monitoring of virtual machine port binding of our SDN software, so we can identify and fix troubled VMs faster in the future

SysEleven Stack API issues in Region DBL

SysEleven Status — Mon, 01 Oct 2018 11:30:00 +0000

Currently we are facing issues with the SysEleven Stack API. We are working on a solution and keep you informed. The outage only affects changing network resources, as well as spawning new virtual machines. Existing virtual machines are not affected. Please contact us, should you have any troubles or need any assistance. We apologise for any inconveniences. **UPDATE 16:15:** We found out that a number of virtual machines lost their network connectivity due to the API issue. We are trying to bring them back online as soon as possible. Impacted customers have been informed by e-mail. **UPDATE 17:25:** We resolved the API issues and connectivity issues for virtual machines. All customers that have been affected by connectivity issues should have been informed by e-mail. Please contact us should you still have issues or need assistance.

SysEleven Managed Hosting Network issues in region BKI1

SysEleven Status — Fri, 21 Sep 2018 12:15:00 +0000

At the moment it is possible that virtual machines face issues with network connectivity. We are working on a solution. We will keep you informed and apologise for any inconveniences. **Update 21/09/2018 14:40** Most systems are accessible and we are working on some remaining problems. **Update 21/09/2018 15:20** Systems experienced another downtime of 10 minutes and are coming online now.

SysEleven Stack emergency maintenance

SysEleven Status — Fri, 14 Sep 2018 08:55:00 +0000

--- **Between September 14th, 2018 11:15 AM and 12:00 noon (UTC +2)** we will perform emergency maintenance in **both Regions** --- We urgently fix the problem that caused the incident yesterday evening to prevent the issue from reoccurring. There should be no impact, however, there is a little risk that similar problems occour one last time. - storage availability **may be** affected - SEOS (S3) availability **may be** affected - virtual machines **may be** affected indirectly by "freezes" accessing their storage - network availability will **not** be affected but services might hang while waiting for storage - the API will **not** be affected --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack issues in both regions

SysEleven Status — Thu, 13 Sep 2018 19:00:00 +0000

Around 21:00 CEST we had an incident which affected storage of SysEleven Stack instances in both regions. The issue disappeared and we are still investigating cause and effect. Affected customers are encouraged to open support tickets with affected resources, obeservations and timestamps, if available. Update 10:40 CEST: The cause has been identified and we are planning an emergency maintenance to fix it to avoid the problem to reoccur. Update 11:15 CEST: We are fixing the root cause in an emergency maintenance. Update 12:00 CEST: The root cause has been fixed. To our knowledge there were some glitches in region DBL, but everything went smooth in region CBK.

SysEleven Stack storage maintenance announcement

SysEleven Status — Wed, 29 Aug 2018 09:29:00 +0000

--- **Between August 29th, 2018 11:30 AM and 17:00 PM (UTC +2)** we will perform planned maintenance in **Region DBL** --- We expect temporary performance impact and marginal impact on some virtual machines due to necessary live-migrations. - virtual machines **may be** affected by short "freezes" (<0.5sec) due to live migration - state of virtual machines will **not** be affected - network availability will **not** be affected - storage availability will **not** be affected - storage performance **can be** temporarily degraded - SEOS (S3) availability will **not** be affected - SEOS (S3) performance **can be** temporarily degraded - the API will **not** be affected --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack network maintainance announcement

SysEleven Status — Fri, 24 Aug 2018 12:35:22 +0000

--- **Between August 27th, 2018 07:00 AM and 08:00 PM (UTC +2)** we will perform planned maintenance in **Region CBK** --- We expect up to three short network interruptions (of less than 15 seconds each) per virtual machine. - network availability **will be** affected (see above) - state of virtual machines will **not** be affected - storage availability will **not** be affected --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack API issues in Region DBL

SysEleven Status — Thu, 19 Jul 2018 08:00:00 +0000

Currently it may not be possible to create Cinder Volumes using the SysEleven Stack API. We are working on a solution and keep you informed. The outage only affects creating new cinder volumes. We apologise for any inconveniences. **UPDATE 16:00:** The Cinder API should be functioning normally again. Please contact cloudsupport@syseleven.de if you are still experiencing issues.

SysEleven Stack Block Storage issues in region DBL

SysEleven Status — Tue, 17 Jul 2018 10:15:00 +0000

At the moment we are facing issues with the distributed file system, a core component of the SysEleven Stack. At the moment it is possible that virtaul machines can't be spawned or have no write access to their volumes(disks). We are working hard to resolving the incident and will keep you informed and apologise for any inconveniences. **Update 12:34:** We fixed the issues and are now investigating the root cause.

SysEleven Stack API issues in all regions

SysEleven Status — Mon, 09 Jul 2018 09:45:00 +0000

SysEleven Stack API issues in Region DBL

SysEleven Status — Wed, 04 Jul 2018 12:31:00 +0000

Currently we are facing issues with the SysEleven Stack API. We are working on a solution and keep you informed. The outage only affects spawning new virtual machines or changing existing resources. We apologize for any inconveniences. **Update 14:44** The API is available again. The reason for the short unavailability was due to a software bug.

SysEleven Stack Object Storage issues in region CBK

SysEleven Status — Tue, 03 Jul 2018 16:30:00 +0000

At the moment we are facing issues with the Object Storage in Region CBK. At the moment it is possible that objects can't be read or written. We will keep you informed and apologise for any inconveniences. **UPDATE 19:54:** We fixed the issues. Every service is up and running again.

SysEleven Stack Compute issues in Region CBK

SysEleven Status — Tue, 03 Jul 2018 12:50:00 +0000

Currently we are facing issues with the SysEleven Stack Compute service in Region CBK. Thus it is not possible to create new instances at the moment. Also it is not possible to create new Volumes in this region. Running VMs are not affected. We apologise for any inconviniences. **UPDATE 16:00:** We fixed the issues and are now investigating the root cause.

Cinder API issues in the DBL region

SysEleven Status — Tue, 03 Jul 2018 09:45:00 +0000

We are currently facing issues with the cinder volume API. Only creating, updating or attaching cinder volumes in the DBL region is affected. Running VMs with cinder volumes are NOT affacted. **UPDATE 12:15:** We solved the issues and are watching the systems closely. **UPDATE 13:00:** The issues are solved and we are working on a root cause analysis.

Maintenance work in our storage infrastructure in Region CBK

SysEleven Status — Tue, 03 Jul 2018 08:18:00 +0000

**Between July 3rd, 2018 10:00 AM and 18:00 PM (UTC +1)** we will perform planned maintenance in **Region CBK** --- We do **not** expect impact on running virtual machines or the S3 service. --- Should you have any trouble following the maintenance work please contact us.

Release maintenance announcement - SysEleven Managed Kubernetes (Kubermatic)

SysEleven Status — Thu, 07 Jun 2018 15:27:15 +0000

**Between June 11th, 2018 2:00 PM (UTC +1) and 2:30 PM (UTC +1)** we will perform planned maintenance --- we would like to inform you about the change of your login details from your GitHub to SysElevenStack credentials on 11.06.2018 at 2:00 PM UTC+1. After the change the registration will be easier and more secure, because you can log in the [K8s-Dashboard](https://k8s.01.syseleven.de/) using your SysElevenStack account. --- **Please note:** Your K8s cluster will not be visible in the frontend after the change, nevertheless, the clusters will still be available and reachable via API. --- **Action required:** Please send us your GitHub and SysElevenStack e-mail address so that we can migrate your cluster and SSH-keys in order to make them visible again.

SysEleven Stack API issues in region DBL

SysEleven Status — Wed, 06 Jun 2018 13:26:00 +0000

Currently we are facing issues with the SysEleven Stack API. We are working on a solution. - The outage only affects spawning new virtual machines or changing existing ressources. --- **Update: 16:59** The issue is solved, the API is up and responding properly again. The problem was also caused by a network fabric failure. We apologise for any inconveniences.

SysEleven Stack Block and Object Storage issues in region DBL

SysEleven Status — Wed, 06 Jun 2018 12:18:00 +0000

At the moment we are facing issues with the distributed file system, a core component of the SysEleven Stack. * Virtul machines might have trouble spawning or have no access to their volumes (disks) * SEOS/S3 access might be slow or interrupted --- **Update: 14:34** We identified the cause and are working on a fix. **Update: 14:46** The issue is solved, everything is up and running properly again. The problem was caused by a network fabric failure. --- We will keep you informed and apologise for any inconveniences.

SysEleven Stack maintainance announcement

SysEleven Status — Tue, 05 Jun 2018 13:01:05 +0000

--- **Between June 5th, 2018 10:15 (CET) and 11:45 (CET)** we will perform planned maintenance on the SysEleven Stack * Some instances may become unreachable for a few seconds during this timeframe * The short interruption of instances network connectivity will take place randomly * The instances will keep running; the unavailability is only related to the network component --- Should you have any trouble following the maintenance work please contact us.

SysEleven Stack API issues

SysEleven Status — Fri, 18 May 2018 08:20:00 +0000

Currently we are facing an outage inside the SysEleven Stack cluster components. We are working on a solution and will keep you informed. - Affected are the regions DBL and CBK. Since the API is non functional also the Dashboard service is affected. - Spawning new virtual machines is not possible - Running instances or other OpenStack resources are not affected. **Update 10:30** - Any issues are resolved now and every component of the SysEleven Stack is working again. We apologise for any inconviniences.

SysEleven Stack maintainance announcement

SysEleven Status — Tue, 15 May 2018 13:28:10 +0000

--- **Between May 23, 2018 14:00 (CET) and May 23, 2018 15:00 (CET)** we will perform planned maintenance on the SysEleven Stack. Our plan is to have no customer-facing impact, but there is a possibility that the API will not be available in both regions - leading to a short loss of control. --- During this period the API and Dashboard will not be available in both regions. Existing resources like virtual machines will not be affected in any way. SEOS (S3) will not be affected in any way.

Wartungsarbeiten an redundanter Stromversorgung

SysEleven Status — Mon, 14 May 2018 12:17:00 +0000

**Update** Gegen 5:00 Uhr wurden die Wartungsarbeiten erfolgreich und ohne Unterbrechung abgeschlossen. **Ankündigung** Am Dienstag, den 15.05.2018 zwischen 00:00 und 06:00 Uhr finden in unserem Rechenzentrum Wartungsarbeiten an der redundanten Stromversorgung eines Serverschranks statt. In diesem Zuge wird die Redundanz der Stromversorgung der in Server in diesem Rack unterbrochen werden müssen, was jedoch keinen Einfluss auf die Verfügbarkeit Ihrer Dienste haben wird. Vorsorglich haben wir alle direkt betroffenen Kunden direkt kontaktiert und informiert.

SysEleven PVC Störung einer Hardware Node

SysEleven Status — Fri, 11 May 2018 07:50:00 +0000

**Update 10:45** Es sind keine weiteren Auffälligkeiten festgestellt worden. Damit beenden wir den Vorfall. **Update 10:27** Alle Systeme arbeiten wieder wie gewohnt. Wir haben die Services geprüft und konnten keine weiteren Auffälligkeiten feststellen. **Update 10:18** Die Betroffene Hardware startete wieder wie gewohnt. Derzeit werden noch alle Gastsysteme gestartet und überprüft. **Update 10:10** Betroffene Hardware wird neugestartet **Ankündigung 09:50** Unsere Überwachung hat eine Hardware Node identifiziert die in einem Zustand ist, der außerhalb der normalen Parameter liegt. Die entsprechenden Kunden wurde per E-Mail kontaktiert und das weitere Vorgehen wird mit ihnen ggf. individuell abgestimmt. Sollten Sie Probleme identifiziert haben, aber nicht per E-Mail kontaktiert sein, wenden Sie sich bitte an support@syseleven.de oder rufen Sie uns an.

SysEleven Datacenter BKI - NFS partial outage

SysEleven Status — Wed, 09 May 2018 17:25:00 +0000

**Update 20:20 Uhr** All systems are working in the expected parameters. We apologize for any inconvenience. **Update 19:55 Uhr** We fixed the problem and monitor all systems. **Announcement 19:25** We are currently facing a service failure in our central storage appliance in our datacenter Berlin BKI. This leads to problems in access to NFS shares. We are working on a solution with high pressure.