We are currently experiencing issues with the OpenStack infrastructure, MetaKube and some MetaKube clusters are running on. At the moment we are fixing the underlying issue and analyze which MetaKube clusters are affected and bring them back online again.
2019-04-17 09:22 am
The first effected Clusters are back online. Our MetaKube Engineering team is still working on a solution for every Cluster. If you mention any issues please feel free to contact our customer Support.
2019-04-17 10:04 am
Most of the effected Clusters are up and running in a normal stage. We are still working on this incident and come back with more information asap.
2019-04-17 10:32 am
Our monitoring and our personal checks show, that all customer clusters are up and running in a normale state. We close this incident and work on the root cause.
We will send out a RFO Report (Reason For Outage) as soon as we have identified all aspects of this issue.
If you still have problems with your cluster or identify unnormal behaviour in your cluster, please contact out customer Support.
]]>Update: We have temporarily disabled one of our upstream traffic partners who seemed to cause the issues. We expect no further impact on our reachability. We already have request the RFO (Reason for outage).
]]>Between April 17th, 2019 0:00 and 6:00 AM (UTC +2) we will perform planned maintenance in Region DBL
During this period we will update the storage system.
During the maintenance window the following services will be affected:
Should you have any trouble following the maintenance work please contact us.
]]>It is not possible to start, stop or create virtual machines. It is also not possible to create or delete cinder volumes.
Running VMs and the object storage (S3) are not affected.
We will keep you informed and apologise for any inconveniences.
UPDATE 05:45: We are still working on resolving these issues together with our technology partners.
UPDATE 06:25: All issues are now resolved.
]]>Between April 4th, 2019 0:00 and 5:00 AM (UTC +2) we will perform planned maintenance in Region CBK
During this period we will update the network and storage system.
During the maintenance window the following services will be affected:
Should you have any trouble following the maintenance work please contact us.
]]>We will keep you informed and apologise for any inconveniences.
Update 2019-03-06 17:12: The issue has been resolved. We applied a bug fix and now the Neutron Network API in region CBK is fully available again.
]]>Between March 7th, 2019 0:00 AM and 01:00 AM (UTC +1) we will perform planned maintenance in Region CBK
During this period the Cinder (Volume) API in the CBK region is not available - leading to a loss of control.
During this period there will be:
After the maintenance is over, the storage availability zone name will be changed from "nova" to "cbk1". No Heat stack or Terraform state changes are required, if the availability zone was not explicitly specified.
Should you have any trouble following the maintenance work please contact us.
]]>We have noticed, that when creating a new Cinder Volume, it will be in "error" state immediately.
Cinder volumes that have been created before already are not affected.
We are working on a solution and keep you informed.
We apologise for any inconveniences.
UPDATE 16:56: The issues have been resolved. We are now investigating the root cause.
]]>Engineers are now looking into the problem and we will keep you informed here.
UPDATE 15:45: Creating L1.* virtual machines currently requires unusual amounts of disk space on the hypervisors. The elevated error rate was because not enough space was left, for short amounts of time. Our engineers have resolved this issue for now, and are still working on a permanent solution.
Please contact us at cloudsupport@syseleven.de if you are still encountering any issues.
]]>We will keep you informed and apologise for any inconveniences.
]]>Between February 27th 2019 00:00 AM and 05:00 AM (UTC +1) we will perform planned maintenance in Region CBK
During this period the API in the CBK region is not available - leading to a loss of control.
During this period there will be:
Should you have any trouble following the maintenance work please contact us.
]]>The outage only affects spawning new virtual machines or changing existing resources.
We apologise for any inconveniences.
UPDATE 00:30:
We have identified the cause of the API issues in DBL and resolved the issues for most API operations.
Due to a bug in OpenStack API components, the DBL API services used CBK identity URLs for validating tokens, which did not work due to the planned maintenance.
We changed all internal identity URLs in the endpoint registry to region-agnostic URLs.
We are now working on fully resolving this issue for all API operations in DBL.
UPDATE 00:50: All issues are resolved.
]]>Should you have any trouble following the maintenance work please contact us.
]]>Should you have any trouble following the maintenance work please contact us.
]]>Should you have any trouble following the maintenance work please contact us.
]]>This is due to a mitigation we put in place that solves sporadic performance problems, when a virtual machine has been created or deleted. We are right now working on a better mitigation, that will not affect creating servers.
The issue only affects spawning new virtual machines with the L1.* flavor (Local SSD Storage).
We apologise for any inconveniences.
UPDATE 2019-02-13: The issues are fully resolved now. Creating and deleting Local SSD Storage instances will only take a few seconds now, and the time for creating and deleting VMs is constant, no matter what disk capacity.
]]>In order to offer the same features in both regions we will roll out new versions of our networking components in the CBK region.
The maintenance window is planned for 2019-02-13 from 00:00-05:00 (CET)
During the maintenance window we will update the networking components in the CBK region.
There will be the following service degradations within the maintenance window:
The DBL region will not be affected.
UPDATE 01:55: The maintenance has been completed.
Should you have any trouble following the maintenance work please contact us.
]]>Between January 23rd, 2018 00:00 AM and 06:00 AM (UTC +1) we will perform planned storage maintenance in Region DBL
During the maintenance window we will update the distributed storage software in our DBL region in order to fix a memory leak.
It is possible that IO requests are slow during the maintenance window.
Affected services:
Not affected:
UPDATE 04:32 AM (CET): The maintenance has been completed successfully.
Should you have any trouble following the maintenance work please contact us at cloudsupport@syseleven.de
]]>We created a oauth service with dex to authenticate our web applications against. Dex writes all of the session information into the etcd cluster. Due to a misconfigured monitoring we created over 500000 sessions in the etcd cluster. The database size grow from about 300MB to 700MB. That slowed down all cluster operations. During the whole incident customer applications were not affected, the MetaKube dashboard including cluster creation were regularly not working correctly when one of the master apiservers ran out of memory.
]]>Between January 17th, 2019 07:00 AM and 08:00 AM (UTC +1) we will perform planned maintenance in Region DBL
During the maintenance window we will update the SDN software in our DBL region in order to fix a memory leak.
During the maintenance window the following services will not be accessible:
The following services will not be affected:
Should you have any trouble following the maintenance work please contact us.
]]>Creating new VMs results in an error since 11:30 AM CET.
We are investigating.
We will keep you informed and apologise for any inconveniences.
UPDATE 11:53 CET: We have resolved the issue, and isolated the root cause.
]]>Network connectivity of VMs were not affected by this issue.
We apologise for any inconveniences.
]]>We will keep you informed and apologise for any inconveniences.
Update 11:23: The issue is fixed. Reason for outage: The SDN agent on one of our compute nodes crashed because of memory pressure. We are investigating the reason for the memory pressure.
]]>Our engineers are currently looking into them and will update this status page post with more information soon.
We apologise for any inconveniences.
UPDATE 14:30: We determined the impact. The errors are impacting a very small number of attempts to launch VMs. So far only one customer is affected. We are now looking for a solution.
UPDATE 15:30: The issue also affects Neutron API performance. If you are experience any troubles, please contact cloudsupport@syseleven.de
UPDATE 16:15: The API performance issues are now resolved.
UPDATE 17:00: The API issues are fully resolved now.
]]>Update 11:05: The problem appears to be currently resolved but we are still investigating to understand the root cause.
Update 11:45: All Instances in our DBL region were affected by a loss of network.
The outage was caused by a maintenance command issued on one isolated and empty compute node that unexpectedly affected all hypervisors in the whole cluster. It raised the log level which led to our SDN agents to be overwhelmed by traffic.
We will conduct a post-mortem to find ways to prevent that error from happening again.
]]>We apologise for any inconveniences.
Update: 15:00 The problem was found and could be fixed immediately.
]]>Between November 20th, 2018 14:00 and 16:00 (UTC +1) we will perform planned maintenance in Region DBL
We plan to update components of the software-defined networking layer.
During the maintenance, the following degradations in service will occur:
The network connectivity of the virtual machines will have multiple interruptions of up to 45 seconds. This will affect all virtual machines on a compute node at the same time. We will try to avoid concurrent connectivity loss of multiple compute nodes.
access to the API/Dashboard will not be affected
SEOS (S3) will not be affected
storage availability will not be affected
Should you have any trouble following the maintenance work please contact us.
]]>We will keep you informed and apologise for any inconveniences.
Update: 14:22
We identified a suspicious hardware node that slowed down the overall cluster performance. We are working to isolate that node and bring back cluster write performance.
Update: 14:23
We isolated the affected hardware and performance is back to normal. If you experience any problems please contact us.
We will keep you informed and apologise for any inconveniences.
UPDATE 4:14: Any network connectivity issues are solved now. Reason for outage was a faulty line card.
]]>Affected times for the region CBK
13:12 – 13:17 (MESZ)
14:32 – 14:37 (MESZ)
Affected times for the region DBL
12:05 – 12:15 (MESZ)
12:51 – 13:00 (MESZ)
13:50 – 14:00 (MESZ)
During these times it was possible that virtual machines did not respond due to IO wait. It might also have affected creating virtual machines and cinder volumes.
We will keep you informed and apologise for any inconveniences.
UPDATE 16:20
We identified that regular maintenance work on storage nodes (reboots) have indirectly caused this incident. The maintenance work is stopped until we find and resolve the root cause in close collaboration with the storage software vendor.
We will keep you informed on the results.
]]>We apologise for any inconveniences.
UPDATE 22:45
We have identified and resolved the networking API troubles.
We found a bad entry in the network state database of our SDN software, that caused a cascading failure: Internal locks could not be acquired anymore, which caused the API errors, which in turn caused networking troubles during live migrations in a small number of virtual machines.
We now understood the root cause and will post more information on actions we will take to prevent this from happening again once we conducted the post-mortem.
UPDATE 15. October
We are currently working the following action points:
Existing virtual machines are not affected.
Please contact us, should you have any troubles or need any assistance.
We apologise for any inconveniences.
UPDATE 16:15:
We found out that a number of virtual machines lost their network connectivity due to the API issue. We are trying to bring them back online as soon as possible. Impacted customers have been informed by e-mail.
UPDATE 17:25:
We resolved the API issues and connectivity issues for virtual machines.
All customers that have been affected by connectivity issues should have been informed by e-mail.
Please contact us should you still have issues or need assistance.
]]>Update 21/09/2018 14:40
Most systems are accessible and we are working on some remaining problems.
Update 21/09/2018 15:20
Systems experienced another downtime of 10 minutes and are coming online now.
Between September 14th, 2018 11:15 AM and 12:00 noon (UTC +2) we will perform emergency maintenance in both Regions
We urgently fix the problem that caused the incident yesterday evening to prevent the issue from reoccurring. There should be no impact, however, there is a little risk that similar problems occour one last time.
Should you have any trouble following the maintenance work please contact us.
]]>Update 10:40 CEST: The cause has been identified and we are planning an emergency maintenance to fix it to avoid the problem to reoccur.
Update 11:15 CEST: We are fixing the root cause in an emergency maintenance.
Update 12:00 CEST: The root cause has been fixed. To our knowledge there were some glitches in region DBL, but everything went smooth in region CBK.
]]>Between August 29th, 2018 11:30 AM and 17:00 PM (UTC +2) we will perform planned maintenance in Region DBL
We expect temporary performance impact and marginal impact on some virtual machines due to necessary live-migrations.
Should you have any trouble following the maintenance work please contact us.
]]>Between August 27th, 2018 07:00 AM and 08:00 PM (UTC +2) we will perform planned maintenance in Region CBK
We expect up to three short network interruptions (of less than 15 seconds each) per virtual machine.
Should you have any trouble following the maintenance work please contact us.
]]>We are working on a solution and keep you informed.
The outage only affects creating new cinder volumes.
We apologise for any inconveniences.
UPDATE 16:00: The Cinder API should be functioning normally again. Please contact cloudsupport@syseleven.de if you are still experiencing issues.
]]>We are working hard to resolving the incident and will keep you informed and apologise for any inconveniences.
Update 12:34:
We fixed the issues and are now investigating the root cause.
We apologize for any inconveniences.
Update: 12:20
We resolved all issues.
We apologize for any inconveniences.
Update 14:44
The API is available again. The reason for the short unavailability was due to a software bug.
We will keep you informed and apologise for any inconveniences.
UPDATE 19:54: We fixed the issues. Every service is up and running again.
]]>Thus it is not possible to create new instances at the moment. Also it is not possible to create new Volumes in this region.
Running VMs are not affected.
We apologise for any inconviniences.
UPDATE 16:00:
We fixed the issues and are now investigating the root cause.
Only creating, updating or attaching cinder volumes in the DBL region is affected.
Running VMs with cinder volumes are NOT affacted.
UPDATE 12:15:
We solved the issues and are watching the systems closely.
UPDATE 13:00:
The issues are solved and we are working on a root cause analysis.
We do not expect impact on running virtual machines or the S3 service.
Should you have any trouble following the maintenance work please contact us.
]]>we would like to inform you about the change of your login details from your GitHub to SysElevenStack credentials on 11.06.2018 at 2:00 PM UTC+1. After the change the registration will be easier and more secure, because you can log in the K8s-Dashboard using your SysElevenStack account.
Please note: Your K8s cluster will not be visible in the frontend after the change, nevertheless, the clusters will still be available and reachable via API.
Action required: Please send us your GitHub and SysElevenStack e-mail address so that we can migrate your cluster and SSH-keys in order to make them visible again.
]]>Update: 16:59
The issue is solved, the API is up and responding properly again.
The problem was also caused by a network fabric failure.
We apologise for any inconveniences.
]]>Update: 14:34
We identified the cause and are working on a fix.
Update: 14:46
The issue is solved, everything is up and running properly again.
The problem was caused by a network fabric failure.
We will keep you informed and apologise for any inconveniences.
]]>Between June 5th, 2018 10:15 (CET) and 11:45 (CET) we will perform planned maintenance on the SysEleven Stack
Should you have any trouble following the maintenance work please contact us.
]]>Update 10:30
We apologise for any inconviniences.
]]>Between May 23, 2018 14:00 (CET) and May 23, 2018 15:00 (CET) we will perform planned maintenance on the SysEleven Stack. Our plan is to have no customer-facing impact, but there is a possibility that the API will not be available in both regions - leading to a short loss of control.
During this period the API and Dashboard will not be available in both regions.
Existing resources like virtual machines will not be affected in any way. SEOS (S3) will not be affected in any way.
]]>Ankündigung
Am Dienstag, den 15.05.2018 zwischen 00:00 und 06:00 Uhr finden in unserem Rechenzentrum Wartungsarbeiten an der redundanten Stromversorgung eines Serverschranks statt.
In diesem Zuge wird die Redundanz der Stromversorgung der in Server in diesem Rack unterbrochen werden müssen, was jedoch keinen Einfluss auf die Verfügbarkeit Ihrer Dienste haben wird.
Vorsorglich haben wir alle direkt betroffenen Kunden direkt kontaktiert und informiert.
]]>Update 10:27
Alle Systeme arbeiten wieder wie gewohnt. Wir haben die Services geprüft und konnten keine weiteren Auffälligkeiten feststellen.
Update 10:18
Die Betroffene Hardware startete wieder wie gewohnt. Derzeit werden noch alle Gastsysteme gestartet und überprüft.
Update 10:10
Betroffene Hardware wird neugestartet
Ankündigung 09:50
Unsere Überwachung hat eine Hardware Node identifiziert die in einem Zustand ist, der außerhalb der normalen Parameter liegt.
Die entsprechenden Kunden wurde per E-Mail kontaktiert und das weitere Vorgehen wird mit ihnen ggf. individuell abgestimmt.
Sollten Sie Probleme identifiziert haben, aber nicht per E-Mail kontaktiert sein, wenden Sie sich bitte an support@syseleven.de oder rufen Sie uns an.
Update 19:55 Uhr
We fixed the problem and monitor all systems.
Announcement 19:25
We are currently facing a service failure in our central storage appliance in our datacenter Berlin BKI. This leads to problems in access to NFS shares.
We are working on a solution with high pressure.