We will keep you informed and apologise for any inconveniences.
Update 2020-12-04 00:40: The storage system recovered. We are still watching.
Update 2020-12-04 01:35: The problem is solved.
]]>Update: The service is back online
We will keep you informed and apologise for any inconveniences.
]]>The problem is identified and we work on a fix.
Update 15:08 : We fixed the issue accordingly
We apologise for any inconveniences.
]]>Between December 2nd, 2020 00:00 AM and 07:00 AM (UTC +1) we will perform planned maintenance in Region CBK
We expect two short network interruptions up to one minute and degradation in storage performance.
Should you have any trouble following the maintenance work please contact us.
]]>We apologise for any inconveniences.
Update 16:30 : We identified the issue and fixed it accordingly
]]>Between November 19th, 2020 00:00 AM and 07:00 AM (UTC +1) we will perform planned maintenance in Region CBK
We will perform a planned maintenance on our network storage system.
Should you have any trouble following the maintenance work please contact us.
]]>The problem was identified. We are currently observing.
The issues were fixed.
We will keep you informed and apologise for any inconveniences.
]]>Between November 10th, 2020 00:00 AM and 07:00 AM (UTC +1) we will perform planned maintenance in Region DBL
We will perform a planned maintenance on our network storage system.
Should you have any trouble following the maintenance work please contact us.
]]>Performance of virtual machines using M1 flavors or cinder volumes might be degraded.
UPDATE 15:45: The issues are now resolved.
If you are still experiencing issues, please contact us at cloudsupport@syseleven.de
]]>We will keep you informed and apologise for any inconveniences.
UPDATE 03:00 CET: The issues are mostly resolved now. There might be still some performance issues left. We are working hard to resolve those too.
UPDATE 04:20 CET: The performance issues are now resolved.
If you are still experiencing issues, please contact us at cloudsupport@syseleven.de
]]>We apologise for any inconveniences.
UPDATE 11:57: The issues are now resolved. If you are still experiencing issues, please contact us at cloudsupport@syseleven.de
]]>Between November 05, 2020 23:00 and 00:00 (UTC +1) we will perform planned maintenance in Region CBK
We expect short network interruptions of up to 1 minute.
Should you have any trouble following the maintenance work please contact us.
]]>Between November 02, 2020 23:00 and 00:00 (UTC +1) we will perform planned maintenance in Region DBL
UPDATE: The appointment for the maintenance has been moved to Nov. 2nd
We expect short network interruptions of up to 1 minute.
Should you have any trouble following the maintenance work please contact us.
]]>Update 09:55: Service has been recovered.
]]>We will keep you informed and apologise for any inconveniences.
Update 12:05: The node could not be recovered, all affected VMs are gone and will be removed from your list.
]]>We apologise for any inconveniences.
]]>We will keep you informed and apologize for any inconveniences.
Update: 18:48 The issues were identified and fixed
]]>We apologise for any inconveniences.
Update: We solved the problem with the Nova API
]]>We will perform a planned maintenance on our network storage system.
Should you have any trouble following the maintenance work please contact us.
]]>We will keep you informed and apologise for any inconveniences.
Update 17:40 : We identified the root cause and fixed the problem, the APIs are working again.
]]>Between September 17, 2020 00:00 and 01:00 AM (UTC +2) we will perform planned maintenance in Region DBL
We expect short network interruptions of up to 15 minutes.
Should you have any trouble following the maintenance work please contact us.
]]>During a check we noticed a config error on a router interface. Contrary to expectations, the correction of this error was associated with a service impact on the route Berlin <-> Frankfurt. The reason for this is that RSVP/MPLS on the interface in question was set to a faulty state by the Change. This must be corrected as soon as possible. Therefore we plan for tonight (starting from 0 o'clock the 11.09.) appropriate maintenance work to correct this.
-> The issue on our route Berlin <-> Frankfurt has been resolved
]]>For urgent requests or to report an incident, please also contact our telephone hotline
Support Hotline: +49 30 233 20 12 30 (Mo-Fr 9-18 hrs) Emergency Hotline: +49 30 609 8922 11 (to report incidents outside business hours)
[Update 09/09/2020 15:30]
Zendesk seems to recover. We assume that email updates to tickets have been delivered during the downtime. Please anyway check if your requests have been acknowledged by logging in to https://helpdesk.syseleven.de and checking your incoming mail.
]]>The outage affects connectivity of virtual machines and load balancers.
We apologise for any inconveniences.
UPDATE 00:45: The issues are now resolved.
If you are still experiencing issues, please contact us at cloudsupport@syseleven.de
]]>Between September 3, 2020 00:00 and 01:00 AM (UTC +2) we will perform planned maintenance in Region CBK
We expect short network interruptions (of less than 1 minute each).
Should you have any trouble following the maintenance work please contact us.
]]>We apologise for any inconveniences.
]]>We will keep you informed and apologise for any inconveniences.
UPDATE 14:45 We were able to resolve the problem. If you are still experiencing issues, please contact us at cloudsupport@syseleven.de
]]>We will keep you informed and apologise for any inconveniences.
]]>Between August 21th, 2020 00:00 AM and 02:00 AM CEST we will perform planned maintenance in Region CBK
During that maintenance window the API in region CBK will not be available, leading to a loss of control.
During this period there will be:
Should you have any trouble following the maintenance work please contact us.
]]>Between August 19th, 2020 00:00 AM and 02:00 AM CEST we will perform planned maintenance in Region DBL
During that maintenance window the API in region DBL will not be available, leading to a loss of control.
During this period there will be:
Should you have any trouble following the maintenance work please contact us.
]]>We will keep you informed and apologise for any inconveniences.
]]>We apologise for any inconveniences.
UPDATE 10:05: The issues are now resolved. If you are still experiencing a problem, please contact us at cloudsupport@syseleven.de
]]>Between July 24th, 2020 00:00 AM and 07:00 AM (CEST) we will perform planned maintenance in Region DBL
During this interval we are going to perform maintenance on our storage and network systems.
During this period there will be:
Should you have any trouble following the maintenance work please contact us.
]]>We start our maintenance work at 20.07.2020 08:00 am.
]]>We apologise for any inconveniences.
UPDATE 11:20 CEST: The API functionality is mostly restored. We are still investigating one issue. If you are experiencing problems, please contact us at cloudsupport@syseleven.de
UPDATE 12:00 CEST: We are working on issues with controller database cluster. The API issues are still ongoing. We will keep you updated here.
UPDATE 16:00 CEST: We announced an emergency maintenance window on 2020-07-17 0:00–3:00 CEST. We will perform important work on our controller database cluster to restore the API reliability. We sent a notification to affected customers. If you have any questions, please don't hesitate to contact us.
UPDATE 12:15 CEST: The maintenance from 2020-07-17, 0:00-3:00 (CEST) was successful. The services are available again.
If you have questions or encounter problems please do not hesitate to contact us.
]]>We apologise for any inconveniences.
UPDATE 09:55 CEST: The issues are now resolved.
]]>We will keep you informed and apologise for any inconveniences.
UPDATE 11:45: We were able to resolve the problem. If you are still experiencing issues, please contact us at cloudsupport@syseleven.de
]]>Should you have any trouble with DNS resolution using the SysEleven DNS Resolvers following the maintenance work, please contact us at support@syseleven.de.
]]>We start our maintenance work at 06.07.2020 08:00 am.
]]>Between July 3rd, 2020 0:00 and 1:00 AM (UTC +2) we will perform planned maintenance in Region DBL
We will update the network software (SDN) and expect interruptions of network connectivity of up to 30 seconds per virtual machine.
Should you have any trouble following the maintenance work please contact us.
]]>Between July 2nd, 2020 0:00 and 1:00 AM (UTC +2) we will perform planned maintenance in Region CBK
We will update the network software (SDN) and expect interruptions of network connectivity of up to 30 seconds per virtual machine.
Should you have any trouble following the maintenance work please contact us.
]]>Servers are now being restored by re-balancing power connections. Afterwards we will analyse the PDU/Power event.
Please contact us via e-mail or hotline if you encountered problems but did not receive any mail.
Update: 99% of impact has been fixed/workaround-ed. We are recovering remaining systems. Due the power outage we lost one last server completely. The rest is back online.
]]>We apologise for any inconveniences.
Update 14:34 The SysEleven Stack API is fully operational again
]]>We start our maintenance work at 22.06.2020 08:00 am.
]]>We apologise for any inconveniences.
Update 11:40: The API is available again. The issues are now resolved.
]]>It was unable to create new volumes. Existing attached volumes were working without any problems.
]]>Update: quay.io is available again and all affected customer clusters have recovered.
]]>Between April 29th, 2020 0:00 and 1:00 AM (UTC +2) we will perform planned maintenance in Region DBL
We expect a short network interruption of about 30 seconds.
Should you have any trouble following the maintenance work please contact us.
]]>Between April 28th, 2020 0:00 and 1:00 AM (UTC +2) we will perform planned maintenance in Region CBK
We expect a short network interruption of about 30 seconds.
Should you have any trouble following the maintenance work please contact us.
]]>The issues seem to be resolved now. We are investigating the issue and will improve our infrastructure.
We will keep you informed and apologise for any inconveniences.
]]>UPDATE 11:30: We resolved issues. We are aware of the root cause and will work to resolve it. If you still experience any issues, please contact us at cloudsupport@syseleven.de
UPDATE 11:52: We are facing new issues with designate DNS. We continue to investigate.
UPDATE 13:13: We could solve all issues.
We will keep you informed and apologise for any inconveniences.
]]>We also noticed that a number of VMs are in the ERROR state right now. Some of them lost their network connectivity. We are sending separate notifications for these customers.
UPDATE 12:22: We notified affected customers, where this affected the availability of virtual machines. The virtual machines should be back online. The API continues to be degraded. We are still investigating that.
UPDATE 14:30: We resolved the API issues. We are aware of the root cause and will work to resolve it. If you still experience any issues, please contact us at cloudsupport@syseleven.de
We will keep you informed and apologise for any inconveniences.
]]>Between April 1st, 2020 00:00 and 05:00 AM (UTC +2) we will perform planned maintenance in Region DBL
In order to provide new features and improve stability we will upgrade all OpenStack components in DBL to "Queens".
There will be the following service degradations during the maintenance window:
Network connectivity of the VMs and S3 are not affected during the update. The CBK region will not be affected.
Should you have any trouble following the maintenance work please contact us.
]]>We will keep you informed and apologise for any inconveniences.
UPDATE 16:30: The issue is resolved since 16:00.
]]>A small number of requests to our S3 object storage (SEOS) service in the region DBL might progress slowly or time out.
We will keep you informed and apologise for any inconveniences.
UPDATE 10:00: The issues are resolved.
]]>We will keep you informed and apologise for any inconveniences.
UPDATE 4:30: The performance issues are now resolved.
]]>We apologise for any inconveniences.
]]>We apologise for any inconveniences.
]]>Between March 6th 2020 00:00 AM and 02:00 AM (UTC +1) we will perform planned maintenance in global region.
During this time there can be short interruptions of DNSaaS API (Designate). DNS resolving won't be affected.
Should you have any trouble following the maintenance work please contact us.
]]>Between February 26, 2020 00:00 and 05:00 AM (UTC +1) we will perform planned maintenance in Region CBK
In order to provide new features and improve stability we will upgrade all Openstack components in region CBK to "Queens".
During this period there will be:
Should you have any trouble following the maintenance work please contact us.
]]>We apologise for any inconveniences.
Update 15:45: The issues are resolved and the DNS API is working again.
]]>We apologise for any inconveniences.
Update 13:45: The API is currently available again. Temporary performance degradations may reoccur during the afternoon, while we are fixing the problem.
Update 16:30: We expect no further performance degradations anymore.
]]>We will keep you informed and apologise for any inconveniences.
UPDATE 1:20: The issues are now fully resolved.
]]>Some virtual VMs might experience storage performance issues.
We will keep you informed and apologise for any inconveniences.
EDIT 22:20: We are seeing some stuck IO in CBK. We are working hard to resolve the issues as quickly as possible.
EDIT 23:00: The issues are fully resolved now. We know the root cause and will implement measures to prevent this from happening again.
]]>Between February 7th, 2020 00:00 AM and 01:00 AM (UTC +1) we will perform planned maintenance in Region CBK
During this period there will be:
Should you have any trouble following the maintenance work please contact us.
]]>Between February 6th, 2020 00:00 AM and 03:00 AM (UTC +1) we will perform planned maintenance in Region DBL
During this period there will be:
Should you have any trouble following the maintenance work please contact us.
]]>We apologise for any inconveniences.
UPDATE 12:00: The issues are now resolved.
]]>We apologise for any inconveniences.
UPDATE 16:25: The issues are now resolved.
]]>We will keep you informed and apologise for any inconveniences.
Update 12:00h: problems are mitigated for the moment.
Update 13:30h: we consider this incident over now.
]]>Furthermore, there were performance issues affecting distributed storage of other virtual machines that had not to be migrated.
We apologise for any inconveniences.
]]>There was an issue with our DNS. The issue has been resolved, but it may take some time for the issues to fully resolve for everyone.
The outage only affects spawning new virtual machines or changing existing resources.
We apologise for any inconveniences.
UPDATE 15:00: The issues are fully resolved.
]]>Between Dec 19th, 2019 00:00 AM and 01:00 AM (UTC +1) we will perform planned maintenance in Region DBL
During the maintenance window we will update our core components of the underlying infrastructure in region DBL.
Availability of the following services will be reduced:
Should you have any trouble following the maintenance work please contact us.
]]>Between Dec 17th, 2019 00:00 AM and 01:00 AM (UTC +1) we will perform planned maintenance in Region CBK
During the maintenance window we will update our core components of the underlying infrastructure in region CBK.
Availability of the following services will be reduced:
Should you have any trouble following the maintenance work please contact us.
]]>During the maintenance, the following degradations in service will occur:
We apologize for any inconvenience this might cause .
]]>We will keep you informed and apologise for any inconveniences.
UPDATE 10:45 CET: The issues are now resolved.
]]>Between November 27th, 2019 00:00 AM and 06:00 AM (UTC +1) we will perform planned network hardware maintenance in Region DBL
Our plan is to have no customer-facing impact, but there is a possibility that reachability and the networking performance of virtual machines will be impacted for a short period of time within maintenance window.
Should you have any trouble following the maintenance work please contact us.
]]>We will keep you informed and apologise for any inconveniences.
Update 10:10: The issues are identified and resolved.
]]>During the maintenance window we will update our storage system in region DBL.
Our plan is to have no customer-facing impact, but there is a possibility that the IO performance will be degraded. No other services will be affected.
Should you have any trouble following the maintenance work please contact us.
]]>Between November 15th, 2019 00:00 AM and 02:00 AM (UTC +1) we will perform planned maintenance in Region CBK
Our plan is to have no customer-facing impact, but there is a possibility that the IO performance will be degraded. No other services will be affected.
Should you have any trouble following the maintenance work please contact us.
]]>We will restart the affected VMs running on this compute node and inform the affected when the VMs are running again.
We will keep you informed and apologise for any inconveniences.
]]>Between November 14th, 2019 00:00 AM and 06:00 AM (UTC +1) we will perform planned maintenance in Region CBK
Our plan is to have no customer-facing impact, but there is a possibility that reachability and the networking performance of virtual machines will be impacted for a short period of time within maintenance window.
Should you have any trouble following the maintenance work please contact us.
]]>Sehr geehrte Damen und Herren,
heute gegen 12:05 Uhr kam es zu massiven Paketverlusten auf einem unserer Transit Connections. Davon betroffen waren ca. 5(%) des SysEleven Traffics. Der betroffene Traffic wurde umgehend auf andere Verbindungen umgeleitet, so dass gegen 12:15 Uhr kein Paketverlust mehr zu beobachten war. Zur Klärung der Störung stehen wir bereits mit dem entsprechnden Transit Provider in Verbindung.
Bei Fragen und Problemen stehen wir Ihnen gerne zur Verfügung, Ihr SysEleven-Team
--
Dear Mrs. / Mr.,
today around 12:05 o'clock it came to massive package loss at one of our transit providers. About 5% of the SysEleven traffic was affected. We then redirected the affected traffic immediately to other links, so that around 12:15 o'clock no more packetloss appeared. We are already in contact with the corresponding transit provider to clarify the problem.
If you encounter any problems, do not hesitate to contact us, Your SysEleven-Team
]]>Between November 12th, 2019 0:00 and 1:00 AM (UTC +1) we will perform planned maintenance in Region DBL
During the maintenance window we will update our network service in the DBL region. We expect an interruption of network connectivity of 30 seconds.
Should you have any trouble following the maintenance work please contact us.
]]>Between November 5th, 2019 00:00 and 01:00 AM (UTC +1) we will perform planned maintenance in Region CBK
During the maintenance window we will update our network service in the CBK region. We expect an interruption of network connectivity of 30 seconds.
Should you have any trouble following the maintenance work please contact us.
]]>Update 16:43: The issues are identified and resolved.
]]>We are looking for a solution under high pressure.
We will keep you informed and apologise for any inconveniences.
Update 16:00: The issues are now resolved. Please contact cloudsupport@syseleven.de if you are still experiencing issues.
Update 16:20: We received reports from some customers that they are still experiencing network issues. We are investigating.
Update 17:00: We are still investigating connectivity issues for some VMs in the CBK region.
Update 17:13: We experienced another major downtime between 17:13 and 17:14. We are still investigating and working on a solution under high pressure.
Update 17:15: The issues are now resolved. Please contact cloudsupport@syseleven.de if you are still experiencing issues.
Update 17:41: The issues are identified and resolved. We will conduct a post-mortem and post a more detailed description of the problems until the end of the next week.
]]>This led to 30 second downtime for some VMs and VPN services in DBL.
Update 16:00: We have found the root cause and will make sure that this will not happen again.
]]>Update: The issue has been resolved.
]]>Between October 18th, 2019 00:00 AM and 01:00 AM (UTC+2) we will perform planned maintenance in Region DBL
During the maintenance window we will update our network service in region DBL.
Availability of the following services will be reduced:
Should you have any trouble following the maintenance work please contact us.
]]>Between October 16th, 2019 00:00 AM and 04:00 AM (UTC +2) we will perform planned maintenance in Region CBK
During the maintenance window we will update our storage system and a network service in region CBK.
Availability of the following services will be reduced:
Should you have any trouble following the maintenance work please contact us.
]]>The issues are resolved already. We are looking into what caused these issues.
Update 10:36: We identified an issue in our switching fabric. The issues were caused by the short time the network took to reconverge.
Update 10:42: We observed slow IO in our distributed storage layer. We are investigating.
Update 11:11: All systems are operational again. We are watching closely and continue investigating.
Update 11:28: We observed slow IO in our distributed storage layer. We are investigating.
Update 11:38: We identified the root cause in our storage layer. All issues are now resolved. We will conduct a post-mortem to make sure this will not happen again.
]]>Between September 19th, 2019 00:00 AM and 04:00 AM (UTC +2) we will perform planned maintenance in Region DBL
During the maintenance window we will update our storage system in region DBL.
There may be a degradation in storage performance.
Should you have any trouble following the maintenance work please contact us.
]]>We apologise for any inconveniences.
UPDATE 18:05: There was a problem with controller hardware. It is solved and we are working on a longterm solution.
Please contact us at cloudsupport@syseleven.de if you are still encountering any issues.
]]>Between September 6th, 2019 0:00 AM and 04:00 AM (UTC +2) we will perform planned maintenance in Region DBL
During the maintenance window we will update our storage system in region DBL.
There may be a degradation in storage performance.
Should you have any trouble following the maintenance work please contact us.
]]>Between September 3rd, 2019 00:00 AM and 01:00 AM (UTC +2) we will perform planned maintenance in Region DBL
We will update our network service and expect network interruptions of less than 30 seconds per virtual machine.
Should you have any trouble following the maintenance work please contact us.
]]>We apologise for any inconveniences.
UPDATE 12:55: There was a problem with our Designate Infrastructure. It is solved and we are working on a longterm solution.
Please contact us at cloudsupport@syseleven.de if you are still encountering any issues.
]]>Zabbix wird nun in der Version 4.2 betrieben.
Bei Darstellungsproblemen leeren sie bitte den Cache ihres Browsers für die Seite https://zabbix.syseleven.de/.
]]>Während der Wartung treten die folgenden Beeinträchtigungen im Betrieb auf:
Between August 23rd, 2019 00:00 AM and 01:00 AM (CEST) we will perform planned maintenance in Region CBK
During the maintenance window we will perform a configuration update on our networking components in our region CBK.
During the maintenance window the following services will not be accessible:
Should you have any trouble following the maintenance work please contact us.
]]>Between August 15th, 2019 0:00 AM and 04:00 AM (UTC +2) we will perform planned maintenance in Region DBL
During the maintenance window we will update our storage system and a network service in region DBL.
Availability of the following services will be reduced:
Should you have any trouble following the maintenance work please contact us.
]]>Between August 1st, 2019 0:00 AM and 02:00 PM (CEST) we perform planned maintenance in Region DBL
We expect temporary performance impact.
Should you have any trouble following the maintenance work please contact us.
]]>There might be troubles when creating new VPN connections. Existing connections should continue to work.
Please contact our support team at cloudsupport@syseleven.de if you are experiencing any issues.
Update 2019-07-26 14:35: Due to a misbehavior of the SDN software, scheduling of new VPN services was unreliable. Now the service is fully functional again.
Between July 17th, 2018 12:00 (noon) and 12:30 PM (CEST, UTC +2) we will perform emergency maintenance in Region CBK and Region DBL
During the maintenance window we will roll back the configuration change of the previous maintenance on our networking components in our region CBK and DBL due to unpredicted side effects for some customers.
During the maintenance window the following services will be affected:
Update 12:20: The emergency maintenance has been carried out as planned.
Should you have any trouble following the maintenance work please contact us.
]]>Between July 16, 2019 0:00 AM and 06:00 AM (CEST) we will perform a planned maintenance in Region CBK and Region DBL
During the maintenance window we will perform a configuration update on our networking components in our region CBK and DBL, and update our storage system only in the DBL region.
During the maintenance window the following services will not be accessible:
Should you have any trouble following the maintenance work please contact us.
]]>Between June 27, 2019 1:00 AM and 02:00 AM (CEST) we will perform a planned maintenance in Region CBK and Region DBL
During the maintenance window we will perform a configuration update on our networking components in our region CBK and DBL.
During the maintenance window the following services will not be accessible:
Should you have any trouble following the maintenance work please contact us.
]]>Between May 24, 2019 0:00 AM and 01:00 AM (UTC +2) we will perform planned maintenance in Region CBK
During the maintenance window we will update the SDN software in our DBL region in order to fix an issue with the management of LBaaS load balancer health monitors.
During the maintenance window the following services will not be accessible:
Because of that network components cannot be created or re-configured during this time.
Should you have any trouble following the maintenance work please contact us.
]]>Between May 23, 2019 0:00 AM and 01:00 AM (UTC +2) we will perform planned maintenance in Region DBL
During the maintenance window we will update the SDN software in our DBL region in order to fix an issue with the management of LBaaS load balancer health monitors.
During the maintenance window the following services will not be accessible:
Because of that network components cannot be created or re-configured during this time.
Should you have any trouble following the maintenance work please contact us.
]]>We are currently experiencing issues with the OpenStack infrastructure, MetaKube and some MetaKube clusters are running on. At the moment we are fixing the underlying issue and analyze which MetaKube clusters are affected and bring them back online again.
2019-04-17 09:22 am
The first effected Clusters are back online. Our MetaKube Engineering team is still working on a solution for every Cluster. If you mention any issues please feel free to contact our customer Support.
2019-04-17 10:04 am
Most of the effected Clusters are up and running in a normal stage. We are still working on this incident and come back with more information asap.
2019-04-17 10:32 am
Our monitoring and our personal checks show, that all customer clusters are up and running in a normale state. We close this incident and work on the root cause.
We will send out a RFO Report (Reason For Outage) as soon as we have identified all aspects of this issue.
If you still have problems with your cluster or identify unnormal behaviour in your cluster, please contact out customer Support.
]]>Update: We have temporarily disabled one of our upstream traffic partners who seemed to cause the issues. We expect no further impact on our reachability. We already have request the RFO (Reason for outage).
]]>Between April 17th, 2019 0:00 and 6:00 AM (UTC +2) we will perform planned maintenance in Region DBL
During this period we will update the storage system.
During the maintenance window the following services will be affected:
Should you have any trouble following the maintenance work please contact us.
]]>It is not possible to start, stop or create virtual machines. It is also not possible to create or delete cinder volumes.
Running VMs and the object storage (S3) are not affected.
We will keep you informed and apologise for any inconveniences.
UPDATE 05:45: We are still working on resolving these issues together with our technology partners.
UPDATE 06:25: All issues are now resolved.
]]>Between April 4th, 2019 0:00 and 5:00 AM (UTC +2) we will perform planned maintenance in Region CBK
During this period we will update the network and storage system.
During the maintenance window the following services will be affected:
Should you have any trouble following the maintenance work please contact us.
]]>We will keep you informed and apologise for any inconveniences.
Update 2019-03-06 17:12: The issue has been resolved. We applied a bug fix and now the Neutron Network API in region CBK is fully available again.
]]>Between March 7th, 2019 0:00 AM and 01:00 AM (UTC +1) we will perform planned maintenance in Region CBK
During this period the Cinder (Volume) API in the CBK region is not available - leading to a loss of control.
During this period there will be:
After the maintenance is over, the storage availability zone name will be changed from "nova" to "cbk1". No Heat stack or Terraform state changes are required, if the availability zone was not explicitly specified.
Should you have any trouble following the maintenance work please contact us.
]]>We have noticed, that when creating a new Cinder Volume, it will be in "error" state immediately.
Cinder volumes that have been created before already are not affected.
We are working on a solution and keep you informed.
We apologise for any inconveniences.
UPDATE 16:56: The issues have been resolved. We are now investigating the root cause.
]]>Engineers are now looking into the problem and we will keep you informed here.
UPDATE 15:45: Creating L1.* virtual machines currently requires unusual amounts of disk space on the hypervisors. The elevated error rate was because not enough space was left, for short amounts of time. Our engineers have resolved this issue for now, and are still working on a permanent solution.
Please contact us at cloudsupport@syseleven.de if you are still encountering any issues.
]]>We will keep you informed and apologise for any inconveniences.
]]>Between February 27th 2019 00:00 AM and 05:00 AM (UTC +1) we will perform planned maintenance in Region CBK
During this period the API in the CBK region is not available - leading to a loss of control.
During this period there will be:
Should you have any trouble following the maintenance work please contact us.
]]>The outage only affects spawning new virtual machines or changing existing resources.
We apologise for any inconveniences.
UPDATE 00:30:
We have identified the cause of the API issues in DBL and resolved the issues for most API operations.
Due to a bug in OpenStack API components, the DBL API services used CBK identity URLs for validating tokens, which did not work due to the planned maintenance.
We changed all internal identity URLs in the endpoint registry to region-agnostic URLs.
We are now working on fully resolving this issue for all API operations in DBL.
UPDATE 00:50: All issues are resolved.
]]>Should you have any trouble following the maintenance work please contact us.
]]>Should you have any trouble following the maintenance work please contact us.
]]>Should you have any trouble following the maintenance work please contact us.
]]>This is due to a mitigation we put in place that solves sporadic performance problems, when a virtual machine has been created or deleted. We are right now working on a better mitigation, that will not affect creating servers.
The issue only affects spawning new virtual machines with the L1.* flavor (Local SSD Storage).
We apologise for any inconveniences.
UPDATE 2019-02-13: The issues are fully resolved now. Creating and deleting Local SSD Storage instances will only take a few seconds now, and the time for creating and deleting VMs is constant, no matter what disk capacity.
]]>In order to offer the same features in both regions we will roll out new versions of our networking components in the CBK region.
The maintenance window is planned for 2019-02-13 from 00:00-05:00 (CET)
During the maintenance window we will update the networking components in the CBK region.
There will be the following service degradations within the maintenance window:
The DBL region will not be affected.
UPDATE 01:55: The maintenance has been completed.
Should you have any trouble following the maintenance work please contact us.
]]>Between January 23rd, 2018 00:00 AM and 06:00 AM (UTC +1) we will perform planned storage maintenance in Region DBL
During the maintenance window we will update the distributed storage software in our DBL region in order to fix a memory leak.
It is possible that IO requests are slow during the maintenance window.
Affected services:
Not affected:
UPDATE 04:32 AM (CET): The maintenance has been completed successfully.
Should you have any trouble following the maintenance work please contact us at cloudsupport@syseleven.de
]]>We created a oauth service with dex to authenticate our web applications against. Dex writes all of the session information into the etcd cluster. Due to a misconfigured monitoring we created over 500000 sessions in the etcd cluster. The database size grow from about 300MB to 700MB. That slowed down all cluster operations. During the whole incident customer applications were not affected, the MetaKube dashboard including cluster creation were regularly not working correctly when one of the master apiservers ran out of memory.
]]>Between January 17th, 2019 07:00 AM and 08:00 AM (UTC +1) we will perform planned maintenance in Region DBL
During the maintenance window we will update the SDN software in our DBL region in order to fix a memory leak.
During the maintenance window the following services will not be accessible:
The following services will not be affected:
Should you have any trouble following the maintenance work please contact us.
]]>Creating new VMs results in an error since 11:30 AM CET.
We are investigating.
We will keep you informed and apologise for any inconveniences.
UPDATE 11:53 CET: We have resolved the issue, and isolated the root cause.
]]>Network connectivity of VMs were not affected by this issue.
We apologise for any inconveniences.
]]>We will keep you informed and apologise for any inconveniences.
Update 11:23: The issue is fixed. Reason for outage: The SDN agent on one of our compute nodes crashed because of memory pressure. We are investigating the reason for the memory pressure.
]]>Our engineers are currently looking into them and will update this status page post with more information soon.
We apologise for any inconveniences.
UPDATE 14:30: We determined the impact. The errors are impacting a very small number of attempts to launch VMs. So far only one customer is affected. We are now looking for a solution.
UPDATE 15:30: The issue also affects Neutron API performance. If you are experience any troubles, please contact cloudsupport@syseleven.de
UPDATE 16:15: The API performance issues are now resolved.
UPDATE 17:00: The API issues are fully resolved now.
]]>Update 11:05: The problem appears to be currently resolved but we are still investigating to understand the root cause.
Update 11:45: All Instances in our DBL region were affected by a loss of network.
The outage was caused by a maintenance command issued on one isolated and empty compute node that unexpectedly affected all hypervisors in the whole cluster. It raised the log level which led to our SDN agents to be overwhelmed by traffic.
We will conduct a post-mortem to find ways to prevent that error from happening again.
]]>We apologise for any inconveniences.
Update: 15:00 The problem was found and could be fixed immediately.
]]>Between November 20th, 2018 14:00 and 16:00 (UTC +1) we will perform planned maintenance in Region DBL
We plan to update components of the software-defined networking layer.
During the maintenance, the following degradations in service will occur:
The network connectivity of the virtual machines will have multiple interruptions of up to 45 seconds. This will affect all virtual machines on a compute node at the same time. We will try to avoid concurrent connectivity loss of multiple compute nodes.
access to the API/Dashboard will not be affected
SEOS (S3) will not be affected
storage availability will not be affected
Should you have any trouble following the maintenance work please contact us.
]]>We will keep you informed and apologise for any inconveniences.
Update: 14:22
We identified a suspicious hardware node that slowed down the overall cluster performance. We are working to isolate that node and bring back cluster write performance.
Update: 14:23
We isolated the affected hardware and performance is back to normal. If you experience any problems please contact us.
We will keep you informed and apologise for any inconveniences.
UPDATE 4:14: Any network connectivity issues are solved now. Reason for outage was a faulty line card.
]]>Affected times for the region CBK
13:12 – 13:17 (MESZ)
14:32 – 14:37 (MESZ)
Affected times for the region DBL
12:05 – 12:15 (MESZ)
12:51 – 13:00 (MESZ)
13:50 – 14:00 (MESZ)
During these times it was possible that virtual machines did not respond due to IO wait. It might also have affected creating virtual machines and cinder volumes.
We will keep you informed and apologise for any inconveniences.
UPDATE 16:20
We identified that regular maintenance work on storage nodes (reboots) have indirectly caused this incident. The maintenance work is stopped until we find and resolve the root cause in close collaboration with the storage software vendor.
We will keep you informed on the results.
]]>We apologise for any inconveniences.
UPDATE 22:45
We have identified and resolved the networking API troubles.
We found a bad entry in the network state database of our SDN software, that caused a cascading failure: Internal locks could not be acquired anymore, which caused the API errors, which in turn caused networking troubles during live migrations in a small number of virtual machines.
We now understood the root cause and will post more information on actions we will take to prevent this from happening again once we conducted the post-mortem.
UPDATE 15. October
We are currently working the following action points:
Existing virtual machines are not affected.
Please contact us, should you have any troubles or need any assistance.
We apologise for any inconveniences.
UPDATE 16:15:
We found out that a number of virtual machines lost their network connectivity due to the API issue. We are trying to bring them back online as soon as possible. Impacted customers have been informed by e-mail.
UPDATE 17:25:
We resolved the API issues and connectivity issues for virtual machines.
All customers that have been affected by connectivity issues should have been informed by e-mail.
Please contact us should you still have issues or need assistance.
]]>Update 21/09/2018 14:40
Most systems are accessible and we are working on some remaining problems.
Update 21/09/2018 15:20
Systems experienced another downtime of 10 minutes and are coming online now.
Between September 14th, 2018 11:15 AM and 12:00 noon (UTC +2) we will perform emergency maintenance in both Regions
We urgently fix the problem that caused the incident yesterday evening to prevent the issue from reoccurring. There should be no impact, however, there is a little risk that similar problems occour one last time.
Should you have any trouble following the maintenance work please contact us.
]]>Update 10:40 CEST: The cause has been identified and we are planning an emergency maintenance to fix it to avoid the problem to reoccur.
Update 11:15 CEST: We are fixing the root cause in an emergency maintenance.
Update 12:00 CEST: The root cause has been fixed. To our knowledge there were some glitches in region DBL, but everything went smooth in region CBK.
]]>Between August 29th, 2018 11:30 AM and 17:00 PM (UTC +2) we will perform planned maintenance in Region DBL
We expect temporary performance impact and marginal impact on some virtual machines due to necessary live-migrations.
Should you have any trouble following the maintenance work please contact us.
]]>Between August 27th, 2018 07:00 AM and 08:00 PM (UTC +2) we will perform planned maintenance in Region CBK
We expect up to three short network interruptions (of less than 15 seconds each) per virtual machine.
Should you have any trouble following the maintenance work please contact us.
]]>We are working on a solution and keep you informed.
The outage only affects creating new cinder volumes.
We apologise for any inconveniences.
UPDATE 16:00: The Cinder API should be functioning normally again. Please contact cloudsupport@syseleven.de if you are still experiencing issues.
]]>We are working hard to resolving the incident and will keep you informed and apologise for any inconveniences.
Update 12:34:
We fixed the issues and are now investigating the root cause.
We apologize for any inconveniences.
Update: 12:20
We resolved all issues.
We apologize for any inconveniences.
Update 14:44
The API is available again. The reason for the short unavailability was due to a software bug.
We will keep you informed and apologise for any inconveniences.
UPDATE 19:54: We fixed the issues. Every service is up and running again.
]]>Thus it is not possible to create new instances at the moment. Also it is not possible to create new Volumes in this region.
Running VMs are not affected.
We apologise for any inconviniences.
UPDATE 16:00:
We fixed the issues and are now investigating the root cause.
Only creating, updating or attaching cinder volumes in the DBL region is affected.
Running VMs with cinder volumes are NOT affacted.
UPDATE 12:15:
We solved the issues and are watching the systems closely.
UPDATE 13:00:
The issues are solved and we are working on a root cause analysis.
We do not expect impact on running virtual machines or the S3 service.
Should you have any trouble following the maintenance work please contact us.
]]>we would like to inform you about the change of your login details from your GitHub to SysElevenStack credentials on 11.06.2018 at 2:00 PM UTC+1. After the change the registration will be easier and more secure, because you can log in the K8s-Dashboard using your SysElevenStack account.
Please note: Your K8s cluster will not be visible in the frontend after the change, nevertheless, the clusters will still be available and reachable via API.
Action required: Please send us your GitHub and SysElevenStack e-mail address so that we can migrate your cluster and SSH-keys in order to make them visible again.
]]>Update: 16:59
The issue is solved, the API is up and responding properly again.
The problem was also caused by a network fabric failure.
We apologise for any inconveniences.
]]>Update: 14:34
We identified the cause and are working on a fix.
Update: 14:46
The issue is solved, everything is up and running properly again.
The problem was caused by a network fabric failure.
We will keep you informed and apologise for any inconveniences.
]]>Between June 5th, 2018 10:15 (CET) and 11:45 (CET) we will perform planned maintenance on the SysEleven Stack
Should you have any trouble following the maintenance work please contact us.
]]>Update 10:30
We apologise for any inconviniences.
]]>Between May 23, 2018 14:00 (CET) and May 23, 2018 15:00 (CET) we will perform planned maintenance on the SysEleven Stack. Our plan is to have no customer-facing impact, but there is a possibility that the API will not be available in both regions - leading to a short loss of control.
During this period the API and Dashboard will not be available in both regions.
Existing resources like virtual machines will not be affected in any way. SEOS (S3) will not be affected in any way.
]]>Ankündigung
Am Dienstag, den 15.05.2018 zwischen 00:00 und 06:00 Uhr finden in unserem Rechenzentrum Wartungsarbeiten an der redundanten Stromversorgung eines Serverschranks statt.
In diesem Zuge wird die Redundanz der Stromversorgung der in Server in diesem Rack unterbrochen werden müssen, was jedoch keinen Einfluss auf die Verfügbarkeit Ihrer Dienste haben wird.
Vorsorglich haben wir alle direkt betroffenen Kunden direkt kontaktiert und informiert.
]]>Update 10:27
Alle Systeme arbeiten wieder wie gewohnt. Wir haben die Services geprüft und konnten keine weiteren Auffälligkeiten feststellen.
Update 10:18
Die Betroffene Hardware startete wieder wie gewohnt. Derzeit werden noch alle Gastsysteme gestartet und überprüft.
Update 10:10
Betroffene Hardware wird neugestartet
Ankündigung 09:50
Unsere Überwachung hat eine Hardware Node identifiziert die in einem Zustand ist, der außerhalb der normalen Parameter liegt.
Die entsprechenden Kunden wurde per E-Mail kontaktiert und das weitere Vorgehen wird mit ihnen ggf. individuell abgestimmt.
Sollten Sie Probleme identifiziert haben, aber nicht per E-Mail kontaktiert sein, wenden Sie sich bitte an support@syseleven.de oder rufen Sie uns an.
Update 19:55 Uhr
We fixed the problem and monitor all systems.
Announcement 19:25
We are currently facing a service failure in our central storage appliance in our datacenter Berlin BKI. This leads to problems in access to NFS shares.
We are working on a solution with high pressure.