SysEleven Status Page

Frankfurt1 Outage

2018-04-09T19:37:00+00:00

Sehr geehrte Damen und Herren,

Um ca. 21:37 Uhr meldeten unsere Monitoringsysteme den Ausfall des Standorts Frankfurt. In Folge dessen kam es für einige Minuten zu Beeinträchtigungen des Traffics der über diesen Standort abgewickelt wurde. Ein überwiegender Teil der Dienste ist nach automatischen Re-Routing nun wieder ohne Probleme erreichbar. Wir befinden uns noch im Troubleshooting gehen aber aktuell von einem Stromausfall aus.

Sobald wir weitere Erkenntnisse haben werden wir die Site https://www.syseleven-status.net/ entsprechend updaten.

Bei Fragen und Problemen stehen wir Ihnen gerne zur Verfügung, Ihr SysEleven-Team

Update:
Mittlerweile hat der DataCenter Betreiber den Stromausfall bestätigt und arbeitet mit Hochdruck an der Behebung.

Update: 10.04.2018 - 00:07 Uhr
Wir stehen im ständigen Kontakt mit dem RZ Betreiber um Status Updates einzuholen. Aktuell gehen wir von einem längeren Ausfall aus der sich mindestens bis spät in die Nacht ziehen wird. Ein verbindlicher ETF ("Estimated Time to Fix") liegt noch nicht vor.

Update: 10.04.2018 - 01:58 Uhr
Der Stromversorgung in Frankfurt wurde wiederhergestellt. Wir überprüfen nun den Standort und beobachten den langsam anlaufenden Betrieb.

Update: 10.04.2018 - 02:17 Uhr
Wir haben nun die Bestätigung erhalten das ein Teil der Stromversorgung wiederhergestellt wurde. Der Standort ist wieder erreichbar und SysEleven hat entsprechend Zugriff auf alle Devices Vorort. Der Incident kann jedoch noch nicht als beendet erklärt werden da immer noch die Root Cause Analyse läuft. Erneute Ausfälle können zum aktuellen Zeitpunkt nicht ausgeschlossen werden.

Update: 10.04.2018 - 09:52 Uhr
Gegen 04:30 Uhr kam es zu einem erneuten Ausfall der Stromversorgung - diese wurde gegen 05:45 Uhr wiederhergstellt. Seit dem ist der Standort wieder erreichbar und alle Services verfügbar. Laut dem DataCenter Betreiber ist die Stromversorgung wieder stabil.
Wir beobachten den Standort weiterhin.

Update: 11.04.2018 - 09:30 Uhr
Seit dem letzten Update kam es zu keinen weiteren Ausfällen der Stromversorung. Der DataCenter Betreiber hat bestätigt, dass die Stromversorung inkl. Redundanzen wieder voll hergestellt ist. An der Analyse der Hauptursache wird derzeit seitens des DataCenter Betreibers gearbeitet.

NFS Wartungsarbeiten im Rechenzentrum BKI - Dienstag, 10.04.2018 10:00 bis 16:00 Uhr

2018-04-04T14:37:50+00:00

--- German Version (English below) ---

am Dienstag, 10.04.2018 zwischen 10:00 und 16:00 Uhr werden wir unaufschiebbare Wartungsarbeiten an unserer NFS Cluster im Rechenzentrum BKI durchführen.

In der Zeit werden Techniker von uns vor Ort eine Node aus dem Cluster heraus nehmen - die anderen Nodes übernehmen die Aufgaben der entfernten Node selbstständig. An dieser Node muss der Austausch von RAM-Riegeln durchgeführt werden. Nach der überprüfung der Funktionalität durch unsere Techniker wird die Node wieder in den Verbund aufgenommen und nimmt selbstständig ihre Arbeit wieder auf.

Während dieser Zeit kann es zur Performanceengpässen kommen. Sobald die Arbeiten abgeschlossen sind, benachrichtigen wir Sie gesondert dazu.

Wir bitten um Ihr Verständnis.

--- English Version ---

on Tuesday, 10.04.2018 between 10:00 and 16:00 o'clock we will perform urgent maintenance work on our NFS cluster in the data center BKI.

During this time, technicians from us will take a node out of the cluster - the other nodes will automatically take over the tasks of the removed node. RAM bars must be replaced on this node. After our technicians have checked the functionality, the node is rejoined the cluster and resumes its work again.

During this time, performance issues can occur. As soon as the work is completed, you will be notified separately.

We ask for your patience.

Bei Fragen stehen wir gerne zur Verfügung! Ihr SysEleven-Team

SysEleven Stack API/Dashboard

2018-03-29T14:00:00+00:00

Currently we are facing performance issues with the SysEleven Stack API in the region DBL. The CBK Region is unaffected. We are working on a solution and keep you informed.

The performance issues do not affect running systems.

We apologise for any inconviniences.

UPDATE 22:45 CEST:

Because of the neutron performance issues, creating or updating network resources may fail. Existing virtual machines and networks are NOT affected. Three of our engineers were constantly working on isolating the neutron performance issues during the last 7 hours.

UPDATE 11:05 CEST:

We identified the perormance issue. The Networking API is fully operational again. Creating and updating network resources is now working reliably.

SysEleven Stack API/Dashboard

2018-03-26T08:32:00+00:00

Currently we are facing performance issues with the SysEleven Stack API and Dashboard.
We are analyzing the root cause and keep you informed. The performance issues do not affect running systems.

We apologise for any inconviniences.

Update: 10:56
Launching stacks is possible but might be interrupted. The API is available and works.
We are still investigating and will keep you informed.

Update: 11:10
The performance issues have been fixed. The SysElevenStack is fully operational again.

SysEleven Stack maintainance window: 2018-02-22, 8:00-10:00 CET

2018-02-20T10:03:00+00:00

During the maintenance window the following services will be affected:

No access to the SysEleven Stack API
It won't be possible to start/stop virtual instances or control any other OpenStack resources

Running instances or other OpenStack ressources won't be affected during the maintenance window.

Update:
The maintenance tasks have been finished successfully. The authentification service has been changed successfully and is now ready for other OpenStack regions. After completion of the maintenance tasks every service is now available again.

SysEleven Stack partial storage outage

2018-02-06T09:06:00+00:00

Currently we are facing an outage inside the SysEleven Stack storage components. We are analyzing the root cause and keep you informed. We apologize for any inconveniences.

Update: 10:12
We have a partial storage outage and are working on that.
The outage affects only some running systems. We will inform customers accordingly.

Update: 12:15
Unfortunately the issues are still ongoing. We are debugging the storage issues with the storage vendor and will inform you of any changes ASAP.

Update: 12:50
The issues with the compute storage are solved. The cluster rebalancing is still ongoing. Therefore slow writes in the SEOS/S3 storage might occur.

Update: 15:00 The rebalancing finished. Write speed in the SEOS/S3 is back to normal.

Thank you for your patience!

SysEleven Stack Quobyte issues

2018-02-01T15:32:00+00:00

At the moment we are facing issues with the distributed file system, a core component of the SysEleven Stack. It is possible that some virtual machines can't be spawned or have no write access to their disks. We will keep you informed and apologise for any inconveniences.

17:15 Update:
We fixed the issue. Please contact us should you experience any further issues.

SysEleven Stack network issues

2018-01-25T14:30:00+00:00

At the moment we are facing issues within our networking stack. Virtual machines lost their networking connection. We are currently working on a solution.

Update 15:35: The API is affected as well. We are working on the machine connectivity with high priority.

Update 15:38: Network connectivity of the virtual machines is restored. We are now working on the API issues.

Update 16:13: The API is available again. We are investigating the root cause of the issue.

Störung einer Hardware Node

2018-01-23T11:19:00+00:00

Unsere Überwachung hat eine Hardware Node identifiziert die in einem Zustand ist, der außerhalb der normalen Parameter liegt. Die entsprechenden Kunden wurde per E-Mail kontaktiert und das weitere Vorgehen wird mit ihnen ggf. individuell abgestimmt. Sollten Sie Probleme identifiziert haben, aber nicht per E-Mail kontaktiert sein, wenden Sie sich bitte an support@syseleven.de oder rufen Sie uns an.

Update: 14:45 Uhr: Das Problem wurde behoben. Einzelne Kunden wurden Evakuiert. Weitere Kommunikation wird direkt mit den Kunden geführt.

Einspielen der Sicherheitsupdates gegen "Meltdown & Spectre"

2018-01-17T23:00:00+00:00

Wir werden in der Zeit von 18.01.2018 00:00 Uhr bis 18.01.2018 08:00 Uhr einen Teil unserer Infrastruktur mit aktuellen Sicherheitsupdates gegen die bekannten Sicherheitslücken "Meltdown & Spectre" ausstatten.

Es ist mit einer mehrminütigen Downtime einzelner Systeme zu rechnen.

Alle Kunden, die von dieser Maßnahme betroffen sind wurden separat per Mail informiert.

Einspielen der Sicherheitsupdates gegen "Meltdown & Spectre"

2018-01-16T23:00:00+00:00

Wir werden in der Zeit von 17.01.2018 00:00 Uhr bis 17.01.2018 08:00 Uhr einen Teil unserer Infrastruktur mit aktuellen Sicherheitsupdates gegen die bekannten Sicherheitslücken "Meltdown & Spectre" ausstatten.

Es ist mit einer mehrminütigen Downtime einzelner Systeme zu rechnen.

Alle Kunden, die von dieser Maßnahme betroffen sind wurden separat per Mail informiert.

SysEleven Stack Midonet outage

2018-01-16T09:57:00+00:00

At the moment we are facing issues with the software defined networking, a core component of the SysEleven Stack. It is possible that virtual machines have no working network connectivity. We will keep you informed and apologise for any inconveniences.

Update 11:02
The network connectivity is up and running again.

Einspielen der Sicherheitsupdates gegen "Meltdown & Spectre"

2018-01-15T23:00:00+00:00

Wir werden in der Zeit von 11.01.2018 00:00 Uhr bis 11.01.2018 08:00 Uhr einen Teil unserer Infrastruktur mit aktuellen Sicherheitsupdates gegen die bekannten Sicherheitslücken "Meltdown & Spectre" ausstatten.

Es ist mit einer mehrminütigen Downtime einzelner Systeme zu rechnen.

Alle Kunden, die von dieser Maßnahme betroffen sind wurden separat per Mail informiert.

Partial outage due to core router issues

2018-01-12T15:20:00+00:00

There is an incident with one of our Core Routers that impacts connectivity. We have identified the issue and are currently resolving it.

Update 16:30

Due to the convergence to other routes in the network, traffic starts to be routed normally again.

We are closely watching the issue currently while the network is converging.

Update 16:55

Full network connectivity is restored.

Reason for Outage

Due to a configuration change, the routing engine on one of the core routers in the SysEleven Datacenter Berlin 1 unintentionally restarted its routing daemon.

This lead to an drop of traffic from and to this router and impacted the overall connectivity of the network.

SysEleven Stack API outage

2018-01-10T13:55:00+00:00

At the moment the SysEleven Stack API and Horizon Dashboard are down, due to a database issue.

This means it is currently not possible to manage virtual machines from the OpenStack API or Horizon Dashboard.

Running virtual machines are not affected.

We are already working on resolving the issues and will keep you posted.

Update 15:15 CET

We have identified the cause and fixed the database issues. The API and Horizon dashboard are available again. If you have any questions or are facing any issues, please let us know.

Einspielung der Sicherheitspatches gegen "Meltdown & Spectre" auf nicht Live-Hardware

2018-01-08T12:00:00+00:00

Uns stehen erste Patches zur Verfügung die wir jetzt auf nicht Live Systeme einspielen. Wir beginnen ab ca. 13:00 Uhr mit den Wartungsarbeiten an den nicht Live Systemen. Ihre Systeme müssen für die Wartung einmal neu gestartet werden. Es ist mit einer Ausfallzeit von wenigen Minuten zu rechnen. Sobald die Wartung abgeschlossen ist, werden wir Sie darüber informieren.

Einspielen der Sicherheitsupdates gegen "Meltdown & Spectre"

2018-01-08T10:56:00+00:00

Es ist mit einer mehrminütigen Downtime einzelner Systeme zu rechnen.

Alle Kunden, die von dieser Maßnahme betroffen sind wurden separat per Mail informiert.

Notfallwarnung an Staging- und Dev Systemen zur Behebung der Sicherheitslücken Meltdown & Spectre

2018-01-05T10:39:57+00:00

Vorausgesetzt, dass es bis zum Montag den 08.01.2017 bereits Patches für die Genannten Sicherheitslücken Meltdown & Spectre gibt werden wir kurzfristig mit dem Einspielen dieser Patches auf Nicht Live Hardware beginnen.

Wir werden Sie noch ein mal gesondert, aber kurzfristig Informieren wann Ihre Systeme dran sind.

SysEleven PVC Evacuation Hardware Node

2017-12-14T10:40:00+00:00

Unsere Überwachung hat eine Hardware Node identifiziert die in einem Zustand ist, die eine umgehende Evakuierung nötig macht. Die entsprechenden Kunden wurde per E-Mail kontaktiert und das weitere Vorgehen wird mit ihnen ggf. individuell abgestimmt. Sollten Sie Probleme identifiziert haben, aber nicht per E-Mail kontaktiert sein, wenden Sie sich bitte an support@syseleven.de oder rufen Sie uns an.

Update 15:50 Uhr

Die Arbeiten an dem Problem wurden nach erfolgreicher Untersuchung der Ursache beendet. Alle Systeme arbeiten wieder wie gewohnt.

Einschränkung in der Bearbeitung von Änderungen am 15.12.2017 ab 15:00 Uhr

2017-12-13T15:13:32+00:00

Aufgrund einer internen Veranstaltung werden wir am Freitag, 15.12.2017, ab 15 Uhr Änderungsanfragen nur eingeschränkt bearbeiten können. Die Bearbeitung von Incidents ist sichergestellt.

F5 Load Balancer Maintenance - Software Update

2017-12-13T07:53:00+00:00

German Version

Zwecks Verbesserung und Optimierung unserer F5 Load Balancer im RZ Berlin Kitzingstrasse planen wir für Mittwochnacht (am 13.12.2017) zwischen 0:00 und 6:00 Uhr ein Update

Das Update ermöglicht einige neue Funktionen, u.a.:

Final support of HTTP2 spec.
Expanded cookie parsing
HTTP Strict Transport Security (HSTS) functionality
TCP Fast Open

Die vollständigen Releasenotes finden Sie hier: https://support.f5.com/kb/en-us/products/big-ip_ltm/releasenotes/product/relnote-ltm-12-0-0.html#rn_new

Wie üblich wird das Update nacheinander nur auf der jeweiligen standby Clusternode durchgeführt (Gefolgt von einem Clusterschwenk), um die Auswirkung auf Ihr Setup so gering wie möglich zu halten. Während der Arbeiten kann es jedoch mehrfach zu mehrminütigen Unterbrechungen der F5 Dienste kommen.

English Version

To enhance and optimize our F5 Load Balancers we're planning to perform a software update in our datacenter Berlin Kitzingstr. The update is planned on the 13th of December 2017 between 00:00 and 06:00 am.

Among others, the update includes the following enhancements:

Final support of HTTP2 spec.
Expanded cookie parsing
HTTP Strict Transport Security (HSTS) functionality
TCP Fast Open

You can find the complete release notes here: https://support.f5.com/kb/en-us/products/big-ip_ltm/releasenotes/product/relnote-ltm-12-0-0.html#rn_new

As usual we will perform the update on the standby clusternodes only (following by a manual cluster failover). It is possible that we will experience small outages for some minutes during the maintenance.

Abkündigung - F5 Load Balancer Maintenance - Software Update

2017-12-13T00:35:00+00:00

German Version

Das Update wurde erfolgreich ausgerollt, unser Monitoring hatte keine Einschränkungen in der Erreichbarkeit Ihrer Services registriert.

English Version

The update was successful, according to our monitoring none of your services was interrupted in any way.

DNS Issues

2017-12-07T17:00:00+00:00

The DNS related issues have been fixed. All systems are operating as desired again.

SysEleven Datacenter 1+2 NFS partial outage

2017-12-07T15:34:00+00:00

We have verified that all services with problems related to the NFS outage have recovered.

SysEleven Datacenter 1+2 NFS partial outage

2017-12-07T14:55:00+00:00

We are seeing widespread recovery throughout both Datacenters Berlin 1 and 2. We're looking into all services which are not fully recovered yet.

DNS Issues

2017-12-07T14:50:00+00:00

We are currently facing DNS issues for domains where our name servers are authoritative. We have identified the cause of the issue and are working on the resolution.

SysEleven Datacenter 1+2 NFS partial outage

2017-12-07T14:49:00+00:00

Remounting of NFS Shares in Datacenter Berlin 2 is finished, we are closely monitoring the recovery of related services.

SysEleven Datacenter 1+2 NFS partial outage

2017-12-07T14:35:00+00:00

Remounting of NFS Shares in Datacenter Berlin 1 is finished, we are closely monitoring the recovery of related services.

SysEleven Datacenter 1+2 NFS partial outage

2017-12-07T14:20:00+00:00

Our Engineers are remounting all NFS shares for the affected systems. When this is finished, all services should start to work as expected again.

SysEleven Datacenter 1+2 NFS partial outage

2017-12-07T13:40:00+00:00

We are currently facing a service impact regarding the central storage appliances in datacenters Berlin 1 and 2. This leads to problems in access to NFS shares.

SysEleven Stack: SDN network outage

2017-11-30T10:30:00+00:00

Our monitoring reported a network outage at 11:30 CET. This affected all virtual machines. The APIs and SEOS/S3 object storage services were not affected.

The outage coincided with a scheduled maintenance of one of our controller nodes, which under normal circumstances does not have any customer impact. We are now looking into the root cause to avoid this in the future.

Update 11:33:
The network is available again.

SysEleven Stack: SDN network outage

2017-11-16T13:45:00+00:00

Our monitoring reported a network outage between 14:43 and 14:48 CET. This affected all virtual machines. The APIs and S3 object storage services were not affected.

The outage coincided with a scheduled maintenance of one of our controller nodes, which under normal circumstances does not have any customer impact.

We are now looking into the root cause to avoid this in the future.

Update 15:15: We previously reported a 2 minutes outage. Our data tells us the outage actually was between 14:43 – 14:48 CET.

SysEleven Stack outage

2017-10-20T14:26:00+00:00

We are currently facing a degradation of the SysEleven Stack cluster. We found the root cause of the problem and are working on restoring the service.
The outage does only affect some running systems.

We apologise for any inconveniences.

Update - 17:15
The SDS (Storage Cluster) was degraded and is currently rebalancing to provide the data in proper redundancy.

Update - 21:03
Our storage cluster is fully functional again and affected virtual machines have been evacuated.

SysEleven Stack partial degradation of services

2017-09-26T08:00:00+00:00

We are currently facing a degradation of the SysEleven Stack cluster API. We are working on restoring the service.
The outage affects spawning new virtual machines as well as deleting or rebooting systems/stacks.
This does not affect running systems.

We apologise for any inconviniences.

Update: 11:55
Service is restored. We identified the cause and keep you informed.

SysEleven Stack NTP issues, may lead to storage issues

2017-09-12T09:00:00+00:00

At the moment we are looking into clock synchronization issues, which may lead to storage performance issues. We are very sorry for the inconvenience and will keep you informed here.

Update:
Any clock synchronisation problems are now solved. We are investigating why this happened. Storage latencies related to this problem should not occur since 10:13 UTC.

Issues with one of our compute hosts

2017-08-29T05:50:00+00:00

We encountered issues with one of our compute hosts. Some customers also reported hanging VMs. Currently we are in the process of evacuating the compute host. This can lead to a brief downtime of a small number of virtual machines. They are expected to be up and running again soon. Affected customers and affected VMs will be informed by email. After that we will be investigating why this happened.

Update 08:40
We evacuated any affected VM. Customers have been informed by email.
Customers facing ongoing problems should respond to the Helpdesk ticket that was created for them.

We apologize for any inconvenience.

Issues with some of our compute hosts

2017-08-26T06:16:00+00:00

Following a compute node failure we had shared storage mount crashes on 2 other compute nodes. Some virtual machines may have been unreachable or unreactive since 08:16 CEST.

Update 11:00
Service is restored for all VMs. If you are still experiencing issues, please contact cloudsupport@syseleven.de

Issues with one of our compute hosts

2017-08-25T08:30:00+00:00

We encountered issues with one of our compute hosts. Libvirtd, the software that controls the virtual machine processes, does not react to commands anymore. Some customers also reported hanging VMs.

We are currently in the process of evacuating the compute host. This can lead to a brief downtime of a small number of virtual machines. They are expected to be up and running again soon. Affected customers and affected VMs will be informed by email.

After that we will be investigating why this happened.

Update 11:35
We evacuated any affected VM. Customers have been informed by email.
Customers facing ongoing problems should respond to the Helpdesk ticket that was created for them.

We apologize for any inconvenience.

SysEleven Datacenter 1 NFS partial outage

2017-07-23T02:10:00+00:00

We are currently facing a service failure in our central storage appliance in our datacenter Berlin Lützowstraße (BLU1). This leads to problems in access to NFS shares. We are in contact with the supplier and are working on a solution with high pressure.

Update 11:30 The Problems seem to be mitigated for the moment, the cleanup, investigation and observance are still ongoing.

SysEleven Stack issues

2017-07-20T13:13:00+00:00

We implemented measures to avoid problems like this in the future. We sent a notifier with more detailed information to every customer.

Please contact us at cloudsupport@syseleven.de for further questions and feedback.

SysEleven Stack issues

2017-07-18T08:40:00+00:00

We identified and fixed the problem. All issues should be resolved now.

We are now working on the root cause analysis and will also share a brief version of it with you.

If you are still experiencing issues please contact cloudsupport@syseleven.de

SysEleven Stack issues

2017-07-18T08:30:00+00:00

We are experiencing I/O wait issues in our software defined storage component and are working hard to resolve them. Other issues like network timeouts seem to be connected to the I/O issues.

SysEleven Stack issues

2017-07-18T07:30:00+00:00

We are currently looking into different reports indicating issues with SysEleven Stack. As soon as we know more we will update the status here and get in touch with affected customers directly.

SysEleven Stack overlay network outage

2017-06-02T13:41:00+00:00

Update: We fixed the issue with our SDN, there should be no issues left.

SysEleven Stack API downtime

2017-05-23T09:50:42+00:00

We were facing issues with the SysEleven Stack API the last hour. This was caused by a partial DNS outage. No server instances where affected.

SysEleven Stack VM Storage issues

2017-05-16T16:15:00+00:00

We have isolated the cause of the issues and implemented a fix and are watching the system closely. All instances should be running again now.

It may still be the case that it won't come up again on its own, if we had to shut down the VM hard. Then there will be manual intervention from your side required.

If you have still stuck VMs then please contact us.

SysEleven Stack VM Storage issues

2017-05-16T12:00:00+00:00

We have identified a problem in our shared distributed storage layer that was causing some IO-Requests to get stuck. Because of this, some KVM-Processes of VMs locked up and got hung causing the VM-Instances to stall.

Our Engineers are currently in the process of bringing the affected VMs online again. This may lead to restarting single Instances if hotmigration doesn't work because of the failure.

We will keep you informed on this issue and want to apologise for the inconveniences.

SysEleven Stack VM issues

2017-05-16T09:00:00+00:00

At the moment we are facing issues with the distributed file system, a core component of the SysEleven Stack. At the moment it is possible that virtual machines can't be spawned, are hanging or have no write access to their disks. We will keep you informed and apologise for any inconveniences.

SysEleven Stack network issues

2017-04-26T12:27:00+00:00

We are still investigating the network connectivity / DHCP issue. Only a small number of VMs is affected.

We found a workaround to resolve this issue for affected VMs and are working closely with our vendors to find the root cause of this issue.

Please contact cloudsupport@syseleven.de if you are experiencing this.

SysEleven Stack network issues

2017-04-26T10:00:00+00:00

We are investigating another issue, where DHCP fails for a small number of VMs.

The previous issue we were working on is resolved.

SysEleven Stack network issues

2017-04-25T14:09:00+00:00

We identified the problem and fixed the affected machines.

We are now working on fixing the root cause.

SysEleven Stack network issues

2017-04-25T13:00:00+00:00

We are currently investigating issues with the network component. A small number of virtual machines failed to attach their virtual networking port. We will keep you informed and apologise for any inconveniences.

Storage problems

2017-04-11T18:44:00+00:00

We fixed the root cause of the storage problem. Newly created VMs will work as expected. VMs with filesystem issues are still affected, they need to be repaired manually. We will provide instructions on how to do that.

Storage problems

2017-04-11T12:15:00+00:00

We identified the cause of the filesystem corruptions inside of the VMs. We are looking at methods to restore the affected VMs and their filesystems.

From our current knowledge, the base operating system files are affected. Customer data should not be affected.

Storage problems

2017-04-10T20:00:00+00:00

We've found and resolved numerous issues in both our network and our storage infrastructure. We've also started a full diagnostic of the aforementioned storage infrastructure that will be running throughout the night, this diagnostic should be done running by tomorrow morning.

We'll continue our investigation into the root cause tomorrow.

Storage problems

2017-04-10T10:00:00+00:00

We are having issues with our storage infrastructure since Friday 07/04/2017. We are working closely with the vendor to identify and resolve these problems and we will notify you as soon as the situation changes.

partial storage outage

2017-04-07T13:17:00+00:00

We identified the problem. The outage started yesterday 14:59 (UTC). A small number of virtual instances ran into kernel panics. Rebooting these instances fixed the problems.

partial storage outage

2017-04-07T10:19:00+00:00

At the moment we are facing issues with the distributed file system, a core component of the SysEleven Stack. It is possible that virtual machines have no write or read access to their disks.

SysEleven Stack maintenance end

2017-03-21T10:24:00+00:00

We're happy to notify you that the maintenance has successfully concluded.

Now that the maintenance has been completed the following issues should be resolved:

The temporary loss of connectivity of virtual machines
The temporary loss of internet connectivity
Problems with MTU-size under specific circumstances

We'd like to thank you for your patience.

SyselEven Stack maintainance update

2017-03-21T09:34:00+00:00

Update: The scheduled network update runs as expected. We finished updating our outbound traffic components and also our compute nodes. The last component will be our controller infrastructure before we have finished the whole update of the SysEleven Stack.

SysEleven Stack maintenance start

2017-03-21T06:59:21+00:00

As previously announced, we're starting the maintenance on the SysEleven Stack now. Assuming everything goes according to plan, we should be done by 12:00 CET (11:00 UTC).

During the maintenance, the following degradations in service will occur:

The SysEleven Stack API and webinterface will be unavailable.
The network connectivity of the virtual machines will have multiple interruptions of up to 90 seconds. This will affect all virtual machines on a compute node at the same time. We will try to avoid concurrent connectivity loss of multiple compute nodes.

We apologize for any inconvenience this might cause and we will notify you once the maintenance has completed.

SysEleven Stack API Slowdown

2017-02-22T13:05:00+00:00

Currently we are fixing an Issue with the SysEleven Stack API. The current state is that HTTP-Requests take so long that they are running into an timeout. Any running instance is not affected, but creating or deleteting new infrastructure is might be affected.

Issues with our network

2016-12-29T08:15:00+00:00

Update: Problem is fixed

At the moment we have network issues which could remain in higher packet-loss. We are working on the problem and will inform you as soon as it's been resolved.

SysEleven Stack maintainance update

2016-12-15T16:17:00+00:00

Right now we are performing planned maintenance on our OpenStack compute nodes in order to update several Components on the cluster. Customers may experience service interruptions. We apologize for any inconvenience this maintenance work may cause. Affected customers will be notified by e-mail individually.

We have finished our maintenance.

Important maintenance in the SysEleven Stack

2016-12-14T15:52:00+00:00

Between December 15th 2016 09:00 (CET) and December 15th 2016 18:00 (CET) we will perform planned maintenance on our OpenStack Compute Nodes in order to update the several Components on the Cluster. Customers will experience service interruptions during this maintenance window. We apologize for any inconvenience this maintenance work may cause. Affected customers will be notified by e-mail individually.

Network connection problems

2016-10-31T07:00:00+00:00

At the moment we are facing issues with our network-connection. It could be possible that response times are higher then normal and instances aren't available for few seconds. Update 9:30: Problem is fixed

Issues with our distributed file system

2016-10-27T14:10:00+00:00

At the moment we are facing issues with the distributed file system, a core component of the SysEleven Stack. At the moment it is possible that virtual machines can't be spawned or have no write access to their disks.

We are working on a solution.

Problems with SysEleven-DNS

2016-03-23T11:30:00+00:00

Update 15:15 CET: DNS services are back again, InternetX will update their status page soon.

Update 15:00 CET: DNS services recovered but outage is still listed on the status page of InternetX (see below).

Our external hosted DNS-Server are currently experiencing reachability problems. You can see can the system status at anytime on http://www.internetx-status.com/de/. Customers will receive the next update via e-mail at least at 15.00 CET. We apologize for any inconvenience caused.

QFabric lost connectivity during upgrade

2016-02-23T05:32:00+00:00

-- 24-02-2016 --

Update 04:00 CET -- Connectivity and services restored. End of incident. We'll continue monitoring and quality assurance. Customers have been notified by e-mail.

Update 03:30 CET -- Connectivity partially restored. Engineers are working on this in the datacenter with vendor support. Next update at 2016-02-24 04:00 CET

Update 03:00 CET -- During last switchover on network node group we've lost connectivity on the platform. We're investigating with the vendor. Next update at 24-02-2016 03:30 CET.

Update 02:38 CET -- During last switchover on network node group we've lost connectivity on the platform. We're investigating with the vendor. Next update at 24-02-2016 03:00 CET.

Update 02:30 CET -- All services are operational, we're still performing software update on network node group. Next update at 24-02-2016 03:00 CET.

Update 02:00 CET -- All services are operational, we're still performing software update on network node group. Next update at 24-02-2016 02:30 CET.

-- 23-02-2016 --

Update 23:53 CET -- All services operational, update of director device has been completed successfully. We will now continue with upgrading software version on interconnect devices. Next update at 24-02-2016 02:00 CET.

Update 22:00 CET -- All services operational, starting maintenance with vendor specialists as scheduled. Next update at 24-02-2016 00:00 CET.

Update 17:30 CET -- All services operational, maintenance window together with vendor specialists is scheduled (see Notifier). Next update at 22:00 CET.

Update 09:00 CET -- All services back up. System is operational but degraded. We are planning an additional service window with the vendor starting at 22:00 CET.

Update 08:00 CET -- All services back up. We are monitoring the Fabric and are working closely with the vendor on underlying issues. Highest priority is to keep customer systems online and operational. Next update 09:00 CET

Update 07:30 CET -- Services are still coming up. We are currently monitoring the Fabric to converge and investigate further. Next update 08:00 CET

Update 07:15 CET -- Failover completing. Services are coming up. Investigation continues. Next update 07:30 CET

Update 07:00 CET -- we have initiated failover to the secondary director. We are currently investigating with the vendor. Next update 07:15 CET

During upgrade of the 2nd QFabric director we lost connectivity to the core routers. We are currently investigating with the vendor. Next update at 07:00 CET

Maintenance Network-Infrastructure 17.02.2016

2016-02-16T11:53:00+00:00

Between February 16th 2016 00:00 (CET) and February 16th 2016 06:00 (CET) and on February 17th within the same timeframe we will perform planned maintenance on our core routers in order to update the firmware on one device and exchange configuration network-wide. Customers might experience short traffic interruptions and re-routings during this maintenance window. We apologize for any inconvenience this maintenance work may cause. Affected customers will be notified by e-mail individually.

Maintenance Network-Infrastructure 23.02.2016

2016-02-15T09:43:00+00:00

Between February 23th 2016 00:00 (CET) and February 23th 2016 18:00 (CET) we will perform planned maintenance on our central switching platform at our datacenter Berlin 2. The Juniper QFabric (QFX3000-M) used on site connects all rack switches, interconnects and directors to a single fabric. In order to ensure continuous vendor support we will perform a nonstop software update to the current software release. We do not expect traffic interruptions for your services during the update due to redundancy of all components and servers connected to two rack switches. The update has been laboratory-confirmed and vendor specialists will accompany the maintenance window. The update will be carried out in multiple stages and lasts serveral hours due to the high amount of devices. Therefore, the maintenance window is scheduled for 18 hours. Nine hours for the update process and another nine hours for monitoring and quality assurance. We will keep you updated on a regular basis - at least at the beginning of the update, if the update process has been completed and at the end of the maintenance window. Affected customers will be notified by e-mail individually.

Maintenance Network-Infrastructure 16.02.2016

2016-02-12T17:16:00+00:00

Routing issues from Deutsche Telekom towards SysEleven

2016-02-12T07:00:00+00:00

Deutsche Telekom, one of our upstream providers, performed a scheduled maintenance tonight and updated the software on one of their core routers in Berlin. Due to a linecard error routing from Deutsche Telekom (AS3320) towards SysEleven (AS25291) was partially disrupted between 08.00 CET and 09:00 CET. Traffic has been rerouted by SysEleven and connectivity has been finally restored by Deutsche Telekom.