SysEleven Status and Incidents

INCIDENT: SysEleven STACK API issues, region FES

Wed, 04 Dec 2024 17:07:00 +0000

Affected Components: SysEleven Stack API, region FES

Incident Start: 2024-12-04 19:07 UTC+01:00 (CET)

Incident End: 2024-12-04 20:10 UTC+01:00 (CET)

Description:

Accessibility of the SysEleven Stack API is not ensured.

Customer Impact:

Requests on OpenStack API may return an error code
Spawning new virtual machines (VMs) or changing existing resources may fail

Update: 2024-12-04 20:00 UTC+01:00 (CET)

We identified the likely root cause and a fix is being applied.

Update: 2024-12-04 20:10 UTC+01:00 (CET)

OpenStack API is now working again as expected.

INCIDENT: SysEleven STACK issues in region dus2

Fri, 06 Dec 2024 15:45:00 +0000

Affected Components: SysEleven Stack, region dus2

Incident Start: **2024-12-06 17:45 UTC+01:00 (CET)

Description:

We are seeing some network connectivity issues in the region and are investigating.

Customer Impact:

Connectivity is degraded for MetaKube Services in dus2
Connectivity to Database as a Service in dus2 is degraded

Update: 2024-12-06 18:45 UTC+01:00 (CET)

We identified an issue with one of our gateways and are working on a fix

INCIDENT: Partial outage of Control Planes in Regions FES, DBL, CBK

Mon, 09 Dec 2024 16:30:00 +0000

Affected Components: Control Planes in Regions FES, DBL, CBK

Incident Start: 2024-12-09 18:30 UTC+01:00 (CET)

Description:

Some cluster control planes are not reachable.

Update (22:20 CET):

We have found a way to mitigate the issues temporarily and start applying the fix.

Update (22:50 CET):

We applied the fix everywhere. We don't see any broken clusters anymore.

Update 2024-12-10 12:30 UTC+01:00 (CET):

We identified the root cause: An unintended side-effect of an upgrade to a kube-proxy setting changed the proxy mode. This created iptables rules which were not cleaned up and through which traffic was dropped.

We are taking measures to prevent this in the future.

INCIDENT: SysEleven STACK network issues in region DBL

Fri, 13 Dec 2024 12:09:00 +0000

Affected Components: SysEleven Stack, region DBL

Incident Start: 2024-12-13 14:09 UTC+01:00 (CET)

Incident End: 2024-12-13 16:09 UTC+01:00 (CET)

Description:

DBL network has performance issue

Customer Impact:

Network performance degradation

Update: 2024-12-13 15:17 UTC+01:00 (CET)

Situation is back to normal.

Update: 2024-12-13 15:57 UTC+01:00 (CET)

We notice performance degradation again.

Update: 2024-12-13 16:09 UTC+01:00 (CET)

Situation is back to normal.

INCIDENT: SysEleven STACK issues in region DBL

Tue, 17 Dec 2024 22:40:00 +0000

Affected Components: SysEleven Stack, region DBL

Incident Start: 2024-18-12 00:47 Incident Start: 2024-18-12 01:27

Description:

Occurring errors are currently being investigated.

Customer Impact:

Connectivity is restricted

Update: 2024-18-12 01:27

We could see problems with cross region connectivity outgoing from the DBL region, between 00:35 - 01:10, at the moment traffic seems to have normalized again, we are still investigating

Update: 2024-18-12 02:00

Device causing the network issues was identified, root cause will be further investigated

INCIDENT: SysEleven STACK issues in region CBK

Mon, 06 Jan 2025 08:35:12 +0000

Affected Components: SysEleven Stack, region CBK

Incident Start: 2025-06-01 08:18

Incident Start: 2025-06-01 08:50

Description:

Occurring errors are currently being investigated.

Customer Impact:

Connectivity is restricted

INCIDENT: SysEleven STACK issues in region CBK

Mon, 06 Jan 2025 10:14:32 +0000

Affected Components: SysEleven Stack, region CBK

Incident Start: 2025-06-01 10:00

Incident End: 2025-06-01 10:30

Description:

Occurring errors are currently being investigated.

Customer Impact:

Connectivity is restricted

INCIDENT: SysEleven STACK issues in region CBK

Tue, 07 Jan 2025 09:25:38 +0000

Affected Components: SysEleven Stack, region CBK

Incident Start: 2025-01-07 09:16

Incident End: 2025-01-07 09:45

Description:

Occurring errors are currently being investigated.

Customer Impact:

Connectivity is restricted

Update : 09:45

Situation stabilized again, we are investigating the situation at the moment

INCIDENT: minor outage of Database as a Service

Tue, 21 Jan 2025 14:41:37 +0000

Affected Components: Database as a Service, all regions

Incident Start: 2025-01-21 10:30 UTC+01:00 (CET) Incident End: 2025-01-21 14:45 UTC+01:00 (CET)

Description:

Database as a Service is currently only available via the API and terraform, the UI is not working

Update 2025-01-21 14:45

We fixed the underlying issue, therefore Database as a Service is also working again in the UI

INCIDENT: MetaKube Control Plane issues, region FES

Tue, 11 Feb 2025 08:04:01 +0000

Affected Components: MetaKube Control Planes, region FES

Incident Start: 2025-02-11 06:00 UTC+01:00 (CET)

State: Resolved

Description:

Accessibility of the MetaKube API is not ensured.
After a scheduled maintenance to the network in FES, the MetaKube control cluster (which is hosting the customer control planes) has problems reaching DNS. This is causing issues to the customer control planes.
This also affects Database as a Service and Observability as a Service
All times below are CET

UPDATE 2025-02-13 12:20 - We consider all service disruptions of the incident to be mitigated - Although we are not expecting any more service disruptions, we are still watching all systems closly

Previous Updates in reverse chronological order

UPDATE 2025-02-11 10:00

Still investigating the DNS issue. We sent out a notifier to all potentially affected customers.

UPDATE 2025-02-11 11:15 - We have used the time since the last update to narrow down the root cause of the incident. We excluded some possibilites but did not find the root-cause. We are now preparing a partial rollback to downgrade the SDN again.

UPDATE 2025-02-11 12:05 - We completed a partial OVN/SDN downgrade, however this has not yet resolved the incident. - We are investigating further

UPDATE 2025-02-11 12:25 - We’re exploring further downgrade approaches (previous rollbacks were, as announced, partial) and are in parallel investigating further.

UPDATE 2025-02-11 13:00 - As we originally updated the SDN due to a critical security gap, it was decided that we will not perform a full OVN/SDN rollback to the initial state. - We have now activated several teams who will be developing and evaluating different solutions until 1.30 pm. An update on how we proceed will follow then.

UPDATE 2025-02-11 13:50 - Our Teams will continue developing and evaluating solutions in break out session as there are further leads but no breakthrough, yet. - In parallel we are preparing a failover for IAM and Alloy to DUS/HAM

UPDATE 2025-02-11 15:30 - Our teams investigation in SDN traffic loss is ongoing - Our teams continue developing and evaluating solutions and possible workarounds - In parallel we are evaluating a rebuild of the SDN (software defined network)

UPDATE 2025-02-11 17:47 - Part of the services are still not functional - We are still working hard to resolve the issues but we will roll back the update of the SDN if no progress is made. - The planned maintenance period begins today, 11 February 2025, at 23:00 and is expected to last until around 06:00 CET on 12 February 2025, during which time there may be repeated interruptions to services. - The maintenance is also announced via notifier. You will get an info of the end of the maintenance also via notifier.

UPDATE 2025-02-11 20:15 - IAM, Alloy and Observability as a Service are restored to full functionality - The Database as a Service API has also been restored, but the API still has some issues which are related to the wider SDN problem. The Databases themselves were at no point affected by the incident.

UPDATE 2025-02-11 23:00 - The situation with the API improved. We will continue watching it.

UPDATE 2025-02-12 10:15 - The previous maintenance work did not achieve the desired success. - Workarounds have been implemented, so operations should be able to continue without disruptions. - Maintainance work will continue during the upcoming night to fully resolve the incident.

UPDATE 2025-02-12 14:55 - We scheduled another maintenance window for this night, February 12th, from 11:00 PM to 6:00 AM the following day. - During this maintenance window, there may be brief interruptions or limited availability of certain services. - The goal is the complete resolution of the incident caused by Monday's update. - An RfO will be available in our Helpdesk after the incident is mitigated.

UPDATE 2025-02-13 15:30 - Incident is resolved

INCIDENT: SysEleven STACK issues in region FES

Sat, 08 Mar 2025 14:29:22 +0000

Affected Components: SysEleven Stack, region FES

Incident Start: 2025-03-08 13:00

Incident End: 2025-03-08 15:30

Description:

Occurring errors are currently being investigated.

Customer Impact:

IPv6 Connectivity is restricted

Update: 16:25

Flaky ipv6 connectivity was repaired around 15:30, a mismatch in configuration was identified

INCIDENT: SysEleven STACK API issues, region CBK

Thu, 13 Mar 2025 14:43:01 +0000

Affected Components: SysEleven Stack API, region CBK

Incident Start: 2025-03-13 14:30 Incident End: 2025-03-13 17:15

Description:

Accessibility of the SysEleven Stack API is not ensured.

Customer Impact:

Spawning new virtual machines (VMs) or changing existing resources is not possible.

INCIDENT: Partial degradation of SysEleven IAM services

Mon, 17 Mar 2025 10:39:16 +0000

We're currently investigating an issue in our IAM systems that prevents creation and deletion of projects.

INCIDENT: all platforms

Fri, 11 Apr 2025 00:22:15 +0000

Affected Components: All regions

Incident Start: 2025-04-10 23:40 (CETS)

Incident End: 2025-04-11 01:08 (CETS)

Description:

We have major outage in all regions

Update: 01:01 am

Issue identified and we are working on it.

Update: 01:08 am

Issue is resolved and we are whatching it.

INCIDENT: SysEleven STACK Object Storage issues, region FES, HAM1, DUS2

Mon, 14 Apr 2025 16:51:56 +0000

Affected Components: SysEleven Stack Object Storage, region FES, DUS2, HAM1

Incident Start: 2025-04-14 16:26 UTC+01:00 (CET)

Description:

At the moment we are facing issues with the Object Storage in Region FES, DUS2, HAM1.

Customer Impact:

Some objects might be corrupted regarding latest known issues in Ceph. https://docs.ceph.com/en/latest/releases/squid/#v19-2-2-squid
We are investigating the situation and working on updates to mitigate the issues

Update: 18:40

The Ceph S3 endpoint of the DUS2 region is currently being updated, after successful testing we will proceed with the HAM1 and FES region

Update: 19:45

The update for the DUS2 region went through without problems, we are proceeding to update HAM1 and the FES region
Affected customers will be contacted individually once we got a complete overview of potential affected buckets and objects

Update: 21:10

Update of HAM1 region also finished, FES region is in progress

Update: 22:52

Update of FES region is still ongoing

Update: 15-04-2025 09:36

Update of FES region is still ongoing and everything works as intended

Update: 16:40

Update of FES region is expected to be finished to the end of the business day

Update: 16-04-2025 03:08

Update of FES is over

INCIDENT: SysEleven STACK issues in region DBL

Thu, 17 Apr 2025 10:11:22 +0000

Affected Components: SysEleven Stack, region DBL

Incident Start: 2025-04-17 10:00

Incident End: 2025-04-17 10:45

Description:

Occurring errors are currently being investigated.

Update [2025-04-17 10:28 am]: Alloy (GUI) & API Authentication HAM & DUS, Metakube API in DBL & CBK are also affected

Update [2025-04-17 10:40 am]: DBL Cloud Gateways have been identified as possible source of malfunction, we are bringing them back to function, first services are back online, next Update at 11:00 am

Update [2025-04-17 11:00 am]: All services are up and running, we are still watching all environments, incident is declared as ended

INCIDENT: SysEleven STACK issues in region CBK

Fri, 25 Apr 2025 17:18:12 +0000

Affected Components: SysEleven Stack, region CBK

Incident Start: 2025-04-25 15:55:00 UTC+01:00 (CET)

Description:

Occurring errors are currently being investigated.

Customer Impact:

We are expirencing a partial outage of the Network where some connection might get dropped.

INCIDENT: SysEleven Managed Hosting Network partial outage, region BLU1

Fri, 25 Apr 2025 17:46:54 +0000

Affected Components: SysEleven Managed Hosting Network partial outage, region BLU1

Incident Start: 2025-04-25 15:55:00 UTC+01:00 (CET)

Description: - Occurring errors are currently being investigated.

Customer Impact: - We are expirencing a partial outage of the Network where some connection might get dropped.

INCIDENT: SysEleven STACK issues in region CBK (Network)

Tue, 29 Apr 2025 18:01:27 +0000

Affected Components: SysEleven Stack Network, region CBK

Incident Start: 2025-04-29 17:30 UTC+02:00 (CEST) Incident End: 2025-04-29 17:50 UTC+02:00 (CEST)

Description:

Network connectivity to cloud region CBK was impaired

Customer Impact:

Connectivity is restricted

Update: 17:59

Network connectivity issues were resolved around 17:50

INCIDENT: Partial outage of Metakube control plane in DUS

Wed, 30 Apr 2025 16:10:27 +0000

Affected Components: metakube controle plane, region DUS

Incident Start: 2025-04-30 16:00 UTC+02:00

Description:

The loadbalancer for the kubernetes apiserver does not properly work for some clusters.

Customer Impact: - Potential connection loss to the apiserver

Update: 2025-04-30 16:30 UTC+02:00

We have recovered the affected clusters but haven't found the root cause. Right now we can't see any other affected clusters but will continue to monitor the situation.

Update: 2025-04-30 16:52 UTC+02:00 We have identified and fixed the root-cause of the incident. All clusters are working properly again.