Past Incidents

Affected Components: metakube controle plane, region DUS

Incident Start: 2025-04-30 16:00 UTC+02:00

Description:

The loadbalancer for the kubernetes apiserver does not properly work for some clusters.

Customer Impact:

• Potential connection loss to the apiserver

Update: 2025-04-30 16:30 UTC+02:00

We have recovered the affected clusters but haven't found the root cause. Right now we can't see any other affected clusters but will continue to monitor the situation.

Update: 2025-04-30 16:52 UTC+02:00 We have identified and fixed the root-cause of the incident. All clusters are working properly again.

Affected Components: SysEleven Stack Network, region CBK

Incident Start: 2025-04-29 17:30 UTC+02:00 (CEST) Incident End: 2025-04-29 17:50 UTC+02:00 (CEST)

Description:

• Network connectivity to cloud region CBK was impaired

Customer Impact:

• Connectivity is restricted

Update: 17:59

Network connectivity issues were resolved around 17:50

Affected Components: SysEleven Stack, region CBK

Incident Start: 2025-04-25 15:55:00 UTC+01:00 (CET)

Description:

• Occurring errors are currently being investigated.

Customer Impact:

• We are expirencing a partial outage of the Network where some connection might get dropped.

Affected Components: SysEleven Stack, region DBL

Incident Start: 2025-04-17 10:00

Incident End: 2025-04-17 10:45

Description:

• Occurring errors are currently being investigated.

Update [2025-04-17 10:28 am]: Alloy (GUI) & API Authentication HAM & DUS, Metakube API in DBL & CBK are also affected

Update [2025-04-17 10:40 am]: DBL Cloud Gateways have been identified as possible source of malfunction, we are bringing them back to function, first services are back online, next Update at 11:00 am

Update [2025-04-17 11:00 am]: All services are up and running, we are still watching all environments, incident is declared as ended

Affected Components: SysEleven Stack Object Storage, region FES, DUS2, HAM1

Incident Start: 2025-04-14 16:26 UTC+01:00 (CET)

Description:

At the moment we are facing issues with the Object Storage in Region FES, DUS2, HAM1.

Customer Impact:

• Some objects might be corrupted regarding latest known issues in Ceph. https://docs.ceph.com/en/latest/releases/squid/#v19-2-2-squid
• We are investigating the situation and working on updates to mitigate the issues

Update: 18:40

• The Ceph S3 endpoint of the DUS2 region is currently being updated, after successful testing we will proceed with the HAM1 and FES region

Update: 19:45

• The update for the DUS2 region went through without problems, we are proceeding to update HAM1 and the FES region
• Affected customers will be contacted individually once we got a complete overview of potential affected buckets and objects

Update: 21:10

• Update of HAM1 region also finished, FES region is in progress

Update: 22:52

• Update of FES region is still ongoing

Update: 15-04-2025 09:36

• Update of FES region is still ongoing and everything works as intended

Update: 16:40

• Update of FES region is expected to be finished to the end of the business day

Update: 16-04-2025 03:08

• Update of FES is over

Affected Components: All regions

Incident Start: 2025-04-10 23:40 (CETS)

Incident End: 2025-04-11 01:08 (CETS)

Description:

We have major outage in all regions

Update: 01:01 am

Issue identified and we are working on it.

Update: 01:08 am

Issue is resolved and we are whatching it.