Past Incidents

Affected Components: SysEleven Stack Object Storage, region FES, DUS2, HAM1

Incident Start: 2025-04-14 16:26 UTC+01:00 (CET)

Description:

At the moment we are facing issues with the Object Storage in Region FES, DUS2, HAM1.

Customer Impact:

• Some objects might be corrupted regarding latest known issues in Ceph. https://docs.ceph.com/en/latest/releases/squid/#v19-2-2-squid
• We are investigating the situation and working on updates to mitigate the issues

Update: 18:40

• The Ceph S3 endpoint of the DUS2 region is currently being updated, after successful testing we will proceed with the HAM1 and FES region

Update: 19:45

• The update for the DUS2 region went through without problems, we are proceeding to update HAM1 and the FES region
• Affected customers will be contacted individually once we got a complete overview of potential affected buckets and objects

Update: 21:10

• Update of HAM1 region also finished, FES region is in progress

Update: 22:52

• Update of FES region is still ongoing

Update: 15-04-2025 09:36

• Update of FES region is still ongoing and everything works as intended

Update: 16:40

• Update of FES region is expected to be finished to the end of the business day

Update: 16-04-2025 03:08

• Update of FES is over

Affected Components: All regions

Incident Start: 2025-04-10 23:40 (CETS)

Incident End: 2025-04-11 01:08 (CETS)

Description:

We have major outage in all regions

Update: 01:01 am

Issue identified and we are working on it.

Update: 01:08 am

Issue is resolved and we are whatching it.

We're currently investigating an issue in our IAM systems that prevents creation and deletion of projects.