SysEleven Status and Incidents

MAINTENANCE: MetaKube Platform

Tue, 30 Jan 2024 14:12:00 +0000

Affected Components: MetaKube Platform, all regions

Scheduled Start: 05. Feb. 2024 / 6:00 pm

Scheduled End: 05. Feb. 2024 / 7:00 pm

State: FINISHED

Description:

Scheduled maintenance of MetaKube platform infrastructure

Customer Impact during the maintenance:

Short interruptions (<1min) of the control plane possible (e.g. apiserver, controller-manager, scheduler), but not MetaKube API
Running workloads will not be affected
Network availability will not be affected
Storage availability will not be affected

Customer Actions:

No customer actions needed
Please inform us if you notice any irregularities

UPDATE 19:15

Maintenance finished succesfully.

INCIDENT: SysEleven STACK DNS issues

Mon, 05 Feb 2024 03:03:00 +0000

Affected Components: SysEleven Stack DNS Issues

Incident Start: 5 Feb 2024 03:03

Description:

DNS servers overloaded
ns01.cloud.syseleven.net, ns02.cloud.syseleven.net, ns03.cloud.syseleven.net, ns04.cloud.syseleven.net

Customer Impact:

availability of dns servers is reduced

Update 5 Feb 2024 03:22

Connectivity recovered

INCIDENT: SysEleven STACK DNS issues

Tue, 06 Feb 2024 17:44:00 +0000

Affected Components: SysEleven Stack DNS Issues

Incident Start: 6 Feb 2024 17:44 Incident End: 6 Feb 2024 17:59

Description:

DNS servers overloaded
ns01.cloud.syseleven.net, ns02.cloud.syseleven.net, ns03.cloud.syseleven.net, ns04.cloud.syseleven.net

Customer Impact:

availability of dns servers is reduced

Update 18:00

Connectivity recovered

INCIDENT: SysEleven STACK DNS issues

Wed, 07 Feb 2024 12:15:00 +0000

Affected Components: SysEleven Stack DNS Issues

Incident Start: 7 Feb 2024 12:15 Incident End: 7 Feb 2024 12:35

Description:

DNS servers overloaded
ns01.cloud.syseleven.net, ns02.cloud.syseleven.net, ns03.cloud.syseleven.net, ns04.cloud.syseleven.net

Customer Impact:

availability of dns servers is reduced

Update 12:35

Connectivity recovered

MAINTENANCE: MetaKube announcement

Wed, 07 Feb 2024 07:00:00 +0000

Affected Components: all regions

Scheduled Start: 07. Feb. 2024/06:00 pm Scheduled End: 07. Feb. 2024/09:00 pm

State: COMPLETED

Description:

Infrastructure security update in all regions.

Customer Impact during the Maintenance:

Short interruptions (<5min) of the control plane possible (e.g. controller-manager, scheduler), but not API
Running workloads will not be affected
Network availability will not be affected
Storage availability will not be affected

Customer Actions:

No customer actions needed
Please inform us if you notice any irregularities

MAINTENANCE: Dringende Wartungsarbeiten / Urgent maintenance work network component Fortigate

Sun, 11 Feb 2024 20:00:00 +0000

Affected Components: Network Component Fortigate

Scheduled Start: Feb. 11 2024 / 08:00 p.m.

Scheduled End: Feb. 11 2024 / 11: p.m.

State: Scheduled Maintainance

Description:

Urgent security update to Fortigate network component

(FortiOS - Out-of-bound Write in sslvpnd (RCE) (CVE-2024-21762))

Expected Impact:

About 1-2 short disconnections when the services are panning.

Customer Impact during the Maintenance:

Network Service might be interrupted when actual maintainance takes place

Update 4:30 p.m.

After further evaluating options to mitigate or patch the disclosed vulnerabilities, we learned that patches were anounced but not yet available for download. We continue to monitor the situation closely and will act as soon as patches and upgrades are available.

Update Sunday Feb. 11 7:00 p.m.

After evaluating and testing the provided FortiOS Updates in our datacenter Lankwitzer Straße (BLA) we will be rolling out the patches today between 8 and 11 p.m. in our datacenters BKI and BLU. We have contacted affected customers separately. As the maintenance work affects our VPN routers, short outages of VPN connections are to be expected during the maintenance window mentioned.

Update Sunday Feb. 11 11:00 p.m. All instances of FortiOS in our datacenters have been updated to versions secure against CVE-2024-21762.

INCIDENT: Partial outage of MetaKube apiserver reachability DBL/CBK

Sat, 10 Feb 2024 11:00:00 +0000

Affected Components: MetaKube apiserver external availability, some clusters, DBL/CBK

Incident Start: 10.Feb 2024/11:00 am Incident End: 10. Feb. 2024/12:35 am

Description:

An expired client certificate caused some load balancer configuration to go stale. Thus, a small number of clusters lost external availability of their apiservers.

Customer Impact:

External apiserver availability disrupted for at most 8 clusters at peak for a maximum of 95min

Customer Actions:

Examples: - No customer actions needed

Update 12:35 The certificate was renewed and a failover of the load balancer triggered. This restored availability of all clusters again.

INCIDENT: SysEleven STACK Object Storage issues, region FES

Mon, 12 Feb 2024 13:54:00 +0000

Affected Components: SysEleven Stack Object Storage, region FES

Incident Start: 12th Feb. 2024/13:00 pm (CET - UTC +1)

Incident End: 12th Feb. 2024/14:15 pm (CET - UTC +1)

Description:

At the moment, we are facing issues with the Object Storage (Ceph) in Region FES.

Customer Impact:

API is not working correctly

Update 14:18 p.m.

Issue is fixed and service is operational

MAINTENANCE: MetaKube Platform

Mon, 19 Feb 2024 18:00:00 +0000

Affected Components: MetaKube Platform, all regions

Scheduled Start: 19. Feb 2024 / 06:00 pm

Scheduled End: 19. Feb. 2024 / 9:00 pm

State: FINISHED

Description:

Scheduled maintenance of the MetaKube platform in all regions

Customer Impact during the maintenance:

Short interruptions (<5min) of the control plane possible (e.g. controller-manager, scheduler), but not API
Running workloads will not be affected
Network availability will not be affected
Storage availability will not be affected

Customer Actions:

No customer actions needed
Please inform us if you notice any irregularities

INCIDENT: SysEleven STACK DNS issues

Mon, 19 Feb 2024 08:30:00 +0000

Affected Components: SysEleven Stack DNS Issues

Incident Start: 19 Feb 2024 08:30 CET

Incident End: 19 Feb 2024 08:50 CET

Description:

DNS servers overloaded
ns01.cloud.syseleven.net, ns02.cloud.syseleven.net, ns03.cloud.syseleven.net, ns04.cloud.syseleven.net

Customer Impact:

availability of dns servers is reduced

Update 19 Feb 2024 08:50 CET

Connectivity recovered

INCIDENT: PVC Failure of a hardware node, region blu1

Fri, 23 Feb 2024 22:35:00 +0000

Affected Components: Hardware node failure, region blu

Incident Start: 2024-02-23 22:35 UTC+01:00 (CET) Incident End: 2024-02-23 23:45 UTC+01:00 (CET)

Description:

Malfunction of a hardware node
Restart of the hardware node is necessary
Evacuation of custmer system was necessary

Customer Impact:

During the period of restart, there will be a short interruption in the availability of the systems.
Affected customers were notified via E-Mail.
Please check the affected systems for their full functionality.

MAINTENANCE: MetaKube Platform

Tue, 27 Feb 2024 15:29:56 +0000

Affected Components: MetaKube Platform, all regions

Scheduled Start: **2024-02-29 18:00 UTC+01:00 (CET)

Scheduled End: **2024-02-29 21:00 UTC+01:00 (CET)

State: COMPLETED

Description:

Scheduled maintenance of the MetaKube platform in all regions

Customer Impact during the maintenance:

Short interruptions (<5min) of the control plane possible (e.g. controller-manager, scheduler).
Running workloads will not be affected
Network availability will not be affected
Storage availability will not be affected

Customer Actions:

No customer actions needed
Please inform us if you notice any irregularities

MAINTENANCE: MetaKube Platform Customer Cluster LB maintenance

Tue, 05 Mar 2024 13:02:00 +0000

Affected Components: MetaKube Platform, all regions

Scheduled Start: **2024-03-07 18:00 UTC+01:00 (CET)

Scheduled End: **2024-03-07 19:00 UTC+01:00 (CET)

State: COMPLETED

Description:

Scheduled maintenance of the MetaKube platform in all regions
Small update to MetaKube infrastructure

Customer Impact during the maintenance:

Short interruptions (<5min) of the control plane possible (e.g. controller-manager, scheduler), but not API
Running workloads will not be affected
Network availability will not be affected
Storage availability will not be affected

Customer Actions:

No customer actions needed
Please inform us if you notice any irregularities

INCIDENT: SysEleven MetaKube Dashboard API issues

Tue, 12 Mar 2024 16:00:00 +0000

Affected Components: MetaKube Dashboard API

Incident Start: 2024-03-012 16:00 UTC+01:00 (CET)

Description:

Accessibility of the SysEleven MetaKube Dashboard API is not ensured.
Clusters may not be reconciled currently

Customer Impact:

Access to MetaKube Dashboard and API (including Terraform) might be not possible
Clusters may not be reconciled currently

Update: 2024-03-012 17:30 UTC+01:00 (CET)

Issues are fixed

MAINTENANCE: MetaKube Platform

Mon, 25 Mar 2024 18:00:00 +0000

Affected Components: MetaKube Platform, all regions

Scheduled Start: 2024-03-25 18:00 UTC+01:00 (CET)

Scheduled End: 2024-03-25 21:30 UTC+01:00 (CET)

State: COMPLETED

Description:

Scheduled maintenance of the MetaKube platform in all regions

Customer Impact during the maintenance:

Short interruptions (<5min) of the control plane possible (e.g. controller-manager, scheduler), but not API
Running workloads will not be affected
Network availability will not be affected
Storage availability will not be affected

Customer Actions:

No customer actions needed
Please inform us if you notice any irregularities

INCIDENT: SysEleven STACK Object Storage issues, region FES

Sun, 24 Mar 2024 19:14:00 +0000

Affected Components: SysEleven Stack Object Storage (Quobyte), region FES

Incident Start: 2024-03-24 19:14 UTC+01:00 (CET)

Incident End: 2024-03-24 19:28 UTC+01:00 (CET)

Description:

The Object Storage (Quobyte) API in region FES may have been intermittently unreachable.

Customer Impact:

Writing or reading of objects maybe restricted.

INCIDENT: SysEleven STACK issues in region DBL

Mon, 25 Mar 2024 10:18:00 +0000

Affected Components: SysEleven Stack, MetaKube clusters, MetaKube API, region DBL

Incident Start: 2024-03-25 10:18 UTC+01:00 (CET)

Incident End: 2024-03-25 13:00 UTC+01:00 (CET)

Description:

Major Outage of multiple Hardware Nodes
Occurring errors are currently being investigated.

Customer Impact:

Storage and Compute services in the DBL region are disrupted

Update : 10:39

One Rack in the DBL Datacenter seems to be down
Rack is getting investigated by remote hands

Update : 10:50

Datacenter operator confirmed power outage
Hardware nodes are coming back online
Follow up problems are being tackled

Update : 11:03

We are still stabilizing the affected systems
We see some recoveries coming in already

Update : 11:17

We still see issues on software loadbalancers (Octavia) which we are being investigated at the minute
We see a few CRC problems and will contact the affected customers individually after doing a deep scrub on the storage

Update : 11:40

Octavia loadbalancer recoveries and storage scrub tasks are ongoing

INCIDENT: SysEleven STACK issues in region CBK

Mon, 08 Apr 2024 09:02:00 +0000

Affected Components: SysEleven Stack, region CBK

Incident Start: 2024-04-08 09:02 UTC+01:00 (CET) Incident End: 2024-04-08 09:30 UTC+01:00 (CET)

Description:

Occurring errors are currently being investigated.

Customer Impact:

Connectivity is restricted

MAINTENANCE: MetaKube Platform

Mon, 08 Apr 2024 15:34:00 +0000

Affected Components: MetaKube Platform, all regions

Scheduled Start: 2024-04-09 06:00 UTC+01:00 (CET)

Scheduled End: 2024-04-09 07:00 UTC+01:00 (CET)

State: COMPLETED

Description:

To ensure long-term resiliency and scalability, we migrate all MetaKube clusters' Kubernetes API endpoint to new load balancers. The endpoint will resolve to new IPs, but the migration itself will be seamless.

Customer Impact during the maintenance:

Certain API actions won't propagate to clusters for up to 30min
Interruptions of existing connections to the Kubernetes API during a brief amount of time (5m)
Running workloads will not be affected
Network availability will not be affected
Storage availability will not be affected

Customer Actions:

No customer actions needed
Please inform us if you notice any irregularities

INCIDENT: SysEleven STACK issues in region CBK

Tue, 09 Apr 2024 10:34:00 +0000

Affected Components: SysEleven Stack, region CBK

Incident Start: 2024-04-09 10:34 UTC+01:00 (CET) Incident End: 2024-04-09 11:00 UTC+01:00 (CET)

Description:

Occurring errors are currently being investigated.

Customer Impact:

Connectivity is restricted