Past Incidents

Affected Components: SysEleven Stack Networking in region FES

Scheduled Start: 25 Apr 2023, 22:00 CEST

Scheduled End: 26 Apr 2023, 01:00 CEST

State: COMPLETED

Description:

We will perform maintenance on networking gateways.

Customer Impact during the Maintenance:

• Interruption of external connectivity of virtual machines of up to 2 minutes
• VPN connections could be interrupted for up to 2 minutes

Customer Actions:

• no customer actions needed
• please inform us if you notice any irregularities

Affected Components: SysEleven Stack API, region CBK

Incident Start: 13th Apr. 2023/02:12 pm

Description:

• Performance of the SysEleven Stack API is degraded.
• We are investigating the situation currently.

Customer Impact:

• Spawning new virtual machines (VMs) or changing existing resources can be slower.

Update:02:32 pm

The situation is worse than a performance degradation, correcting the status towards a partial outage, we are further investigating

Update:03.20 pm

We pinned down the issue around 02:56 and are now observing the situation. In the end not only API but also Compute and storage (Block+Object) storage were affected. Changes to the monitoring and alerting are being planned.

Affected Components: CloudObserver/Grafana

Incident Start: 5th April 2023/19:00 CEST
Incident Start: 5th April 2023/22:00 CEST

Description:

• We are currently experiencing issues with the CloudObserver, causing some missing trending data. We identified the problem and are working to resolve the issue.

Customer Impact:

• Metrics from dashboards in your Grafana might be missing

Update 21:30

• We identified and fixed the issue, we were also able to recover most metrics. In some rare cases you might experience a gap between ~19:00 and ~20:00
• We are still closely monitoring the system

Update 22:00

• All systems are back and fully operational

Affected Components: SysEleven Stack API, region CBK

Incident Start: 4th April. 2023/15:30 pm Incident End: 4th April. 2023/15:40 pm

Description:

• Accessibility of the SysEleven Stack API is not ensured.

Customer Impact:

• Spawning new virtual machines (VMs) or changing existing resources is not possible.

Update: 15:42 pm The problem has been solved. The API is available as before.

Affected Components: MetaKube Platform

Scheduled Start: 03. Apr. 2023 / 07:00 pm

Scheduled End: 03. Apr. 2023 / 10:00 pm

State: COMPLETED

Description:

Scheduled maintenance of the MetaKube platform in all regions.

No customer impact expected during the maintenance.

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities

Update 08:10 pm

Maintenance completed successfully.

Affected Components: Control Planes for clusters in FES

Scheduled Start: 31. Mar. 2023/07:00 pm Scheduled End: 31. Mar. 2023/09:00 pm

State: ABORTED

Description:

We will replace our load balancer in front of the control planes of MetaKube clusters in the FES region in this maintenance.

Recent incidents indicate that components related to the load balancer may have been corrupted.

Customer Impact during the Maintenance:

During this period, the public IP of the load balancer will change. We do however not expect any disruptions.

The maintenance is only related to clusters in the FES region and on AWS.

Update (08:20 pm):

We had to abort the maintenance after we saw that the new load balancer behaved faulty under load. We reverted to the previous load balancer and the situation is stable. We will investigate the problem further.

Update 05. April 2023

Reason for Outage (RfO) (de)

Affected Components: MetaKube, region FES

Scheduled Start: 30.03. 2023/07:00 pm CEST Scheduled End: 30.03. 2023/08:00 pm CEST

State: COMPLETED

Description:

• Scheduled maintenance on the Load Balancers for MetaKube in Region FES
• It will happen in the time frame but not take the whole hour

Expected Customer Impact during the Maintenance:

• The Kubernetes-API-Server for your cluster might be unavailable for a short time
• No impact to running services in your cluster

Customer Actions:

• No customer actions needed
• Please inform us if you notice any irregularities

Affected Components: SysEleven Stack Networking in region CBK

Scheduled Start: 28 Mar 2023, 22:00 CEST

Scheduled End: 29 Mar 2023, 02:00 CEST

State: COMPLETED

Description:

We will perform maintenance on networking gateways.

Customer Impact during the Maintenance:

• Interruption of connectivity of virtual machines of up to 2 minutes
• VPN connections could be interrupted for up to 2 minutes

Customer Actions:

• no customer actions needed
• please inform us if you notice any irregularities

Update : 01:55

There was an unplanned delay during an update procedure causing the maintenance to take longer as planned

Affected Components: MetaKube, region FES

Incident Start: 22st March. 2022/10:45 am

Description:

The load balancer to the cluster control planes did not behave correctly and caused slow or dropped connections.

This in turn caused nodes to show as NotReady and problems for kubernetes controllers in the clusters.

Update at 11:30am

Services are operational again.

Update at 15:15am

The load balancer caused problems again for a few minutes.

Incident over at 15:30

If you still experience issues, please open a ticket.

Update: 24st March. 2023/13:45 pm

Reason for Outage (RfO) (eng)

Affected Components: MetaKube Control Plane, region FES

Incident Start: 21st March. 2022/08:30 am

Description:

We're investigating sporadic issues with out MetaKube Control Plane / API in our FES region that may have started already before.

Customer Impact:

Some API commands / changes are not properly executed.

Update: 09:10 am

We created a workaround for an issue in a networking component but affected pods are not restarting automatically, we are triggering that manually.

Update: 09:25 am

Most affected pods have recovered but managing the cluster is still impaired.

Update: 10:13 am

A fix has been implemented but will take some time to be rolled out.

Actions using the Cloud Controller Manager (i.e. creating Load Balancers, getting IPs for Nodes...) do not yet work for the affected clusters.

Update: 12:38 pm

The fix has been rolled out and clusters are running as expected again.

Update: 23st March. 2023/16:25 pm

Reason for Outage (RfO) (eng)