All systems are operational

About This Site

Timezone: Europe/Berlin (CET - UTC +1/CEST - UTC +2)

We provide current information about infrastructure and service availability below. If you experience service impacts or performance issues please contact our helpdesk or our Service Desk.

Past Incidents

Tuesday 9th May 2023

MAINTENANCE: SysEleven STACK announcement (DBL), scheduled 11 months ago

Affected Components: SysEleven Stack in region DBL

Scheduled Start: 9 May 2023, 22:00 CEST

Scheduled End: 10 May 2023, 01:00 CEST

State: SCHEDULED

Description:

We will update some components of the SysEleven Stack.

Customer Impact during the Maintenance:

  • During the update the SysEleven Stack API is unavailable
  • Object Storage API is not affected

Customer Actions:

  • no customer actions needed
  • please inform us if you notice any irregularities

Tuesday 2nd May 2023

MAINTENANCE: SysEleven STACK announcement (CBK), scheduled 1 year ago

Affected Components: SysEleven Stack in region CBK

Scheduled Start: 2 May 2023, 22:00 CEST

Scheduled End: 3 May 2023, 01:00 CEST

State: SCHEDULED

Description:

We will update some components of the SysEleven Stack.

Customer Impact during the Maintenance:

  • During the update the SysEleven Stack API is unavailable
  • Object Storage API is not affected

Customer Actions:

  • no customer actions needed
  • please inform us if you notice any irregularities

Friday 28th April 2023

No incidents reported

Thursday 27th April 2023

No incidents reported

Wednesday 26th April 2023

No incidents reported

Tuesday 25th April 2023

MAINTENANCE: SysEleven STACK announcement (FES), scheduled 1 year ago

Affected Components: SysEleven Stack Networking in region FES

Scheduled Start: 25 Apr 2023, 22:00 CEST

Scheduled End: 26 Apr 2023, 01:00 CEST

State: COMPLETED

Description:

We will perform maintenance on networking gateways.

Customer Impact during the Maintenance:

  • Interruption of external connectivity of virtual machines of up to 2 minutes
  • VPN connections could be interrupted for up to 2 minutes

Customer Actions:

  • no customer actions needed
  • please inform us if you notice any irregularities
API INCIDENT: SysEleven STACK designate API Issues

Affected Components: SysEleven Stack DNSaaS API

Incident Start: 25th Sept. 2022/20:00 pm Incident End: 25th Sept. 2022/21:00 pm

Description:

  • Updating and creating DNS zones is not possible

Customer Impact:

  • Creating new DNS Zones and updating records for existing zones will fail

Update: 21:52$ pm

The API is available again but there are sometimes performance issues. Some requests can take some time until they are active on our name servers.

Monday 24th April 2023

No incidents reported

Sunday 23rd April 2023

No incidents reported

Saturday 22nd April 2023

No incidents reported

Friday 21st April 2023

Network INCIDENT: SysEleven STACK issues in region FES

Affected Components: SysEleven Stack, region FES

Incident Start: 21st Apr. 2023/07:00 am Incident End: 21st Apr. 2023/17:50 pm

Description:

  • There is an increased packet loss up to 8% for the IPv6 connections. IPv4 connections are not affected.

Customer Impact:

  • Network connectivity has some issues for IPv6.

Thursday 20th April 2023

No incidents reported

Wednesday 19th April 2023

No incidents reported

Tuesday 18th April 2023

No incidents reported

Monday 17th April 2023

No incidents reported

Sunday 16th April 2023

No incidents reported

Saturday 15th April 2023

No incidents reported

Friday 14th April 2023

No incidents reported

Thursday 13th April 2023

API INCIDENT: SysEleven STACK API issues, region CBK

Affected Components: SysEleven Stack API, region CBK

Incident Start: 13th Apr. 2023/02:12 pm

Description:

  • Performance of the SysEleven Stack API is degraded.
  • We are investigating the situation currently.

Customer Impact:

  • Spawning new virtual machines (VMs) or changing existing resources can be slower.

Update:02:32 pm

The situation is worse than a performance degradation, correcting the status towards a partial outage, we are further investigating

Update:03.20 pm

We pinned down the issue around 02:56 and are now observing the situation. In the end not only API but also Compute and storage (Block+Object) storage were affected. Changes to the monitoring and alerting are being planned.

Wednesday 12th April 2023

No incidents reported

Tuesday 11th April 2023

No incidents reported

Monday 10th April 2023

No incidents reported

Sunday 9th April 2023

No incidents reported

Saturday 8th April 2023

No incidents reported

Friday 7th April 2023

No incidents reported

Thursday 6th April 2023

No incidents reported

Wednesday 5th April 2023

SysEleven CloudObserver INCIDENT: SysEleven CloudObserver missing metrics

Affected Components: CloudObserver/Grafana

Incident Start: 5th April 2023/19:00 CEST
Incident Start: 5th April 2023/22:00 CEST

Description:

  • We are currently experiencing issues with the CloudObserver, causing some missing trending data. We identified the problem and are working to resolve the issue.

Customer Impact:

  • Metrics from dashboards in your Grafana might be missing

Update 21:30

  • We identified and fixed the issue, we were also able to recover most metrics. In some rare cases you might experience a gap between ~19:00 and ~20:00
  • We are still closely monitoring the system

Update 22:00

  • All systems are back and fully operational

Tuesday 4th April 2023

API INCIDENT: SysEleven STACK API issues, region CBK

Affected Components: SysEleven Stack API, region CBK

Incident Start: 4th April. 2023/15:30 pm Incident End: 4th April. 2023/15:40 pm

Description:

  • Accessibility of the SysEleven Stack API is not ensured.

Customer Impact:

  • Spawning new virtual machines (VMs) or changing existing resources is not possible.

Update: 15:42 pm The problem has been solved. The API is available as before.

code.syseleven.de INCIDENT: Issues with merge request list on Gitlab code.syseleven.de

Affected Components: code.syseleven.de

Incident Start: 4th April 2023/11:30 CEST Incident End: 4th April 2023/15:30 CEST

Description:

  • Earlier today we discovered that sometimes the merge request list view returns with a http error 500 (e.g. https://code.syseleven.de/my-project/my-repo/-/merge_requests)
  • We are still investigating why this happens

Workaround:

  • It is still possible to view merge requests, if the ID is known, e.g. https://code.syseleven.de/my-project/my-repo/-/merge_requests/15
  • If the ID is unknown, it can be obtained by viewing the branch list (e.g. https://code.syseleven.de/my-project/my-repo/-/branches) and clicking on the commit hash. There will then be a link to the associated merge request

Customer Impact:

  • merge request overview on code.syseleven.de is unavailable sometimes

Update:

  • The problem could be resolved by updating Gitlab to the latest version

Monday 3rd April 2023

MAINTENANCE: MetaKube Platform, scheduled 1 year ago

Affected Components: MetaKube Platform

Scheduled Start: 03. Apr. 2023 / 07:00 pm

Scheduled End: 03. Apr. 2023 / 10:00 pm

State: COMPLETED

Description:

Scheduled maintenance of the MetaKube platform in all regions.

No customer impact expected during the maintenance.

Customer Actions:

  • No customer actions needed
  • Please inform us if you notice any irregularities

Update 08:10 pm

Maintenance completed successfully.

Sunday 2nd April 2023

No incidents reported

Saturday 1st April 2023

No incidents reported

Friday 31st March 2023

MAINTENANCE: MetaKube Core announcement, scheduled 1 year ago

Affected Components: Control Planes for clusters in FES

Scheduled Start: 31. Mar. 2023/07:00 pm Scheduled End: 31. Mar. 2023/09:00 pm

State: ABORTED

Description:

We will replace our load balancer in front of the control planes of MetaKube clusters in the FES region in this maintenance.

Recent incidents indicate that components related to the load balancer may have been corrupted.

Customer Impact during the Maintenance:

During this period, the public IP of the load balancer will change. We do however not expect any disruptions.

The maintenance is only related to clusters in the FES region and on AWS.

Update (08:20 pm):

We had to abort the maintenance after we saw that the new load balancer behaved faulty under load. We reverted to the previous load balancer and the situation is stable. We will investigate the problem further.

Update 05. April 2023

Reason for Outage (RfO) (de)

Control Plane Services (SysEleven - FES) MetaKube cluster apiserver connectivity issues in FES

We're currently experiencing issues with the cluster apiserver Load Balancer in the FES region.

Update (13:08):

A failover of the Load Balancer mitigated the issue. We're watching the situation and will investigate the root cause.

Update (13:30):

It seems, the issues were caused by a failover to a faulty machine. We stopped it and will replace the Load Balancer during a emergency maintenance window, which we will announce later.

Thursday 30th March 2023

No incidents reported