All systems are operational

About This Site

Timezone: Europe/Berlin (CET - UTC +1/CEST - UTC +2)

We provide current information about infrastructure and service availability below. If you experience service impacts or performance issues please contact our helpdesk or our Service Desk.

Past Incidents

Thursday 29th February 2024

MAINTENANCE: MetaKube Platform, scheduled 1 month ago

Affected Components: MetaKube Platform, all regions

Scheduled Start: **2024-02-29 18:00 UTC+01:00 (CET)

Scheduled End: **2024-02-29 21:00 UTC+01:00 (CET)

State: COMPLETED


Description:

  • Scheduled maintenance of the MetaKube platform in all regions

Customer Impact during the maintenance:

  • Short interruptions (<5min) of the control plane possible (e.g. controller-manager, scheduler).
  • Running workloads will not be affected
  • Network availability will not be affected
  • Storage availability will not be affected

Customer Actions:

  • No customer actions needed
  • Please inform us if you notice any irregularities

Wednesday 28th February 2024

No incidents reported

Tuesday 27th February 2024

No incidents reported

Monday 26th February 2024

No incidents reported

Sunday 25th February 2024

No incidents reported

Saturday 24th February 2024

No incidents reported

Friday 23rd February 2024

Virtual Machines INCIDENT: PVC Failure of a hardware node, region blu1

Affected Components: Hardware node failure, region blu

Incident Start: 2024-02-23 22:35 UTC+01:00 (CET) Incident End: 2024-02-23 23:45 UTC+01:00 (CET)


Description:

  • Malfunction of a hardware node
  • Restart of the hardware node is necessary
  • Evacuation of custmer system was necessary

Customer Impact:

  • During the period of restart, there will be a short interruption in the availability of the systems.
  • Affected customers were notified via E-Mail.
  • Please check the affected systems for their full functionality.

Thursday 22nd February 2024

No incidents reported

Wednesday 21st February 2024

No incidents reported

Tuesday 20th February 2024

No incidents reported

Monday 19th February 2024

SysEleven Stack Authoritative DNS INCIDENT: SysEleven STACK DNS issues

Affected Components: SysEleven Stack DNS Issues

Incident Start: 19 Feb 2024 08:30 CET

Incident End: 19 Feb 2024 08:50 CET


Description:

  • DNS servers overloaded
  • ns01.cloud.syseleven.net, ns02.cloud.syseleven.net, ns03.cloud.syseleven.net, ns04.cloud.syseleven.net

Customer Impact:

  • availability of dns servers is reduced

Update 19 Feb 2024 08:50 CET

Connectivity recovered

Sunday 18th February 2024

MAINTENANCE: MetaKube Platform, scheduled 2 months ago

Affected Components: MetaKube Platform, all regions

Scheduled Start: 19. Feb 2024 / 06:00 pm

Scheduled End: 19. Feb. 2024 / 9:00 pm

State: FINISHED


Description:

  • Scheduled maintenance of the MetaKube platform in all regions

Customer Impact during the maintenance:

  • Short interruptions (<5min) of the control plane possible (e.g. controller-manager, scheduler), but not API
  • Running workloads will not be affected
  • Network availability will not be affected
  • Storage availability will not be affected

Customer Actions:

  • No customer actions needed
  • Please inform us if you notice any irregularities

Saturday 17th February 2024

No incidents reported

Friday 16th February 2024

No incidents reported

Thursday 15th February 2024

No incidents reported

Wednesday 14th February 2024

No incidents reported

Tuesday 13th February 2024

No incidents reported

Monday 12th February 2024

Object storage (S3 / SEOS) INCIDENT: SysEleven STACK Object Storage issues, region FES

Affected Components: SysEleven Stack Object Storage, region FES

Incident Start: 12th Feb. 2024/13:00 pm (CET - UTC +1)

Incident End: 12th Feb. 2024/14:15 pm (CET - UTC +1)


Description:

At the moment, we are facing issues with the Object Storage (Ceph) in Region FES.


Customer Impact:

  • API is not working correctly

Update 14:18 p.m.

  • Issue is fixed and service is operational

Sunday 11th February 2024

SysEleven Network MAINTENANCE: Dringende Wartungsarbeiten / Urgent maintenance work network component Fortigate

Affected Components: Network Component Fortigate

Scheduled Start: Feb. 11 2024 / 08:00 p.m.

Scheduled End: Feb. 11 2024 / 11: p.m.

State: Scheduled Maintainance


Description:

Urgent security update to Fortigate network component

(FortiOS - Out-of-bound Write in sslvpnd (RCE) (CVE-2024-21762))

Expected Impact:

About 1-2 short disconnections when the services are panning.


Customer Impact during the Maintenance:

  • Network Service might be interrupted when actual maintainance takes place

Update 4:30 p.m.

After further evaluating options to mitigate or patch the disclosed vulnerabilities, we learned that patches were anounced but not yet available for download. We continue to monitor the situation closely and will act as soon as patches and upgrades are available.


Update Sunday Feb. 11 7:00 p.m.

After evaluating and testing the provided FortiOS Updates in our datacenter Lankwitzer Straße (BLA) we will be rolling out the patches today between 8 and 11 p.m. in our datacenters BKI and BLU. We have contacted affected customers separately. As the maintenance work affects our VPN routers, short outages of VPN connections are to be expected during the maintenance window mentioned.


Update Sunday Feb. 11 11:00 p.m. All instances of FortiOS in our datacenters have been updated to versions secure against CVE-2024-21762.

Saturday 10th February 2024

Control Plane Services (SysEleven - CBK) INCIDENT: Partial outage of MetaKube apiserver reachability DBL/CBK

Affected Components: MetaKube apiserver external availability, some clusters, DBL/CBK

Incident Start: 10.Feb 2024/11:00 am Incident End: 10. Feb. 2024/12:35 am


Description:

An expired client certificate caused some load balancer configuration to go stale. Thus, a small number of clusters lost external availability of their apiservers.


Customer Impact:

  • External apiserver availability disrupted for at most 8 clusters at peak for a maximum of 95min

Customer Actions:

Examples:

  • No customer actions needed

Update 12:35 The certificate was renewed and a failover of the load balancer triggered. This restored availability of all clusters again.

Friday 9th February 2024

No incidents reported

Thursday 8th February 2024

No incidents reported

Wednesday 7th February 2024

MAINTENANCE: MetaKube announcement, scheduled 2 months ago

Affected Components: all regions

Scheduled Start: 07. Feb. 2024/06:00 pm Scheduled End: 07. Feb. 2024/09:00 pm

State: COMPLETED


Description:

Infrastructure security update in all regions.


Customer Impact during the Maintenance:

  • Short interruptions (<5min) of the control plane possible (e.g. controller-manager, scheduler), but not API
  • Running workloads will not be affected
  • Network availability will not be affected
  • Storage availability will not be affected

Customer Actions:

  • No customer actions needed
  • Please inform us if you notice any irregularities

Network INCIDENT: SysEleven STACK DNS issues

Affected Components: SysEleven Stack DNS Issues

Incident Start: 7 Feb 2024 12:15 Incident End: 7 Feb 2024 12:35


Description:

  • DNS servers overloaded
  • ns01.cloud.syseleven.net, ns02.cloud.syseleven.net, ns03.cloud.syseleven.net, ns04.cloud.syseleven.net

Customer Impact:

  • availability of dns servers is reduced

Update 12:35

Connectivity recovered

Tuesday 6th February 2024

Network INCIDENT: SysEleven STACK DNS issues

Affected Components: SysEleven Stack DNS Issues

Incident Start: 6 Feb 2024 17:44 Incident End: 6 Feb 2024 17:59


Description:

  • DNS servers overloaded
  • ns01.cloud.syseleven.net, ns02.cloud.syseleven.net, ns03.cloud.syseleven.net, ns04.cloud.syseleven.net

Customer Impact:

  • availability of dns servers is reduced

Update 18:00

Connectivity recovered

Monday 5th February 2024

MAINTENANCE: MetaKube Platform, scheduled 2 months ago

Affected Components: MetaKube Platform, all regions

Scheduled Start: 05. Feb. 2024 / 6:00 pm

Scheduled End: 05. Feb. 2024 / 7:00 pm

State: FINISHED


Description:

  • Scheduled maintenance of MetaKube platform infrastructure

Customer Impact during the maintenance:

  • Short interruptions (<1min) of the control plane possible (e.g. apiserver, controller-manager, scheduler), but not MetaKube API
  • Running workloads will not be affected
  • Network availability will not be affected
  • Storage availability will not be affected

Customer Actions:

  • No customer actions needed
  • Please inform us if you notice any irregularities

UPDATE 19:15

Maintenance finished succesfully.

Network INCIDENT: SysEleven STACK DNS issues

Affected Components: SysEleven Stack DNS Issues

Incident Start: 5 Feb 2024 03:03


Description:

  • DNS servers overloaded
  • ns01.cloud.syseleven.net, ns02.cloud.syseleven.net, ns03.cloud.syseleven.net, ns04.cloud.syseleven.net

Customer Impact:

  • availability of dns servers is reduced

Update 5 Feb 2024 03:22

Connectivity recovered

Sunday 4th February 2024

No incidents reported

Saturday 3rd February 2024

No incidents reported

Friday 2nd February 2024

No incidents reported

Thursday 1st February 2024

No incidents reported

Wednesday 31st January 2024

No incidents reported

Tuesday 30th January 2024

Control Plane Services (SysEleven - All regions) INCIDENT: Partial outage of MetaKube API

Affected Components: MetaKube platform, region: FES

Incident Start: 30.01.2024 /11:00 am


Description:

Provisioning of new clusters does not succeed.


Customer Impact:

Currently, no new clusters can be provisioned in the region FES.


Customer Actions:

  • No customer actions needed

UPDATE 12:12 pm

A fix has been implemented and verified, waiting for automatic testing to complete.

UPDATE 12:50 pm

The fix has been rolled out and is being verified.

UPDATE 12:55 pm

The fix has been verified. Clusters created during the incident should automatically come up now. If you encounter any issues, please let us know.