https://syseleven-status.deSysEleven Status and Incidents2024-03-19T08:15:04.994802+00:00SysElevensupport@syseleven.depython-feedgenhttps://www.syseleven.de/wp-content/uploads/2020/10/SysEleven_XL_Logo_quer_RGB.pngGet all incidents by feed475MAINTENANCE: SysEleven STACK announcement (CBK, DBL, FES)2024-03-19T08:15:05.139804+00:00<p>Affected Components: SysEleven Stack API in all regions</p>
<p>Scheduled Start: <strong>30 Jan 2024, 22:00 CET</strong></p>
<p>Scheduled End: <strong>31 Jan 2024, 02:00 CET</strong></p>
<p>State: <strong>SCHEDULED</strong></p>
<hr />
<p>Description:</p>
<p>We will perform maintenance on the central authentication service of the SysEleven Stack.</p>
<hr />
<p>Customer Impact during the Maintenance:</p>
<ul>
<li>The SysEleven Stack APIs might be unavailable for a few seconds.</li>
<li>Running VMs are not affected.</li>
</ul>
<hr />
<p>Customer Actions:</p>
<ul>
<li>no customer actions needed</li>
<li>please inform us if you notice any irregularities</li>
</ul>
<hr />2024-01-22T17:16:07+00:00476MAINTENANCE: MetaKube Platform2024-03-19T08:15:05.137071+00:00<p>Affected Components: <strong>MetaKube Platform, all regions</strong></p>
<p>Scheduled Start: <strong>24. Jan. 2024 / 07:00 pm</strong></p>
<p>Scheduled End: <strong>24. Jan. 2024 / 10:00 pm</strong></p>
<p>State: <strong>COMPLETED</strong></p>
<hr />
<p>Description:</p>
<ul>
<li>Scheduled maintenance of the MetaKube platform in all regions</li>
<li>Small update to MetaKube infrastructure</li>
</ul>
<hr />
<p>Customer Impact during the maintenance:</p>
<ul>
<li>Short interruptions (<5min) of the control plane possible (e.g. controller-manager, scheduler), but not API</li>
<li>Running workloads will <strong>not</strong> be affected</li>
<li>Network availability will <strong>not</strong> be affected</li>
<li>Storage availability will <strong>not</strong> be affected</li>
</ul>
<hr />
<p>Customer Actions:</p>
<ul>
<li>No customer actions needed</li>
<li>Please inform us if you notice any irregularities</li>
</ul>
<hr />2024-01-24T13:55:00+00:00477INCIDENT: SysEleven STACK network issues in region CBK2024-03-19T08:15:05.135216+00:00<p>Affected Components: <strong>SysEleven Stack Networking, region CBK</strong></p>
<p>Incident Start: <strong>27th Jan. 2024/06:57 am</strong></p>
<hr />
<p>Description:</p>
<ul>
<li>We currently experience issues with our SDN in cbk</li>
</ul>
<hr />
<p>Customer Impact:</p>
<ul>
<li>Connectivity is restricted</li>
</ul>
<hr />
<p>Update 07:50 :
The situation has been resolved.</p>2024-01-27T07:00:00+00:00478INCIDENT: SysEleven STACK DNS issues2024-03-19T08:15:05.133658+00:00<p>Affected Components: <strong>SysEleven Stack DNS Issues</strong></p>
<p>Incident Start: <strong>27th Jan. 2024/07:00 am</strong></p>
<hr />
<p>Description:</p>
<ul>
<li>DNS servers overloaded</li>
<li>ns01.cloud.syseleven.net,ns02.cloud.syseleven.net,ns03.cloud.syseleven.net,ns04.cloud.syseleven.net</li>
</ul>
<hr />
<p>Customer Impact:</p>
<ul>
<li>availability of dns servers is reduced</li>
</ul>
<hr />
<p>Update <strong>27 Jan 2024/09:50</strong></p>
<p>Connectivity recovered</p>2024-01-27T07:00:00+00:00479INCIDENT: SysEleven STACK DNS issues2024-03-19T08:15:05.132162+00:00<p>Affected Components: <strong>SysEleven Stack DNS Issues</strong></p>
<h2>Incident Start: <strong>28th Jan. 2024/10:48 pm</strong></h2>
<p>Description:</p>
<ul>
<li>DNS servers overloaded</li>
<li>ns01.cloud.syseleven.net, ns02.cloud.syseleven.net, ns03.cloud.syseleven.net, ns04.cloud.syseleven.net</li>
</ul>
<hr />
<p>Customer Impact:</p>
<ul>
<li>availability of dns servers is reduced</li>
</ul>
<hr />
<p>Update <strong>28 Jan 2024/11:07</strong></p>
<p>Connectivity recovered</p>2024-01-28T23:07:00+00:00480INCIDENT: Partial outage of MetaKube API2024-03-19T08:15:05.130584+00:00<p>Affected Components: <strong>MetaKube platform, region: FES</strong></p>
<p>Incident Start: <strong>30.01.2024 /11:00 am</strong></p>
<hr />
<p>Description:</p>
<p>Provisioning of new clusters does not succeed.</p>
<hr />
<p>Customer Impact:</p>
<p>Currently, no new clusters can be provisioned in the region FES.</p>
<hr />
<p>Customer Actions:</p>
<ul>
<li>No customer actions needed</li>
</ul>
<hr />
<p><strong>UPDATE 12:12 pm</strong></p>
<p>A fix has been implemented and verified, waiting for automatic testing to complete.</p>
<p><strong>UPDATE 12:50 pm</strong></p>
<p>The fix has been rolled out and is being verified.</p>
<p><strong>UPDATE 12:55 pm</strong></p>
<p>The fix has been verified.
Clusters created during the incident should automatically come up now.
If you encounter any issues, please let us know.</p>2024-01-30T11:00:00+00:00481MAINTENANCE: MetaKube Platform2024-03-19T08:15:05.127217+00:00<p>Affected Components: <strong>MetaKube Platform, all regions</strong></p>
<p>Scheduled Start: <strong>05. Feb. 2024 / 6:00 pm</strong></p>
<p>Scheduled End: <strong>05. Feb. 2024 / 7:00 pm</strong></p>
<p>State: <strong>FINISHED</strong></p>
<hr />
<p>Description:</p>
<ul>
<li>Scheduled maintenance of MetaKube platform infrastructure</li>
</ul>
<hr />
<p>Customer Impact during the maintenance:</p>
<ul>
<li>Short interruptions (<1min) of the control plane possible (e.g. apiserver, controller-manager, scheduler), but not MetaKube API</li>
<li>Running workloads will <strong>not</strong> be affected</li>
<li>Network availability will <strong>not</strong> be affected</li>
<li>Storage availability will <strong>not</strong> be affected</li>
</ul>
<hr />
<p>Customer Actions:</p>
<ul>
<li>No customer actions needed</li>
<li>Please inform us if you notice any irregularities</li>
</ul>
<hr />
<p><strong>UPDATE 19:15</strong></p>
<p>Maintenance finished succesfully.</p>2024-01-30T14:12:00+00:00482INCIDENT: SysEleven STACK DNS issues2024-03-19T08:15:05.125842+00:00<p>Affected Components: <strong>SysEleven Stack DNS Issues</strong></p>
<p>Incident Start: <strong>5 Feb 2024 03:03</strong></p>
<hr />
<p>Description:</p>
<ul>
<li>DNS servers overloaded</li>
<li>ns01.cloud.syseleven.net, ns02.cloud.syseleven.net, ns03.cloud.syseleven.net, ns04.cloud.syseleven.net</li>
</ul>
<hr />
<p>Customer Impact:</p>
<ul>
<li>availability of dns servers is reduced</li>
</ul>
<hr />
<p>Update <strong>5 Feb 2024 03:22</strong></p>
<p>Connectivity recovered</p>2024-02-05T03:03:00+00:00483INCIDENT: SysEleven STACK DNS issues2024-03-19T08:15:05.123818+00:00<p>Affected Components: <strong>SysEleven Stack DNS Issues</strong></p>
<p>Incident Start: <strong>6 Feb 2024 17:44</strong>
Incident End: <strong>6 Feb 2024 17:59</strong></p>
<hr />
<p>Description:</p>
<ul>
<li>DNS servers overloaded</li>
<li>ns01.cloud.syseleven.net, ns02.cloud.syseleven.net, ns03.cloud.syseleven.net, ns04.cloud.syseleven.net</li>
</ul>
<hr />
<p>Customer Impact:</p>
<ul>
<li>availability of dns servers is reduced</li>
</ul>
<hr />
<p>Update 18:00</p>
<p>Connectivity recovered</p>2024-02-06T17:44:00+00:00484INCIDENT: SysEleven STACK DNS issues2024-03-19T08:15:05.121555+00:00<p>Affected Components: <strong>SysEleven Stack DNS Issues</strong></p>
<p>Incident Start: <strong>7 Feb 2024 12:15</strong>
Incident End: <strong>7 Feb 2024 12:35</strong></p>
<hr />
<p>Description:</p>
<ul>
<li>DNS servers overloaded</li>
<li>ns01.cloud.syseleven.net, ns02.cloud.syseleven.net, ns03.cloud.syseleven.net, ns04.cloud.syseleven.net</li>
</ul>
<hr />
<p>Customer Impact:</p>
<ul>
<li>availability of dns servers is reduced</li>
</ul>
<hr />
<p>Update <strong>12:35</strong></p>
<p>Connectivity recovered</p>2024-02-07T12:15:00+00:00485MAINTENANCE: MetaKube announcement2024-03-19T08:15:05.118109+00:00<p>Affected Components: <strong>all regions</strong></p>
<p>Scheduled Start: <strong>07. Feb. 2024/06:00 pm</strong>
Scheduled End: <strong>07. Feb. 2024/09:00 pm</strong></p>
<p>State: <strong>COMPLETED</strong></p>
<hr />
<p>Description:</p>
<p>Infrastructure security update in all regions.</p>
<hr />
<p>Customer Impact during the Maintenance:</p>
<ul>
<li>Short interruptions (<5min) of the control plane possible (e.g. controller-manager, scheduler), but not API</li>
<li>Running workloads will <strong>not</strong> be affected</li>
<li>Network availability will <strong>not</strong> be affected</li>
<li>Storage availability will <strong>not</strong> be affected</li>
</ul>
<hr />
<p>Customer Actions:</p>
<ul>
<li>No customer actions needed</li>
<li>Please inform us if you notice any irregularities</li>
</ul>
<hr />2024-02-07T07:00:00+00:00486MAINTENANCE: Dringende Wartungsarbeiten / Urgent maintenance work network component Fortigate2024-03-19T08:15:05.114323+00:00<p>Affected Components: <strong>Network Component Fortigate</strong></p>
<p>Scheduled Start: <strong>Feb. 11 2024 / 08:00 p.m.</strong></p>
<p>Scheduled End: <strong>Feb. 11 2024 / 11: p.m.</strong></p>
<p>State: <strong>Scheduled Maintainance</strong></p>
<hr />
<p>Description:</p>
<p>Urgent security update to Fortigate network component</p>
<p>(FortiOS - Out-of-bound Write in sslvpnd (RCE) (CVE-2024-21762))</p>
<p>Expected Impact:</p>
<p>About 1-2 short disconnections when the services are panning.</p>
<hr />
<p>Customer Impact during the Maintenance:</p>
<ul>
<li>Network Service might be interrupted when actual maintainance takes place</li>
</ul>
<hr />
<p>Update <strong>4:30 p.m.</strong></p>
<p>After further evaluating options to mitigate or patch the disclosed vulnerabilities, we learned that patches were anounced but not yet available for download.
We continue to monitor the situation closely and will act as soon as patches and upgrades are available.</p>
<hr />
<p>Update <strong>Sunday Feb. 11 7:00 p.m.</strong></p>
<p>After evaluating and testing the provided FortiOS Updates in our datacenter Lankwitzer Straße (BLA) <strong>we will be rolling out the patches today between 8 and 11 p.m. in our datacenters BKI and BLU</strong>. We have contacted affected customers separately. As the maintenance work affects our VPN routers, short outages of VPN connections are to be expected during the maintenance window mentioned.</p>
<hr />
<p>Update <strong>Sunday Feb. 11 11:00 p.m.</strong>
All instances of FortiOS in our datacenters have been updated to versions secure against CVE-2024-21762.</p>2024-02-11T20:00:00+00:00487INCIDENT: Partial outage of MetaKube apiserver reachability DBL/CBK2024-03-19T08:15:05.112107+00:00<p>Affected Components: <strong>MetaKube apiserver external availability, some clusters, DBL/CBK</strong></p>
<p>Incident Start: <strong>10.Feb 2024/11:00 am</strong>
Incident End: <strong>10. Feb. 2024/12:35 am</strong></p>
<hr />
<p>Description:</p>
<p>An expired client certificate caused some load balancer configuration to go stale.
Thus, a small number of clusters lost external availability of their apiservers.</p>
<hr />
<p>Customer Impact:</p>
<ul>
<li>External apiserver availability disrupted for at most 8 clusters at peak for a maximum of 95min</li>
</ul>
<hr />
<p>Customer Actions:</p>
<p>Examples:
- No customer actions needed</p>
<hr />
<p><strong>Update 12:35</strong>
The certificate was renewed and a failover of the load balancer triggered.
This restored availability of all clusters again.</p>2024-02-10T11:00:00+00:00489INCIDENT: SysEleven STACK Object Storage issues, region FES2024-03-19T08:15:05.109600+00:00<p>Affected Components: <strong>SysEleven Stack Object Storage, region FES</strong></p>
<p>Incident Start: <strong>12th Feb. 2024/13:00 pm (CET - UTC +1)</strong></p>
<p>Incident End: <strong>12th Feb. 2024/14:15 pm (CET - UTC +1)</strong></p>
<hr />
<p>Description:</p>
<p>At the moment, we are facing issues with the Object Storage (Ceph) in Region FES.</p>
<hr />
<p>Customer Impact:</p>
<ul>
<li>API is not working correctly</li>
</ul>
<hr />
<p>Update <strong>14:18 p.m.</strong></p>
<ul>
<li>Issue is fixed and service is operational</li>
</ul>
<hr />2024-02-12T13:54:00+00:00490MAINTENANCE: MetaKube Platform2024-03-19T08:15:05.106265+00:00<p>Affected Components: <strong>MetaKube Platform, all regions</strong></p>
<p>Scheduled Start: <strong>19. Feb 2024 / 06:00 pm</strong></p>
<p>Scheduled End: <strong>19. Feb. 2024 / 9:00 pm</strong></p>
<p>State: <strong>FINISHED</strong></p>
<hr />
<p>Description:</p>
<ul>
<li>Scheduled maintenance of the MetaKube platform in all regions</li>
</ul>
<hr />
<p>Customer Impact during the maintenance:</p>
<ul>
<li>Short interruptions (<5min) of the control plane possible (e.g. controller-manager, scheduler), but not API</li>
<li>Running workloads will <strong>not</strong> be affected</li>
<li>Network availability will <strong>not</strong> be affected</li>
<li>Storage availability will <strong>not</strong> be affected</li>
</ul>
<hr />
<p>Customer Actions:</p>
<ul>
<li>No customer actions needed</li>
<li>Please inform us if you notice any irregularities</li>
</ul>
<hr />2024-02-19T18:00:00+00:00491INCIDENT: SysEleven STACK DNS issues2024-03-19T08:15:05.104415+00:00<p>Affected Components: <strong>SysEleven Stack DNS Issues</strong></p>
<p>Incident Start: <strong>19 Feb 2024 08:30 CET</strong></p>
<p>Incident End: <strong>19 Feb 2024 08:50 CET</strong></p>
<hr />
<p>Description:</p>
<ul>
<li>DNS servers overloaded</li>
<li>ns01.cloud.syseleven.net, ns02.cloud.syseleven.net, ns03.cloud.syseleven.net, ns04.cloud.syseleven.net</li>
</ul>
<hr />
<p>Customer Impact:</p>
<ul>
<li>availability of dns servers is reduced</li>
</ul>
<hr />
<p>Update <strong>19 Feb 2024 08:50 CET</strong></p>
<p>Connectivity recovered</p>2024-02-19T08:30:00+00:00492INCIDENT: PVC Failure of a hardware node, region blu12024-03-19T08:15:05.102395+00:00<p>Affected Components: <strong>Hardware node failure, region blu</strong></p>
<p>Incident Start: <strong>2024-02-23 22:35 UTC+01:00 (CET)</strong>
Incident End: <strong>2024-02-23 23:45 UTC+01:00 (CET)</strong></p>
<hr />
<p>Description:</p>
<ul>
<li>Malfunction of a hardware node</li>
<li>Restart of the hardware node is necessary </li>
<li>Evacuation of custmer system was necessary</li>
</ul>
<hr />
<p>Customer Impact:</p>
<ul>
<li>During the period of restart, there will be a short interruption in the availability of the systems.</li>
<li>Affected customers were notified via E-Mail.</li>
<li>Please check the affected systems for their full functionality.</li>
</ul>2024-02-23T22:35:00+00:00493MAINTENANCE: MetaKube Platform2024-03-19T08:15:05.098338+00:00<p>Affected Components: <strong>MetaKube Platform, all regions</strong></p>
<p>Scheduled Start: **2024-02-29 18:00 UTC+01:00 (CET) </p>
<p>Scheduled End: **2024-02-29 21:00 UTC+01:00 (CET)</p>
<p>State: <strong>COMPLETED</strong></p>
<hr />
<p>Description:</p>
<ul>
<li>Scheduled maintenance of the MetaKube platform in all regions</li>
</ul>
<hr />
<p>Customer Impact during the maintenance:</p>
<ul>
<li>Short interruptions (<5min) of the control plane possible (e.g. controller-manager, scheduler).</li>
<li>Running workloads will <strong>not</strong> be affected</li>
<li>Network availability will <strong>not</strong> be affected</li>
<li>Storage availability will <strong>not</strong> be affected</li>
</ul>
<hr />
<p>Customer Actions:</p>
<ul>
<li>No customer actions needed</li>
<li>Please inform us if you notice any irregularities</li>
</ul>
<hr />2024-02-27T15:29:56+00:00494MAINTENANCE: MetaKube Platform Customer Cluster LB maintenance2024-03-19T08:15:05.094365+00:00<p>Affected Components: <strong>MetaKube Platform, all regions</strong></p>
<p>Scheduled Start: **2024-03-07 18:00 UTC+01:00 (CET) </p>
<p>Scheduled End: **2024-03-07 19:00 UTC+01:00 (CET) </p>
<p>State: <strong>COMPLETED</strong></p>
<hr />
<p>Description:</p>
<ul>
<li>Scheduled maintenance of the MetaKube platform in all regions</li>
<li>Small update to MetaKube infrastructure</li>
</ul>
<hr />
<p>Customer Impact during the maintenance:</p>
<ul>
<li>Short interruptions (<5min) of the control plane possible (e.g. controller-manager, scheduler), but not API</li>
<li>Running workloads will <strong>not</strong> be affected</li>
<li>Network availability will <strong>not</strong> be affected</li>
<li>Storage availability will <strong>not</strong> be affected</li>
</ul>
<hr />
<p>Customer Actions:</p>
<ul>
<li>No customer actions needed</li>
<li>Please inform us if you notice any irregularities</li>
</ul>
<hr />2024-03-05T13:02:00+00:00495INCIDENT: SysEleven MetaKube Dashboard API issues2024-03-19T08:15:05.091335+00:00<p>Affected Components: MetaKube Dashboard API</p>
<p>Incident Start: <strong>2024-03-012 16:00 UTC+01:00 (CET)</strong></p>
<hr />
<p>Description:</p>
<ul>
<li>Accessibility of the SysEleven MetaKube Dashboard API is not ensured.</li>
<li>Clusters may not be reconciled currently</li>
</ul>
<hr />
<p>Customer Impact:</p>
<ul>
<li>Access to MetaKube Dashboard and API (including Terraform) might be not possible</li>
<li>Clusters may not be reconciled currently</li>
</ul>
<hr />
<p>Update: <strong>2024-03-012 17:30 UTC+01:00 (CET)</strong></p>
<ul>
<li>Issues are fixed</li>
</ul>2024-03-12T16:00:00+00:00