https://syseleven-status.de 2024-09-10T10:32:59.483417+00:00 SysEleven support@syseleven.de python-feedgen https://www.syseleven.de/wp-content/uploads/2020/10/SysEleven_XL_Logo_quer_RGB.png Get all incidents by feed 501 2024-09-10T10:32:59.622547+00:00 <p>Affected Components: <strong>MetaKube Platform, all regions</strong></p> <p>Scheduled Start: <strong>2024-04-09 06:00 UTC+01:00 (CET)</strong></p> <p>Scheduled End: <strong>2024-04-09 07:00 UTC+01:00 (CET)</strong></p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <p>To ensure long-term resiliency and scalability, we migrate all MetaKube clusters' Kubernetes API endpoint to new load balancers. The endpoint will resolve to new IPs, but the migration itself will be seamless.</p> <hr /> <p>Customer Impact during the maintenance:</p> <ul> <li>Certain API actions won't propagate to clusters for up to 30min</li> <li>Interruptions of existing connections to the Kubernetes API during a brief amount of time (5m)</li> <li>Running workloads will <strong>not</strong> be affected</li> <li>Network availability will <strong>not</strong> be affected</li> <li>Storage availability will <strong>not</strong> be affected</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>No customer actions needed</li> <li>Please inform us if you notice any irregularities</li> </ul> <hr /> 2024-04-08T15:34:00+00:00 502 2024-09-10T10:32:59.620597+00:00 <p>Affected Components: <strong>SysEleven Stack, region CBK</strong></p> <p>Incident Start: <strong>2024-04-09 10:34 UTC+01:00 (CET)</strong> Incident End: <strong>2024-04-09 11:00 UTC+01:00 (CET)</strong></p> <hr /> <p>Description:</p> <ul> <li>Occurring errors are currently being investigated.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Connectivity is restricted</li> </ul> 2024-04-09T10:34:00+00:00 503 2024-09-10T10:32:59.619169+00:00 <p>Affected Components: Apiserver connectivity MetaKube clusters in FES</p> <p>Incident Start: <strong>2024-04-30 15:48 UTC+01:00 (CET)</strong> Incident End: <strong>2024-04-30 16:23 UTC+01:00 (CET)</strong></p> <hr /> <p>Description:</p> <p>A corrupt NAT entry in the OpenStack overlay following a restart of a neutron server disrupted connectivity to MetaKube control plane load balancers.</p> <hr /> <p>Customer Impact:</p> <ul> <li>Loss of connectivity to MetaKube apiservers for all clusters in the FES region</li> </ul> <hr /> 2024-04-30T15:48:00+00:00 504 2024-09-10T10:32:59.617723+00:00 <p>Affected Components: SysEleven Stack Networking in region CBK</p> <p>Scheduled Start: <strong>21 May 2024, 22:00 CEST</strong></p> <p>Scheduled End: <strong>22 May 2024, 01:00 CEST</strong></p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <p>We will perform an maintenance on networking components.</p> <hr /> <p>Customer Impact during the Maintenance:</p> <ul> <li>Interruption of connectivity of virtual machines of up to 2 minutes</li> <li>VPN connections could be interrupted for up to 2 minutes</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>no customer actions needed</li> <li>please inform us if you notice any irregularities</li> </ul> <hr /> 2024-05-13T13:14:00+00:00 505 2024-09-10T10:32:59.614975+00:00 <p>Affected Components: SysEleven Stack Networking in region DBL</p> <p>Scheduled Start: <strong>23 May 2024, 22:00 CEST</strong></p> <p>Scheduled End: <strong>24 May 2024, 01:00 CEST</strong></p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <p>We will perform an maintenance on networking components.</p> <hr /> <p>Customer Impact during the Maintenance:</p> <ul> <li>Interruption of connectivity of virtual machines of up to 2 minutes</li> <li>VPN connections could be interrupted for up to 2 minutes</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>no customer actions needed</li> <li>please inform us if you notice any irregularities</li> </ul> <hr /> 2024-05-13T13:16:00+00:00 506 2024-09-10T10:32:59.612582+00:00 <p>Affected Components: <strong>SysEleven Stack network, region CBK</strong></p> <p>Incident Start: <strong>2024-05-16 14:33 UTC+02:00 (CEST)</strong></p> <p>Incident Start: <strong>2024-05-16 15:27 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <h2>- There are network connectivity issues in Region CBK</h2> <p>Customer Impact:</p> <ul> <li>Connectivity is impacted</li> </ul> <hr /> <p><strong>update</strong> 14:55 connectivity is restored</p> <p><strong>update</strong> 15:05 connectivity issues again</p> <p><strong>update</strong> 15:27 connectivity is restored</p> 2024-05-14T14:30:00+00:00 507 2024-09-10T10:32:59.610556+00:00 <p>Affected Components: <strong>SysEleven Stack API, region CBK</strong></p> <p>Incident Start: <strong>2024-05-21 11:22</strong> Incident End: <strong>2024-05-21 11:40</strong></p> <hr /> <p>Description:</p> <ul> <li>Nova and Cinder API have problems, we are investigating the situation currently</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Spawning new virtual machines (VMs) or changing existing resources is not possible.</li> </ul> 2024-05-21T11:22:00+00:00 509 2024-09-10T10:32:59.608199+00:00 <p>Affected Components: <strong>MetaKube Platform, all regions</strong></p> <p>Scheduled Start: **2024-05-22 20:30 UTC+02:00 (CET)</p> <p>Scheduled End: **2024-05-22 21:30 UTC+02:00 (CET)</p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <ul> <li>Minor update on MetaKube infrastructure</li> </ul> <hr /> <p>Customer Impact during the maintenance:</p> <ul> <li>No MetaKube services are impacted</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>No customer actions needed</li> <li>Please inform us if you notice any irregularities</li> </ul> <hr /> 2024-05-22T20:57:28+00:00 510 2024-09-10T10:32:59.605255+00:00 <p>Affected Components: <strong>MetaKube, all regions</strong></p> <p>Incident Start: <strong>2024-05-23 23:07 UTC+01:00 (CEST)</strong></p> <hr /> <p>Description:</p> <ul> <li>The problem has been identified, a solution is currently being worked on.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>MetaKube Dashboard (https://metakube.syseleven.de/) not functional</li> <li>Clusters and their control planes are not impaired</li> </ul> <hr /> <p>Update <strong>2024-05-24 01:08 UTC+01:00 (CEST)</strong></p> <ul> <li>There is an emergency cable rerouting to one of our datacenters. We need to wait for it to be done. It is expected to be done by 2024-05-24 06:00 UTC+01:00 (CEST)</li> </ul> <hr /> <p>Update <strong>2024-05-24 02:19 UTC+01:00 (CEST)</strong></p> <ul> <li>The problem was solved</li> </ul> 2024-05-24T00:32:00+00:00 511 2024-09-10T10:32:59.604035+00:00 <p>Affected Components: <strong>SysEleven Stack</strong></p> <p>Incident Start: <strong>2024-28-05 11:00</strong> Incident End: <strong>2024-28-05 11:44</strong></p> <hr /> <p>Description:</p> <ul> <li>Occurring errors are currently being investigated.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>DNS zones are in ERROR state, we are investigating</li> </ul> <hr /> <p><strong>Update: 11:44</strong></p> <ul> <li>The root cause was identified, monitoring will be extended to speed up resolve time for the future</li> </ul> 2024-05-28T11:00:00+00:00 512 2024-09-10T10:32:59.601973+00:00 <p>Affected Components: SysEleven Stack Networking in region CBK</p> <p>Scheduled Start: <strong>11 Jun 2024, 22:00 CEST</strong></p> <p>Scheduled End: <strong>12 Jun 2024, 00:00 CEST</strong></p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <p>We will perform maintenance on networking gateways.</p> <hr /> <p>Customer Impact during the Maintenance:</p> <ul> <li>Interruption of connectivity of virtual machines of up to 2 minutes</li> <li>VPN connections could be interrupted for up to 2 minutes</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>no customer actions needed</li> <li>please inform us if you notice any irregularities</li> </ul> <hr /> 2024-05-30T15:58:00+00:00 513 2024-09-10T10:32:59.599923+00:00 <p>Affected Components: <strong>SysEleven Stack API, region FES</strong></p> <p>Incident Start: <strong>2024-06-27 10:55:00</strong> Incident End: <strong>2024-06-27 11:08:00</strong></p> <hr /> <p>Description:</p> <ul> <li>Accessibility of the SysEleven Stack API is not ensured.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Spawning new virtual machines (VMs) or changing existing resources is not possible.</li> </ul> 2024-06-27T10:55:00+00:00 514 2024-09-10T10:32:59.597291+00:00 <p>Affected Components: <strong>SysEleven Stack Authoritative DNS</strong></p> <p>Incident Start: <strong>2024-07-27 09:37 UTC+02:00 (CEST)</strong></p> <p>Incident End: <strong>2024-07-27 10:40 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <ul> <li>ns01.cloud.syseleven.net and ns02.cloud.syseleven.net authoritative DNS servers are unavailable</li> <li>(ns03.cloud.syseleven.net and ns04.cloud.syseleven.net are still working)</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>DNS resolution may fail for zones hosted in SysEleven Stack DNSaaS</li> </ul> <hr /> <p><strong>Update: 2024-07-27 10:40 UTC+02:00 (CEST)</strong></p> <ul> <li>We have implemented a workaround meaning that ns01.cloud.syseleven.net and ns02.cloud.syseleven.net are available again and can answer DNS queries</li> <li>We are working on a more sustainable solution</li> </ul> <hr /> <p><strong>Update: 2024-07-27 12:15 UTC+02:00 (CEST)</strong></p> <ul> <li>A more sustainable solution is now in place</li> </ul> 2024-07-27T09:37:00+00:00 515 2024-09-10T10:32:59.595000+00:00 <p>Affected Components: <strong>MetaKube Platform, all regions</strong></p> <p>Scheduled Start: <strong>2024-07-30 18:00 UTC+02:00 (CEST)</strong></p> <p>Scheduled End: <strong>2024-07-30 20:00 UTC+02:00 (CEST)</strong></p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <ul> <li>Scheduled maintenance of the MetaKube platform in all regions</li> <li>Small update to MetaKube infrastructure</li> </ul> <hr /> <p>Customer Impact during the maintenance:</p> <ul> <li>Short interruptions (&lt;5min)</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>No customer actions needed</li> <li>Please inform us if you notice any irregularities</li> </ul> <hr /> 2024-07-30T11:52:00+00:00 517 2024-09-10T10:32:59.592921+00:00 <p>Affected Components: <strong>SysEleven Stack, region CBK</strong></p> <p>Incident Start: <strong>2024-12-08 11:20</strong> Incident End: <strong>2024-12-08 11:38</strong></p> <hr /> <p>Description:</p> <ul> <li>Occurring errors are currently being investigated.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Connectivity is restricted</li> </ul> <hr /> <p>Update : <strong>2024-12-08 11:38</strong> </p> <ul> <li>Network problems were identified, we fixed the underlying issue and are now observing the situation</li> </ul> 2024-08-12T11:20:00+00:00 518 2024-09-10T10:32:59.590325+00:00 <p>Affected Components: SysEleven Stack Networking in region DBL</p> <p>Scheduled Start: <strong>20 Aug 2024, 22:00 CEST</strong></p> <p>Scheduled End: <strong>21 Aug 2024, 00:00 CEST</strong></p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <p>We will perform an maintenance on networking components.</p> <hr /> <p>Customer Impact during the Maintenance:</p> <ul> <li>Interruption of connectivity of virtual machines of up to 2 minutes</li> <li>VPN connections could be interrupted for up to 2 minutes</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>no customer actions needed</li> <li>please inform us if you notice any irregularities</li> </ul> <hr /> 2024-08-12T17:17:00+00:00 519 2024-09-10T10:32:59.588241+00:00 <p>Affected Components: <strong>SysEleven Stack API </strong></p> <p>Incident Start: <strong>2024-08-13 13:33</strong></p> <hr /> <p>Description:</p> <ul> <li>We are currently experiencing a loss of control for DNS zones managed by the DNSaaS API</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Changes towards DNSaaS zones and records are not propagated</li> </ul> <hr /> <p>Update: <strong>2024-08-13 16:10</strong></p> <ul> <li>Root cause was identified and fixed, we are observing the situation</li> </ul> 2024-08-13T13:33:00+00:00 520 2024-09-10T10:32:59.585325+00:00 <p>Affected Components: Setups that route traffic through Cloudflare experience issues</p> <p>Investigation Start: <strong>2024-09-02 13:20 UTC+01:00 (CET)</strong></p> <hr /> <p>Description:</p> <p>We are investigating network problems reported by customers that route traffic through Cloudflare</p> <hr /> <p>Customer Impact:</p> <p>Customers that are routing traffic to SysEleven networks through Cloudflare experience high retransmission rates and traffic loss.</p> <hr /> <p><strong>Update: 2024-09-02 14:20 UTC+01:00 (CET)</strong></p> <p>We are still investigating.</p> <hr /> <p><strong>Update: 2024-09-02 14:49 UTC+01:00 (CET)</strong></p> <p>We are observing significant improvements of the retransmit rates on some affected loadbalancers but will continue to observe and analyse the issues.</p> <hr /> <p><strong>Update: 2024-09-02 16:25 UTC+01:00 (CET)</strong></p> <p>Since the Problem seems to be solved for most customers, we declare the incident as solved. Further investigations into the reasons for the problems will continue.</p> 2024-09-02T14:20:00+00:00 521 2024-09-10T10:32:59.581900+00:00 <p>Affected Components: <strong>MetaKube Platform, all regions</strong></p> <p>Scheduled Start: <strong>2024-09-03 18:00 UTC+01:00 (CET)</strong></p> <p>Scheduled End: <strong>2024-09-03 21:00 UTC+01:00 (CET)</strong></p> <p>State: <strong>SCHEDULED</strong></p> <hr /> <p>Description:</p> <ul> <li>Scheduled maintenance of the MetaKube platform in all regions</li> <li>Small update to MetaKube infrastructure</li> </ul> <hr /> <p>Customer Impact during the maintenance:</p> <ul> <li>Short interruptions (&lt;5min) of the control plane possible (e.g. controller-manager, scheduler), but not API</li> <li>Running workloads will <strong>not</strong> be affected</li> <li>Network availability will <strong>not</strong> be affected</li> <li>Storage availability will <strong>not</strong> be affected</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>No customer actions needed</li> <li>Please inform us if you notice any irregularities</li> </ul> <hr /> 2024-09-03T09:53:42+00:00 522 2024-09-10T10:32:59.579365+00:00 <p>Affected Components: <strong>SysEleven Stack API, region HAM1 and DUS2</strong></p> <p>Incident Start: <strong>2024-09-04 15:20 UTC+02:00 (CEST)</strong></p> <p>Incident End: <strong>2024-09-04 16:30 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <ul> <li>New VMs could not be created</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Spawning new virtual machines (VMs) </li> </ul> <hr /> <p><strong>Update: 16:20</strong></p> <p>We rolled out a configuration change and are observing.</p> <p><strong>Update: 16:30</strong></p> <p>We see that VMs can be created again in our automated tests. We are closing the incident</p> 2024-09-04T15:20:00+00:00