https://syseleven-status.de 2024-12-03T02:01:02.329358+00:00 SysEleven support@syseleven.de python-feedgen https://www.syseleven.de/wp-content/uploads/2020/10/SysEleven_XL_Logo_quer_RGB.png Get all incidents by feed 523 2024-12-03T02:01:02.454280+00:00 <p>Affected Components: <strong>SysEleven Stack Designate API</strong></p> <p>Incident Start: <strong>2024-09-17 20:42 UTC+02:00 (CEST)</strong></p> <p>Incident End: <strong>2024-09-17 22:15 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <ul> <li>Managing DNS zones and records through the Designate API may fail.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Loss of control for DNS resources.</li> </ul> <hr /> <p><strong>Update: 2024-09-17 21:20 UTC+02:00 (CEST)</strong></p> <p>At 21:15, we reverted a configuration change that was rolled out earlier today, which fixed the issue. We are watching the situation.</p> <hr /> <p><strong>Update: 2024-09-17 22:15 UTC+02:00 (CEST)</strong></p> <p>Underlying issues that were persistent, but not impacting the API, are now fully resolved. Incident is over.</p> <hr /> 2024-09-17T20:42:00+00:00 525 2024-12-03T02:01:02.452703+00:00 <p>Affected Components: <strong>SysEleven Stack API, region DBL</strong></p> <p>Incident Start: <strong>2024-09-19 08:23</strong> Incident End: <strong>2024-09-19 08:35</strong></p> <hr /> <p>Description:</p> <ul> <li>Loss of control situation for volume and compute services</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Creating new virtual machines (VMs) or changing existing resources is not possible.</li> </ul> <hr /> <p>Update: <strong>2024-09-18 08:35</strong></p> <p>Some nova services also seem to be affected, we investigated the situation and bring the services back up</p> 2024-09-19T08:23:00+00:00 526 2024-12-03T02:01:02.450837+00:00 <p>Affected Components: <strong>SysEleven Stack Block Storage, region CBK</strong></p> <p>Incident Start: <strong>2024-09-24 10:55 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <p>At the moment we are facing issues with the Block Storage in Region CBK.</p> <hr /> <p>Customer Impact:</p> <ul> <li>Volumes may be unavailable</li> <li>Instances booted from a volume may be unavailable</li> </ul> <hr /> <p><strong>Update: 2024-09-24 11:00 UTC+02:00 (CEST)</strong></p> <p>Issue identified and we are fixing the problem.</p> <hr /> <p><strong>Update: 2024-09-24 12:15 UTC+02:00 (CEST)</strong></p> <p>Problem is fixed. Instances using volumes had to be restarted.</p> 2024-09-24T10:55:00+00:00 528 2024-12-03T02:01:02.448327+00:00 <p>Affected Components: <strong>MetaKube Platform, all regions</strong></p> <p>Scheduled Start: <strong>2024-09-26 20:30 UTC+01:00 (CET)</strong></p> <p>Scheduled End: <strong>2024-09-26 22:30 UTC+01:00 (CET)</strong></p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <ul> <li>Scheduled maintenance of the MetaKube platform in all regions</li> <li>Small update to MetaKube infrastructure</li> </ul> <hr /> <p>Customer Impact during the maintenance:</p> <ul> <li>Short interruptions (&lt;5min) of the control plane possible (e.g. controller-manager, scheduler), but not API</li> <li>Running workloads will <strong>not</strong> be affected</li> <li>Network availability will <strong>not</strong> be affected</li> <li>Storage availability will <strong>not</strong> be affected</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>No customer actions needed</li> <li>Please inform us if you notice any irregularities</li> </ul> <hr /> 2024-09-26T20:33:00+00:00 529 2024-12-03T02:01:02.446406+00:00 <p>Affected Components: <strong>Hardware node failure, region XXX</strong></p> <p>Incident Start: <strong>2024-10-01 09:15 UTC+02:00 (CEST)</strong> Incident End: <strong>2024-10-01 09:36 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <ul> <li>Malfunction of a hardware node</li> <li>Restart of the hardware node is necessary </li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>During the period of restart, there will be a short interruption in the availability of the systems.</li> <li>Affected customers were notified via E-Mail.</li> <li>Please check the affected systems for their full functionality.</li> </ul> 2024-10-01T09:15:00+00:00 530 2024-12-03T02:01:02.444716+00:00 <p>Affected Components: <strong>SysEleven Stack Network, region DBL</strong></p> <p>Incident Start: <strong>2024-10-01 14:17 UTC+02:00 (CEST)</strong></p> <p>Incident End: <strong>2024-10-01 14:34 UTC+02:00 (CEST)</strong></p> <hr /> <p>Description:</p> <p>At the moment, we are facing issues with the Network in Region DBL.</p> <hr /> <p>Customer Impact:</p> <ul> <li>Network performance issue</li> </ul> <hr /> <p><strong>Update: 2024-10-01 14:34 UTC+02:00 (CEST)</strong></p> <p>The incident is over, and all services are operational.</p> <hr /> 2024-10-01T14:17:00+00:00 531 2024-12-03T02:01:02.442923+00:00 <p>Affected Components: SysEleven Stack Networking in region CBK</p> <p>Scheduled Start: <strong>16 Oct 2024, 22:00 CEST</strong></p> <p>Scheduled End: <strong>16 Oct 2024, 23:00 CEST</strong></p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <p>We will perform maintenance on networking switches.</p> <hr /> <p>Customer Impact during the Maintenance:</p> <ul> <li>No impact is expected</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>no customer actions needed</li> <li>please inform us if you notice any irregularities</li> </ul> <hr /> 2024-10-08T16:09:00+00:00 532 2024-12-03T02:01:02.440295+00:00 <p>Affected Components: <strong>SysEleven Stack, region HAM1</strong></p> <p>Incident Start: <strong>2024-10-10 23:55</strong> Incident End: <strong>2024-10-11 00:00</strong></p> <hr /> <p>Description:</p> <ul> <li>Occurring errors were investigated. </li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Connectivity was restricted</li> </ul> <hr /> <p><strong>Update: 01:00</strong></p> <ul> <li>We can observe further short term issues with the HAM1 region connectivity, the provider is aware of the issues and is currently investigating the situation</li> </ul> <hr /> <p><strong>Update: 01:30</strong></p> <ul> <li>The network provider is proceeding with a network maintenance until 05:00, we are on standby</li> </ul> 2024-10-10T23:55:00+00:00 533 2024-12-03T02:01:02.434530+00:00 <p>Affected Components: <strong>MetaKube Platform, all regions</strong></p> <p>Scheduled Start: <strong>2024-10-29 17:00 UTC+01:00 (CET)</strong></p> <p>Scheduled End: <strong>2024-10-29 20:00 UTC+01:00 (CET)</strong></p> <p>State: <strong>DONE</strong></p> <hr /> <p>Description:</p> <ul> <li>Scheduled maintenance of the MetaKube platform in all regions</li> <li>Small update to MetaKube infrastructure</li> </ul> <hr /> <p>Customer Impact during the maintenance:</p> <ul> <li>Short interruptions (&lt;5min) of the control plane possible (e.g. controller-manager, scheduler), but not API</li> <li>Running workloads will <strong>not</strong> be affected</li> <li>Network availability will <strong>not</strong> be affected</li> <li>Storage availability will <strong>not</strong> be affected</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>No customer actions needed</li> <li>Please inform us if you notice any irregularities</li> </ul> <hr /> 2024-10-29T15:01:00+00:00 534 2024-12-03T02:01:02.431383+00:00 <p>Affected Components: <strong>SysEleven Stack API</strong></p> <p>Incident Start: <strong>2024-10-30 09:30 CET</strong></p> <p>Incident End: <strong>2024-10-30 12:05 CET</strong></p> <hr /> <p>Description:</p> <ul> <li>Accessibility of the SysEleven Stack API is not ensured.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Spawning new virtual machines (VMs) or changing existing resources is not possible.</li> </ul> <hr /> <p><strong>Update: 10:40</strong></p> <p>We are still investigating the situation and are in contact with our external network provider to further analyze the problems.</p> <hr /> <p><strong>Update: 11:30</strong></p> <p>The issue has been identified, we are waiting for our external network provider to further fix the situation.</p> <hr /> <p><strong>Update: 12:05</strong></p> <p>The issue has been resolved.</p> 2024-10-30T09:30:00+00:00 535 2024-12-03T02:01:02.429440+00:00 <p>Affected Components: SysEleven Stack Networking in region CBK</p> <p>Scheduled Start: <strong>12 Nov 2024, 22:00 CET</strong></p> <p>Scheduled End: <strong>13 Nov 2024, 00:00 CET</strong></p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <p>We will perform maintenance on networking gateways.</p> <hr /> <p>Customer Impact during the Maintenance:</p> <ul> <li>Interruption of connectivity of virtual machines of up to 2 minutes</li> <li>VPN connections could be interrupted for up to 2 minutes</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>no customer actions needed</li> <li>please inform us if you notice any irregularities</li> </ul> <hr /> 2024-11-04T11:56:00+00:00 536 2024-12-03T02:01:02.427239+00:00 <p>Affected Components: SysEleven Stack Networking in region DBL</p> <p>Scheduled Start: <strong>19 Nov 2024, 22:00 CET</strong></p> <p>Scheduled End: <strong>20 Nov 2024, 00:00 CET</strong></p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <p>We will perform maintenance on networking gateways.</p> <hr /> <p>Customer Impact during the Maintenance:</p> <ul> <li>Interruption of connectivity of virtual machines of up to 2 minutes</li> <li>VPN connections could be interrupted for up to 2 minutes</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>no customer actions needed</li> <li>please inform us if you notice any irregularities</li> </ul> <hr /> 2024-11-07T14:19:00+00:00 537 2024-12-03T02:01:02.425246+00:00 <p>Affected Components: <strong>SysEleven Stack Object Storage, region DBL</strong></p> <p>Incident Start: <strong>2024-11-12 17:45 UTC+01:00 (CET)</strong></p> <p>Incident End: <strong>2024-11-12 18:40 UTC+01:00 (CET)</strong></p> <hr /> <p>Description:</p> <p>At the moment we are facing issues with the Object Storage in Region DBL.</p> <hr /> <p>Customer Impact:</p> <ul> <li>Writing or reading of objects maybe restricted.</li> </ul> <hr /> <p>Update: <strong>2024-11-12 18:40 UTC+01:00 (CET)</strong></p> <p>We mitigated the problem and do further investigation</p> 2024-11-12T17:45:00+00:00 538 2024-12-03T02:01:02.422408+00:00 <p>Affected Components: <strong>SysEleven Stack, region CBK</strong></p> <p>Incident Start: <strong>2024-11-12 22:35</strong> Incident End: <strong>2024-11-13 00:00</strong></p> <hr /> <p>Description:</p> <ul> <li>Occurring errors are currently being investigated.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Connectivity is restricted</li> </ul> <hr /> <p><strong>Update: 23:08</strong></p> <p>The announced maintenance is having a bigger impact than expected, we are investigating the situation</p> <hr /> <p><strong>Update: 23:45</strong></p> <p>We were able to pin down the rootcause and prepare a fix to mitigate the problems</p> <hr /> <p><strong>Update: 12:00</strong></p> <p>The network problems were mitigated. If you still encounter issues please contact us!</p> 2024-11-12T22:35:00+00:00 539 2024-12-03T02:01:02.420322+00:00 <p>Affected Components: SysEleven Stack Networking in region CBK</p> <p>Scheduled Start: <strong>14 Nov 2024, 22:00 CET</strong></p> <p>Scheduled End: <strong>15 Nov 2024, 01:00 CET</strong></p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <p>We will perform an emergency maintenance on networking gateways in order to improve performance settings.</p> <hr /> <p>Customer Impact during the Maintenance:</p> <ul> <li>Interruption of connectivity of virtual machines of up to 2 minutes</li> <li>VPN connections could be interrupted for up to 2 minutes</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>no customer actions needed</li> <li>please inform us if you notice any irregularities</li> </ul> <hr /> 2024-11-14T17:14:00+00:00 541 2024-12-03T02:01:02.417732+00:00 <p>Affected Components: <strong>MetaKube Control Plane Services, region FES</strong></p> <p>Incident Start: <strong>2024-11-18 11:30 UTC+01:00 (CET)</strong></p> <hr /> <p>Description:</p> <p>Infrastructure hosting the MetaKube Control Plane Services has problems.</p> <hr /> <p>Customer Impact:</p> <ul> <li>MetaKube Control Plane might be slow or not answering</li> </ul> <hr /> <p><strong>UPDATE 2024-11-18 12:30 UTC+01:00 (CET)</strong></p> <p>We have identified networking problems as the cause, currently working to resolve them.</p> <p><strong>UPDATE 2024-11-18 13:27 UTC+01:00 (CET)</strong></p> <p>We have increased conntrack table size on hardware nodes to avoid networking problems.</p> <p>We continue to have issues with overloaded pods which we are working on.</p> <p><strong>UPDATE 2024-11-18 14:00 UTC+01:00 (CET)</strong></p> <p>We managed to get the overloaded pods running by isolating them on dedicated nodes and raising the resource limits. This stopped other issues as well.</p> <p>We still need to investigate what caused the overloading of certain pods.</p> <p>Incident is over.</p> 2024-11-18T11:30:00+00:00 542 2024-12-03T02:01:02.414151+00:00 <p>Affected Components: <strong>MetaKube Platform, all regions</strong></p> <p>Scheduled Start: <strong>2024-11-27 18:00 UTC+01:00 (CET) </strong></p> <p>Scheduled End: <strong>2024-11-27 23:00 UTC+01:00 (CET) </strong></p> <p>State: <strong>COMPLETED</strong></p> <hr /> <p>Description:</p> <ul> <li>Scheduled maintenance of the MetaKube platform in all regions</li> <li>Small update to MetaKube infrastructure</li> </ul> <hr /> <p>Customer Impact during the maintenance:</p> <ul> <li>Short interruptions (&lt;5min) of the control plane possible (e.g. controller-manager, scheduler), but not API</li> <li>Running workloads will <strong>not</strong> be affected</li> <li>Network availability will <strong>not</strong> be affected</li> <li>Storage availability will <strong>not</strong> be affected</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>No customer actions needed</li> <li>Please inform us if you notice any irregularities</li> </ul> <hr /> 2024-11-27T14:47:00+00:00 543 2024-12-03T02:01:02.411838+00:00 <p>Affected Components: <strong>SysEleven IAM, regions DUS and HAM</strong></p> <p>Incident Start: <strong>2024-11-28 12:00 UTC+01:00 (CET)</strong></p> <hr /> <p>Description:</p> <ul> <li>We're currently investigating a service degradation in the SysEleven IAM. Inviting users to an organization is currently not possible.</li> </ul> <hr /> <p>Customer Impact:</p> <ul> <li>Inviting users to an organization is currently not possible.</li> </ul> <hr /> <p><strong>UPDATE 2024-11-28 13:10 UTC+01:00 (CET)</strong></p> <p>The issue has been resolved and inviting users to organizations is possible again</p> 2024-11-28T12:00:00+00:00 544 2024-12-03T02:01:02.409580+00:00 <p>Affected Components: <strong>metakube control plane services, region ham1</strong></p> <p>Incident Start: <strong>2024-11-29 11:00 UTC+01:00 (CET)</strong></p> <p>Incident End: <strong>2024-11-29 13:40 UTC+01:00 (CET)</strong></p> <hr /> <p>Description:</p> <p>The metakube control plane services in ham1 can't be reached currently due to slow i/o</p> <hr /> <p>Customer Impact:</p> <ul> <li>Metakube services e.g. clusters in ham1 can't be reached</li> </ul> <hr /> <p>Customer Actions:</p> <ul> <li>Please inform us if you notice any irregularities</li> </ul> <hr /> <p>Update 13:23</p> <p>The situation improved.</p> 2024-11-29T11:00:00+00:00 545 2024-12-03T02:01:02.407069+00:00 <p>Affected Components: <strong>SysEleven Stack, Storage, region HAM1</strong></p> <p>Incident Start: <strong>2024-11-29 11:00 UTC+01:00 (CET)</strong></p> <p>Incident End: <strong>2024-11-29 13:40 UTC+01:00 (CET)</strong></p> <hr /> <p>Description:</p> <p>We are facing issues with the distributed file system, a core component of the SysEleven Stack.</p> <hr /> <p>Customer Impact:</p> <ul> <li>Starting of virtual machines (VMs) partially not possible.</li> <li>Writing Access to volumes (VM disks) maybe restricted.</li> </ul> <hr /> <p>Update 13:23</p> <p>The situation improved, storage access latencies are back to normal.</p> 2024-11-29T11:00:00+00:00