Currently you may experience troubles with the network API . Spawning and deleting instances or other network resources is not possible. If you experience any unexpected behaviour don't hesitate to contact us.
We apologise for any inconveniences.
UPDATE 22:45
We have identified and resolved the networking API troubles.
We found a bad entry in the network state database of our SDN software, that caused a cascading failure: Internal locks could not be acquired anymore, which caused the API errors, which in turn caused networking troubles during live migrations in a small number of virtual machines.
We now understood the root cause and will post more information on actions we will take to prevent this from happening again once we conducted the post-mortem.
UPDATE 15. October
We are currently working the following action points:
- We identified and are working to resolve the root cause in our SDN software
- We will improve the monitoring of our networking APIs, to identify and correlate future troubles earlier
- We will improve monitoring of virtual machine port binding of our SDN software, so we can identify and fix troubled VMs faster in the future
