The issues were related with the linux bridge on a specific compute node. This caused an error state which ended up throwing DHCP requests in a loop causing big spikes in traffic and latency. This already happened last night but the root cause was found this morning. Our cloud provider is tightening up our monitoring to detect such instances earlier in the future and we’re taking the necessary steps to prevent this from happening again. Our apologies for the inconvenience this may have caused.
Posted Mar 17, 2020 - 13:25 CET
Resolved
The issue has been resolved. Post mortem will follow.
Posted Mar 17, 2020 - 13:19 CET
Monitoring
The issues seem to have been resolved but we are still closely monitoring the situation and working on a root cause analysis. We'll be providing further information when we have it.
Posted Mar 17, 2020 - 10:32 CET
Investigating
Last night we experienced some issues at our cloud provider which caused lag and downtime, this has since been resolved however we are seeing similar issues this morning. Unfortunately this means you may have slowdowns on your Hypernode, so far no actual downtime is occurring.
Our tech team is working with the network engineers to isolate this issue. We are sorry for this disruption in service and we'll keep you updated through this incident.
Posted Mar 17, 2020 - 09:32 CET
This incident affected: Hypernode Platform (Hypernode Servers).