Last night we experienced a sequence of events that led to intermittent degraded performance across many OpenSRS services. At 1 AM UTC we were the target of a sophisticated DNS attack that was followed by an unrelated double failure of core network equipment at our main Canadian data center, caused by an undocumented software limitation. We were able to quickly recover from the equipment failure but continued to experience the DNS attack until 13:10 UTC when the attack was stopped and systems started responding reliably again. The network equipment failure made it more difficult for us to identify that we were under a DNS attack and impacted our response time.
This complex combination of events impacted the following services:
- OpenSRS API services
- Cluster A and B mail services
- opensrs.com, which includes [email protected] email and live chat
- DNS service through systemdns.com and mdnsservice.com
- Hosted Registrar Services (HRS)
We are working on creating more separation between components of our DNS infrastructure that will improve resilience to similar attacks in the future.
We encourage you to check for orders that may have failed during the service interruption. We also expect to see inbound mail delivery delays, as third-party email providers will have queued up mail for delivery to our system. Delayed mail will be delivered throughout the day.
We have been monitoring the stability of our services and now consider the issue resolved. We will continue to monitor the problem and provide updates if the status changes. More information can be found on the OpenSRS Status page.
If you have any technical difficulties with any OpenSRS services, please contact our support team so we can investigate the problem. We apologize for any inconvenience this issue has caused.