On Sunday and Monday (July 22nd & 23rd) our DNS Service Provider Zerigo was a target of a massive DDos attack and 4 of their 6 public name servers stopped responding to DNS queries. Starting at around 10PM PST on Sunday, Freshdesk customers from around the world were facing difficulties intermittently, in accessing Freshdesk servers. Though the Freshdesk website and application were up and running, customers were not able to access the website or their Support Portals as the DNS servers were not responding.
By around midnight PST, we had moved our DNS over to Amazon Route 53 and some of our customers were able to access Freshdesk. But DNS propagation is a painfully slow process and we had to wait for over several hours before full propagation happened. In the meantime our website and blog were not accessible to customers, and our emails were also being delivered late. Our only mode of communication to our customers was through our Twitter account (@freshdesk).
As a workaround we enabled customers to be able to access their agent portal by having a static route in their local hosts file (and thereby bypass DNS lookup).
By around 1:45 am PST on 24th we were seeing that Freshdesk was fully accessible for all customers and things were returning to normal.
Lessons and Action
I would like to apologize personally to all our customers for the inconvenience caused by this outage. The Zerigo DDoS attack was a tough lesson, and we’d like to thank all our customers who stood by us during this crisis.
DDoS attacks are nasty and unpredictable, and are the act of unscrupulous criminals who want to destroy good companies like Zerigo. We believe it is important for us to stand by Zerigo during this time, just like our customers have stood by us. We’ve used Zerigo since we launched Freshdesk and have loved them for their awesome service. In fact, up until the DDoS attack we have never had a single issue with them.
We plan to continue using Zerigo as our primary DNS. However to avoid situations like these in the future, we will be using Amazon Route 53 as our secondary DNS. In hindsight we should have planned for redundancy in our DNS before this happened. Once again please accept our apologies. We hope to learn, improve and serve you better in future.