The Freshdesk blog

Your daily dose of
peppermints, orange juice and oatmeal cookies...
Freshly squeezed droplets of customer support...

Girish

Impact of this week’s DDoS attack on our DNS Provider Zerigo – Learnings and Post Mortem

July 25th, 2012
11 Comments

On Sunday and Monday (July 22nd & 23rd)  our DNS Service Provider Zerigo was a target of a massive DDos attack and 4 of their 6 public name servers stopped responding to DNS queries. Starting at around 10PM PST on Sunday, Freshdesk customers from around the world were facing difficulties intermittently, in accessing Freshdesk servers. Though the Freshdesk website and application were up and running, customers were not able to access the website or their Support Portals as the DNS servers were not responding.

By around midnight PST, we had moved our DNS over to Amazon Route 53 and some of our customers were able to access Freshdesk. But DNS propagation is a painfully slow process and we had to wait for over several hours before full propagation happened. In the meantime our website and blog were not accessible to customers, and our emails were also being delivered late. Our only mode of communication to our customers was through our Twitter account (@freshdesk).

As a workaround we enabled customers to be able to access their agent portal by having a static route in their local hosts file (and thereby bypass DNS lookup).

By around 1:45 am PST on 24th we were seeing that Freshdesk was fully accessible for all customers and things were returning to normal.

Lessons and Action
I would like to apologize personally to all our customers for the inconvenience caused by this outage. The Zerigo DDoS attack was a tough lesson, and we’d like to thank all our customers who stood by us during this crisis.

DDoS attacks are nasty and unpredictable, and are the act of unscrupulous criminals who want to destroy good companies like Zerigo. We believe it is important for us to stand by Zerigo during this time, just like our customers have stood by us. We’ve used Zerigo since we launched Freshdesk and have loved them for their awesome service. In fact, up until the DDoS attack we have never had a single issue with them.

We plan to continue using Zerigo as our primary DNS. However to avoid situations like these in the future, we will be using Amazon Route 53 as our secondary DNS. In hindsight we should have planned for redundancy in our DNS before this happened. Once again please accept our apologies. We hope to learn, improve and serve you better in future.

Girish Mathrubootham

  • Pushpak Rangaiah

    i suggest you guys to try out dyndns
    http://dyn.com/ :)

  • Pushpak Rangaiah

    i suggest you guys to use dyndns
    http://dyn.com/ :)

    • http://twitter.com/mrgirish Girish Mathrubootham

      Thanks Pushpak. We will talk to them.

      • Dave

        DNSMadeEasy is a leader in enterprise DNS without crazy pricing. Would suggest them.

      • Milo

        HostGator currently uses them and their website never goes down regardless of how many attacks they get :) .

  • caithriellesam

    These web attacks are really becoming a problem. Good thing you were able to prevent it.

  • dave

    It is a real worry that you didn’t have secondary dns in place before this happened. How many more lessons will have to be learnt the hard way?

  • Jack

    DNS does not take that long to propagate. Seems like there was other issues since you were down far longer after Zerigo was up. Also… Route53 does not do secondary DNS so unless you do something special with your setups you will still have downtime if Zerigo goes down again. It seems like you guys you making decisions on your DNS based on what you think will happen (and it is not correct). I would strongly suggest bringing in some DNS expertise. It does not appear your providers are giving you the expertise that you need.
    It is actually shocking that Zerigo has not had larger issues with their unicast network. The reports online are showing how vulnerable they are. I would take this opportunity to warn them to invest more than 6 servers into their DNS infrastructure. Times are changing…. if Zerigo is not willing to invest more than 6 systems for the prices they are charging…. you are not doing your job as a business owner to stay with them… Loyalty is stupid if you are following a company that is making bad decisions.
    Either way, nice post (Zerigo did not do anything like this) but as stated… you guys really appear to need to get some DNS expertise on your side or you will have more downtime. And your clients will not be so likely to stick by you if you make the same mistake over again. So if you stick with a provider… you better make sure they are working on making fixes… or you need to move providers… or go out of business.

    • http://twitter.com/mrgirish Girish Mathrubootham

      Thanks for your inputs. We are already talking to a couple of providers. This time we are seeking some expert advice as well.

      • Jack

        Deciding to move to Dyn means you guys must have a lot of money…. We better expect lower pricing if you have money to waste like this.

  • Pingback: Flowdock is now integrated with UserVoice – and you can get 3 months of it for free + MORE | WordPress Support Plugin