How we scaled Freshdesk (Part II) – The Sharding and Rebalancing techniques we used

Written by on June 9, 2014

This is the second part of the How We Scaled Freshdesk story. In my previous post, I talked about how we tried to vertically scale Freshdesk before we decided to implement sharding with our database. You can find it here.

After using a bunch of methods to scale like R/W split, MySQL partitioning and witnessing them being inadequate for our rapid growth, we came to the conclusion that scaling vertically can only get you so far. The more we thought about it, the more it made sense for us to shard our database.

It was kind of inevitable because sharding was pretty much the only cost-effective way to increase write scalability beyond the instance size. We could also horizontally scale our DB infrastructure in terms of database size, backups, schema changes and more. In our case, we had two main concerns before we took the final call on sharding:

1. No distributed transaction – We wanted all tenant details to be in one shard

2. Rebalancing the shards should be easy – We wanted control over which tenant sits in which shard and to be able to move them around when needed

A little research showed us that there was only one way to go, directory based sharding.

Directory based sharding

Directory based sharding suited our requirements much better than hash key based or range based mainly because it’s simpler to implement. And rebalancing the shards was far easier than with other methods. So, we started caching the directory lookups for fast access and maintaining multiple copies of directory database. We take regular backups of it to avoid a single point of failure.

A typical directory entry looks like this:

tenant_info

shard_details

shard_status

Stark Industries

shard 1

Read & Write

Where, tenant_info is the unique key referring to the DB entry, Shard_details is the shard in which that tenant exists and shard_status tells what kind of activity the tenant is ready for. We have multiple shard statuses like Not Ready, Only Reads, Read & Write etc.

How the directory lookup works

Data can be accessed through multiple entry points like web, background jobs, analytics etc. When a request comes in, an API wrapper accesses the directory to get the appropriate shard and status of the tenant. We tuned the API wrapper to accept the tenant information in multiple forms like tenant URL, tenant ID etc. The shard information returned by the API wrapper contains the shard_details and shard_status of the data. The sharding API even acts as a unique ID generator so that the tenant ID generated is unique across shards.

Why we care about rebalancing

There are instances when a user who initially started out with 1000 tickets per day grows to processing about 10,000 tickets per day. This user could sorely affect the performance of the whole shard. However, we can’t solve the problem by splitting up his data into multiple shards because we didn’t want the mess of distributed transactions.

So, in these cases, we decided that we’d move the noisy customer to a shard of his own. That way, everybody wins.

How rebalancing works

1. Every shard has its own slave to scale the reads. For example, say Wayne Enterprises and Stark industries are in Shard 1. The directory entry looks like this

Wayne Enterprises

shard1

Read & Write

Stark Industries

shard1

Read & Write

2. Seeing the rate at which Wayne enterprises is growing,  moving it to another shard is for the best (averting the danger of Bruce Wayne and Tony Stark being mad at us the same time).

3. So we boot up a new slave to Shard 1, call it Shard 2, attach a read replica to the new slave and wait for it to sync with the master.

4. We’d then stop the writes for Wayne Enterprises by changing the shard status in the directory.

Wayne Enterprises

shard1

Read only

Stark Industries

shard1

Read & Write

5. Then we’d stop the replication of master data in shard 2 and promote it to master.

6. Now the directory entry would be updated accordingly

Wayne Enterprises

shard2

Read & Write

Stark Industries

shard1

Read & Write

7. This effectively moves Wayne Enterprises to its own shard. Batman’s happy and so is Iron Man.

 

How freshdesk scaled

Now, the only thing left to do is to clean up the duplicate Wayne enterprises data in Shard 1. But that can be done leisurely because it will not affect the writes in either of the shards. And all this can be done through API in Amazon RDS and a few clicks via the Amazon RDS console.

Word of caution

If you are thinking about sharding for your DB, I have some advice for you,

1. Don’t do it unless it’s absolutely necessary. You will have to rewrite code for your whole app, and maintain it.

2. You could use Functional partitioning (moving an over-sized table to another DB altogether) to completely avoid sharding if writes are not a problem.

3. Choosing the right sharding algorithm is a bit tricky as each has its own benefits and drawbacks. You need to make a thorough study of all your requirements while picking one.

4. You will need to take care of the Unique ID generation across shards.

What’s next for Freshdesk

We get 250,000 tickets across Freshdesk every day and 100 M queries during the same time (with a peak of 3-4k QPS). We have a separate shard now for all new sign ups. And each shard can roughly carry 20,000 tenants.

In the future, we’d like to move beyond sharding and explore Multi-pod architecture and also look at Proxy architecture using Mysql Fabric, Scalebase etc.

Further reading

Tumblr. Massively Sharded MySqQL

Database Sharding at Netlog

Why don’t you want to shard

Subscribe for blog updates

  • pete_didonato

    Great post and great work. It’s really cool to see a company growing so quickly and using engineering brilliance to meet the challenges.

  • Would it really not be easier to have multiple instances of the application and send users out to the different instances based on login?

    • Kiran Darisi

      We are also thinking in terms of Multi pod architecture there are couple of missing pieces uniqueness across pods and communication between pods but pod architecture is on our roadmap.

      • Hi Karin.

        I’m just thinking that you’d be able to handle the migrations within a single pod. If you put a front end on logins and first entry which allows clients to have access to multiple pods in case they have multiple accounts, it would seem to me (as a very active client) that you’d be golden.

        You’d just have to work on a very clean deployment routine so that deploying from development server to multiple pods is no more difficult than deploying to a single pod.

        You could also isolate FreshTalk and FreshChat (as examples) so that multiple pods could talk to those services.

  • alcalbg

    Great tech insight. I often do the same and write about the challenges keeping my startup transparent for customers. Keep up the good work!

  • Thanks

  • Paulo Gonzaga Lopes

    Thank you!

  • Abhay Dandekar

    Hi, What really happens over the write transactions that are about to
    enter in for shard1 ( before switch ) and shard2 is ready to take
    writes and we mark shard1 as read-only. Wont there be any transaction
    failure ? How much millis does it take for switchover to take place ?
    Just wanted to understand if this is a failure proof ~Abhay

    • Kiran Darisi

      There won’t be any write transactions because we are marking the shard1 read only at the application level not at the DB level the moment when we are marking the current transaction will complete.

  • cruisemaniac

    Great article Kiran. From someone coming from the Oracle RAC side of the pond, I have a few questions – Sorry for the long mailish style comment 😉 :

    1. What is the latency across steps 3 through 5 of your rebalancing routine? Considering the 3-4k QPS you’ve mentioned, this has to be scarily fast no?

    2. Following @aleck:disqus in his comment below, why not have thresholds / triggers and move customers to dedicated shards from the get go instead of the delicate rebalancing dance that you do here? (This is a great great way to handle things btw, hands down!) I ask this question because you “know” the reason your database grows: More tickets -> more action -> more data

    3. Does directory based Sharding not introduce a Single point of failure for both Iron Man and Batman? How is this *time bomb* handled?

    Would love to hear your views on this when you find time!

    Thanks once again for a GREAT post!

    • Kiran Darisi

      Answers as below

      1. There is hardly any latency as we are marking read only after the current transaction is done there will be nothing to write to shard1. It’s just a config change from app side.

      2. Deciding about the shard allocation even before the customer goes beyond some point will be a bit of hit on resource utilisation there are lot of customers who are with us for more than 2 years still only have some handful of tickets and are happy to stay in the same shard.

      3. As i mentioned in the post we have multiple copies and we cache it heavily so there is no SPOF.

      • cruisemaniac

        Kiran,

        Thanks a ton for the answers! Sorts this out clean and clear!

        Great write up! Looking forward to meeting you in person 🙂

  • Ananth Srinivasan

    Thanks for Sharing Kiran. Is it a decision not to go for distributed transaction. Will further partitioning based on the business logic of the data and handling the distributed transactions through transaction manager work. In my current company we employ those architecture and proved to work quite well.

  • GK

    Hi Kiran,
    Though directory approach to scale out is well documented, I’ve also found out it is really difficult to get it running in practice. Good that you’ve got it right!

    1. I’m curious if you’ve tried Aurora (on AWS) which is in preview/beta now, any thoughts on how easier/simpler it is at your scale ?

    2. If you have significant majority reads compared to writes, have you tried CQRS pattern (dedicated read/write instances) at the service level ? Looking at your scale/growth I’m assuming you have some sort of a micro-service based architecture, supported by your underlying PaaS.

    3. Are your typical queries long-tail (a set of “recent important records” as head/first-page by some characteristic, plus a list of “less necessary” records that can be queried/refreshed as necessary, and which probably >50% of users never scroll for.

    Really curious to learn how you’re squeezing performance from your architecture at scale.

    Great blog! Sorry for the late questions since I discovered your blog just now. I’m passionate about scalable computing, and would love to receive your response.

    Thanks,
    Ganesh