The SocialToo Blog

Today I received some disturbing news on the Twitter Developers mailing list.  In a post on the list, Alex Payne, the API Lead for Twitter, informed developers they would be placing new limits on the API which will take place at the end of this week.  In the note, he said:

“Starting later this week we’ll be limiting those on the whitelist to
20,000 requests per hour. Yes, you read that right: twenty THOUSAND
requests per hour. According to our logs, this accounts for all but
the very largest consumers of our API. This is essentially a
preventative measure to ensure that no one API client, even a
whitelisted account or IP, can consume an inordinate amount of our
resoures.”

By capitalizing the word THOUSAND, Alex scares us into thinking 20,000 requests in a single hour seem like a lot. But I argue as apps grow this is going to hurt every app out there. I’m arguing that 20,000, or any request-rate limit for that matter, limits any app out there from being able to develop on the Twitter platform, and I don’t see why any able-minded entrepreneur would want to build on it if there’s such a rate limit in place. Here’s why:

As a user of SocialToo, you’re aware that we provide services to enable others to follow those that follow them, as well as other services to enable users to stop following those that stop following them.  We’ve also enabled anti-spam measures, and were working to build more on this premise, which we detect when spammers follow you, and then stop following you in the same day or short amount of time, and we stop following them for you.  In addition, we’re providing very informative e-mails for our users enabling them to track those that stop following them, along with approximately what Tweets they posted when the person left them.  For any user concerned about building a following, these are powerful tools, and we were working to only make them better for you.  Twitter provides nothing like this.

However, there is a caveat to the Twitter API that makes this process quite difficult.  We’ve become very good at it, but it’s no easy task.  In order to retrieve a list of a user’s followers, the Twitter API currently requires any developer to go through the follower list, 100 people at a time, no matter how large the following is.  So, for instance, for Guy Kawasaki, we have to traverse through over 350 pages of followers in order to get his entire list and determine if anyone new followed him, or stopped following him.  That’s 350 requests just to get his list of followers – that doesn’t include the requests we have to make to follow each new follower.  There is no better way to get this information.

Now, let’s go back to the request rate limit issue above. I’m going to get into some math here to show you why this is a problem. Imagine we process him, Chris Pirillo, Robert Scoble, and others all with over 30,000 followers.  It’s going to take quite a long time to get through each user’s list of followers, and we have near 10,000 users to process!  So, the only way to scale this process is to split these users up and process them concurrently so we get through the users faster and can check for new followers again.  So, if we split that into two, making 5,000 users we have to process, that makes the potential of 2 requests (at least) per second, equalling 7,200 requests in an hour.  However, processing that many would mean we take over a day, possibly several days to get through all your followers! (Remember, we have to traverse through your entire list of friends)

So, let’s double that, or even triple that to make it a little more bearable.  At triple, we’re at 1,250 users for each process we run at the same time, making it 8 concurrent processes, a potential of at least 8 requests per second.  Already, we’ve exceeded the limit, as 8 requests * 60 seconds * 60 minutes = 28,800 requests an hour.  I can already tell you that even that number is pretty hard to go through in a short amount of time, and where we’re growing at near 5,000 users or more a month that number is only going to grow.

Twitter is venturing into dangerous territory here with this new rate limit.  The 20,000 requests Alex mentions is a hard cap, which means no other developer can grow above that, ever, no matter how big they get.  In essence, Twitter just limited their developers even more, keeping us from ever getting big enough to build a viable business model.

I’ve talked with Alex about this, and to make matters worse, he stated that in addition to the rate request, Twitter has no intention to improve the API in order to reduce the requests we have to make to make their load any lighter. From our conversation, his words were, “We want people to be able to follow and unfollow without any social repercussions. So it’s extremely unlikely that we’re going to make changes to the API that enable this kind of application or behavior.”  Therefore, expect Qwitter to go out of business as well as they grow, as well as FriendOrFollow, or TwitterKarma.  I would also add TwitterGradr and TweetStats to this list.  Essentially, any app that needs to get following data could potentially, as they grow, be affected by this.  I guarantee so long as these apps keep growing they will eventually hit this limit and be forced out of business.

I agree they need to protect their servers, but with the rigorous funding and expected business plan they are expected to initiate soon, along with Jeff Bezos as an investor, getting a sweetheart deal on scalable EC2 servers ought to be easy. By implementing this limit, it makes it near impossible for SocialToo, or any Twitter-based business for that matter, to grow and build on top of Twitter. Any app that retrieves a user’s followers should be scared out of their minds by this limit, and I would argue the same goes with other aspects of the API.

What are we going to do about this at SocialToo?  I have some plans – we are called SocialToo and not TwitterToo, after all, but if this limit is enforced, as of the end of this week we will probably remove your nightly e-mails, remove auto-unfollow and unfollow filtering, and possibly have to temporarily disable auto-follow due to the short time-frame we’re being given to comply.  Your surveys will still work, as will the Facebook profile redirect, and we hope to have auto-follow back up shortly after.  However, this change is going to affect thousands of you users, and there’s nothing we can do about it unfortunately. Unfortunately having everyone migrate to other services due to this rate limit is going to cause the same problem for those services as well.

I suggest Twitter remove the rate limit and instead work on fixing their API to reduce the need for so many requests to get your friends and followers.  I suggest you, our users, spread the word about this and write to biz@twitter.com and ev@twitter.com stating your concern on this matter.  SocialToo will not be shut down, but many of the services you have come to love for Twitter, including those of our competitors, and many other Twitter-based services are in jeopardy.  This is scary news as an entrepreneur and Twitter developer.  Twitter has basically just limited how big any Twitter-based business can grow. This makes it worthless to build a business on top of Twitter now.