Today I received some disturbing news on the Twitter Developers mailing list. In a post on the list, Alex Payne, the API Lead for Twitter, informed developers they would be placing new limits on the API which will take place at the end of this week. In the note, he said:
“Starting later this week we’ll be limiting those on the whitelist to
20,000 requests per hour. Yes, you read that right: twenty THOUSAND
requests per hour. According to our logs, this accounts for all but
the very largest consumers of our API. This is essentially a
preventative measure to ensure that no one API client, even a
whitelisted account or IP, can consume an inordinate amount of our
resoures.”
By capitalizing the word THOUSAND, Alex scares us into thinking 20,000 requests in a single hour seem like a lot. But I argue as apps grow this is going to hurt every app out there. I’m arguing that 20,000, or any request-rate limit for that matter, limits any app out there from being able to develop on the Twitter platform, and I don’t see why any able-minded entrepreneur would want to build on it if there’s such a rate limit in place. Here’s why:
As a user of SocialToo, you’re aware that we provide services to enable others to follow those that follow them, as well as other services to enable users to stop following those that stop following them. We’ve also enabled anti-spam measures, and were working to build more on this premise, which we detect when spammers follow you, and then stop following you in the same day or short amount of time, and we stop following them for you. In addition, we’re providing very informative e-mails for our users enabling them to track those that stop following them, along with approximately what Tweets they posted when the person left them. For any user concerned about building a following, these are powerful tools, and we were working to only make them better for you. Twitter provides nothing like this.
However, there is a caveat to the Twitter API that makes this process quite difficult. We’ve become very good at it, but it’s no easy task. In order to retrieve a list of a user’s followers, the Twitter API currently requires any developer to go through the follower list, 100 people at a time, no matter how large the following is. So, for instance, for Guy Kawasaki, we have to traverse through over 350 pages of followers in order to get his entire list and determine if anyone new followed him, or stopped following him. That’s 350 requests just to get his list of followers – that doesn’t include the requests we have to make to follow each new follower. There is no better way to get this information.
Now, let’s go back to the request rate limit issue above. I’m going to get into some math here to show you why this is a problem. Imagine we process him, Chris Pirillo, Robert Scoble, and others all with over 30,000 followers. It’s going to take quite a long time to get through each user’s list of followers, and we have near 10,000 users to process! So, the only way to scale this process is to split these users up and process them concurrently so we get through the users faster and can check for new followers again. So, if we split that into two, making 5,000 users we have to process, that makes the potential of 2 requests (at least) per second, equalling 7,200 requests in an hour. However, processing that many would mean we take over a day, possibly several days to get through all your followers! (Remember, we have to traverse through your entire list of friends)
So, let’s double that, or even triple that to make it a little more bearable. At triple, we’re at 1,250 users for each process we run at the same time, making it 8 concurrent processes, a potential of at least 8 requests per second. Already, we’ve exceeded the limit, as 8 requests * 60 seconds * 60 minutes = 28,800 requests an hour. I can already tell you that even that number is pretty hard to go through in a short amount of time, and where we’re growing at near 5,000 users or more a month that number is only going to grow.
Twitter is venturing into dangerous territory here with this new rate limit. The 20,000 requests Alex mentions is a hard cap, which means no other developer can grow above that, ever, no matter how big they get. In essence, Twitter just limited their developers even more, keeping us from ever getting big enough to build a viable business model.
I’ve talked with Alex about this, and to make matters worse, he stated that in addition to the rate request, Twitter has no intention to improve the API in order to reduce the requests we have to make to make their load any lighter. From our conversation, his words were, “We want people to be able to follow and unfollow without any social repercussions. So it’s extremely unlikely that we’re going to make changes to the API that enable this kind of application or behavior.” Therefore, expect Qwitter to go out of business as well as they grow, as well as FriendOrFollow, or TwitterKarma. I would also add TwitterGradr and TweetStats to this list. Essentially, any app that needs to get following data could potentially, as they grow, be affected by this. I guarantee so long as these apps keep growing they will eventually hit this limit and be forced out of business.
I agree they need to protect their servers, but with the rigorous funding and expected business plan they are expected to initiate soon, along with Jeff Bezos as an investor, getting a sweetheart deal on scalable EC2 servers ought to be easy. By implementing this limit, it makes it near impossible for SocialToo, or any Twitter-based business for that matter, to grow and build on top of Twitter. Any app that retrieves a user’s followers should be scared out of their minds by this limit, and I would argue the same goes with other aspects of the API.
What are we going to do about this at SocialToo? I have some plans – we are called SocialToo and not TwitterToo, after all, but if this limit is enforced, as of the end of this week we will probably remove your nightly e-mails, remove auto-unfollow and unfollow filtering, and possibly have to temporarily disable auto-follow due to the short time-frame we’re being given to comply. Your surveys will still work, as will the Facebook profile redirect, and we hope to have auto-follow back up shortly after. However, this change is going to affect thousands of you users, and there’s nothing we can do about it unfortunately. Unfortunately having everyone migrate to other services due to this rate limit is going to cause the same problem for those services as well.
I suggest Twitter remove the rate limit and instead work on fixing their API to reduce the need for so many requests to get your friends and followers. I suggest you, our users, spread the word about this and write to biz@twitter.com and ev@twitter.com stating your concern on this matter. SocialToo will not be shut down, but many of the services you have come to love for Twitter, including those of our competitors, and many other Twitter-based services are in jeopardy. This is scary news as an entrepreneur and Twitter developer. Twitter has basically just limited how big any Twitter-based business can grow. This makes it worthless to build a business on top of Twitter now.
Tags: following, rate limit, stop following, twitter api, unfollowing
[...] California hosting facility snags sizable energy-savings rebate The SocialToo Blog: Twitter Limits Potential App Growth – How This Hurts Our UsersRed Hat version 5.3 released The Open Road: Red Hat set to surpass Sun in market capitalization New [...]
[...] much dismay amongst the digerati Twitter yesterday announced a limit of 20,000 calls per hour to their API for [...]
[...] today received an email from Twitter API head honcho, Alex Payne. The e-mail, courtesy of SocialToo blog, looks like this: “Starting later this week we’ll be limiting those on the whitelist to 20,000 [...]
[...] application can poll the service using the API. The SocialToo blog carries an indepth discussion of why the Twitter API rate limit will hamper development, saying: “This is scary news as an entrepreneur and Twitter developer. Twitter has basically [...]
[...] strange and puzzling why Twitter has decided to impose limits to calls on its API – a move that SocialToo contends “is going go hurt every app out [...]
i asked alex about removing the rate limit. he asnwered that that their new year resolution. I was under impression that there wont be any rlimit on APIs.. though 20K is much more cmpared to existing 100 limits… however as applications grows and traffic increases 20K will look like a small figure
Well, I don't know. But I think that this (throttling API calls) will cause a good disruption in the twitter application community. If a developer is not able to find an audience to support their product, they'll either have to dig deep and cover their costs until they do, or rethink the relevance of their product in the market.
The bar gets set higher, and I think we're all the better for it as both developers and end users.
[...] news, though, is that Twitter is even going to limit these outside services to 20,000 requests per hour. That might seem like a lot for a personal user, but when talking about a public website used by a [...]
A couple of things:
1. XMPP. if they provided XMPP pub/sub nodes for users feeds and data…. data could be pushed, if people were smart about things, it would scale better than repeated HTTP polling, and it would be awesome.
2. Open Microblogging, Laconica is compatible with the twitter api, provides XMPP *and* services could build implementations of open microblogging that let them subscribe to their clients via that network, and not have to deal with polling limits.
The way you present is makes Twitter look cold towards their eco-sphere — which is surprising given how many partners and clients they have; maybe they've tried and are coming back from it, but their success is heavily wired to that strategy, and they appear not too ignorant of that fact. Therefore, I would let them the benefit of the doubt and imagine they have good reasons to do what they seem to have decided. Money appeared tight recently, and saving on the few rather wasteful services (albeit the only way to have Guy's new followers, you method is a real bleeding).
Regarding the specific features that you want, I would make a distinction between:
* new followers (an information users can get by e-mail, and that they might agree to forward to you, with a simple IMAP filter); and
* people who stopped following a certain user.
As pointed out by the extensive literature, interrupted relations or more generally most negative news impair greatly a social service. It's the first question I get asked about Facebook or Twitter, and I can give you examples of services whose usage dropped days after implementing a look-back feature. By empowering Guy, you remove my ability (and some would say 'right') to stop being fed his every move without him knowing. Guy is a great guy, and he knows he twits too much, and probably wouldn't be offended. Others might like that with less detachment, and drama will not only happen, but get front-page treatment. That is threatening to Twitter because, with services such as yours and your competitor's, they cannot guarantee the same freedom to browse feeds.
Twitter, or Facebook who recently banned the Whooper Sacrifice application, do not want the balance of information tilt that way. Given Facebook massive lack of personnel, any of they move is most likely triggered by users. I do not think they decided to remove such a popular app without good reasons, namely users offended either by the fact that they were un-friended, or who forgot they had the application active and un-friended someone who took offense. It could be that non-important friends, aka 'weak-ties' are essential to their success (they are) but I doubt ten of those were worth the ban.
It's a matter of opinion, and would be hard to prove, but the success of a technology such as yours demands that you balance the right to know with the right to remain discreet; a majority of experts would agree with me.
Finally, I would love to have an alert saying that the recent user that friended me is apparently a spambot — provided you give the people behind those the ability to challenge your call. Twitter would love to have the ability to list those, and find out which one are useless, or do not respect the terms of services to ban them, and save some money; maybe they already do.
I'm not sure the competition is between closed and open-source services. It is an important aspect — but adoption of the service is the first step. Compatibility with other service is the only way out of a monopoly, and then, you have to argue in favor of open code.
I say that because the source behind Twitter is not the hardest asset to copy: leveraging a community of practice is. The people behind the Open Stack and DiSo realized that their arch-enemy, Facebook Connect, not only offers a better experience (thanks to good UI and market-base) but also drive massive press and attention to their solutions. Just like them, you need to push Twitter (and its equivalents/competitors) to learn about how to do micro-blogging properly, and let people learn about micro-blogging.
Rather then considering the hard-to-estimate value-added, you might want to consider costs to Twitter; this would make tour service neutral to them, and you'd be in a safer position later to negotiate the (then better known) value created by being the largest information network.
Otherwise, advertising might be the better solution (like spam-filtering, your service is valuable, but hard to put a price on). Were would you be able to put some?
None of this helps getting a user's friends and followers, which is the
heart of this issue.
[...] data on behalf of its users. (Another product that might be affected: Mr. Tweet.) The problem, according to SocialToo developer Jesse Stay, is that for some of its individual users, getting the basic information they need requires not one [...]
If microblogging were implemented as an XMPP service, your friends/follwers would be the “roster,” (or abstracted as such) or as members of a multi-user-chat (or abstracted as such), and getting a list of users, would be a single request/stream from the server.
OMB doesn't directly solve the API problem, but it does so indirectly: no OMB server needs to “hold” the entire microblogging world (like twitter needs to), so the scaling problems that twitter has with its API limits aren't likely to be as much of a sore point. Secondly, you can set up a local/private OMB instance that would allow you to write your App/site against a “local” platform/site, rather than against a third party site…
—
Having said that, these kinds of problem where “crowd sourced” platforms/sites end up moving things in a direction that helps their business, but hurts the crowd, are *very frustrating* but also not surprising. Alex is a great guy, twitter is a very inspiring platform (that secedes because of the number of users, and the tools that Alex has helped provide for), but it's still a business that really has to serve it's own interest, and if this doesn't illustrate the worth and utility of open source, and the software freedom approach, then I don't know what would.
The heart of twitter's issue, is that when they thought they were providing a “status message service” they never really expected that they'd be creating what amounts to a public messaging and communications service; combined with the fact that non-niche web-services are incredibly difficult to monetize/collect subscriber fee's for…
Developing for closed platforms isn't a bad thing at all–often–but it does put developers completely at the mercy of another party, and that's pretty tough to stomach.
[...] vemos en SocialToo, en la lista de desarrollo para Twitter, Alex Payne, el líder de API de Twitter ha anunciado lo [...]
[...] today received an email from Twitter API head honcho, Alex Payne. The e-mail, courtesy of SocialToo blog, looks like this: “Starting later this week we’ll be limiting those on the whitelist to 20,000 [...]
[...] an app that offers a list of Twitter-related services like auto-following, surveys and more, quickly responded with their thoughts and experiences from developing their services. Short story: they don’t believe this is a [...]
[...] an otherwise < href=”http://tumblelog.marco.org/72580346″>excellent rebuttal to a rather ill-thought-out complaint about Twitter rate-limiting their API Marco misses what I think is the main reason for not building [...]
Twitter to me is a bunch of snob losers that want you to check out their HOT thing and they never have any intention of reciprocating,if Google bought Twitter and closed it down I wouldn't miss a beat.
It's great to see a QUALITY blog that is Do follow,mine is also and I'm 
2009 is going to be rough people are to picky and don't appreciate JACK
sadly like a old prostitute society on the internet will use you abuse you and walk away
Despite all that
on the LOOKOUT to network with cool people. If you know what I mean Bring IT
Nice blog Stumbled
[...] been a lot of discussion regarding the recent announcement about Twitter’s API limiting. The SocialToo blog has a very good overview of the situation. Granted I am not a fan of the limitations, but that is not the reason for this [...]
[...] much dismay amongst the digerati Twitter yesterday announced a limit of 20,000 calls per hour to their API for [...]
[...] you’re new here, you may want to subscribe to our RSS feed. Thanks for visiting!At SocialToo, it has always been my goal to make following the people you want [...]
Search engine optimization also referred as SEO in the IT industry is basically the science of making your website important to the search engines in order to rank highly in their search results for a strategically defined set of keywords. Search Engine Optimization is the only key to online success in a scenario where competition is rising high. Online businesses have benefited a great deal through competent and efficient search engine optimization solutions.
[...] pesonas que son seguidoras de Guy Kawasaki) merecen desaparecer!. Va Mashable. The SocialToo Blog. [Liga] [etiquetas: [...]
[...] while providing a free service to low-volume players. Twitter recently placed a limit on its API to 20,000 requests/hour. addthis_url = [...]
I'm sure they have good reasons for their new policies.We know too little to pass any judgement
Ecommerce Web Design we are called SocialToo and not TwitterToo,
website design service his change is going to affect thousands of you users, and there’s nothing we can do about it unfortunately.
Using smart caching locally can help.
David, only to an extent. The only solution I've been able to come up with
is scale to lots of servers, which I'm doing now.
Post informative and entertaining post in your Twitter updates. Follow people who interest you, and keep twittering.
platforms/sites end up moving things in a direction that helps their business, but hurts the crowd, are *very frustrating* but also not surprising. Alex is a great guy, twitter is a very inspiring platform (that secedes because of the number of users, and the tools that Alex has helped provide for), but it's still a business that really has to serve it's own interest, and if this doesn't illustrate the worth and utility
yeah, now it's down to 150 requests per hour.. my client's twitter account has 11000+ friends and 11000+ followers.. I can't even do an api request to get their names (just their ids) because trying to get their names from the list of ips gives me the “over the limit” error
I think so.
I don't understand how you can make 2 completely contradictory statements in the *same sentence.* How can “Developing for closed platforms isn't a bad thing at all” be true, while “developers completely at the mercy of another party” (in other words, your business model is in someone else's hands) is also true?
While original comment is 6 months old, and I'm not sure if the events of the past six months are worth anything (and I don't have the time at this moment to consider these implications), here's a clarification:
- I don't mean to present a black and white moral system on this issue. It can be both “not a sin,” and also “not a good thing.” It's not, I think, a transgression to develop for a closed platform in an open way, and also I think it's true that putting users/developers at the mercy of another party is something that should be avoided.
- Perhaps I should have said “isn't neccessarily a bad thing.”
We have examples of people developing tools aplenty. Emacs/Cgywin/Notepad++ and so forth all exist for windows and people put a lot of time into that, and that's not bad, but I think everyone's a bit wary of that kind of work becasue of the way it requires them to have their work exist at the mercy of other people.
Or hell, I think the whole mono debate is contentious *because* of this tension…
Cheers,
sam
You guys should check out this twitter app http://www.sponsoraloser.com It's about posting twitter losers and finding them the answer to their problems, It's very funny! and It's all about helping others.