Twitter Limits Potential App Growth – How This Hurts Our Users
Today I received some disturbing news on the Twitter Developers mailing list. In a post on the list, Alex Payne, the API Lead for Twitter, informed developers they would be placing new limits on the API which will take place at the end of this week. In the note, he said:
“Starting later this week we’ll be limiting those on the whitelist to
20,000 requests per hour. Yes, you read that right: twenty THOUSAND
requests per hour. According to our logs, this accounts for all but
the very largest consumers of our API. This is essentially a
preventative measure to ensure that no one API client, even a
whitelisted account or IP, can consume an inordinate amount of our
resoures.”
By capitalizing the word THOUSAND, Alex scares us into thinking 20,000 requests in a single hour seem like a lot. But I argue as apps grow this is going to hurt every app out there. I’m arguing that 20,000, or any request-rate limit for that matter, limits any app out there from being able to develop on the Twitter platform, and I don’t see why any able-minded entrepreneur would want to build on it if there’s such a rate limit in place. Here’s why:
As a user of SocialToo, you’re aware that we provide services to enable others to follow those that follow them, as well as other services to enable users to stop following those that stop following them. We’ve also enabled anti-spam measures, and were working to build more on this premise, which we detect when spammers follow you, and then stop following you in the same day or short amount of time, and we stop following them for you. In addition, we’re providing very informative e-mails for our users enabling them to track those that stop following them, along with approximately what Tweets they posted when the person left them. For any user concerned about building a following, these are powerful tools, and we were working to only make them better for you. Twitter provides nothing like this.
However, there is a caveat to the Twitter API that makes this process quite difficult. We’ve become very good at it, but it’s no easy task. In order to retrieve a list of a user’s followers, the Twitter API currently requires any developer to go through the follower list, 100 people at a time, no matter how large the following is. So, for instance, for Guy Kawasaki, we have to traverse through over 350 pages of followers in order to get his entire list and determine if anyone new followed him, or stopped following him. That’s 350 requests just to get his list of followers – that doesn’t include the requests we have to make to follow each new follower. There is no better way to get this information.
Now, let’s go back to the request rate limit issue above. I’m going to get into some math here to show you why this is a problem. Imagine we process him, Chris Pirillo, Robert Scoble, and others all with over 30,000 followers. It’s going to take quite a long time to get through each user’s list of followers, and we have near 10,000 users to process! So, the only way to scale this process is to split these users up and process them concurrently so we get through the users faster and can check for new followers again. So, if we split that into two, making 5,000 users we have to process, that makes the potential of 2 requests (at least) per second, equalling 7,200 requests in an hour. However, processing that many would mean we take over a day, possibly several days to get through all your followers! (Remember, we have to traverse through your entire list of friends)
So, let’s double that, or even triple that to make it a little more bearable. At triple, we’re at 1,250 users for each process we run at the same time, making it 8 concurrent processes, a potential of at least 8 requests per second. Already, we’ve exceeded the limit, as 8 requests * 60 seconds * 60 minutes = 28,800 requests an hour. I can already tell you that even that number is pretty hard to go through in a short amount of time, and where we’re growing at near 5,000 users or more a month that number is only going to grow.
Twitter is venturing into dangerous territory here with this new rate limit. The 20,000 requests Alex mentions is a hard cap, which means no other developer can grow above that, ever, no matter how big they get. In essence, Twitter just limited their developers even more, keeping us from ever getting big enough to build a viable business model.
I’ve talked with Alex about this, and to make matters worse, he stated that in addition to the rate request, Twitter has no intention to improve the API in order to reduce the requests we have to make to make their load any lighter. From our conversation, his words were, “We want people to be able to follow and unfollow without any social repercussions. So it’s extremely unlikely that we’re going to make changes to the API that enable this kind of application or behavior.” Therefore, expect Qwitter to go out of business as well as they grow, as well as FriendOrFollow, or TwitterKarma. I would also add TwitterGradr and TweetStats to this list. Essentially, any app that needs to get following data could potentially, as they grow, be affected by this. I guarantee so long as these apps keep growing they will eventually hit this limit and be forced out of business.
I agree they need to protect their servers, but with the rigorous funding and expected business plan they are expected to initiate soon, along with Jeff Bezos as an investor, getting a sweetheart deal on scalable EC2 servers ought to be easy. By implementing this limit, it makes it near impossible for SocialToo, or any Twitter-based business for that matter, to grow and build on top of Twitter. Any app that retrieves a user’s followers should be scared out of their minds by this limit, and I would argue the same goes with other aspects of the API.
What are we going to do about this at SocialToo? I have some plans – we are called SocialToo and not TwitterToo, after all, but if this limit is enforced, as of the end of this week we will probably remove your nightly e-mails, remove auto-unfollow and unfollow filtering, and possibly have to temporarily disable auto-follow due to the short time-frame we’re being given to comply. Your surveys will still work, as will the Facebook profile redirect, and we hope to have auto-follow back up shortly after. However, this change is going to affect thousands of you users, and there’s nothing we can do about it unfortunately. Unfortunately having everyone migrate to other services due to this rate limit is going to cause the same problem for those services as well.
I suggest Twitter remove the rate limit and instead work on fixing their API to reduce the need for so many requests to get your friends and followers. I suggest you, our users, spread the word about this and write to biz@twitter.com and ev@twitter.com stating your concern on this matter. SocialToo will not be shut down, but many of the services you have come to love for Twitter, including those of our competitors, and many other Twitter-based services are in jeopardy. This is scary news as an entrepreneur and Twitter developer. Twitter has basically just limited how big any Twitter-based business can grow. This makes it worthless to build a business on top of Twitter now.
Tags: following, rate limit, stop following, twitter api, unfollowing
[...] out the Social Too entry on this and perhaps Twitter might do something to stop itself from chopping its own head [...]
Pingback by Twitter’s proposed limits could be the death of Twitter | Christopher Kusek, Technology Evangelist — January 21, 2009 @ 12:31 pm
isn't this limit per account. And if it is then why not use your user's credentials to do the api requests.
If it's a per ip limit just get more ips, they are relatively cheap .
Of course this is a short term solution, they are probably going to charge for extra api requests.
But you should have seen this coming, it's impossible to handle any number of api requests from anyone. there's no such thing as scaling a system to infinite and for free too.
Comment by Mihai — January 21, 2009 @ 12:34 pm
Is there a chance they will announce a premium fee based API service with rate tiers? Or perhaps your software could become a client side app?
Comment by sephillips — January 21, 2009 @ 12:35 pm
Great post Jesse.. I think API limits are BS and you are exactly right – this pretty much limits any growth for developers building on top of the Twitter's API.
Comment by abrudtkuhl — January 21, 2009 @ 12:38 pm
Good point – Maybe this is a push towards a premium API service
Comment by abrudtkuhl — January 21, 2009 @ 12:38 pm
How long before they start charging for API access?
Have to agree with you in that they do need to fix their “creaking” API.
The cynic in me does wonder if this is a method of limiting others from building success “out of” the success of Twitter itself.
Possibly very damaging to all Twitter based apps and not good news at all.
Chris
Comment by shinerweb — January 21, 2009 @ 12:41 pm
This does seem a bit backwards. Did they give any more explanation as to their rationale?
Comment by Matik72 — January 21, 2009 @ 12:45 pm
Maybe Twitter is trying to “shake” the apps that help those of us who are trying to truly get folks who WANT to follow us and WANT serious followers. I'm noticing a surge in porn and dating tweets. If Twitter can't come up with a better way to get rid of this type of traffic outside of “block”, (even though this is a free service), I may be better off with one of it's competitors.
I have noticed a difference in my “follows” since I joined ST. I no longer waste time doing my own deletions of folks who simply are looking for “temporary follow time” and “numbers” as they are auto deleted the minute they “unfollow”. Since the porn/dating stuff is more recent, I'm hoping to notice a reduction over the next couple of days of this traffic.
Eventually, I would like to see some sort of statistical information on why others “unfollow” me. It will help me learn if I've posted something potentially offensive or if I've just lost their interest.
Comment by theretrometro — January 21, 2009 @ 12:45 pm
this is an interesting development in light of the uncertainty as to twitter's business model. i can understand their intent to try and disincentivize automation of the twitter experience but i think you're right in suggesting they work on making applications like yours viable by tweaking their API. i wonder if they want more of these types of 'services' under their roof, eventually, which i might suggest if i were on the board looking for a way to make money on the whole venture.
Comment by d — January 21, 2009 @ 12:47 pm
Agreed completely… API limits are BS – especially for twitter where most of these apps are HUGE added value to the service and are a big reason for their continued growth.
There is no reason now that they have Bezos and AWS architecture behind them that they should impose rate limits.
Comment by abrudtkuhl — January 21, 2009 @ 12:51 pm
Just want to second the fact this would be a null issue if the API was improved. I agree with you both and the more the API lacks, the more things like this will continue to creep up.
Comment by jonconley — January 21, 2009 @ 1:08 pm
I am wondering if this is not a move towards a “pay to play” revenue model?
Comment by Matik72 — January 21, 2009 @ 1:08 pm
Seems like there could be a layer of caching that you can do that would alleviate at least some of the pains.
Yes it limits your potential growth should every twitter user decide to use the service, but once you've scraped the 20,000 accounts associated with Scoble/Pirillo or others, you can cache that information for a few hours. Then you can query your internal caches as many times as needed. While the auto follow/un-follow won't be as real time, it will still work and Twitter's API is happier as a result.
I think it's time to get creative with the solutions. I don't expect complaining will solve much.
Erik
Comment by Erik Giberti — January 21, 2009 @ 1:13 pm
Limiting the growth of the services which are based on Twitter will ultimately limit Twitter itself, unless Twitter plans to provide those services itself. But I highly doubt it will come to that. This is a ridiculously stupid act of self-strangulation.
Comment by Chris Charabaruk — January 21, 2009 @ 1:20 pm
Agreed the API limits are just a band-aid on poor API development. But can you expand on who TweetDeck violates this because technically they don't. They aren't being punished because they don't tax the system (as bad as it is) anymore than a normal user can do on their own. Lucky, maybe, but violators I don't think so.
Comment by strangedesign — January 21, 2009 @ 1:21 pm
See where I differ slightly from your thinking is while yes they are a single app they are not using their own API account. They use MY API calls which limit me the individual user. If I choose to have a lot of groups in the client that is my choice. I have to decide how to use my 100/hr and I decide to give them to tweetdeck that is my choice. Where as Socialtoo (disclosure I am a user of SocialToo and LOVE IT) is is using a DEV account. Not saying it is right, but the spirit of the change from twitter IMHO is not about limiting applications, it is about resource preservation pure and simple.
Now I have worked on large implementations (talking millions of concurrent users) and Yes they need to fix their API but also they need to protect themselves from scriptkiddies and hacks who don't know how to code and therefore use up too many API calls. just my $0.02.
Comment by strangedesign — January 21, 2009 @ 1:38 pm
This might be step one of the 'making money by end of the year' plan. One way to do this could be to start building decent features for the power users (twitter lacks, well, everything, and they know it) – but before that they need to gently wind down the tens of services that proliferated around these missing features (doing it after would be a pr disaster).
It's easy to see how socialtoo could solve the problem (at least for a good while) by disabling the services for who has more than, say, 500 followers. The problem with that, even if just as a very temporary measure, is that any possible revenue is likely to come from those very people – the twitter broadcasters. This is true for socialtoo as for any other twitter-based service – and twitter itself; so I think it's quite unlikely they will ever lift the usage cap or address those API issues. Not because it's difficult, not because of design decisions – but because of *business* decisions.
Comment by markmaximum — January 21, 2009 @ 1:38 pm
ok, I see you have a real problem now.
my guess is this is either a temporary problem and it will be fixed once they manage to upgrade their system to scale better
or it's the first step to paid API access
Comment by Mihai — January 21, 2009 @ 1:56 pm
But what if that limit is what get's them through Inauguration and MacWorlds, etc? I would think keeping Jesse and his 10K users being able to auto-follow/unfollow/DM running is not their concern. Keeping their system running is what matters. At this point the only thing you can really be firm about is the communication and timing of this change is BS. Whether it is justified, the right thing to do, etc can't be answered without more information.
Comment by strangedesign — January 21, 2009 @ 2:08 pm
[...] out the Social Too entry on this and perhaps Twitter might do something to stop itself from chopping its own head off. [...]
Pingback by Twitter’s proposed limits could be the death of Twitter | Who is ... On Twitter? — January 21, 2009 @ 2:14 pm
Absolutely is does. They are treating a symptom and not a cause but it is still probably what is driving the decision IMHO. A lot cheaper and quicker to limit the API, slow growth, and piss of what is a relatively small number of users than add unlimited capacity. Bad for business yes. Will it hurt them in the long run, who knows.
Comment by strangedesign — January 21, 2009 @ 2:40 pm
This does seem like a rather short-sited thing to do on the part of Twitter. What if Apple had told people that they could not build accessories designed for the iPod? Not that they could do that, but you see where I'm going. Twitter is limiting the growth potential.
It's my guess that Twitter is doing this so that they can force users down a certain path that will correspond with their future plans for monetization.
Comment by Jason Egan — January 21, 2009 @ 3:18 pm
Totally agree with you again.
That was the cynic within me again wondering (and only wondering since I haven't seen evidence) if in future they would offer a “paid for – unlimited/controlled!” access.
As you say, it would make much more sense than simply slamming the door shut.
They will also be doing themselves no favours either. Were it not for apps like SocialToo, Twitter doesn't have a fraction of the appeal it does at the moment because of those apps.
(Cynic head on again, perhaps developing their own 3rd party apps is on the agenda?)
Chris
Comment by shinerweb — January 21, 2009 @ 3:20 pm
[...] today received an email from Twitter API head honcho, Alex Payne. The e-mail, courtesy of SocialToo blog, looks like this: “Starting later this week we’ll be limiting those on the whitelist to 20,000 [...]
Pingback by Twitter API Gets Limited; Will It Hurt App Growth? — January 21, 2009 @ 3:30 pm
I agree that the API limit is going to hurt larger developments, but then again we don't know the true extent Twitters servers are getting hammered for at present. Considering that this API limit has come after the Inauguration then perhaps they have made the change for a valid reason purely to protect their own servers and to keep Twitter running smoothly for it's users. Of course, paid API access would be a good solution however a paid API access solution is not something that can be implemented in a few days/weeks. If the servers are getting hammered hard right now to the extent that it's causing problems/affecting performance then I can understand the limit. I'd certainly rather see a limit imposed rather than have my Twitter experience ruined by slowdowns and downtime due to overloading.
Just because there are no hints towards a paid API model, or towards improving the API doesn't mean it's not going to happen surely? I would imagine that Twitter are aware of how important the development community is and in the long term I seriously doubt they would want to cut that off. At the same time, they also have a business to run and interests to protect and as such they are likely trying not to give away too much information about what they are doing. I'd imagine they have a lot of ideas and changes to implement which they are keeping tight lipped about until it's been finalised.
Just my 2c anyway. I guess only time will tell!
Comment by Chris — January 21, 2009 @ 3:42 pm
Whilst out on a walk today I was thinking about twitter and how it behaves. A year and a half ago they were struggling with the amounts of tweets people were both sending and receiving. They changed many things around and for a company that has been around for as long as they have (in startup terms) they have done very little to personaly improve the usability of their site. In fact since I started using twitter the biggest change is the replies arrow to avoid typing a name.
Where this will hurt them most is on mobile devices. If you're at a desk you can visit the page and although not as convenient it's not that much of a hinderance. Come to mobile devices though and you see how bad it gets. Several times recently I have hit the api limit on mobile 3rd party apps. As a result I simply didn't tweet as much as I would.
I am moving towards friendfeed, more intersted in more dynamic websites than twitter.
Scoble was speaking about the new twitter for friendfeed api to import friends and how, due to his vast number of followers he was running into problems.
As a side note I must admit to skimming through the post as I am not a developer.
Comment by warzabidul — January 21, 2009 @ 4:07 pm
I'm blogging this post Jesse. It's bad news for you and for me (a socialtoo user). But even other services like Seesmic could be affected. It's going to be painful and widespread if this goes ahead. I understand twitter … kind of. But imagine being able to make only 2 phone calls a day! In the end it just does not work. It will limit the service… and get Dave Winer back on his hobby horse, decentralized twitter which Bezos will hate.
It will do damage that's for sure. For instance I see the BBC following the inauguration yesterday and peoples reactions. The have loads of twitter quotes. So the media is waking up to the concept of 'the back channel'. Except when you are a BBC, NBC etc. and you want to do stuff! 20.000 requests should be peanuts I hope.
Comment by dccrowley — January 21, 2009 @ 4:55 pm
I do not classify it as ridiculously stupid to limit API calls to 20K/hour. In fact, speaking as a web service provider, I think that is a very generous limit. However, I would consider it foolish if Twitter did not introduce more powerful/efficient API methods. That would indeed limit their ability to grow.
Comment by F. Andy Seidl — January 21, 2009 @ 4:58 pm
[...] today received an email from Twitter API head honcho, Alex Payne. The e-mail, courtesy of SocialToo blog, looks like this: “Starting later this week we’ll be limiting those on the whitelist to 20,000 [...]
Pingback by BitTorrent And Everything Technology Related - BBCCCP.com — January 21, 2009 @ 5:01 pm
Thanks for the mention! This is important stuff, for sure.
Comment by jessestay — January 21, 2009 @ 5:38 pm
[...] here, you may want to subscribe to my RSS feed. Thanks for visiting!The twitterati and some Twitter App Makers have been up in arms regarding twitters recent API changes that will mean a 20,000 limit will be [...]
Pingback by Twitter API Limiting will Lead to Revenue Model : Nick Halstead, What is this Tech? — January 21, 2009 @ 5:52 pm
Jesse — no idea if this will help your current bind, but if there are sections of your app that just rely on public twitter data, then Gnip can help tremendously by sending notifications when there is new data to be grabbed. This keeps you from wasting polls on endpoints that have not changed.
Eric Marcoullier
CEO
Gnip (http://www.gnipcentral.com)
Comment by bpm140 — January 21, 2009 @ 7:21 pm
This is crazy. Thanks for publicizing it. Twitter could make their life a lot easier by following your (and Andy's) advice and make the API's more like their own database queries.
I thought Twitter was already on AWS.
-Stiennon
Comment by Stiennon — January 21, 2009 @ 7:39 pm
Yeah, we're not a solution for you in the short term. We're rolling out distributed polling fairly soon, but it will be a while before we release *authenticated* distributed polling.
Comment by bpm140 — January 21, 2009 @ 7:44 pm
I agree with abrudtkuhl. A premium service is needed. You can't expect a business to provide a free service that lets YOU build a business for FREE, and get upset when they lookout for themselves. Jesse I know you would make some changes if you were in their shoes. Maybe not the same choice, but you would protect you business first.
I mean come on, Twitter was never meant to be an API for someone to build a business on. I like a lot of the services that use the Twitter API, but I certainly can't blame them. Maybe a discussion with the development community should have taken place first, just to gage the acceptance level, but you really don't know where they sit.
Mostly I think we should work with them instead of building a negative campaign against them.
Comment by Thom Allen — January 21, 2009 @ 7:44 pm
Jesse's math in the article shows why I think 20K is stupid and unreasonable. All it takes is the likes of Scoble or another power user to max out a service for an entire hour. Would you like it if a service you made could only deal with one customer per hour because of changes by a third party?
Comment by Chris Charabaruk — January 21, 2009 @ 7:47 pm
Yeah, it probably wasn't a good idea to pull the carpet that fast, and like I said, Twitter should have engaged the development community first, that would have made the transition easier. Could have been handled a little better.
But what do you think about external services causing issues with the overall service that affects the average person who just uses Twitter. Should they suffer because companies want complete and unfettered use of the API and it's resources? Doesn't sound too fair too me. I understand you position on timing, they have more to worry about than external services.
Comment by Thom Allen — January 21, 2009 @ 8:09 pm
I guess my twitter mobile app is going to change too! that's bad news!
http://m.mwd.com
Comment by LiveCrunch — January 21, 2009 @ 8:17 pm
I thought this is not “by IP” but over all 20.000 requests ( but to answer your question between 50-200) i would say. now multiply that by 1000 or 2000 users (not that I have that many) I would be screwed too!
I feel bad for SocialToo but I do feel bad for Twitter as well, they dont have any revenue income yet from “Ads” yet their servers suck big time. They could of at least hire nice CCNP to clear some data
Comment by LiveCrunch — January 21, 2009 @ 8:27 pm
But Jesse … of course installing on the client is possible! There are several solutions to work around the limits, and then carefully enroll the folks at twitter about the benefits of updating and enhancing their API.
Whining and complaining will probably not work …
Comment by Scott C. Lemon — January 21, 2009 @ 8:30 pm
[...] Now a service called SocialToo has raised the alarm about the forthcoming limit and says that a similar feature it offers will disappear as well. [...]
Pingback by Is Twitter Strangling its Famous API? | google android os blog — January 21, 2009 @ 8:48 pm
[...] 20,000 calls may seem like a lot but some application builders are up-in-arms about the new limit. SocialToo, which accesses Twitter’s API to provide users with notifications when users un-follow them (along with various other services) says the new cap is a poor way of making up for a bad API. “In order to retrieve a list of a user’s followers, the Twitter API currently requires any developer to go through the follower list, 100 people at a time, no matter how large the following is. So, for instance, for Guy Kawasaki, we have to traverse through over 350 pages of followers in order to get his entire list and determine if anyone new followed him, or stopped following him. That’s 350 requests just to get his list of followers – that doesn’t include the requests we have to make to follow each new follower.” — SocialToo Blog [...]
Pingback by Twitter Caps API Calls « chcameron.com — January 21, 2009 @ 9:13 pm
>>Would you like it if a service you made could only deal with one customer per hour because of changes by a third party?<<
Of course not, but that is not what I am suggesting.
What am suggesting is that the ideal approach is not to raise the limit on the number of requests per hour, but raise the limit on how much work each request can accomplish.
For many reasons, it would be far more efficient to perform many operation right at the service (i.e., at Twitter) rather than by requiring the client to do it in hundreds of bits and pieces. If you had to unload a pick-up truck full of sand, it would be very slow going if you were forced to make thousands of trips with a teaspoon. Far better would be to make a few dozen trips with a wheelbarrel.
From Twitter's point of view, they are allowing each service to make 5.5 requests per second on average, 24 hours per day. That is actually very generous, IMO, for a free service. By offering smarter API methods, they could increase by orders of magnitute the work that a client application could accomplish in the same number of request. And in many cases, this would also result in LESS CPU time AND LESS bandwidth. E.g., getting Guy K.'s new subscribers would result in transfering a relatively short list of results compared to transfering his entire subscriber list every time. It is also likely to take less CPU time (in aggregate) on Twitter's end by eliminating all the overhead of hundreds of API invocations, credential verification, etc.
Comment by F. Andy Seidl — January 21, 2009 @ 9:16 pm
Twitter Limits Potential App Growth – How This Hurts Our Users…
Today I received some disturbing news on the Twitter Developers mailing list. In a post on the list, Alex Payne, the API Lead for Twitter…
Trackback by buzzup.com — January 21, 2009 @ 11:48 pm
[...] The SocialToo Blog « Twitter Limits Potential App Growth – How This Hurts Our Users [...]
Pingback by The Game Must Go On | The SocialToo Blog — January 21, 2009 @ 11:59 pm
I can appreciate the concerns and thank you for going through the numbers for your readers. However, I was hoping that somewhere within your post you would have offered an olive branch in the form of a payment structure you would fine reasonable, given the value twitter and their API offer to you.
What would you find a reasonable starting point?
Comment by Christopher_Ross — January 22, 2009 @ 1:07 am
[...] pengembang menanggapi berita ini dengan kekecewaan dan mengharapkan pembatalan rencana ini. SocialToo melalui blog-nya bahkan menyatakan bahwa banyak startup berbasis Twitter yang akan jatuh atau [...]
Pingback by Twitter Batasi Jumlah Akses API | Daily Social — January 22, 2009 @ 1:27 am
The Elephant in the room
If you are an Open Source Developer or Content creator (If you are a member of either service you are a content creator) you should not use Facebook or Twitter.
By using Facebook or Twitter you are essentially raising the value of their companies and applications. Both Facebook and Twitter are closed source content silos that do not allow you to control the content that you create. Neither Facebook or Twitter put the content creator/members at the top of their pyramids when thinking about revenue models. Each of these companies puts their Companies first above the members and communities that have given them value and money.
If you are a developer you may be able to make some money by creating applications for Facebook or Twitter ; but I do not believe that Facebook or Twitter will ever allow your application to eat into their user base or their revenue. Because they are both closed source companies that have the ability to literally cut you off by changing the code/api or by using their proprietary knowledge to build an application that you can not possibly compete with. As a coder understand that when you build and extend Facebook or Twitters propitiatory platforms that you undermine the longevity of the Open Internet.
Content owners and Developers do not help these closed source companies (Twitter and Facebook) in their goal of creating another closed source content trap that will extract hundreds of Millions on dollars from their member and developer communities and give nothing back in return.
Comment by william — January 22, 2009 @ 3:45 am
Out of curiosity, how do you split requests and “process them concurrently”?
Is there a way to make a cumulative requests to get the list of followers?
Don't you have to make one request for each page of followers?
Anyway I think that relationship data (e.g. who follows who) are crucial for the next generation of
microblogging applications. Twiitter and friends will have to address this issue sooner or later.
Comment by funkyboy — January 22, 2009 @ 4:22 am
[...] Wonderful. Just in time to catch the newly arriving short-end of the stick. I can understand why Twitter is doing this. They are riding a tiger that’s [...]
Pingback by So GAE « Don Park’s Daily Habit — January 22, 2009 @ 4:37 am
[...] California hosting facility snags sizable energy-savings rebate The SocialToo Blog: Twitter Limits Potential App Growth – How This Hurts Our UsersRed Hat version 5.3 released The Open Road: Red Hat set to surpass Sun in market capitalization New [...]
Pingback by Between the Lines mobile edition — January 22, 2009 @ 5:08 am
[...] much dismay amongst the digerati Twitter yesterday announced a limit of 20,000 calls per hour to their API for [...]
Pingback by Twitter lays the foundation for it’s revenue model | The Equity Kicker — January 22, 2009 @ 5:29 am
[...] today received an email from Twitter API head honcho, Alex Payne. The e-mail, courtesy of SocialToo blog, looks like this: “Starting later this week we’ll be limiting those on the whitelist to 20,000 [...]
Pingback by Twitter API Gets Rate Limit; Will It Hurt App Growth? | Webtrendblog.com — January 22, 2009 @ 6:13 am
[...] application can poll the service using the API. The SocialToo blog carries an indepth discussion of why the Twitter API rate limit will hamper development, saying: “This is scary news as an entrepreneur and Twitter developer. Twitter has basically [...]
Pingback by LinkedIn gets big business into the swing of social networking | socialmediainfluence.com — January 22, 2009 @ 6:20 am
[...] strange and puzzling why Twitter has decided to impose limits to calls on its API – a move that SocialToo contends “is going go hurt every app out [...]
Pingback by Is Twitter Shooting Itself in the Foot | Twitterrati — January 22, 2009 @ 6:26 am
i asked alex about removing the rate limit. he asnwered that that their new year resolution. I was under impression that there wont be any rlimit on APIs.. though 20K is much more cmpared to existing 100 limits… however as applications grows and traffic increases 20K will look like a small figure
Comment by Kevin — January 22, 2009 @ 7:58 am
Well, I don't know. But I think that this (throttling API calls) will cause a good disruption in the twitter application community. If a developer is not able to find an audience to support their product, they'll either have to dig deep and cover their costs until they do, or rethink the relevance of their product in the market.
The bar gets set higher, and I think we're all the better for it as both developers and end users.
Comment by Christopher_Ross — January 22, 2009 @ 10:10 am
[...] news, though, is that Twitter is even going to limit these outside services to 20,000 requests per hour. That might seem like a lot for a personal user, but when talking about a public website used by a [...]
Pingback by Twitter API Usage And You | PCMech — January 22, 2009 @ 11:09 am
A couple of things:
1. XMPP. if they provided XMPP pub/sub nodes for users feeds and data…. data could be pushed, if people were smart about things, it would scale better than repeated HTTP polling, and it would be awesome.
2. Open Microblogging, Laconica is compatible with the twitter api, provides XMPP *and* services could build implementations of open microblogging that let them subscribe to their clients via that network, and not have to deal with polling limits.
Comment by tychoish — January 22, 2009 @ 11:29 am
The way you present is makes Twitter look cold towards their eco-sphere — which is surprising given how many partners and clients they have; maybe they've tried and are coming back from it, but their success is heavily wired to that strategy, and they appear not too ignorant of that fact. Therefore, I would let them the benefit of the doubt and imagine they have good reasons to do what they seem to have decided. Money appeared tight recently, and saving on the few rather wasteful services (albeit the only way to have Guy's new followers, you method is a real bleeding).
Regarding the specific features that you want, I would make a distinction between:
* new followers (an information users can get by e-mail, and that they might agree to forward to you, with a simple IMAP filter); and
* people who stopped following a certain user.
As pointed out by the extensive literature, interrupted relations or more generally most negative news impair greatly a social service. It's the first question I get asked about Facebook or Twitter, and I can give you examples of services whose usage dropped days after implementing a look-back feature. By empowering Guy, you remove my ability (and some would say 'right') to stop being fed his every move without him knowing. Guy is a great guy, and he knows he twits too much, and probably wouldn't be offended. Others might like that with less detachment, and drama will not only happen, but get front-page treatment. That is threatening to Twitter because, with services such as yours and your competitor's, they cannot guarantee the same freedom to browse feeds.
Twitter, or Facebook who recently banned the Whooper Sacrifice application, do not want the balance of information tilt that way. Given Facebook massive lack of personnel, any of they move is most likely triggered by users. I do not think they decided to remove such a popular app without good reasons, namely users offended either by the fact that they were un-friended, or who forgot they had the application active and un-friended someone who took offense. It could be that non-important friends, aka 'weak-ties' are essential to their success (they are) but I doubt ten of those were worth the ban.
It's a matter of opinion, and would be hard to prove, but the success of a technology such as yours demands that you balance the right to know with the right to remain discreet; a majority of experts would agree with me.
Finally, I would love to have an alert saying that the recent user that friended me is apparently a spambot — provided you give the people behind those the ability to challenge your call. Twitter would love to have the ability to list those, and find out which one are useless, or do not respect the terms of services to ban them, and save some money; maybe they already do.
Comment by Bertil — January 22, 2009 @ 12:24 pm
I'm not sure the competition is between closed and open-source services. It is an important aspect — but adoption of the service is the first step. Compatibility with other service is the only way out of a monopoly, and then, you have to argue in favor of open code.
I say that because the source behind Twitter is not the hardest asset to copy: leveraging a community of practice is. The people behind the Open Stack and DiSo realized that their arch-enemy, Facebook Connect, not only offers a better experience (thanks to good UI and market-base) but also drive massive press and attention to their solutions. Just like them, you need to push Twitter (and its equivalents/competitors) to learn about how to do micro-blogging properly, and let people learn about micro-blogging.
Comment by Bertil — January 22, 2009 @ 12:30 pm
Rather then considering the hard-to-estimate value-added, you might want to consider costs to Twitter; this would make tour service neutral to them, and you'd be in a safer position later to negotiate the (then better known) value created by being the largest information network.
Otherwise, advertising might be the better solution (like spam-filtering, your service is valuable, but hard to put a price on). Were would you be able to put some?
Comment by Bertil — January 22, 2009 @ 12:34 pm
None of this helps getting a user's friends and followers, which is the
heart of this issue.
Comment by jessestay — January 22, 2009 @ 12:57 pm
[...] data on behalf of its users. (Another product that might be affected: Mr. Tweet.) The problem, according to SocialToo developer Jesse Stay, is that for some of its individual users, getting the basic information they need requires not one [...]
Pingback by Freetracking.org » Twitter puts new limits on API calls: Who’s affected — January 22, 2009 @ 1:07 pm
If microblogging were implemented as an XMPP service, your friends/follwers would be the “roster,” (or abstracted as such) or as members of a multi-user-chat (or abstracted as such), and getting a list of users, would be a single request/stream from the server.
OMB doesn't directly solve the API problem, but it does so indirectly: no OMB server needs to “hold” the entire microblogging world (like twitter needs to), so the scaling problems that twitter has with its API limits aren't likely to be as much of a sore point. Secondly, you can set up a local/private OMB instance that would allow you to write your App/site against a “local” platform/site, rather than against a third party site…
—
Having said that, these kinds of problem where “crowd sourced” platforms/sites end up moving things in a direction that helps their business, but hurts the crowd, are *very frustrating* but also not surprising. Alex is a great guy, twitter is a very inspiring platform (that secedes because of the number of users, and the tools that Alex has helped provide for), but it's still a business that really has to serve it's own interest, and if this doesn't illustrate the worth and utility of open source, and the software freedom approach, then I don't know what would.
The heart of twitter's issue, is that when they thought they were providing a “status message service” they never really expected that they'd be creating what amounts to a public messaging and communications service; combined with the fact that non-niche web-services are incredibly difficult to monetize/collect subscriber fee's for…
Developing for closed platforms isn't a bad thing at all–often–but it does put developers completely at the mercy of another party, and that's pretty tough to stomach.
Comment by tychoish — January 22, 2009 @ 2:58 pm
[...] vemos en SocialToo, en la lista de desarrollo para Twitter, Alex Payne, el líder de API de Twitter ha anunciado lo [...]
Pingback by Limitaciones al acceso por API de Twitter - esTwitter.com — January 22, 2009 @ 6:57 pm
[...] today received an email from Twitter API head honcho, Alex Payne. The e-mail, courtesy of SocialToo blog, looks like this: “Starting later this week we’ll be limiting those on the whitelist to 20,000 [...]
Pingback by Twitter API Gets Rate Limit; Will It Hurt App Growth? | Kawink Blog — January 23, 2009 @ 12:28 am
[...] an app that offers a list of Twitter-related services like auto-following, surveys and more, quickly responded with their thoughts and experiences from developing their services. Short story: they don’t believe this is a [...]
Pingback by Twitter Limits API Access, Community Cries Out | Microblink — January 23, 2009 @ 8:30 am
[...] an otherwise < href=”http://tumblelog.marco.org/72580346″>excellent rebuttal to a rather ill-thought-out complaint about Twitter rate-limiting their API Marco misses what I think is the main reason for not building [...]
Pingback by Binary Code » Build a Business on Twitter? Really? — January 23, 2009 @ 3:35 pm
Twitter to me is a bunch of snob losers that want you to check out their HOT thing and they never have any intention of reciprocating,if Google bought Twitter and closed it down I wouldn't miss a beat.
It's great to see a QUALITY blog that is Do follow,mine is also and I'm 
2009 is going to be rough people are to picky and don't appreciate JACK
sadly like a old prostitute society on the internet will use you abuse you and walk away
Despite all that
on the LOOKOUT to network with cool people. If you know what I mean Bring IT
Nice blog Stumbled
Comment by John Sullivan — January 24, 2009 @ 12:35 am
[...] been a lot of discussion regarding the recent announcement about Twitter’s API limiting. The SocialToo blog has a very good overview of the situation. Granted I am not a fan of the limitations, but that is not the reason for this [...]
Pingback by The Danger of Changing The Baseline | Guilda Blog — January 24, 2009 @ 11:27 am
[...] much dismay amongst the digerati Twitter yesterday announced a limit of 20,000 calls per hour to their API for [...]
Pingback by Finance Geek » Twitter lays the foundation for it’s revenue model — January 24, 2009 @ 5:32 pm
[...] you’re new here, you may want to subscribe to our RSS feed. Thanks for visiting!At SocialToo, it has always been my goal to make following the people you want [...]
Pingback by Get Your Follows and Replies Immediately With E-mail Forwarding | The SocialToo Blog — January 29, 2009 @ 3:29 pm
Search engine optimization also referred as SEO in the IT industry is basically the science of making your website important to the search engines in order to rank highly in their search results for a strategically defined set of keywords. Search Engine Optimization is the only key to online success in a scenario where competition is rising high. Online businesses have benefited a great deal through competent and efficient search engine optimization solutions.
Comment by SEO Company in India — February 9, 2009 @ 11:46 am
[...] pesonas que son seguidoras de Guy Kawasaki) merecen desaparecer!. Va Mashable. The SocialToo Blog. [Liga] [etiquetas: [...]
Pingback by » OLDaily por Stephen Downes, enero 21, 2009 TIC, E/A, PER…: — February 9, 2009 @ 3:48 pm
[...] while providing a free service to low-volume players. Twitter recently placed a limit on its API to 20,000 requests/hour. addthis_url = [...]
Pingback by Five Things Twitter Should Do with $35M | Twitterrati — February 15, 2009 @ 12:10 pm
I'm sure they have good reasons for their new policies.We know too little to pass any judgement
Comment by online dating — February 27, 2009 @ 2:45 pm
Ecommerce Web Design we are called SocialToo and not TwitterToo,
Comment by williyamb — April 15, 2009 @ 8:35 am
website design service his change is going to affect thousands of you users, and there’s nothing we can do about it unfortunately.
Comment by williyamb — April 15, 2009 @ 8:36 am
Using smart caching locally can help.
Comment by david — May 13, 2009 @ 1:12 am
David, only to an extent. The only solution I've been able to come up with
is scale to lots of servers, which I'm doing now.
Comment by jessestay — May 13, 2009 @ 1:33 am
Post informative and entertaining post in your Twitter updates. Follow people who interest you, and keep twittering.
Comment by Amanda Crowe — May 21, 2009 @ 4:09 pm
platforms/sites end up moving things in a direction that helps their business, but hurts the crowd, are *very frustrating* but also not surprising. Alex is a great guy, twitter is a very inspiring platform (that secedes because of the number of users, and the tools that Alex has helped provide for), but it's still a business that really has to serve it's own interest, and if this doesn't illustrate the worth and utility
Comment by sikis — June 19, 2009 @ 6:49 pm
yeah, now it's down to 150 requests per hour.. my client's twitter account has 11000+ friends and 11000+ followers.. I can't even do an api request to get their names (just their ids) because trying to get their names from the list of ips gives me the “over the limit” error
Comment by Jeff Donchez — July 6, 2009 @ 3:59 am
I think so.
Comment by wholesale korean clothing — July 13, 2009 @ 9:27 am
I don't understand how you can make 2 completely contradictory statements in the *same sentence.* How can “Developing for closed platforms isn't a bad thing at all” be true, while “developers completely at the mercy of another party” (in other words, your business model is in someone else's hands) is also true?
Comment by candrewswpi — August 4, 2009 @ 3:38 pm
While original comment is 6 months old, and I'm not sure if the events of the past six months are worth anything (and I don't have the time at this moment to consider these implications), here's a clarification:
- I don't mean to present a black and white moral system on this issue. It can be both “not a sin,” and also “not a good thing.” It's not, I think, a transgression to develop for a closed platform in an open way, and also I think it's true that putting users/developers at the mercy of another party is something that should be avoided.
- Perhaps I should have said “isn't neccessarily a bad thing.”
We have examples of people developing tools aplenty. Emacs/Cgywin/Notepad++ and so forth all exist for windows and people put a lot of time into that, and that's not bad, but I think everyone's a bit wary of that kind of work becasue of the way it requires them to have their work exist at the mercy of other people.
Or hell, I think the whole mono debate is contentious *because* of this tension…
Cheers,
sam
Comment by tychoish — August 4, 2009 @ 4:00 pm
You guys should check out this twitter app http://www.sponsoraloser.com It's about posting twitter losers and finding them the answer to their problems, It's very funny! and It's all about helping others.
Comment by Jhonny — August 6, 2009 @ 1:09 am