The SocialToo Blog

March 8, 2010

Enabling Phishing Protection for All SocialToo Users

Filed under: Uncategorized — Jesse Stay @ 8:00 am

I’m proud to announce that along with Friday’s new design and DM inbox launch, SocialToo has officially enabled protection for all 60,000+ users from malicious and phishing DMs that could lead to spammers and hackers stealing users passwords.  Previously we had only enabled this for our users that were using the DM Filters found under “Filters” in the SocialToo interface.  This move is unprecedented, in that we hope this makes Twitter a much safer place.

In the past, as we’ve reported, Twitter has been subject to very large-scale phishing attacks, causing users that clicked on links in DMs to be tricked into providing their Twitter username and passwords.  These accounts would then become compromised, providing an avenue for more spam, often unknown to those whose accounts were compromised.  We’ve found that no one is immune to this, as even some of the largest accounts on Twitter have been subject to such compromise.

Starting today any SocialToo user that logs in through Twitter will have proactive protection on their Twitter account DMs.  We will scan each DM, show the clean DMs in your SocialToo Inbox, and delete and filter out any DMs we detect are malicious.  No DM filters are necessary to protect from truly malicious dms – it all happens automatically for our users.  In addition to deleting the DM from Twitter, we also send a DM on your behalf to @spam on Twitter notifying the Twitter team of the compromised accounts.  We hope that this can help Twitter identify the compromised accounts more easily.  We will also be communicating compromised accounts via other means to Twitter as well.

So far, SocialToo has blocked a total of 19,814 malicious DMs just amongst the 2,000 or so users that have implemented this feature.  We expect this number to get much, much bigger now that 30 times that number are having their DMs filtered.  A fun thing to do is in your SocialToo Inbox, refresh the page and look at the “Total SocialToo DMs filtered to date” number in the lower-right.  You’ll notice it goes up almost every time – that number is accurate!  We also detect these much, much earlier than Twitter is able to.

As has always been a priority, we feel keeping your stream clean and the web in general a cleaner place is important.  Hopefully this makes a significant change in how clean the streams of Twitter users are.  If you have any suggestions on how we can make it even cleaner, drop us a note!

January 28, 2009

Inaccuracies in Stats E-mail Tonight

Filed under: Uncategorized — Tags: , , , — Jesse Stay @ 12:38 am

Just as forewarning, there will be some missing followers and unfollowers in your stats e-mail tonight due to some limits we hit with our database today. The database problems have been resolved.

As we grow, we expect growing pains such as this to happen, and we’re working to implement measures to prepare for such growing pains, so we hope them to become much less frequent. Starting tomorrow, your stats should be back to normal. Please note that, as we mentioned earlier your “last tweets” will be slightly inaccurate until Twitter updates their API to allow us to update more frequently. We expect that to happen in the next week. We appreciate your patience!

Stay tuned – we have some exciting new features coming your way shortly!

January 23, 2009

Auto Reply Glitches, Incorrect “Last Tweets”

Filed under: Uncategorized — Tags: , , , — Jesse Stay @ 11:25 pm

(UPDATE: Some of you are asking if you actually sent that message in your report. That update was only a typo in the report – you did not actually send that message. Your following/unfollowing stats are accurate, minus the last tweet message.)

Some of you may have noticed this morning and late yesterday that if you had the placeholders, <<firstname>> or <<lastname>> it was placing someone else’s name in it’s place instead of the intended destination’s first and last names. In addition, in your nightly e-mail last night you may have seen the wrong “last tweet” listed.

We Tweeted about this on our Twitter account this morning and disabled the incorrect messages sent, and we have tracked down the problem. It appears the problem was due to a bug in the Twitter libraries we had switched to yesterday. We’ve fixed this, and your auto-reply dms should be correct now. Tonight, you may still have some incorrect “last tweet”‘s in your nightly e-mail, but after that it should be business as usual. We apologize for this inconvenience and have put in place measures to keep this from happening in the future. As always, you may e-mail contact@socialtoo.com if you have any issues or questions.

Stay tuned – as we mentioned we’ve slowed down our auto-following for a few days, but we have a few solutions coming very shortly that will make that much better. Be sure to subscribe to the blog to get updates! Also, if you have problems be sure to check our Twitter @socialtoo account to see if we have posted a resolution.

January 21, 2009

Twitter Limits Potential App Growth – How This Hurts Our Users

Filed under: Uncategorized — Tags: , , , , — Jesse Stay @ 2:18 am

Today I received some disturbing news on the Twitter Developers mailing list.  In a post on the list, Alex Payne, the API Lead for Twitter, informed developers they would be placing new limits on the API which will take place at the end of this week.  In the note, he said:

“Starting later this week we’ll be limiting those on the whitelist to
20,000 requests per hour. Yes, you read that right: twenty THOUSAND
requests per hour. According to our logs, this accounts for all but
the very largest consumers of our API. This is essentially a
preventative measure to ensure that no one API client, even a
whitelisted account or IP, can consume an inordinate amount of our
resoures.”

By capitalizing the word THOUSAND, Alex scares us into thinking 20,000 requests in a single hour seem like a lot. But I argue as apps grow this is going to hurt every app out there. I’m arguing that 20,000, or any request-rate limit for that matter, limits any app out there from being able to develop on the Twitter platform, and I don’t see why any able-minded entrepreneur would want to build on it if there’s such a rate limit in place. Here’s why:

As a user of SocialToo, you’re aware that we provide services to enable others to follow those that follow them, as well as other services to enable users to stop following those that stop following them.  We’ve also enabled anti-spam measures, and were working to build more on this premise, which we detect when spammers follow you, and then stop following you in the same day or short amount of time, and we stop following them for you.  In addition, we’re providing very informative e-mails for our users enabling them to track those that stop following them, along with approximately what Tweets they posted when the person left them.  For any user concerned about building a following, these are powerful tools, and we were working to only make them better for you.  Twitter provides nothing like this.

However, there is a caveat to the Twitter API that makes this process quite difficult.  We’ve become very good at it, but it’s no easy task.  In order to retrieve a list of a user’s followers, the Twitter API currently requires any developer to go through the follower list, 100 people at a time, no matter how large the following is.  So, for instance, for Guy Kawasaki, we have to traverse through over 350 pages of followers in order to get his entire list and determine if anyone new followed him, or stopped following him.  That’s 350 requests just to get his list of followers – that doesn’t include the requests we have to make to follow each new follower.  There is no better way to get this information.

Now, let’s go back to the request rate limit issue above. I’m going to get into some math here to show you why this is a problem. Imagine we process him, Chris Pirillo, Robert Scoble, and others all with over 30,000 followers.  It’s going to take quite a long time to get through each user’s list of followers, and we have near 10,000 users to process!  So, the only way to scale this process is to split these users up and process them concurrently so we get through the users faster and can check for new followers again.  So, if we split that into two, making 5,000 users we have to process, that makes the potential of 2 requests (at least) per second, equalling 7,200 requests in an hour.  However, processing that many would mean we take over a day, possibly several days to get through all your followers! (Remember, we have to traverse through your entire list of friends)

So, let’s double that, or even triple that to make it a little more bearable.  At triple, we’re at 1,250 users for each process we run at the same time, making it 8 concurrent processes, a potential of at least 8 requests per second.  Already, we’ve exceeded the limit, as 8 requests * 60 seconds * 60 minutes = 28,800 requests an hour.  I can already tell you that even that number is pretty hard to go through in a short amount of time, and where we’re growing at near 5,000 users or more a month that number is only going to grow.

Twitter is venturing into dangerous territory here with this new rate limit.  The 20,000 requests Alex mentions is a hard cap, which means no other developer can grow above that, ever, no matter how big they get.  In essence, Twitter just limited their developers even more, keeping us from ever getting big enough to build a viable business model.

I’ve talked with Alex about this, and to make matters worse, he stated that in addition to the rate request, Twitter has no intention to improve the API in order to reduce the requests we have to make to make their load any lighter. From our conversation, his words were, “We want people to be able to follow and unfollow without any social repercussions. So it’s extremely unlikely that we’re going to make changes to the API that enable this kind of application or behavior.”  Therefore, expect Qwitter to go out of business as well as they grow, as well as FriendOrFollow, or TwitterKarma.  I would also add TwitterGradr and TweetStats to this list.  Essentially, any app that needs to get following data could potentially, as they grow, be affected by this.  I guarantee so long as these apps keep growing they will eventually hit this limit and be forced out of business.

I agree they need to protect their servers, but with the rigorous funding and expected business plan they are expected to initiate soon, along with Jeff Bezos as an investor, getting a sweetheart deal on scalable EC2 servers ought to be easy. By implementing this limit, it makes it near impossible for SocialToo, or any Twitter-based business for that matter, to grow and build on top of Twitter. Any app that retrieves a user’s followers should be scared out of their minds by this limit, and I would argue the same goes with other aspects of the API.

What are we going to do about this at SocialToo?  I have some plans – we are called SocialToo and not TwitterToo, after all, but if this limit is enforced, as of the end of this week we will probably remove your nightly e-mails, remove auto-unfollow and unfollow filtering, and possibly have to temporarily disable auto-follow due to the short time-frame we’re being given to comply.  Your surveys will still work, as will the Facebook profile redirect, and we hope to have auto-follow back up shortly after.  However, this change is going to affect thousands of you users, and there’s nothing we can do about it unfortunately. Unfortunately having everyone migrate to other services due to this rate limit is going to cause the same problem for those services as well.

I suggest Twitter remove the rate limit and instead work on fixing their API to reduce the need for so many requests to get your friends and followers.  I suggest you, our users, spread the word about this and write to biz@twitter.com and ev@twitter.com stating your concern on this matter.  SocialToo will not be shut down, but many of the services you have come to love for Twitter, including those of our competitors, and many other Twitter-based services are in jeopardy.  This is scary news as an entrepreneur and Twitter developer.  Twitter has basically just limited how big any Twitter-based business can grow. This makes it worthless to build a business on top of Twitter now.

Powered by WordPress