New Post: Twitter User Network Import issues

This is a known problem with getting large networks from Twitter. There is a long discussion about it in "Partial network problem in Twitter" at http://nodexl.codeplex.com/discussions/447703, but I can summarize it by saying that Twitter seems to arbitrarily reject further requests at random times, and that I don't know of a way to fix this in NodeXL. I would actually like to remove the NodeXL features that can lead to this condition, because they just aren't working reliably today.

Regarding the 15-minute pauses, NodeXL is required to wait 15 minutes after the first request in a 15-minute "window," not 15 minutes after the most recent request. So if the first request is at 1 PM and the limit is reached at 1:05 PM, then Twitter tells NodeXL to wait until 1:15 PM for the next request, not 1:20 PM.

And that's the way it works, by the way--Twitter tells NodeXL when to start again, as opposed to NodeXL starting a timer. For a while I thought that clock synchronization issues might be causing the problem, but then I noticed that Twitter sometimes kicks NodeXL out on the very first request in a network, which it's not supposed to do. That lends weight to my theory that Twitter protects its servers with a rate limiting scheme that is sometimes augmented by outright rejection based on server loads.

-- Tony