I’ve figured out an easy way to get at interesting data. First I fire up the tweet nabber written in ruby, then run this cheap piece of crap and extract only time data from tweets pulled.
After getting the time data, I open the *.csv in Minitab and create a histogram with it. I then have the frequency with which a person tweets over 3k tweets by time, so I can approximate when they’re most active throughout the day based on my relative time zone.
My current tweet dumper iteration dumps everything collected into a flat file. That’s fine for intermittent use, but won’t do much good when we eventually get into the “big leagues” and start grabbing at streams of data that will include above and beyond 3200 tweets. Flat files will quickly swell to sizes that are no longer manageable.
Wat do.
Well, there are hundreds of “database backend” options. Literally
hundreds. We’re inevitably going to be storing tweets from various
sources with multiple goals in mind. A veritable forest of *.csv files
doesn’t neatly organize our data. SQLite3 will provide the backend. It’s
local, zero configuration required, and we can concentrate all our
information into a single file with multiple tables.
I’ve made improvements to the tweet dumper. Added geo information. Also added progress bar, so check gem requirements.
#!/bin/env ruby
# encoding: utf-8
require 'twitter'
require 'csv'
require 'progressbar'
client = Twitter::REST::Client.new do |config|
config.consumer_key = "YoursHere"
config.consumer_secret = "YoursHere"
config.access_token = "YoursHere"
config.access_token_secret = "YoursHere"
end
scrname = String.new ARGV[0]
def collect_with_max_id(collection=[], max_id=nil, &block)
response = yield(max_id)
collection += response
response.empty? ? collection.flatten : collect_with_max_id(collection, response.last.id - 1, &block)
end
def client.get_all_tweets(user)
twtcount = user(user).statuses_count
if twtcount > 3200
twtcount = 3200 / 200
else
twtcount = twtcount / 200
end
pbar = ProgressBar.new("Downloading", twtcount)
collect_with_max_id do |max_id|
pbar.inc
options = {:count => 200, :include_rts => true}
options[:max_id] = max_id unless max_id.nil?
user_timeline(user, options)
end
end
junk = client.get_all_tweets(scrname)
CSV.open("#{scrname}.csv", "w") do |csv|
junk.each do |tweet|
csv << [tweet.id, tweet.created_at, tweet.user.screen_name, tweet.text, tweet.source, tweet.geo]
end
endI don’t comment. Sorry. I guess I can go back through and comment where it’s helpful and repost another time. It works with Ruby 1.9.2 anyways.
Requires the twitter gem. Install it as per usual. Code as follows:
#!/bin/env ruby
# encoding: utf-8
require 'twitter'
require 'csv'
client = Twitter::REST::Client.new do |config|
config.consumer_key = "insert"
config.consumer_secret = "insert"
config.access_token = "insert"
config.access_token_secret = "insert"
end
def collect_with_max_id(collection=[], max_id=nil, &block)
response = yield(max_id)
collection += response
response.empty? ? collection.flatten : collect_with_max_id(collection, response.last.id - 1, &block)
end
def client.get_all_tweets(user)
collect_with_max_id do |max_id|
options = {:count => 200, :include_rts => true}
options[:max_id] = max_id unless max_id.nil?
user_timeline(user, options)
end
end
junk = client.get_all_tweets(ARGV[0])
CSV.open("#{ARGV[0]}.csv", "w") do |csv|
junk.each do |tweet|
csv << [tweet.id, tweet.created_at, tweet.user.screen_name, tweet.text, tweet.source, tweet.geo]
end
endExcellent. I’m going to revise it as necessary, but it’s a most effective scraper. Though I’d love to add some sort of progress bar to it, haven’t succeeded in that yet. I’ll keep you posted and update it as the iterations of this thing change. It was smashed together from the twitter gem’s bare scraper and CSV output added. I’m quite pleased. Going to also consider adding time and date statistics compilation. I might just write an entirely separate script for that. Not sure yet.
Code first, talk later.
#!/usr/bin/env python
# encoding: utf-8
import tweepy #https://github.com/tweepy/tweepy
import unicodecsv
import sys
#Twitter API credentials
consumer_key = ""
consumer_secret = ""
access_key = ""
access_secret = ""
def get_all_tweets(screen_name):
#Twitter only allows access to a users most recent 3240 tweets with this method
#authorize twitter, initialize tweepy
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_key, access_secret)
api = tweepy.API(auth)
#initialize a list to hold all the tweepy Tweets
alltweets = []
#make initial request for most recent tweets (200 is the maximum allowed count)
new_tweets = api.user_timeline(screen_name = screen_name,count=200)
#save most recent tweets
alltweets.extend(new_tweets)
#save the id of the oldest tweet less one
oldest = alltweets[-1].id - 1
#keep grabbing tweets until there are no tweets left to grab
while len(new_tweets) > 0:
print "getting tweets before %s" % (oldest)
#all subsiquent requests use the max_id param to prevent duplicates
new_tweets = api.user_timeline(screen_name = screen_name,count=200,max_id=oldest)
#save most recent tweets
alltweets.extend(new_tweets)
#update the id of the oldest tweet less one
oldest = alltweets[-1].id - 1
print "...%s tweets downloaded so far" % (len(alltweets))
#transform the tweepy tweets into a 2D array that will populate the csv
outtweets = [[tweet.id_str, tweet.created_at, tweet.text.encode('utf-8'), tweet.geo, tweet.source] for tweet in alltweets]
#write the csv
with open('%s_tweets.csv' % screen_name, 'wb') as f:
writer = unicodecsv.writer(f)
writer.writerow(["id","created_at","text","geo","source"])
writer.writerows(outtweets)
pass
if __name__ == '__main__':
#pass in the username of the account you want to download
get_all_tweets(sys.argv[1])The entirety of this script doesn’t belong to me at all. My only contribution is fixing utf-8 issues. Requires tweepy and unicodecsv. Outputs tweets in a comma-delimited text file.