I’ve taken a few implementations apart and I’ve tried to figure out what the intention of Euler’s Phi function is. I think I’ve got it. I found an interesting implementation in Ruby here but it wasn’t satisfactory. Sure, the code was incredibly minimalistic, but it was also recursive and overflowed the stack for anything over 7704 iterations. I didn’t like the inflexibility of it. So I rewrote it in big ugly moron code that isn’t recursive, but is capable of scaling to some decently large totients. I might see if I can code something similar in Rust, just for shits and giggles.

I’ve figured out an easy way to get at interesting data. First I fire up the tweet nabber written in ruby, then run this cheap piece of crap and extract only time data from tweets pulled.

After getting the time data, I open the *.csv in Minitab and create a histogram with it. I then have the frequency with which a person tweets over 3k tweets by time, so I can approximate when they’re most active throughout the day based on my relative time zone.

My current tweet dumper iteration dumps everything collected into a flat file. That’s fine for intermittent use, but won’t do much good when we eventually get into the “big leagues” and start grabbing at streams of data that will include above and beyond 3200 tweets. Flat files will quickly swell to sizes that are no longer manageable.

Wat do.

Well, there are hundreds of “database backend” options. Literally hundreds. We’re inevitably going to be storing tweets from various sources with multiple goals in mind. A veritable forest of *.csv files doesn’t neatly organize our data. SQLite3 will provide the backend. It’s local, zero configuration required, and we can concentrate all our information into a single file with multiple tables.

I’ve made improvements to the tweet dumper. Added geo information. Also added progress bar, so check gem requirements.

#!/bin/env ruby
# encoding: utf-8

require 'twitter'
require 'csv'
require 'progressbar'

client = Twitter::REST::Client.new do |config|
	config.consumer_key = "YoursHere"
	config.consumer_secret = "YoursHere"
	config.access_token = "YoursHere"
	config.access_token_secret = "YoursHere"
end

scrname = String.new ARGV[0]

def collect_with_max_id(collection=[], max_id=nil, &block)
  response = yield(max_id)
  collection += response
  response.empty? ? collection.flatten : collect_with_max_id(collection, response.last.id - 1, &block)
end

def client.get_all_tweets(user)
  twtcount = user(user).statuses_count
  if twtcount > 3200
      twtcount = 3200 / 200
  else
      twtcount = twtcount / 200
  end
  pbar = ProgressBar.new("Downloading", twtcount)
  collect_with_max_id do |max_id|
    pbar.inc
    options = {:count => 200, :include_rts => true}
    options[:max_id] = max_id unless max_id.nil?
    user_timeline(user, options)
  end
end

junk = client.get_all_tweets(scrname)

CSV.open("#{scrname}.csv", "w") do |csv|
	junk.each do |tweet|
		csv << [tweet.id, tweet.created_at, tweet.user.screen_name, tweet.text, tweet.source, tweet.geo]
	end
end

I don’t comment. Sorry. I guess I can go back through and comment where it’s helpful and repost another time. It works with Ruby 1.9.2 anyways.

Requires the twitter gem. Install it as per usual. Code as follows:

#!/bin/env ruby
# encoding: utf-8

require 'twitter'
require 'csv'

client = Twitter::REST::Client.new do |config|
	config.consumer_key = "insert"
	config.consumer_secret = "insert"
	config.access_token = "insert"
	config.access_token_secret = "insert"
end

def collect_with_max_id(collection=[], max_id=nil, &block)
  response = yield(max_id)
  collection += response
  response.empty? ? collection.flatten : collect_with_max_id(collection, response.last.id - 1, &block)
end

def client.get_all_tweets(user)
  collect_with_max_id do |max_id|
    options = {:count => 200, :include_rts => true}
    options[:max_id] = max_id unless max_id.nil?
    user_timeline(user, options)
  end
end

junk = client.get_all_tweets(ARGV[0])

CSV.open("#{ARGV[0]}.csv", "w") do |csv|
	junk.each do |tweet|
		csv << [tweet.id, tweet.created_at, tweet.user.screen_name, tweet.text, tweet.source, tweet.geo]
	end
end

Excellent. I’m going to revise it as necessary, but it’s a most effective scraper. Though I’d love to add some sort of progress bar to it, haven’t succeeded in that yet. I’ll keep you posted and update it as the iterations of this thing change. It was smashed together from the twitter gem’s bare scraper and CSV output added. I’m quite pleased. Going to also consider adding time and date statistics compilation. I might just write an entirely separate script for that. Not sure yet.

Grab the mention counter. Grab the tweet scraper. Point it at four or five “known” associated targets and grab 3k tweets at regular expected intervals. Better yet, use tweepy to regularly grab tweets as they’re sent. Amass a good amount. Once you’ve amassed a good amount of tweets, fire up the mention counter. Make your cutoff large. Make it count.

Fire up Gephi and open your CSV in it. It should automatically generate an interesting map. Use the heat map feature to get even more interesting results. Be amazed that you can OSINT. Most of all, have fun.

Ruby is like the Visual Basic of the scripting world. You can do simple shit in ten lines of code or less. After experimentation, googling, and satisfaction of curiosity I’ve sated myself with a simple counterpart to my python tweet dumper.

Two arguments. Twitter screen name and cutoff for number of mentions. It’s useful to see your data in a new way. I’m going to take it further pretty soon, processing date information to squeeze every bit of usefulness out of it.