# If I Had a Text File, I’d Hack Regexes in the Morning

Yesterday the topic of academic citation counts came up, so I decided that I should write up some tools for exploring cite counts. The first thing I did was to build a cheap screenscraper in Ruby for pulling citation count information from Google scholar. You’ll see the ugly hack I produced below.

 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41  module CitationTools require 'rubygems' require 'open-uri'   def get_ten_most_cited_works_for_author(author_name) # First, let's clean up the author's name before using it in a URL. escaped_author_name = author_name.gsub(/\s+/, '+')   # Let's create a variable we'll place the Google Scholar HTML in. page_content = nil   # Let's figure out the right URL for Google Scholar. url = "http://scholar.google.com/scholar?q=#{escaped_author_name}"   # Let's access that URL using open-uri and get the HTML from the page. open(url) do |page| page_content = page.read() end   # Let's scan the HTML for the names of this author's works. work_titles = page_content.scan(/

.*?>([^<]+)(?:<\/a><\/span>)?(?:(?:)|(?:\s+-\s+)|(?:\s+-\s+ title, :citation_count => cite_counts[index]} end return works else puts "Failed to process HTML for #{author_name}" return nil end   end end

With that in hand, I wrote a simple wrapper to pull information for a list of authors you store in a file called authors.txt from Google Scholar. The wrapper then prints a CSV file to STDOUT that can be redirected to a file for later analysis.

 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22  # Let's include a mix-in with some methods for parsing Google scholar data. require 'CitationTools' include CitationTools   # Let's pick a haphazard sample of authors. authors = File.new('authors.txt', 'r').readlines.map {|line| line.chomp}   # Let's add a header line to our output. puts '"Author","Work","Citations"'   # And then let's iterate over those authors. authors.each do |author_name| cited_work_data = get_ten_most_cited_works_for_author(author_name)   if cited_work_data.nil? print "Skipping #{author_name}" end   cited_work_data.each do |cited_work| puts "\"#{author_name}\",\"#{cited_work[:title]}\",#{cited_work[:citation_count]}" end end

Then I coded up a simple barplot in R to give you a sense of the citation count for the first few authors that came to mind. The result is below.

Now I think the goal should be to put these tools to a good use.

### 2 responses to “If I Had a Text File, I’d Hack Regexes in the Morning”

1. Great experiment, really like the dead on simple use of ruby to answer a basic question. This is where scripting languages thrive!

I think it also shows where an API is incredibly useful. While your example makes the best of Google’s search results, if they suddenly change “Cited by (\d+)” to “Cited (\d+) times”, your script breaks.

A simple fix, of course, but not fun if you gave it to a friend or went back a year later and wondered why its not working. Or maybe they’ll move the results to another page.

Scanning pages can be really powerful to grab a ton of data easily, but susceptible to changes in results. A predictable excuse might be missing out on ad revenue if someone makes a better app/page that ties into the data. However, in my experience with Google Scholar there are little or no advertisements.

Regardless, I enjoyed reading your script- clean code, easy to follow even with my ruby being very rusty. Fantastic presentation of idea -> script -> chart!