Problem

I needed a script to control a batch workflow process. There were a lot of moving parts to the process and dividing it into mutliple scripts or using command line arguments did not seem feasible. Rake looked like a perfect fit for this problem. One particular sub-problem, was I needed to process several files in a particular way. Originally, I thought I only needed to process them all at once, so I wrote a rake task like the following:

First Iteration

Second Iteration

However, in order to debug the process, I needed to process the files one by one. At first I started to create seperate tasks for each file. But then I remembered that there is nothing special about the way tasks are defined, and I can define tasks using some basic metaprogramming:

Conclusion

Really good DSLs like Rake and Sinatra make me forget I am writing Ruby. While I feel restricted (in a good way) to this subset of commands, at any time I can use the full power of Ruby if I need it. Being able to dynamically create tasks let me quickly create a script that is now up to ~40 tasks (and growing).

While this is starting to get a little bit unweidly, the equivalent bash script would be much worse. What keeps the script manageable is that much of the process is specificied as data. Most of the program is processing a hash table and creating a task per entry the hash table.

Problem:

Create an application to check the availability of a large number of URLs. This list is constnatly changing and every url must be checked at a specific interval (e.g., every minute). The actual checking of the URL needs to be distributable over several servers. This is because if a URL is unavialable, we do not want to continue checking other URLs while waiting for the timeout.

Key Advantages of Redis:

Redis had two key features that made writing this code very straightforward.

• rpoplpush: Pop the element on the right side of the list, return it, push onto left side of the list. This simulates a circular list.
• blpop: Block until a new element is available at the list.

Code Design:

Clock:

• Wake up every N seconds.
• rpoplpush the clock queue and remember this first value.
• Push this value onto a task queue. 
• Continue reading values on the clock queue until encountering a value the same as the first value.

Worker(s):

• Read values of the task queue.
• Performs URL availablity check.
• Block until there is a value in the task queue.

Code snippets:

Clock:

This is a presentation I gave at the First Annual VIVO Conference in NYC. I created a crawler that traverses the linked data available at http://vivo.ufl.edu to generate graphs of the academic organization structure. I start at http://vivo.ufl.edu/individual/UniversityOfFlorida and recursively crawl all core#subOrganization links.

Drawning Organization Charts Using VIVO

This is the best non-interactive graph I've created so far. This graph consistes of all colleges, departments, centers, and institutes.

This is the same graph with labels drawn on.

The biggest problem with adding labels to the graph is the labels are so long (20-30+ characters). I created an interactive version at: http://qa.vivo.ufl.edu/infovis/demo.html

This graph consists of only Colleges and Departments (about ~150 ondes). However, seeing the labels is still difficult in spots. Particularly around the bigger clusters, College of Medicine and College of Liberal arts and Sciecnes.

I created a plist file and when I built my project I received the following error:

CopyPlistFile build/Debug-iphonesimulator/TestApp.app/TestArray.plist TestArray.plist
cd /Users/arockwell/dev/TestApp
 setenv PATH "/Developer/Platforms/iPhoneSimulator.platform/Developer/usr/bin:/Developer/usr/bin"
/Developer/Library/Xcode/Plug-ins/CoreBuildTasks.xcplugin/Contents/Resources/copyplist --convert binary1 TestArray.plist --outdir /Users/arockwell/dev/DrinkMixer/build/Debug-iphonesimulator/TestApp.app
 
sh: plutil: command not found
Command /Developer/Library/Xcode/Plug-ins/CoreBuildTasks.xcplugin/Contents/Resources/copyplist failed with exit code 127

ln -s /usr/bin/plutil /Developer/usr/bin/