Using Redis to Control Recurring Sets of Tasks


Create an application to check the availability of a large number of URLs. This list is constnatly changing and every url must be checked at a specific interval (e.g., every minute). The actual checking of the URL needs to be distributable over several servers. This is because if a URL is unavialable, we do not want to continue checking other URLs while waiting for the timeout.

Key Advantages of Redis:

Redis had two key features that made writing this code very straightforward.

  • rpoplpush: Pop the element on the right side of the list, return it, push onto left side of the list. This simulates a circular list.
  • blpop: Block until a new element is available at the list.

Code Design:


  • Wake up every N seconds.
  • rpoplpush the clock queue and remember this first value.
  • Push this value onto a task queue. 
  • Continue reading values on the clock queue until encountering a value the same as the first value.


  • Read values of the task queue.
  • Performs URL availablity check.
  • Block until there is a value in the task queue.

Code snippets:


Code snippet for Worker:

Final Thoughts:

Getting started with Redis is very straightforward. Redis is easy to install and has a small number of commands to manipulate data. The Redis gem is a thin abstraction layer over the core Redis commands. In under an hour, I was able to write code that manipulated Redis data.

I'm looking forward to trying out Redis on some data processing problems at my current job. I need to process data in stages and I either write the data to temporary files are dump the data into MySQL. Temp files are an awkward solution because I need the process to be interuptable/resumable and dumping the data into MySQL is a performance bottleneck. For my needs, Redis looks like a nice sweet spot between writing temporary files and pulling the data into MySQL.

EDIT: Thanks to Ted Naleid for finding a bug in the original version of the clock code where the first_url was never put onto the worker queue.