Create an application to check the availability of a large number of URLs. This list is constnatly changing and every url must be checked at a specific interval (e.g., every minute). The actual checking of the URL needs to be distributable over several servers. This is because if a URL is unavialable, we do not want to continue checking other URLs while waiting for the timeout.
Key Advantages of Redis:
Redis had two key features that made writing this code very straightforward.
- rpoplpush: Pop the element on the right side of the list, return it, push onto left side of the list. This simulates a circular list.
- blpop: Block until a new element is available at the list.
- Wake up every N seconds.
- rpoplpush the clock queue and remember this first value.
- Push this value onto a task queue.
- Continue reading values on the clock queue until encountering a value the same as the first value.
- Read values of the task queue.
- Performs URL availablity check.
- Block until there is a value in the task queue.