Possibilities for future improvement
Instead of using a fixed queue retry period, instead use a TCP-style bounded exponential backoff
- If you have just tried to talk to a particular site and they were down, odds are that they will be down for a while and thereís no sense wasting your time and theirs in needlessly trying them too quickly
- Lots of real-world experience in the TCP algorithm