About the RotSweep crawler

What RotSweep is

RotSweep is a link-rot and content-drift monitor. Our customers give us a list of URLs to watch — normally the outbound links on their own pages — and we re-fetch those URLs on a schedule so they hear about it when one breaks, starts redirecting somewhere else, or quietly stops saying what it used to say.

If you see us in your logs, a customer of ours links to a page on your site and asked us to keep an eye on that link. RotSweep is not a search-engine crawler. It does not spider your site and it does not build an index. It fetches the exact URLs it was pointed at, and nothing else.

How to identify it

By default every request we make carries this User-Agent:

Mozilla/5.0 (compatible; RotSweep/1.0; +https://rotsweep.com/agent)

The contact URL in it is this page. A request that claims to be RotSweep but does not match that string is not ours.

A customer may opt a specific URL into a browser-like fetch when a host refuses this crawler. Those requests impersonate Chrome, Firefox or Safari and will not match the string above.

What it does on your server

• GET only. We never POST, never submit a form and never write anything.
• One page per check. We read the URL we were given and stop there. We do not follow links out of it, so a monitored URL is one request — not a crawl.
• Paced per domain. Roughly one request per second per domain by default, with a burst of two. Checks queued for the same domain wait their turn.
• Crawl-delay is honoured. If your robots.txt asks for a delay, it lowers our rate for your domain accordingly — Crawl-delay: 5 means one request every five seconds.
• Redirects are followed to the end, up to ten hops, and we stop on a loop.
• Bounded reads. A check times out after fifteen seconds and stops reading after a few megabytes.

How to block it

Add a group for RotSweep to your robots.txt:

User-agent: RotSweep
Disallow: /

A User-agent: RotSweep group replaces the * group for us entirely, so anything you put there is the whole of what applies to us.

There is one caveat, and we would rather state it plainly than have you find it out from your logs. RotSweep makes two kinds of request, and robots.txt does not weigh the same in both:

• Scanning — when a customer points RotSweep at a page or a sitemap to discover the links on it. Scanning fully respects robots.txt. A disallow means we do not fetch the page.
• Monitoring checks — re-fetching a specific URL a customer explicitly designated. Here robots.txt is advisory: we record the disallow against the result, and we still perform the check. A customer is paying us to tell them when a link they depend on breaks, and a rule in your robots.txt does not remove that link from their page.

So a disallow stops discovery outright, and leaves one paced request per already-designated URL per sweep. If you want those stopped as well, tell us and we will stop them — see below.

If we are asking too often

Tell us in the response and we listen. This is cheaper for both of us than a block.

• 429 with Retry-After is honoured for the whole domain for as long as you asked, not just for the URL that returned it. Every other check queued for your domain waits it out.
• 403 and 401 are read as “we were refused”, not “this link is broken”. We back off and try again later, and we never report a customer’s link as dead on the strength of a refusal. If you send Retry-After with it, that is honoured too.
• Timeouts and 5xx are retried once, about ten minutes later, before anything is reported. A blip is not an incident.

Talk to us

If you want RotSweep to stop fetching your site entirely, if you think it is misbehaving, or if you want to know why a URL of yours is being checked, email support@rotsweep.com. A person reads it and answers on business days — we are in Athens, Greece. Our other addresses are on the contact page.

If you would rather see what the product actually does first, that is on the how it works page.