Our crawler
WhoTheyFindBot
If you found this page in your access log, that request came from us. This page says what the crawler is for, how it behaves, and how to stop it.
What it does
WhoTheyFind reads a website the way a search engine and an AI assistant read it, then tells the site owner what is holding it back. WhoTheyFindBot is the part that does the reading. It fetches robots.txt, the sitemap, and then pages, and it keeps the title, the description, the headings, the links and the visible text of each one.
It reads two kinds of site: sites whose owners have connected them to WhoTheyFind, and sites we audit once to show the owner what we would find. Nothing it reads is used to train a model.
How fast it goes
Two seconds between requests, and at most two requests in flight at a time. That is a rate across the whole crawl, not per worker, so your server sees at most one new request every two seconds from us however many pages we have left to read.
If your robots.txt states a Crawl-delay, we slow down to it, up to 30 seconds between requests. If your server answers 429 Too Many Requests or a 503 with a Retry-After, we slow down again, to the same ceiling. Past 30 seconds a crawl stops being a crawl, so we read what we can at that rate and stop rather than wait longer.
If your server keeps refusing us, we stop for the rest of that run rather than keep knocking. We read a site about once a week, so the next attempt is a week away.
How to block it
We read robots.txt before anything else and we obey it. To keep us out of the whole site, add this:
User-agent: WhoTheyFindBot
Disallow: /To keep us out of one section, name that path instead. To slow us down without blocking us, set a Crawl-delay in seconds:
User-agent: WhoTheyFindBot
Crawl-delay: 10Blocking us is not a decision about your search ranking. We are not a search engine and we do not index anything: refusing us changes nothing except that the owner of this site cannot use WhoTheyFind on it.
Reaching us
robots.txt is the fastest control and it needs nothing from us. If you want to talk to a person instead, leave us your address and say which domain you are asking about.