About our crawler
If you have found this page in your server logs, this explains what the traffic is and what to do about it.
How it identifies itself
Every request we make carries this user agent, unchanged:
LegalSyndication/1.0 (+http://localhost/crawler.php; feed aggregator)
We do not imitate a browser. The agent names the service and links here precisely so that you can identify the traffic and decide what to allow.
What we fetch, and why
Two things, and only for firms that have asked to be part of the network:
- Your RSS feed, to see which articles are new.
- The article pages themselves, where the feed carries only a summary, so that the full piece can be published rather than a link to it.
We request nothing else. No logins, no forms, no files, no search pages. Articles
are published with a rel="canonical" tag pointing at your original,
so your site keeps the credit in search results.
How often
Rarely, and slowly. Requests to one site are spaced apart deliberately, and when a site declines a request we stop asking it rather than retrying down the list — a refusal is answered by backing off, not by trying harder.
If you would rather we did not
We read robots.txt before requesting anything and follow it, including
any Crawl-delay you set. Disallowing the agent above, or
*, is enough.
Blocking the user agent works too, and we will not work around it: we do not rotate agents, use residential proxies, or attempt to defeat a challenge. Or tell us and we will stop permanently — no arrangement with anyone at the firm overrides that.
If the firm asked for this and it is being blocked
This is the common case. Bot protection — Cloudflare, ModSecurity and similar — challenges us by default because we come from a cloud address, regardless of how we identify ourselves. Allowing the user agent above is usually enough.
Contact
[email protected] — a real address, read by a person.