AgerBot
AgerBot is Ager's crawler. It collects headlines and metadata from sources for our link-first feed: we never host article full text and always link back to the publisher.
User-Agent
AgerBot's requests carry this User-Agent, so you can identify them in your logs:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36 Ager-Bot/1.0 (+https://agerculture.com)
What we respect
Before each fetch we read and record robots.txt, /.well-known/tdmrep.json (text & data mining opt-out), and /.well-known/ai.txt. We honour exclusion directives and the TDM opt-out.
How to exclude AgerBot
To prevent collection, add a rule for "Ager-Bot" in your robots.txt or declare a TDM opt-out in tdmrep.json. We apply changes on the next fetch.
Questions? bot@agerculture.com