About AryehIntelligenceBot
AryehIntelligenceBot is the name our free tool, Aryeh Intelligence, uses when it reads a public website. It reads only the website someone asks it to read, and it follows the rules below.
The name it sends with every request
AryehIntelligenceBot/1.0 (+https://aryehalgorithms.com/bot)
What it reads
It reads the public pages of one website at a time: the home page, the pages linked from it, the sitemap, and the robots.txt file. It reads text, page titles, headings, contact details, links and structured data. It does not read anything behind a login.
Its limits
- Up to 60 pages and 90 seconds for each scan.
- No more than 2 MB for each page.
- Only web addresses that start with http or https, on the standard web ports.
- Only public addresses. It never reads private networks, and it checks this again at every redirect.
- It never logs in anywhere, fills in a form, or submits anything.
- It never reads LinkedIn.
robots.txt
It reads your robots.txt first and follows it. To keep it out, add these two lines to robots.txt: User-agent: AryehIntelligenceBot, then Disallow: /. It will not read your pages after that.
Why it reads your site
Someone entered your website address into Aryeh Intelligence, usually the owner or a person on your team. It uses what it reads to show them what it found and what a first plan could look like. We keep the profile links they confirm, not the pages we read.
Contact us
If it is reading something you would rather it did not, or you want us to stop reading your site, write to aryeh@aryehalgorithms.com and tell us the website address. We will reply and make the change.