About MetricScoutBot

MetricScoutBot is the crawler behind MetricScout. It only visits a website when a MetricScout user runs an assessment of that site — an agency or consultant analysing a site they are authorised to analyse. It is not a search engine and it does not crawl the web at large.

How it identifies itself

Every request carries this user agent:

Mozilla/5.0 (compatible; MetricScoutBot/1.0; +https://metricscout.io/bot)

What it does

• Fetches HTML pages only, following links within the site, up to the plan’s page limit
• Obeys robots.txt, including crawl-delay (capped so a crawl can finish)
• Waits at least half a second between requests and runs a single crawl at a time per site
• Stops at a fixed time budget and reports honestly that it did not cover everything
• If a page refuses the crawler (401, 403, 429) it waits two seconds and asks once more as itself, in case the refusal was only rate limiting
• If it is still refused, it re-checks once with a browser-like user agent to tell bot protection apart from a genuine fault. A refused page is never reported as broken — only as one that could not be checked
• Does not execute JavaScript, submit forms, or fetch images, video or downloads

How to block it

Add this to your robots.txt:

User-agent: MetricScoutBot
Disallow: /

MetricScoutBot checks robots.txt before every crawl and will stop at the next assessment. Bear in mind that the crawl was requested by someone analysing your site on your behalf — blocking the bot will also block their assessment.

Data and contact

What we collect and how long we keep it is set out in our privacy policy and data retention policy.