Our crawler

User agent

GraidlabBot/1.0 (+https://graidlab.com/bot)

Requests

At most one page per second, per site

Pages

At most 1 (free scan) or 30 (full report)

robots.txt

Rules addressed to GraidlabBot are obeyed, crawl-delay included

Rendering

Headless Chromium, JavaScript executed

Forms

Never submitted. We only issue GET requests.

When somebody asks us to analyse a site, we read that site from the outside. We do not maintain a general-purpose index and we do not crawl sites nobody asked us about.

• It reads robots.txt before it reads anything else on the site, and does not fetch what that file disallows for GraidlabBot. If that includes the page it was asked about, the analysis stops and says why.
• A blanket rule aimed at every crawler at once (User-agent: *) does not stop it. We read a site only because its owner asked us to analyse that site, and a line written to keep uninvited scrapers out is not a refusal of an audit somebody ordered. Name GraidlabBot and we stop.
• It waits at least one second between requests to the same site, so a scan is slower than it could be and cheaper for you to serve.
• It follows redirects, re-checking at every hop that the destination is a public address.
• It renders pages with JavaScript, because that is what an AI crawler that can render will see — and comparing the rendered page with the raw HTML is itself one of the things we measure.
• It never logs in, never submits a form, never posts data, and never touches an address that resolves to a private network.

Add this to your robots.txt. We will stop reading your site the next time somebody asks us to.

Blocking us does not block the AI crawlers your buyers' assistants use. If your goal is to keep those out, block those instead — and know that doing so is exactly what makes a site invisible in answers.

User-agent: GraidlabBot
Disallow: /

We keep what each finding needs in order to show its evidence, for as long as the report is available — 14 days for a free report, 24 months for a paid one. When it expires, that is deleted. We do not sell it, republish it, or use it to train models.

If you see traffic claiming to be GraidlabBot/1.0 that is not behaving as described here, write to weare@graidlab.com with the log lines and we will look into it. Anyone can put a string in a user agent; we would like to know when somebody puts ours in.