Bot information

The Sitehistory.fyi crawler

We are a snapshot crawler that loads the pages a site owner has listed, once per schedule (usually once a day), like a desktop browser and like a phone.

Verification is required before any scheduled job run.

We only visit domains whose owner has verified them with us, through Google Search Console, a confirmed company email address, or a DNS TXT record. See how verification works.

What we do

How to recognise us

SignalValue
User-agent tokenSitehistory.fyi/0.1 (+https://www.sitehistory.fyi/bot)

A normal Chrome user agent ending in this token.

robots.txt tokenSiteHistory

We follow the rules in robots.txt for User-agent: SiteHistory (robots.txt product tokens cannot contain a dot, so this is the product name without .fyi; the case does not matter, and User-agent: Sitehistory.fyi is honoured too). Rules written for our former name, User-agent: SnapshotService, are still honoured as an alias; if both are present, the groups are merged.

Static IP addresses
  • 5.75.246.54 (IPv4, Hetzner, Germany): the job run server. Every job run, robots.txt fetch and access check leaves from this address, unless your property is set up with the second address below.
  • 91.99.153.175 (IPv4, Hetzner, Falkenstein, Germany): our first job run server, now a relay. Properties that were allowlisted on this address before the move (for example because a bot protection only knows it) are fetched through it, so the site still sees this address and nothing else. It is used for those properties only.
We do not use IPv6 for job runs. If you allowlist us, add both addresses; a property that uses the relay is told so in the app. See the allowlisting page.
Optional secret headerX-Snapshot-Token: <your project token>

How to allow us

The safest rule is always our IP addresses and the secret header together. Step by step instructions per protection vendor are on the allowlisting page.

How to stop us

Googlebot profiles: what they are not

Our Googlebot profiles are not Googlebot.

If a site owner switches them on, they send Google's published Googlebot user-agent strings from our IP address. They show what the site's own code sends to a request with a Googlebot user agent (dynamic rendering, prerender services, user-agent switches, accidental cloaking). They cannot show what real Googlebot gets where the site or its CDN verifies Googlebot by IP or reverse DNS. Such requests may also trip "fake Googlebot" protections even on an allowlisted IP. Every such job run carries this note.

The wording above is the note stored with every Googlebot-profile job run, so you can quote it when you read the history.

Abuse contact

Report abuse or ask us to stop visiting your site: abuse@sitehistory.fyi (our legal details are in the imprint).