Skip to content
Surfilby Adam Hafez

SurfilBot

SurfilBot is the crawler behind Surfil store scans. It identifies itself on every request and follows your robots.txt.

Last updated 1 November 2026

How to recognize it

Every request carries this user agent:

SurfilBot/1.0 (+https://surfil.com/bot)

Why it visits

Surfil scans online stores for the technical problems that stop products appearing in Google Search and Merchant Center. A store owner starts a scan of their own store, and SurfilBot fetches that store's pages to check them. SurfilBot also visits a small sample of public store fronts to build aggregated benchmarks. Only aggregated results are kept.

How it behaves

  • It reads robots.txt first and does not fetch what the file disallows for SurfilBot.
  • It honours Crawl-delay and fetches at most two pages per second from one store.

How to block it

Add this to your robots.txt:

User-agent: SurfilBot
Disallow: /

To be removed from the public benchmark sample, or if SurfilBot is causing a problem, write tohello@surfil.com with your domain.