Crawler information

eewabot

eewabot is the automated web crawler operated by eewa.io. This page explains what it does, how it identifies itself, how it honours robots.txt, and how to reach a human if you have questions about its traffic on your site.

What the crawler does

eewabot visits publicly accessible web pages and reads their content. It requests pages the way an ordinary browser does — over HTTP(S), following links it finds — and stores the responses for processing on our side.

• It only requests content that is publicly reachable without logging in.
• It does not attempt to bypass paywalls, authentication, CAPTCHAs, or access controls of any kind.
• It does not submit forms, create accounts, post content, or perform any write operations.
• It does not try to reach pages that are blocked for it (see below).

How to identify it

Any traffic claiming to be eewabot from an unrelated source is not ours. If you see requests using our user agent that you believe are being spoofed, please tell us — we want to know.

How it respects robots.txt

eewabot follows the Robots Exclusion Protocol (RFC 9309). In practice:

• robots.txt is fetched first. Before crawling a host, the bot requests /robots.txt from that host and parses it. Nothing else is requested until it has.
• Your rules are applied. It obeys Disallow and Allow rules in the group matching eewabot, and falls back to the User-agent: * group when there is no group for us specifically.
• Crawl-delay is honoured. If your robots.txt sets a Crawl-delay, the bot waits at least that long between requests to your host.
• A fetch failure means caution. If robots.txt returns a server error or cannot be fetched, the bot treats the host as disallowed and backs off rather than crawling anyway.
• Rules are re-checked. The parsed robots.txt is cached for a short period and then re-fetched, so changes you publish take effect promptly.
• Page-level directives are respected too. It honours <meta name="robots"> and the X-Robots-Tag header, including noindex and nofollow.

Blocking the crawler

You are always free to block us. To stop eewabot from crawling your site entirely, add this to the robots.txt at the root of your domain:

User-agent: eewabot
Disallow: /

To allow most of your site but keep certain paths out:

User-agent: eewabot
Disallow: /private/
Disallow: /checkout/
Crawl-delay: 10

The change applies as soon as the bot re-reads your robots.txt. If you would rather not wait, or if you want us to stop immediately and permanently, email us and we will add your domain to our exclusion list by hand.

Being a good neighbour

The crawler is rate-limited per host and spreads its requests out over time so that it does not put meaningful load on a site. It backs off automatically when a server responds slowly or returns 429 or 5xx status codes, and it caches content to avoid re-requesting pages that have not changed.

If eewabot is nonetheless causing problems for your infrastructure, get in touch and we will throttle or stop it.

Contact us