APEspider Bot

What it is

APEspider Bot is the crawler behind APEspider, the SEO audit tool. It visits web pages to analyze their technical SEO (titles, links, response codes, sitemap, robots.txt and so on) and generate reports.

APEspider Bot only crawls sites that their owner (or their SEO consultant) has expressly registered in APEspider in order to audit them. It does not crawl the internet at large and it does not index third-party content.

How it identifies itself

Every request carries this user agent:

APEspider/1.0 (+https://apespider.com/bot)

How it behaves

• It respects robots.txt: it does not visit paths blocked by the site's rules.
• Moderate pace: the crawl is rate-limited so it does not load your server.
• Read only: it makes GET requests; it does not submit forms or change anything.
• Bounded: each audit crawls a limited number of pages according to the user's plan.

Allowing it through your firewall or WAF

If you use an application firewall (Wordfence, Cloudflare, Sucuri and the like) and you want the audit to work, add a rule allowing the APEspider user agent. On most WAFs an entry in the allowed-bots list (allowlist) matching by user agent is enough.

If you would rather we did not visit your site, block it in robots.txt and the bot will respect it:

User-agent: APEspider
Disallow: /

Seeing traffic you did not expect?

If APEspider Bot is visiting your site and you do not know why, the most likely explanation is that someone connected to your site (your SEO consultant, your agency or a colleague) registered it for auditing. If you think that is a mistake, or you want us to stop crawling it, write to hello@apespider.com with your domain and we will sort it out.