Botify

Botify should be evaluated as three progressively riskier systems. Analytics reads crawler, server-log and search/traffic data. Intelligence prioritizes and alerts. Activation can modify delivered pages, submit discovery signals or route bot traffic through a rendered cache.

> Distinctive strength: Botify can connect crawl and log evidence to prioritization and, through Activation, to changes in what search engines receive. > > Where it stops being an advantage: Activation changes the risk class from analytics to production delivery and requires bounded rollout and rollback evidence.

The first layer can improve evidence. The third can change what engines receive. A procurement team that treats both as “an SEO dashboard” will under-test the consequential part.

BenPicks has not crawled a production site with Botify, ingested logs or deployed PageWorkers/SpeedWorkers. We do not claim traffic, indexation or revenue gains. This review focuses on the operating contract, retention and failure modes a buyer can verify.

The decision in 60 seconds

What does Botify actually include?

Botify documents Essential, Accelerate, Growth, Pro and Enterprise plans. Accelerate exposes many features only in view mode. The product spans:

• SiteCrawler and HTML extraction;
• ad-hoc and recurring crawls;
• LogAnalyzer from customer server logs;
• RealKeywords/Search Console overlays;
• engagement/revenue analytics integrations;
• AI visibility;
• ActionBoard, alerts and daily JS monitoring;
• PageWorkers content/link changes;
• SpeedWorkers rendered bot delivery;
• SmartIndex sitemap/IndexNow/Bing submission;
• SmartContent and optional SmartLink.

Not every module is included on every plan. SpeedWorkers is an option on Pro/Enterprise. PageWorkers is included or optional/view-only depending on tier. Ask for a row-by-row entitlement exhibit, not a demo recap.

How does Botify Analytics improve the evidence?

SiteCrawler inventories URLs and reports technical/search properties. LogAnalyzer imports web-server logs daily so a team can see which bots actually requested which URLs. RealKeywords adds Search Console, while analytics providers contribute visits/conversions.

These sources answer different questions:

• crawl: what Botify could discover and render;
• logs: what Googlebot/Bingbot requested;
• GSC: impressions/clicks/queries, with reported retrieval delay;
• analytics: user/conversion outcomes;
• AI visibility: sampled mentions/citations/sentiment.

Do not collapse them into one score. Reconcile by URL template and date window. A page can be crawlable but ignored, crawled but not indexed, indexed but invisible for relevant queries, or visible without conversion.

Current documentation says GSC is retrieved daily with a 2–3 day delay and imports 16 months when connected. Web logs are ingested daily. Define expected freshness before building alerts.

How long does Botify retain evidence?

The plan matrix is unusually specific:

Crawl retention also limits which full crawls remain: lower tiers preserve the last five, Pro ten, Enterprise twenty, plus the largest monthly crawl afterward.

This matters during incidents. If raw HTML is only retained one month, preserve relevant source/render snapshots externally before the window closes. Contract whether exports/API provide enough detail for your audit and continuity needs.

How do crawls consume time and allowance?

Botify lets users change crawl speed, depth and URL limits while running. A crawl can pause/resume. Stopping is final: analysis runs on collected URLs, consumed pages remain debited and the scheduled recurring crawl may need rescheduling.

The vendor says analyses can take minutes for thousands of pages or days for millions, based on render complexity and server performance. Live statistics expose crawl speed, bandwidth, response time, status codes and discovered queue.

Before a large crawl:

1. define allowed hosts/path/query patterns;
2. cap pages and concurrency;
3. protect faceted/infinite URL spaces;
4. coordinate origin capacity;
5. test JavaScript rendering and authentication;
6. preserve robots and noindex boundaries;
7. define stop/pause authority.

A crawler that discovers millions of duplicate URLs can become an origin incident and a quota expense.

What is PageWorkers?

PageWorkers loads a Botify JavaScript file containing scoped optimizations and applies them on top of the original page. Execution logs return to Botify. The source application is not rewritten, so a change can be paused or deactivated and the original page remains.

That reversibility is useful for validated fixes waiting on engineering. It is not a substitute for durable source remediation. If Botify's CDN/API is unavailable, the source page loads but optimizations do not execute. A critical canonical, robots rule or legal correction should not exist only in an optional runtime overlay.

Test every change type for:

• URL scope and exclusions;
• source versus rendered DOM;
• bot and human parity;
• conflicting A/B/personalization scripts;
• Content Security Policy;
• Core Web Vitals and errors;
• pause/outage behaviour;
• audit identity and rollback time.

For SPAs, Botify says direct landing requests receive optimizations while client-side navigation does not reload the snippet. Engines usually request discovered URLs independently, but humans and automated QA can see a different transition path. Test both.

What is SpeedWorkers?

SpeedWorkers pre-renders and caches page mirrors for search/AI crawlers. In a recommended CloudFront design, a viewer-request function identifies eligible GET/HEAD HTML bot traffic, adds routing headers and directs it to a SpeedWorkers origin group. Humans stay on the original origin.

On a cache hit, Botify returns rendered HTML. On a miss or selected error, CloudFront falls back to the website origin. Static assets/API/non-GET requests should bypass this path.

This is infrastructure. It requires security, CDN and SEO owners—not only an SEO login.

What can go wrong in the SpeedWorkers cache/failover path?

The documentation surfaces concrete failure modes:

• a SpeedWorkers cache miss uses HTTP 403 as a failover signal;
• CloudFront error caching must be disabled, or a cached 403 can freeze failure rather than reaching origin;
• bot/human cache keys must be partitioned, or bot HTML could leak to users;
• headers must preserve original host, URL and user agent;
• only eligible HTML GET/HEAD should invoke the route to avoid cost and method failures;
• bot detection must resist spoofing and unexpected crawlers;
• the fallback origin must remain healthy and tested.

Use Botify's validation-only user agent before enabling real bots. Compare status, canonical, robots, hreflang, schema, content, links and legal text across source, browser render and cached bot render.

How do PageWorkers and SpeedWorkers combine?

When SpeedWorkers serves a cached page, it applies current PageWorkers optimizations before returning HTML. Botify says updates usually become servable within 30 seconds–5 minutes without refreshing the underlying page cache.

That separates base-page freshness from overlay freshness. During an incident, identify whether the wrong output came from origin, cached render, PageWorkers rules or routing. Preserve headers and snapshots from all paths.

A non-JavaScript bot on a SpeedWorkers miss falls back to unrendered origin with a PageWorkers snippet it cannot execute, so it may not receive the optimization. Measure delivery efficiency and cache inventory rather than assuming uniform deployment.

Does disabling SpeedWorkers stop everything?

No. Botify says disabling delivery stops serving inventory to search engines, while fetch/render/cache processes remain active. Individual inventory sources have separate controls.

Likewise, force-refresh is not free capacity. Requests enter a priority queue and displace other URLs, potentially making later pages expire. Refresh credits depend on page load time.

An incident runbook must distinguish:

• disable delivery;
• stop/limit fetch and cache sources;
• remove CDN routing;
• fall back to origin;
• purge or retain Botify inventory;
• pause PageWorkers rules;
• implement fixes in source.

Do not discover those differences during an indexing incident.

How should pricing be evaluated?

Botify does not expose a stable self-serve price or public trial in the inspected sources. Require a written quote including:

• sites/domains and URL/crawl allowances;
• crawl/render frequency and ad-hoc capacity;
• log volume/ingestion and retention;
• users, roles, exports and API;
• AI visibility prompts/models/regions/frequency;
• PageWorkers rules/execution;
• SpeedWorkers inventory, refresh credits, bots and CDN onboarding;
• support/SLA, professional services and overages;
• renewal uplift, data export and termination assistance.

The cost model must include internal engineering/CDN/security effort and exit—not just licence.

How strong are security and privacy disclosures?

Botify's public privacy page states SSL transport, access limited to employees who need personal data and reasonable controls. It gives personal-data retention: active service accounts, deletion after three uninterrupted inactive years, prospecting up to three years and cookies up to thirteen months.

That is not a complete product-security dossier for server logs, raw HTML, analytics, revenue and activation credentials. Request current assurance reports, DPA, subprocessors, hosting regions, encryption at rest, key/access controls, SSO/SCIM, audit logs, incident notice and deletion/export evidence.

Server logs may contain IPs, URLs, query strings and identifiers. Minimize before upload and confirm which fields Botify requires.

Who should shortlist Botify?

Shortlist it when a large/complex site needs crawl + real bot logs + search/traffic evidence at URL scale, and when the organisation can govern enterprise integrations. Consider Activation when measured source fixes are blocked and the team can safely own a runtime/CDN layer.

Look elsewhere when the site is small, pricing transparency is required before a demo, a standalone content editor is the actual need, or security/CDN teams cannot support bot-specific delivery.

Continue with the AI SEO software directory, the AI SEO content operations guide and the canonical software comparison workspace.

A product-specific pilot

1. Select representative templates and preserve source/render/log/GSC baselines.
2. Run a bounded crawl with caps and inspect discovered traps, rendering and reports.
3. Reconcile crawler findings with actual bot logs and GSC.
4. Score alerts/recommendations for actionable precision.
5. Deploy one reversible PageWorkers rule; test scope, parity, outage and pause.
6. In a CDN test environment, route only Botify validation traffic to SpeedWorkers.
7. Simulate hit, miss/403, 429, timeout and outage; prove origin failover.
8. Verify bot/human cache separation and all SEO/legal fields.
9. Test rule update timing and force-refresh queue impact.
10. Export data/rules and rehearse complete teardown.

Define failure thresholds before the demo. No activation module should advance because its analytics sibling is useful.

Main unknowns before purchase

• price, overage and self-serve evaluation;
• SmartContent/SmartLink workflow, approval and quotas;
• AI visibility prompt/model sampling contract;
• complete role/audit/API/SLA/security scope;
• real origin/CDN cost and delivery efficiency;
• teardown assistance and exported data/rule completeness.

Official evidence boundary

Twenty-two official Botify platform, knowledge-base, routing and privacy sources were checked on 28 August 2026. No hands-on crawl, activation, indexation or traffic result is claimed. The profile retains `verified_essentials`; price/evaluation and under-documented workflows remain unknown.