// publisher monetization infrastructure
AI crawlers scrape 1,700–73,000 times per referral while sending zero traffic back. Publishers are blocking AI bots, but blocking isn't a business model. Cloudflare's Pay-Per-Crawl changed everything.
The implementation guide for publishers transitioning from robots.txt blocking to monetization infrastructure. Technical setup, pricing strategy, and deal negotiation tactics.
73,000x
Scrapes per referral (Anthropic)
75%
Publishers blocking CCBot
$250M
News Corp-OpenAI deal
$60M/yr
Reddit-Google licensing
From technical setup to multi-million dollar deal structures.
RSL protocol setup, llms.txt specification, Cloudflare Pay-Per-Crawl configuration, and crawler analytics.
Per-crawl vs. per-inference vs. flat licensing models. Industry benchmarks and content valuation frameworks.
What AP, Reddit, News Corp, and Financial Times actually negotiated. Contract clause breakdowns.
Training data vs. retrieval data valuation. Why crawl ratios differ 5,000x between AI companies.
Step-by-step guide to configuring Cloudflare Pay-Per-Crawl. Pricing tiers, Stripe billing, enforcement settings.
Read guide → ImplementationComprehensive directory of AI crawlers. User-agent strings, crawl behaviors, robots.txt blocking instructions.
Read guide → Deal TeardownDeal structure, property valuations, and lessons for publishers from the benchmark AI content deal.
Read guide → Pricing StrategyIndustry benchmarks for per-crawl pricing, content uniqueness scoring, and common pricing mistakes.
Read guide →Victor Romo builds AI-ready search infrastructure for publishers and businesses. Crawler monetization, licensing setup, technical implementation.