// publisher monetization infrastructure
AI crawlers consume publisher content at scale and send almost nothing back as referral traffic. Publishers responded by blocking. Blocking expresses the objection and collects none of the value. Pay-per-crawl infrastructure exists now, and it turns the same bot traffic into a line item.
This site is the implementation guide for the transition: technical setup, pricing strategy, and deal negotiation, written for the publisher doing it rather than the analyst describing it.
13
Live implementation guides
3
Licensing models compared
2
Planning tools, assumptions stated
1
Vendor-documented crawler monitor
From technical setup to multi-million dollar deal structures.
RSL protocol setup, llms.txt specification, Cloudflare Pay-Per-Crawl configuration, and crawler analytics.
Per-crawl vs. per-inference vs. flat licensing models. Pricing models compared, with the assumptions written down.
Public licensing deals broken down clause by clause, from the reporting that disclosed them.
Training data vs. retrieval data valuation. Why crawl-to-referral ratios differ by orders of magnitude between AI companies.
Step-by-step guide to configuring Cloudflare Pay-Per-Crawl. Pricing tiers, Stripe billing, enforcement settings.
Read guide → ImplementationComprehensive directory of AI crawlers. User-agent strings, crawl behaviors, robots.txt blocking instructions.
Read guide → Deal TeardownDeal structure, property valuations, and lessons for publishers from the benchmark AI content deal.
Read guide → Pricing StrategyIndustry benchmarks for per-crawl pricing, content uniqueness scoring, and common pricing mistakes.
Read guide →Victor Valentine Romo and Scale With Search build AI-ready search infrastructure for publishers: crawler monetization, licensing setup, and the technical implementation this site documents. The same system runs the eight-site network this property belongs to, and it published its first article with no human in the loop on July 30, 2026, receipts on file.