The robots.txt honor system collapsed sometime around late 2024. OpenAI, Anthropic, and ByteDance had already scraped billions of pages by the time publishers added disallow rules. The damage was done. Training data was collected. The models were built.
Blocking AI crawlers today stops future scraping but recovers nothing from the past. And blocking generates zero revenue.
Cloudflare launched Pay-Per-Crawl in July 2025 as a third option. Not blocking. Not allowing freely. Charging.
The concept is simple: AI companies that want your content pay for access. Those that refuse get blocked or throttled. Cloudflare handles detection, billing via Stripe, and enforcement. You set the prices.
Publishers running Pay-Per-Crawl report $500 to $5,000 monthly from AI crawler licensing. Not transformational revenue for large news organizations. Meaningful new income for trade publications, technical documentation sites, and niche content producers.
What Cloudflare Pay-Per-Crawl Is
The Collapse of the robots.txt Honor System
robots.txt was never legally binding. It was a social contract. Crawlers agreed to check the file and respect directives because the alternative was web chaos.
AI companies broke that contract at scale.
By 2025, research showed 75% of major publishers had added AI crawler blocks to robots.txt. The blocks came too late. GPT-4 was already trained. Claude was already trained. The archives were already in the models.
AI Companies Willing to Pay vs. Those That Ignore Terms
Compliant crawlers:
- GPTBot (OpenAI): Checks robots.txt, responds to rate limits, negotiates volume discounts
- ClaudeBot (Anthropic): Honors robots.txt, pays published rates without negotiation
- Google-Extended (Google Gemini): Separate from Googlebot search indexing, complies with payment requirements
Non-compliant crawlers:
- Bytespider (ByteDance): Ignores robots.txt, ignores RSL, doesn't respond to licensing outreach
- CCBot (Common Crawl): Technically compliant but feeds training data to multiple AI companies
Pay-Per-Crawl works for compliant crawlers. Non-compliant ones get blocked.
Prerequisites: What You Need Before Starting
Cloudflare Account Requirements
Pay-Per-Crawl requires Cloudflare Pro plan or higher ($20/month minimum).
Before starting configuration, verify:
- Domain is active on Cloudflare (DNS proxied through CF nameservers)
- SSL certificate is active
- You have admin access to the Cloudflare dashboard
Crawler Activity Baseline
Pull 90 days of server logs. Filter for AI crawler user-agents:
GPTBot
ClaudeBot
Bytespider
Google-Extended
CCBot
Applebot-Extended
Meta-ExternalAgent
PerplexityBot
Calculate for each crawler:
- Total requests per day
- Requests per page (which content they target)
- Crawl depth (surface pages vs. deep archives)
This baseline tells you which AI companies value your content most and what your total addressable market looks like.
Step 1: Enable Cloudflare AI Crawler Detection
Navigate: Security > Bot Management > Configure Bot Management
The Bot Management panel shows:
- Detected bot traffic breakdown
- Bot traffic over time
- Top user-agents by request volume
For Pay-Per-Crawl specifically, navigate to:
Security > AI Crawlers > Monetization Settings
This panel lists known AI crawlers with toggles:
- Allow: Crawler accesses freely (not recommended)
- License: Crawler must pay configured rate
- Block: Crawler receives 403 response
- Throttle: Crawler allowed at reduced rate
Set GPTBot, ClaudeBot, and Google-Extended to License. Set Bytespider to Block.
Step 2: Set Per-Crawl Pricing Tiers
Industry Benchmark Pricing
| Content Type | Low Range | Typical | Premium |
|---|---|---|---|
| News (general) | $0.002 | $0.003-$0.005 | $0.008 |
| News (breaking/real-time) | $0.008 | $0.010-$0.015 | $0.020 |
| B2B/Trade publications | $0.006 | $0.008-$0.012 | $0.015 |
| Technical documentation | $0.010 | $0.015-$0.020 | $0.030 |
| Research/proprietary data | $0.015 | $0.020-$0.030 | $0.050 |
These are per-crawl rates. Don't race to the bottom. AI companies pay News Corp $50 million annually. They can afford your $0.01 per crawl.
Volume Discount Structures
Example volume discount structure:
- 0-10,000 crawls/month: Standard rate
- 10,001-50,000 crawls/month: 15% discount
- 50,001-100,000 crawls/month: 25% discount
- 100,000+ crawls/month: Negotiate directly
Configure in Cloudflare under AI Crawlers > Pricing > Volume Discounts
Dynamic Pricing Based on Content
Cloudflare supports path-based pricing:
AI Crawlers > Pricing > Path Rules
/breaking/*: $0.015
/news/*: $0.005
/research/*: $0.020
/archive/*: $0.002
Step 3: Configure Payment and Enforcement
Connecting Stripe for Automated Billing
Navigate: AI Crawlers > Billing > Connect Stripe
Follow the OAuth flow to authorize Cloudflare to create charges on your behalf.
Stripe handles:
- Credit card processing for AI companies
- Invoicing for enterprise accounts
- Payout to your bank account (standard 2-day rolling)
- Tax documentation
Cloudflare takes a processing fee (currently 5% of AI licensing revenue).
Setting Grace Periods
Recommended settings:
- Grace period: 24 hours (allows time for payment setup)
- Rate limit during grace: 100 requests
- Post-grace behavior: Full billing or block
Configure under: AI Crawlers > Billing > Grace Settings
Blocking vs. Throttling Non-Paying Crawlers
Block (403 response): Clearest enforcement. No content delivered.
Throttle (rate-limited access): Maintains some access as negotiation leverage.
Challenge (CAPTCHA/verification): Blocks crude scraping scripts.
Most publishers start with Block for known non-compliant crawlers and Challenge for unknown crawlers.
Step 4: Deploy and Monitor
First-Week Analytics
After deployment, monitor daily for the first week:
In Cloudflare dashboard:
- AI crawler requests by user-agent
- Challenge completion rates
- Billing events
- Block events
In Stripe dashboard:
- Charges by AI company
- Payment success/failure rates
Red flags to investigate:
- Crawler requests dropping to zero
- High challenge failure rate
- No billing events despite crawler activity
Expect a 1-2 week ramp-up period. Revenue by week four should reflect steady-state.
Adjusting Pricing Based on Response
After 30 days, analyze crawler behavior:
If compliant crawlers are paying without complaint: Your pricing is at market or below. Consider testing higher rates on premium content.
If compliant crawlers stopped crawling: Your pricing may be above market. Test lower rates or add volume discounts.
If non-compliant crawlers are bypassing blocks: User-agent spoofing likely occurring. Add IP-based blocking.
Troubleshooting Common Issues
Crawlers Bypassing Detection
Some crawlers lie about their identity. Detection strategies:
- IP range blocking: AI companies operate from known IP ranges
- Behavioral analysis: AI crawlers request pages faster than humans
- TLS fingerprinting: Available on Business and Enterprise plans
Conflicts with Existing robots.txt Rules
If your robots.txt blocks AI crawlers, Pay-Per-Crawl sees no traffic to monetize.
Resolution options:
Option 1: Remove robots.txt blocks, rely on Cloudflare for access control.
Option 2: Keep blocks for non-compliant crawlers (Bytespider), remove blocks for compliant crawlers (GPTBot, ClaudeBot).
Most publishers choose Option 1 or 2. Relying on Cloudflare for enforcement is simpler than managing parallel systems.
Pay-Per-Crawl represents a shift in how publishers think about AI company relationships. Blocking is protection without compensation. Allowing freely is contribution without compensation. Pay-Per-Crawl is commerce.
The setup takes four to six hours. The monitoring takes an hour weekly for the first month. The revenue starts in 30 to 60 days.
For related implementation guides, see RSL Protocol Implementation and AI Crawler Directory.