Everything you need to know about integrating Cloudflare's Pay Per Crawl into your AI applications. From basic concepts to advanced optimization techniques.
Choose your learning path based on your experience level
Learn the basics of Pay Per Crawl and how to integrate it into your AI applications.
Step-by-step guide to integrate our API into your application.
Understand the data formats and schemas returned by our crawlers.
Set up real-time notifications for crawl completions and updates.
Optimize your crawling strategy and ensure compliance.
Deep dive into advanced features and optimization techniques.
Sign up for an account and generate your API key from the dashboard.
curl -X POST https://aipaypercrawl.com/api/auth/register \
-H "Content-Type: application/json" \
-d '{"email": "you@example.com", "password": "secure_password"}'Request a quote for the URLs you want to crawl.
curl -X POST https://aipaypercrawl.com/api/quote \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"urls": ["https://example.com/page1", "https://example.com/page2"]}'Accept the quote and execute the crawl request.
curl -X POST https://aipaypercrawl.com/api/crawl \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"quoteId": "quote_abc123", "accept": true}'Pay Per Crawl is a new standard that allows AI companies to pay publishers directly for crawling their content. It ensures fair compensation while enabling AI training on high-quality data.
Publishers set their own per-request prices. You only pay for successful crawls, with transparent pricing shown upfront. Bulk discounts are available for high-volume usage.
We support JSON, HTML, and structured data extraction. All responses include metadata like timestamps, content-type, and extraction quality scores.
Most requests complete in under 50ms thanks to Cloudflare's global edge network. Complex JavaScript-rendered pages may take up to 2-3 seconds.
Only websites that have enabled Pay Per Crawl through Cloudflare are available. Our directory shows all available domains with their pricing.
API limits vary by plan. Free tier allows 1,000 requests/day, while enterprise plans offer unlimited requests with custom SLAs.
Still have questions?
Join thousands of AI companies using Pay Per Crawl for their data needs.