The Web Data API for
Deterministic AI Pipelines

Perplexity Computer and Exa are great for exploration — you ask a question, an AI decides which websites to read, and you get a synthesized answer. That's probabilistic: the AI picks the sources, the output varies run to run, and nothing lands in your database.

CrawlHQ is deterministic. You specify exactly which URLs to crawl. You define the exact JSON schema you want back. You get the same structured output every run. It goes straight to your database, your dashboard, or your LLM context — with a full audit trail showing which source URL produced which data point.

If you need to explore a topic, use Perplexity. If you need to run a competitor pricing check every morning at 6 AM and have the results in your Postgres database, use CrawlHQ.

It means: same URL + same schema = same structured output, every time.

With AI search tools, ask "what are Firecrawl's pricing plans?" twice and you might get different answers. One run it finds the pricing page, another run it reads a blog post about pricing. You can't build a production pipeline on that.

With CrawlHQ, you point at firecrawl.dev/pricing, define {plans: [{name, price, credits}]}, and every run returns exactly that schema populated with exactly that page's data. Your monitoring dashboard, your competitive intelligence feed, your daily report — all deterministic. Auditable. Trustworthy.

Every API call costs credits based on the module and complexity. Scrape and Read cost 1-2 credits per URL, Search costs 1 credit per query, Extract costs 5 credits per extraction (uses LLM), Breach and Darkweb cost 3 credits each, and Media transcription costs 5 credits. You're only charged on success — failed requests don't consume credits. Credits never expire.

Firecrawl and ScrapingBee are excellent at getting raw data — HTML and Markdown from any page. But they stop there. You still need to write the parsing logic, handle the schema transformation, build the storage pipeline, and manage retries.

CrawlHQ adds the intelligence layer: /v1/extract uses an LLM to apply your JSON schema to any webpage without writing parsing code. /v1/enrich turns a domain into verified emails. /v1/breach monitors credential exposure. /v1/search gives you real-time web results. It's the full pipeline, not just the fetch step.

Passive monitoring of publicly accessible dark web content (paste sites, forums, .onion directories) for threat intelligence purposes is legal in most jurisdictions. We don't facilitate any illegal activity — the /darkweb module is read-only intelligence gathering, the same function performed by SpyCloud, Recorded Future, and other enterprise security vendors. Consult your legal team for your specific jurisdiction.

Yes — CrawlHQ is built India-first. We accept UPI, NEFT, credit/debit cards in INR via Razorpay. International teams can pay in USD via Stripe. INR pricing is shown by default on our pricing page.

Your API calls will return a 402 error with a clear message. Nothing breaks silently. You can top up anytime from the dashboard with a one-time credit purchase, or upgrade your plan for a higher monthly allocation.

Yes. All plans support concurrent requests. Free tier is rate-limited to 2 req/sec. Starter is 10 req/sec. Growth is 50 req/sec. Scale is uncapped (contact us for dedicated infrastructure).

Not yet — and honestly, you probably don't need one. CrawlHQ is a simple HTTP API. A requests.post() in Python or fetch() in JavaScript is all you need. We may release official SDKs for Python and JavaScript in Q3 2026. Follow our changelog.