What is Website Crawler?
Crawl a website and read the collected pages as clean readable text for research, notes or a content archive.
Settings Website Crawler takes
You describe the task in plain words and your AI assistant fills these in. In code you pass them yourself.
| Setting | What it controls | Tip |
|---|---|---|
| URL | URL | Required |
| PDF parsing controls. | Optional | |
| Zdr | Zdr | Optional · one of enabled, disabled |
| Country | Country | Optional · up to 2 characters |
| Max age ms | Reuse a cached result younger than this many milliseconds. Default 86400000 (1 day), max 2592000000 (30 days). Set 0 to always fetch fresh. | Optional |
| Max depth | Maximum link depth from the starting URL (0 = only the starting page). Unlimited when omitted. | Optional |
| Max pages | Maximum number of pages to crawl. Default 100, hard cap 500. This is also what sizes the admission hold, one credit per page. | Optional · 1 to 500 |
| URL regex | Only URLs matching this pattern are followed and scraped. | Optional |
| Wait for ms | Extra browser wait in milliseconds after page load before the content is captured (0-30000). Useful for JavaScript-heavy pages. | Optional · 0 to 30000 |
| Stop after ms | Soft time budget for the whole crawl in milliseconds (10000-110000, default 80000). When exceeded, the pages collected so far are returned instead of continuing. | Optional · 10000 to 110000 |
| Include links | Preserve hyperlinks in the Markdown output. Default true. | Optional |
| Include frames | Render iframe contents into the output. Default false. | Optional |
| Include images | Include image references in the Markdown output. Default false. | Optional |
| Exclude URL regex | URLs matching this pattern are never followed or scraped, even when they match urlRegex. The starting URL is always scraped. | Optional · up to 256 characters |
| Exclude selectors | CSS selectors to remove before conversion. Applied after includeSelectors; exclusion wins on a double match. | Optional |
| Follow subdomains | Follow links on subdomains of the starting URL's domain (e.g. docs.example.com from example.com). www and apex are always equivalent. Default false. | Optional |
| Include selectors | CSS selectors. When provided, only matching subtrees are kept before each page is converted to Markdown. | Optional |
| Settle animations | Wait briefly for animations to settle before capturing each page. Default false. | Optional |
| Source preference | Markdown sources in order of preference: 'html-conversion' converts the page HTML, 'markdown-direct' uses Markdown the site publishes. Default both, in that order. | Optional |
| Use main content only | Keep only the page's main content, dropping headers, footers, sidebars, and navigation when detectable. Default false. | Optional |
| Shorten base64images | Truncate inline base64 image payloads to keep the output small. Default true. | Optional |
How to use Website Crawler with your AI assistant
Connect AI/ML API
Click Try it now, pick your client and add the MCP server. You sign in with your AI/ML API account once.
Describe the task
Say what you need in one sentence, including the url.
Get the result
Your assistant runs Website Crawler and shows what came back.
Keep going in the chat
Ask it to filter, compare or summarize the result, or to save it as a table.
Ways to use it
- Collect a website’s pages for a content review.
Website Crawler in your code
In your code
REST · your AI/ML API key, JSON records back
The same tool runs over REST with your AI/ML API key, so a backend, a script or a scheduled job can call Website Crawler directly. Start with the Toolbox REST quickstart.
Pricing
Pay as you goBilled per run from your AI/ML API balance
- No subscription
- No seat fee
- One balance for every tool
Each run is billed from your AI/ML API balance, with no subscription or seat fee. Check the current price in your dashboard before large runs.
FAQ
Do I need a separate account for Website Crawler?
No. It runs on your AI/ML API account and balance, from your AI assistant over MCP or from your code over REST.
Which AI assistants can use it?
Claude, ChatGPT, Cursor, Hermes, OpenClaw and other MCP clients. Add the AI/ML API MCP server once with the MCP quickstart.
Run Website Crawler from your assistant
Connect once, then ask for it in plain words.