Website Crawler

Crawl a website and read the collected pages as clean Markdown for research, notes or a content archive.

  • 21 settings
  • REST API + MCP
  • Billed per run
  • Checked

1Your request

Use the AI/ML API MCP Website Crawler tool. Crawl a website and read the collected pages as clean readable text for research, notes or a content archive.

2What the tool takes21 settings · 1 required

URLPdfZdrCountryMax age msMax depthMax pagesURL regexWait for msStop after msInclude linksInclude framesInclude imagesExclude URL regexExclude selectorsFollow subdomainsInclude selectorsSettle animationsSource preferenceUse main content onlyShorten base64images
Source: Settings come from the tool's live input schema, checked on 17 Nov 2026. Your assistant fills them from your request.

What is Website Crawler?

Crawl a website and read the collected pages as clean readable text for research, notes or a content archive.

Settings Website Crawler takes

You describe the task in plain words and your AI assistant fills these in. In code you pass them yourself.

SettingWhat it controlsTip
URLURLRequired
PdfPDF parsing controls.Optional
ZdrZdrOptional · one of enabled, disabled
CountryCountryOptional · up to 2 characters
Max age msReuse a cached result younger than this many milliseconds. Default 86400000 (1 day), max 2592000000 (30 days). Set 0 to always fetch fresh.Optional
Max depthMaximum link depth from the starting URL (0 = only the starting page). Unlimited when omitted.Optional
Max pagesMaximum number of pages to crawl. Default 100, hard cap 500. This is also what sizes the admission hold, one credit per page.Optional · 1 to 500
URL regexOnly URLs matching this pattern are followed and scraped.Optional
Wait for msExtra browser wait in milliseconds after page load before the content is captured (0-30000). Useful for JavaScript-heavy pages.Optional · 0 to 30000
Stop after msSoft time budget for the whole crawl in milliseconds (10000-110000, default 80000). When exceeded, the pages collected so far are returned instead of continuing.Optional · 10000 to 110000
Include linksPreserve hyperlinks in the Markdown output. Default true.Optional
Include framesRender iframe contents into the output. Default false.Optional
Include imagesInclude image references in the Markdown output. Default false.Optional
Exclude URL regexURLs matching this pattern are never followed or scraped, even when they match urlRegex. The starting URL is always scraped.Optional · up to 256 characters
Exclude selectorsCSS selectors to remove before conversion. Applied after includeSelectors; exclusion wins on a double match.Optional
Follow subdomainsFollow links on subdomains of the starting URL's domain (e.g. docs.example.com from example.com). www and apex are always equivalent. Default false.Optional
Include selectorsCSS selectors. When provided, only matching subtrees are kept before each page is converted to Markdown.Optional
Settle animationsWait briefly for animations to settle before capturing each page. Default false.Optional
Source preferenceMarkdown sources in order of preference: 'html-conversion' converts the page HTML, 'markdown-direct' uses Markdown the site publishes. Default both, in that order.Optional
Use main content onlyKeep only the page's main content, dropping headers, footers, sidebars, and navigation when detectable. Default false.Optional
Shorten base64imagesTruncate inline base64 image payloads to keep the output small. Default true.Optional

How to use Website Crawler with your AI assistant

  1. Connect AI/ML API

    Click Try it now, pick your client and add the MCP server. You sign in with your AI/ML API account once.

  2. Describe the task

    Say what you need in one sentence, including the url.

  3. Get the result

    Your assistant runs Website Crawler and shows what came back.

  4. Keep going in the chat

    Ask it to filter, compare or summarize the result, or to save it as a table.

Ways to use it

  • Collect a website’s pages for a content review.

Website Crawler in your code

In your AI assistant

MCP · sign in once with your AI/ML API account

MCP setup guide ↗

In your code

Backend, script or scheduled job

REST · your AI/ML API key, JSON records back

The same tool runs over REST with your AI/ML API key, so a backend, a script or a scheduled job can call Website Crawler directly. Start with the Toolbox REST quickstart.

Pricing

Pay as you goBilled per run from your AI/ML API balance

  • No subscription
  • No seat fee
  • One balance for every tool

Each run is billed from your AI/ML API balance, with no subscription or seat fee. Check the current price in your dashboard before large runs.

See pricing

FAQ

Do I need a separate account for Website Crawler?

No. It runs on your AI/ML API account and balance, from your AI assistant over MCP or from your code over REST.

Which AI assistants can use it?

Claude, ChatGPT, Cursor, Hermes, OpenClaw and other MCP clients. Add the AI/ML API MCP server once with the MCP quickstart.

Run Website Crawler from your assistant

Connect once, then ask for it in plain words.