WebCrawlerAPI for AI agents
Agents read WebCrawlerAPI at once. Changes and sends are held for a person's approval by default.
About WebCrawlerAPI
WebCrawlerAPI provides crawling, page scraping, AI web agents, and monitored website feeds through a unified API.
What an agent can do in WebCrawlerAPI
Every WebCrawlerAPI action is classed by the platform as a read, a change or a send. Reads run at once. Changes and sends are held for a person by default.
Runs at once (reads)
Looks things up in WebCrawlerAPI without asking first. 6 actions.
- Get Crawl JobGet a crawl job's status, configuration, per-page results, content URLs, errors, and observed costs. Respect recommended_pull_delay_ms when polling.
- Get FeedGet one feed's configuration, lifecycle status, and recent run history, including per-run crawl counts and cost. Canceled feeds remain directly readable while retained by the provider.
- Get Organization CostsReturn current spendable balance plus request count and USD usage for a date range. The connected API key is used for both organization routes.
- List Feed ChangesReturn one page of detected feed changes as structured JSON, with an opaque continuation cursor for older pages.
- List FeedsList active and paused feeds for the connected organization, newest first. Logically canceled feeds are omitted.
- Scrape PageScrape one web page synchronously and return requested content or structured extraction. This consumes metered credits; supplying a prompt adds $0.002 per request.
Held for approval by default (changes and sends)
Held for a person's approval by default. The approval is recorded and the action is sent once, never twice, with 10 seconds to undo. 4 actions.
- Create CrawlStart a metered asynchronous crawl over a site and return its job ID. The crawl may visit and charge for up to items_limit pages; use GET_CRAWL_JOB to monitor it.
- Create FeedCreate a recurring website-change feed. Creation immediately schedules an initial crawl that may consume metered credits, and future scheduled runs may also consume credits; delete the feed when it is
- Delete FeedPermanently cancel a feed so it cannot be resumed. The feed is removed from feed listings, but retained history may remain directly readable for 30 days.
- Set Feed StatePause an active feed's future scheduled runs or resume a paused feed. Pausing does not stop a run that has already started, and resuming schedules future crawls that may consume metered credits.
Changes wait in Needs you
Changes and sends in WebCrawlerAPI are held for a person's approval by default. The approval is recorded, and the action is sent once, never twice, with 10 seconds to undo.
WebCrawlerAPI questions
Can an agent change data in WebCrawlerAPI on its own?
Not by default. Changes and sends in WebCrawlerAPI are held for a person's approval. The approval is recorded, and the action is sent once, never twice, with 10 seconds to undo.
What can an agent read in WebCrawlerAPI?
This connector lists 6 read actions. Reads run at once, without waiting for approval.
How does WebCrawlerAPI connect?
It connects with API key. Connectors are built and maintained by third-party developers; availability is confirmed during scoping.