Crawlbase for AI agents
Agents read Crawlbase at once. Changes and sends are held for a person's approval by default.
About Crawlbase
Crawlbase provides APIs for crawling web pages, managing crawler queues, storing crawl results, and monitoring account usage.
What an agent can do in Crawlbase
Every Crawlbase action is classed by the platform as a read, a change or a send. Reads run at once. Changes and sends are held for a person by default.
Runs at once (reads)
Looks things up in Crawlbase without asking first. 6 actions.
- Generate User AgentsGenerate one to ten realistic randomized User-Agent strings for a selected device class.
- Get Crawling UsageReturn current Crawling API usage and optionally previous-month statistics for the connected Crawlbase account.
- Get Storage CountReturn the number of pages currently held in Crawlbase Cloud Storage.
- Get Stored PageRetrieve the latest Crawlbase Cloud Storage page by exactly one request ID or original URL.
- Get Stored PagesRetrieve up to 100 Crawlbase Cloud Storage pages by request ID without deleting them. Returns decoded page content and identifies requested IDs that were not found.
- List Stored Page IDsReturn one page of Crawlbase Cloud Storage request IDs and a short-lived continuation cursor.
Held for approval by default (changes and sends)
Held for a person's approval by default. The approval is recorded and the action is sent once, never twice, with 10 seconds to undo. 2 actions.
- Crawl URLFetch a public HTTP or HTTPS URL through Crawlbase using the connected Normal token and return its content with separate target and Crawlbase statuses.
- Delete Stored PagesIrreversibly delete explicitly selected pages from Crawlbase Cloud Storage by request ID and return each page's deletion outcome.
Changes wait in Needs you
Changes and sends in Crawlbase are held for a person's approval by default. The approval is recorded, and the action is sent once, never twice, with 10 seconds to undo.
Related connectors
Crawlbase questions
Can an agent change data in Crawlbase on its own?
Not by default. Changes and sends in Crawlbase are held for a person's approval. The approval is recorded, and the action is sent once, never twice, with 10 seconds to undo.
What can an agent read in Crawlbase?
This connector lists 6 read actions. Reads run at once, without waiting for approval.
How does Crawlbase connect?
It connects with API key. Connectors are built and maintained by third-party developers; availability is confirmed during scoping.