ScrapeField

Limits and concurrency

We limit how many calls you have in flight at once, not how many you make per second or per day. The limit rises with what you've bought.

Calls in flight

Your account may have this many calls running at the same moment. It’s set by the credits you’ve bought in total, over the life of the account, and it only ever goes up.

Credits bought, in totalCalls at once
None yet: trial credits5
100,000 or more25
1,000,000 or more100
5,000,000 or more250

One more call than that is refused at once with concurrency_limit (HTTP 429) and a Retry-After of one second. It costs nothing. Retry it, or run fewer calls side by side. There’s no limit per second, per minute or per day on top of this.

Demo calls, made with demo=true and no key, are the one exception: they’re limited to 60 a minute from one address, and the next one gets demo_limit with a Retry-After saying how long until the minute is up.

How long a call may take

Each call has 25 seconds. Most answer in a few. If we can’t get the data in time, the call fails with upstream_unavailable and the credits are refunded at once. Set your client’s timeout a little above 25 seconds so you see our answer rather than your own timeout.

Batch jobs

  • A job runs one endpoint over up to 10,000 inputs, 4 at a time. The credits for every input are reserved when the job starts, and each input that fails is refunded.
  • Results are kept for 7 days after the job finishes, then deleted.
  • A job currently runs for up to about five minutes. Inputs it doesn’t reach in that time come back with job_interrupted and their credits refunded; send them again as a new job. For more than a few hundred inputs, split them into several jobs.

How to run one: Batch jobs and webhooks.

Everything else

  • Up to 10 active API keys per account.
  • A list returns at most the limit on its endpoint’s page per call. Walk the rest with cursors.

Errors

CodeHTTPWhen, and what to do
concurrency_limit429More requests are in flight than the account allows. Retry after the seconds in `Retry-After`. Concurrency is what we limit, not requests per second, and it rises with credits purchased.
demo_limit429More than 60 demo calls a minute came from one address. Wait the seconds in `Retry-After`, or add an API key: keyed calls are limited by concurrency, not by rate.
upstream_unavailable502We could not complete the fetch in time. Retry. This call was not charged, and the refund is already in your usage ledger.
job_interrupted500A batch job stopped before it reached some of its inputs. Those inputs were not run, and their credits were refunded. Send them again in a new job.
insufficient_credits403The account is out of credits. Buy a pack on the dashboard’s billing page; calls work again at once. Nothing was charged for this call.