xyte

Appearance

API reference ​

Everything xyte exports, with its doc comment and its TypeScript declaration (fields and parameters are documented inside it). Types are TypeScript-only (import type).

ts
import { Xyte, oxylabs, decodeBody } from '@xtandard/xyte';
  • Client: Xyte, createXyte, XyteOptions, ExtractOptions, BatchResult, decodeBody, Transport, TransportRequest, SessionPoolOptions
  • Requests, responses and errors: ExtractRequest, ExtractResponse, ExtractionField, EXTRACTION_FIELDS, Action, ActionName, ActionResult, ActionSelector, Cookie, HttpHeader, CapturedResponse, NetworkCaptureFilter, CountryCode, Problem, ZyteSchemas, XyteError, XyteProblem, UpstreamProblem, UpstreamInfo, isLikelyBan, BanCheck
  • Providers: direct, proxyUrl, template, custom, proxyEndpointUrl, oxylabs, brightData, decodo, smartproxy, iproyal, nodeMaven, soax, netNut, webshare, rayobyte, packetStream, evomi, dataImpulse, massive, proxySeller, anyIp, thordata, abcProxy, proxy922, lunaProxy, proxyEmpire, rapidProxy, swiftProxy, zenrows, scrapingBee, scraperApi, crawlbase, scrapeDo, scrapingAnt, scrapeOps, scrapfly, oxylabsUnblocker, zyteProxy, zyteApi, xyteApi, ZyteApiOptions, scrappey, scrapingBot, brightDataBrowser, browserless, zenrowsBrowser, oxylabsBrowser, nodeMavenBrowser, PROVIDER_NAMES, ProxyProvider, ProxyEndpoint, ApiEndpoint, ApiResult, ProxyContext, EndpointOf, AttemptOutcome
  • Combining providers: rotate, fallback, FallbackOptions, route, adaptive, byKind, byCountry, byIpType, weighted, healthy, named, budget, watch, AdaptiveOptions, AdaptiveProvider, HealthyOptions, BudgetOptions, WatchOptions, WatchedProvider
  • Provider checks: scrapingBeeUsage, zenrowsUsage, scraperApiUsage, oxylabsUsage, zyteSpend, statusPage, allChecks, STATUS_PAGES, ProviderCheck, CheckResult
  • Shared state: StateStore, memoryStore, redisStore, redisStoreFromUrl, resilient, RedisSend
  • Actions: ActionHandlers, ActionHandler, ActionHandlerContext, ActionEntry, PerSite, ById, SearchKeywordAction, SetLocationAction, InteractionAction, ZyteOnlyAction, searchBox, isZyteOnlyAction
  • Browser: patchright, rebrowser, camoufox, cloakBrowser, STEALTH_BROWSERS, capsolver, twoCaptcha, createTaskSolver, CaptchaSolver, CaptchaTask, CaptchaKind, CaptchaOutcome, BrowserOptions, BrowserHookInfo, BrowserTypeLike, RemoteBrowser
  • Extraction: Extractor, llmExtractors, openAICompatible, Complete, CompletionRequest, Completion, LlmExtractorOptions, parseJson, htmlToMarkdown, MarkdownOptions, OUTPUT_SCHEMAS, OutputSchemaField
  • Reports, costs, cache and pacing: RequestReport, AttemptInfo, ProviderUsage, usageByProvider, Rate, Rates, rateFor, attemptCost, CacheOptions, requestKey, RateLimits, SiteLimit, parseLimit
  • Server: createHandler, HandlerOptions, FetchHandler, toNodeHandler, serve, isPrivateTarget, Jobs, Job, JobsOptions, verifyWebhook, StatsRecorder, StatsOptions, StatsQuery, StatsResponse, StatsQueryError, serveProxyMode, attachProxyMode, ProxyModeOptions, proxiedRequest, createCertificateAuthority, issueCertificate, CertificateAuthority
  • Configuration from the environment: proxyFromEnv, browserFromEnv, extractorsFromEnv, cacheFromEnv, rateLimitsFromEnv, ratesFromEnv, sessionsFromEnv, jobsFromEnv, statsFromEnv, caFromEnv, closeSharedStores
  • Tools: doctor, formatDoctor, DoctorOptions, DoctorResult

Client ​

new Xyte(options) and its methods. Topic pages: configuration, the client's lifecycle.

Xyte ​

The client: Zyte API's POST /v1/extract through the providers you choose.

ts
const xyte = new Xyte({ proxy: oxylabs({ username, password }) });
const page = await xyte.extract({ url, httpResponseBody: true });

Close it when done if it rendered pages (see docs/lifecycle.md).

ts
class Xyte {
  constructor(options?: XyteOptions);
  /**
   * Same contract as `POST https://api.zyte.com/v1/extract`: resolves with the 200 body, rejects
   * with an `XyteError` carrying Zyte's status and problem type.
   */
  extract(request: ExtractRequest, options?: ExtractOptions): Promise<ExtractResponse>;
  /**
   * Many requests, as they finish: `for await (const r of xyte.extractMany(urls.map(…)))`. Takes
   * any iterable or async iterable (a list, a file read line by line, a queue), keeps at most
   * `concurrency` in flight (default: the client's, else 16), and yields `{ index, request,
   * response }` or `{ index, request, error }`: one failure doesn't stop the batch. The client's
   * `concurrency`, `rateLimits`, cache and reports apply to each request as usual. Leaving the
   * loop early (`break`, `return`, an exception) cancels the requests still in flight.
   */
  extractMany(requests: Iterable<ExtractRequest> | AsyncIterable<ExtractRequest>, options?: {
    concurrency?: number;
    signal?: AbortSignal;
  }): AsyncGenerator<BatchResult>;
  /** `extractMany()` collected, in the order of `requests`. */
  extractAll(requests: Iterable<ExtractRequest> | AsyncIterable<ExtractRequest>, options?: {
    concurrency?: number;
    signal?: AbortSignal;
  }): Promise<BatchResult[]>;
  /**
   * Cancels the requests in flight (they reject with a `/client/closed` error), waits for them to
   * settle, then closes the shared browser, if one was started. Safe to call more than once, and
   * the client keeps working afterwards: the next render launches a new browser.
   */
  close(): Promise<void>;
  /** What makes `await using xyte = new Xyte(…)` work: calls `close()` when the block ends (docs/lifecycle.md). */
  [Symbol.asyncDispose](): Promise<void>;
}

createXyte ​

Shorthand for new Xyte(options).

ts
function createXyte(options?: XyteOptions): Xyte;

XyteOptions ​

Options of new Xyte(). Everything is optional: new Xyte() makes direct requests.

ts
interface XyteOptions {
  /** Where requests exit. Default: `direct()` (no proxy). */
  proxy?: ProxyProvider;
  /**
   * Retries after a ban, a proxy failure or a network error, each through a fresh session
   * (a new exit IP). Default 3, like Zyte's own clients.
   */
  retries?: number;
  /** Per-attempt timeout, seconds. Default 60. */
  timeout?: number;
  /** Max requests in flight; extra calls queue. Default: unlimited. */
  concurrency?: number;
  /**
   * Decides whether a downloaded page is a ban (retried, then a 520) rather than the real page.
   * Default: `isLikelyBan`. Pass `false` to return every page as is.
   */
  isBan?: ((page: BanCheck) => boolean) | false;
  /** Browser rendering for `browserHtml`, `screenshot`, `actions`, `networkCapture`. */
  browser?: BrowserOptions;
  /**
   * Handlers for Zyte's own actions outside Zyte: `searchKeyword` (default: a heuristic that
   * finds the search box), `setLocation` (per site), `interaction` (scripts by id). With
   * `zyteApi()` as the provider, Zyte runs them instead.
   */
  actions?: ActionHandlers;
  /**
   * What happens to an action that can't run where the attempt landed: a Zyte-only action with
   * no handler on a local-browser provider (proxies, `direct()`, remote browsers), or any action
   * on an HTML-only API (Scrappey, ScrapingBot), once there's no provider to pass over to.
   * `warn` (default) skips it (`notExecuted` in the response's `actions`, plus a warning) and runs
   * the rest; `error` passes over to `zyteApi()` when a combinator has it, else rejects with a
   * 400; `zyte` passes over to `zyteApi()` when a combinator has it (the built-in search
   * heuristic's actions too), else like `warn`. Actions you have a handler for always run locally.
   * See docs/actions.md.
   */
  unsupportedActions?: "warn" | "error" | "zyte";
  /**
   * Implementations for Zyte's extraction outputs (`product`, `article`, `markdown`, …), keyed by
   * field. Requests for a field without an extractor are rejected with a 400, except `markdown`,
   * which has a built-in (non-AI, like Zyte's) converter.
   */
  extractors?: Partial<Record<ExtractionField, Extractor>>;
  /** Server-managed sessions (`sessionContext`): pool size, max age, bans before retiring. */
  sessions?: SessionPoolOptions;
  /** Low-level HTTP sender; replace to add logging, or for tests. */
  transport?: Transport;
  /** Called before each retry. */
  onRetry?: (info: {
    attempt: number;
    url: string;
    reason: string;
  }) => void;
  /**
   * Called after every attempt with the provider that served it (the one a combinator picked),
   * the outcome, latency and page size: per-provider success rates and costs, in your metrics.
   */
  onAttempt?: (info: AttemptInfo) => void;
  /**
   * Called once per `extract()`, success or failure, with what it took: every attempt (provider,
   * route, result, latency, bytes), totals per provider, and the outcome. For logs, metrics and
   * analytics; the response itself stays Zyte's.
   */
  onRequest?: (report: RequestReport) => void;
  /**
   * Keep responses for a while and answer repeats from them (and let identical requests in flight
   * share one run): `{ ttl: 3600 }`, or per site `{ ttl: { 'wolt.com': 600, '*': 3600 } }`.
   * Requests with a `session` or `sessionContext` are never cached. Off by default.
   */
  cache?: CacheOptions;
  /**
   * How fast, and how many at once, attempts go to each site: `{ 'yelp.co.uk': '2/s', '*': '10/s' }`
   * (each site under `'*'` on its own), or `{ rate, per, concurrency }`. Per process. Off by default.
   */
  rateLimits?: RateLimits;
  /**
   * Your prices, per provider (its name, or a prefix: `oxylabs` covers `oxylabs-residential`):
   * `{ oxylabs: { perGB: 6 }, 'zyte-api': { perRequest: 0.0013 } }`. Attempts and reports then
   * carry `costUsd`. Plans and volume discounts differ, so there are no built-in prices.
   */
  rates?: Rates;
  /**
   * When a plain HTTP request (GET, no body) is banned, retry it in a browser, which gets past
   * JavaScript challenges, and return the page's raw HTTP response as `httpResponseBody`, as
   * Zyte does when it picks a browser itself. Needs Playwright (or `browser.connect`). Default false.
   */
  browserFallback?: boolean;
  /**
   * Requests that run but not as asked, e.g. a `geolocation` the exit can't apply (`direct()`,
   * a plain proxy URL). Default: `console.warn`, once per distinct message.
   */
  onWarning?: (message: string, info: {
    url: string;
  }) => void;
}

ExtractOptions ​

Per-call options of extract().

ts
interface ExtractOptions {
  /** Aborts the request (all attempts). Rejects with the signal's reason. */
  signal?: AbortSignal;
  /** This request's warnings, in addition to the client's `onWarning`. */
  onWarning?: (message: string) => void;
  /** This request's report once it's done, in addition to the client's `onRequest`. */
  onReport?: (report: RequestReport) => void;
}

BatchResult ​

One request of extractMany(): its position, and its response or the error it failed with.

ts
interface BatchResult {
  index: number;
  request: ExtractRequest;
  response?: ExtractResponse;
  error?: unknown;
}

decodeBody ​

An httpResponseBody (or a networkCapture body) as UTF-8 text. Bodies are base64 in Zyte's API, because they can be anything (an image, a PDF):

ts
const page = await xyte.extract({ url, httpResponseBody: true });
const html = decodeBody(page.httpResponseBody!);

For binary content, decode to bytes instead: Buffer.from(body, 'base64').

ts
function decodeBody(base64: string): string;

Transport ​

Sends one HTTP request, never following redirects (the HTTP engine does that itself).

ts
type Transport = (request: TransportRequest) => Promise<Response>;

TransportRequest ​

One HTTP request as a Transport gets it: URL, method, headers, body, the proxy to go through, and a signal.

ts
interface TransportRequest {
  url: string;
  method: string;
  headers: Headers;
  body?: Uint8Array<ArrayBuffer>;
  proxy: ProxyEndpoint | null;
  signal: AbortSignal;
}

SessionPoolOptions ​

Zyte's server-managed sessions (sessionContext): requests with the same context share a small pool of sessions, each initialized once with sessionContextParameters (a sticky exit plus the cookies the init actions produced). Sessions expire after 4 hours or 3 bans.

ts
interface SessionPoolOptions {
  /** Sessions per context. Default 8. */
  size?: number;
  /** Seconds before a session is retired. Default 4 h (Zyte's). */
  maxAge?: number;
  /** Bans (or failed attempts) before a session is retired. Default 3 (Zyte's). */
  maxBans?: number;
  /**
   * Shared state: every instance draws from the same sessions per context, with their cookies,
   * bans and whether they're initialized (a session another instance logged in isn't logged in
   * again). Default: this process only.
   */
  store?: StateStore;
  /** Seconds between re-reads of a context's shared sessions (before an acquire). Default 5. */
  syncEvery?: number;
}

Requests, responses and errors ​

Zyte's request and response bodies, generated from Zyte's OpenAPI spec, and the errors extract() rejects with. Field by field: coverage; errors: the README.

ExtractRequest ​

The body of POST https://api.zyte.com/v1/extract.

Loosened from the spec where Zyte itself is lenient: geolocation in any case, the network capture filters as the union the spec describes in prose, and echoData as any JSON.

ts
type ExtractRequest = Omit<ZyteSchemas["ExtractRequest"], "geolocation" | "networkCapture" | "echoData" | "tags"> & {
  geolocation?: CountryCode | Lowercase<CountryCode>;
  networkCapture?: NetworkCaptureFilter[];
  echoData?: unknown;
  /** Labels for the request. "Values must be strings or `null`" (Zyte's docs; the schema says strings only). */
  tags?: Record<string, string | null>;
};

ExtractResponse ​

The 200 body of POST /v1/extract.

ts
type ExtractResponse = Omit<ZyteSchemas["Response200"], "echoData"> & {
  echoData?: unknown;
};

ExtractionField ​

One of EXTRACTION_FIELDS: product, article, … markdown.

ts
type ExtractionField = (typeof EXTRACTION_FIELDS)[number];

EXTRACTION_FIELDS ​

Zyte's extraction outputs. markdown has a built-in converter; the others need an extractor (extractors option, or llmExtractors()).

ts
const EXTRACTION_FIELDS: readonly ["product", "productList", "productNavigation", "article", "articleList", "articleNavigation", "forumThread", "jobPosting", "jobPostingNavigation", "pageContent", "serp", "customAttributes", "markdown"];

Action ​

One browser action (click, type, scrollBottom, waitForSelector…), as in actions.

ts
type Action = ZyteSchemas["ActionSequence"][number];

ActionName ​

The name of an action: click, type, searchKeyword…

ts
type ActionName = Action["action"];

ActionResult ​

What happened to one action, in the response's actions: status, error, elapsedTime.

ts
type ActionResult = ZyteSchemas["ActionResult"];

ActionSelector ​

How an action finds its element: { type: 'css' | 'xpath', value }.

ts
type ActionSelector = ZyteSchemas["ActionSelector"];

A cookie, as in requestCookies and responseCookies.

ts
type Cookie = ZyteSchemas["Cookie"];

HttpHeader ​

A header as a { name, value } pair, as Zyte lists them (httpResponseHeaders, customHttpRequestHeaders).

ts
type HttpHeader = ZyteSchemas["HTTPHeader"];

CapturedResponse ​

One response a browser render captured (networkCapture): its URL, status, headers and body.

ts
type CapturedResponse = ZyteSchemas["CapturedResponse"];

NetworkCaptureFilter ​

Which responses networkCapture keeps: by URL, or by resource type.

ts
type NetworkCaptureFilter = ZyteSchemas["UrlFilter"] | ZyteSchemas["ResourceTypeFilter"];

CountryCode ​

An ISO 3166-1 alpha-2 country code, as geolocation takes (US, DE…).

ts
type CountryCode = ZyteSchemas["CountryCode"];

Problem ​

Zyte's error body as the spec has it. XyteProblem is the same with the fields xyte always sets.

ts
type Problem = ZyteSchemas["Problem"];

ZyteSchemas ​

Every schema of the Zyte API OpenAPI spec, as generated.

ts
type ZyteSchemas = components["schemas"];

XyteError ​

A Zyte API error response (RFC 7807 problem). status and type use the values Zyte uses (https://docs.zyte.com/zyte-api/usage/errors.html), so code written against Zyte keeps working.

ts
class XyteError extends Error {
  /** The HTTP status Zyte answers with: 400, 422, 429, 500, 520, 521… */
  readonly status: number;
  /** The problem type, the stable thing to branch on: `/download/website-ban`, `/request/invalid`… */
  readonly type: string;
  /** A short human-readable summary of the type. */
  readonly title: string;
  /** What went wrong for this request, when there's more to say. */
  readonly detail?: string;
  /** `retryable` overrides the status-based rule (a misconfiguration's 500 is not). */
  constructor(problem: XyteProblem, options?: {
    cause?: unknown;
    retryable?: boolean;
  });
  /** Worth retrying later, the same way Zyte's own clients retry. */
  get retryable(): boolean;
  /** The problem as Zyte's error body; what `xyte serve` answers with. */
  toJSON(): XyteProblem;
}

XyteProblem ​

A Zyte error body (an RFC 7807 problem): type, status, title and an optional detail.

ts
type XyteProblem = Required<Pick<Problem, "type" | "status" | "title">> & {
  detail?: string;
};

UpstreamProblem ​

A failure an upstream Zyte-compatible API answered with its own problem (a 520 ban, a 429, a suspended account). Retried like any download error; if it is the last one, that problem is returned as is.

ts
class UpstreamProblem extends Error {
  /** Always `UpstreamProblem`. */
  override name: string;
  /** The upstream's problem, returned to the caller as is if no retry succeeds. */
  readonly problem: XyteProblem;
  /**
   * The provider can't serve any request until someone fixes it (a bad key, a suspended account):
   * inside a combinator the request moves on and the provider isn't tried again for it; alone, the
   * problem is the answer, with no retries.
   */
  readonly fatal: boolean;
  /** What the upstream reported about its own attempts (another xyte server's `X-Xyte-Attempts`). */
  readonly upstream?: UpstreamInfo;
  constructor(problem: XyteProblem, options?: {
    fatal?: boolean;
    upstream?: UpstreamInfo;
  });
}

UpstreamInfo ​

An upstream xyte server's report of a request, from its X-Xyte-* headers.

ts
interface UpstreamInfo {
  /** Its attempts, in order: `provider` is its leaf (`oxylabs-residential`), `result` its verdict. */
  attempts: {
    provider: string;
    result: string;
  }[];
  /** Bytes its attempts moved (`X-Xyte-Transfer-Bytes`). */
  transferBytes?: number;
  /** Its request id (`X-Xyte-Request-Id`), to find the request in its logs. */
  requestId?: string;
}

isLikelyBan ​

The default ban check: a 403, 429 or 503, or a 200 that is an anti-bot challenge page (Cloudflare, DataDome, PerimeterX, Akamai…). Replace it with the isBan option.

ts
function isLikelyBan(page: BanCheck): boolean;

BanCheck ​

What the ban check (isBan) looks at: the target's answer to one attempt.

ts
interface BanCheck {
  url: string;
  statusCode: number;
  headers: {
    name: string;
    value: string;
  }[];
  html?: string;
}

Providers ​

Where requests exit. Every preset's credentials and options: providers.

direct ​

No proxy: requests go straight from this machine.

ts
function direct(): ProxyProvider<ProxyEndpoint>;

proxyUrl ​

Any proxy (or list of proxies) given as URLs. With several, attempts rotate through them, so a retry uses the next one. For providers that encode geo/session in the URL, use custom.

ts
function proxyUrl(urls: string | readonly string[], options?: {
  insecure?: boolean;
}): ProxyProvider<ProxyEndpoint>;

template ​

Any provider, from its URL format. Placeholders:

  • {country} / {country:lower}: ISO-2 country from geolocation
  • {session} / {session:8}: sticky-session token (alphanumeric, given length, default 12)
  • {session:number} / {session:number:99999}: numeric session id (below the given max)
  • {ipType}, {attempt}
  • {env:NAME}: an environment variable (e.g. the password)

Square brackets mark an optional part, dropped when a placeholder inside has no value (no geolocation on the request, say). Substituted values are URL-encoded; literal text is not.

template('http://user-{env:USER}[-country-{country:lower}]-session-{session}:{env:PASS}@gate.example.com:7000')
ts
function template(format: string, options?: {
  insecure?: boolean;
  name?: string;
}): ProxyProvider<ProxyEndpoint>;

custom ​

Build the proxy URL yourself from the context.

ts
function custom(name: string, resolve: (context: ProxyContext) => string | ProxyEndpoint | null | Promise<string | ProxyEndpoint | null>): ProxyProvider<ProxyEndpoint>;

proxyEndpointUrl ​

A proxy URL with the credentials percent-encoded: proxyEndpointUrl({ host: 'pr.oxylabs.io', port: 7777, username, password }) → http://user:p%[email protected]:7777.

ts
function proxyEndpointUrl(args: {
  host: string;
  port: number | string;
  username?: string;
  password?: string;
  protocol?: "http" | "https";
}): string;

oxylabs ​

Oxylabs (https://developers.oxylabs.io/proxies/residential-proxies). Residential: customer-USER-cc-US-sessid-…-sesstime-10 on pr.oxylabs.io:7777. Datacenter: user-USER-country-US on dc.oxylabs.io:8000.

ts
function oxylabs(options: {
  username: string;
  password: string;
  type?: "residential" | "datacenter";
  host?: string;
  port?: number;
  /** Sticky session lifetime, minutes (residential, max 1440). Default 10. */
  sessionMinutes?: number;
}): ProxyProvider<ProxyEndpoint>;

brightData ​

Bright Data (https://docs.brightdata.com/proxy-networks/config-options). brd-customer-ID-zone-ZONE-country-us-session-… on brd.superproxy.io:33335. Bright Data intercepts TLS unless your account is verified; pass insecure: true (or install their CA) if you see certificate errors.

ts
function brightData(options: {
  customerId: string;
  zone: string;
  password: string;
  host?: string;
  port?: number;
  insecure?: boolean;
}): ProxyProvider<ProxyEndpoint>;

decodo ​

Decodo, formerly Smartproxy (https://help.decodo.com/docs/residential-proxy-parameters). user-USER-country-us-session-…-sessionduration-10 on gate.decodo.com:7000.

ts
function decodo(options: {
  username: string;
  password: string;
  host?: string;
  port?: number;
  /** Sticky session lifetime, minutes (1–1440). Default 10. */
  sessionMinutes?: number;
}): ProxyProvider<ProxyEndpoint>;

smartproxy ​

@deprecated Smartproxy is now Decodo.

ts
const smartproxy: typeof decodo;

iproyal ​

IPRoyal residential (https://docs.iproyal.com/proxies/residential/proxy). Targeting goes in the password: PASS_country-us_session-XXXXXXXX_lifetime-10m.

ts
function iproyal(options: {
  username: string;
  password: string;
  host?: string;
  port?: number;
  /** Sticky session lifetime, e.g. `10m`, `1h`. Default `10m`. */
  lifetime?: string;
}): ProxyProvider<ProxyEndpoint>;

nodeMaven ​

NodeMaven (https://docs.nodemaven.com): USER-country-us-sid-…-ttl-10m on gate.nodemaven.com:8080. type picks the pool (residential / mobile), filter the IP quality.

ts
function nodeMaven(options: Credentials & {
  type?: "residential" | "mobile";
  filter?: "low" | "medium" | "high";
  ttl?: string;
}): ProxyProvider<ProxyEndpoint>;

soax ​

SOAX (https://helpcenter.soax.com): user package-ID-country-us-sessionid-…-sessionlength-600, password = package key.

ts
function soax(options: {
  packageId: string;
  packageKey: string;
  sessionSeconds?: number;
  host?: string;
  port?: number;
}): ProxyProvider<ProxyEndpoint>;

netNut ​

NetNut (https://help.netnut.io): USER-res-us-sid-123456789 on gw.netnut.net:5959. Verify for non-residential plans.

ts
function netNut(options: Credentials): ProxyProvider<ProxyEndpoint>;

webshare ​

Webshare rotating residential (https://apidocs.webshare.io/proxy-connection): USER-us-1234 (numeric sticky session) on p.webshare.io:80. sticky: false uses -rotate (a new IP per request).

ts
function webshare(options: Credentials & {
  sticky?: boolean;
}): ProxyProvider<ProxyEndpoint>;

rayobyte ​

Rayobyte (https://docs.rayobyte.com): targeting in the password, PASS-country-US-session-…-duration-10.

ts
function rayobyte(options: Credentials & {
  durationMinutes?: number;
}): ProxyProvider<ProxyEndpoint>;

packetStream ​

PacketStream (https://packetstream.io/docs): password KEY_country-US_session-… on proxy.packetstream.io:31112.

ts
function packetStream(options: Credentials): ProxyProvider<ProxyEndpoint>;

evomi ​

Evomi (https://docs.evomi.com): password PASS_country-US_session-…_lifetime-30 on rp.evomi.com:1000.

ts
function evomi(options: Credentials & {
  lifetimeMinutes?: number;
}): ProxyProvider<ProxyEndpoint>;

dataImpulse ​

DataImpulse (https://docs.dataimpulse.com): user LOGIN__cr.us;sessid.… on gw.dataimpulse.com:823.

ts
function dataImpulse(options: Credentials): ProxyProvider<ProxyEndpoint>;

massive ​

Massive (https://docs.joinmassive.com): USER-country-US-session-…-sessionttl-15 on network.joinmassive.com:65534.

ts
function massive(options: Credentials & {
  sessionTtlMinutes?: number;
}): ProxyProvider<ProxyEndpoint>;

proxySeller ​

Proxy-Seller residential (https://docs.proxy-seller.com): USER_c_US_s_…_ttl_15m on res.proxy-seller.com:10000.

ts
function proxySeller(options: Credentials & {
  ttl?: string;
}): ProxyProvider<ProxyEndpoint>;

anyIp ​

Anyip (https://anyip.io/docs): user_ID,country_US,session_…,sesstime_10 on portal.anyip.io:1080 (verify host).

ts
function anyIp(options: Credentials & {
  sessionMinutes?: number;
}): ProxyProvider<ProxyEndpoint>;

thordata ​

Thordata (https://doc.thordata.com): td-customer-USER-country-us-sessid-…-sesstime-10 on t.pr.thordata.net:9999.

ts
function thordata(options: Credentials & {
  sessionMinutes?: number;
}): ProxyProvider<ProxyEndpoint>;

abcProxy ​

ABCProxy (https://docs.abcproxy.com): ACCOUNT-zone-abc-region-us-session-…-sessTime-10 on proxy.abcproxy.com:4950. Verify.

ts
function abcProxy(options: Credentials & {
  sessionMinutes?: number;
}): ProxyProvider<ProxyEndpoint>;

proxy922 ​

922 S5 Proxy (https://doc.922proxy.com): USER-region-FR-sessid-…-sessTime-10; the host is per account. Verify.

ts
function proxy922(options: Credentials & {
  host: string;
  port: number;
  sessionMinutes?: number;
}): ProxyProvider<ProxyEndpoint>;

lunaProxy ​

LunaProxy (https://www.lunaproxy.com/help): user-SUB-region-us-sessid-…-sesstime-10; the host is per account. Verify.

ts
function lunaProxy(options: Credentials & {
  host: string;
  port: number;
  sessionMinutes?: number;
}): ProxyProvider<ProxyEndpoint>;

proxyEmpire ​

ProxyEmpire: USER-country-us-sid-… on v2.proxyempire.io:5000 (session lifetime is set in the dashboard). Verify.

ts
function proxyEmpire(options: Credentials): ProxyProvider<ProxyEndpoint>;

rapidProxy ​

RapidProxy: USER-residential-US-session-12345678-stime-10 on eu.rapidproxy.io:5001. From an API example; verify.

ts
function rapidProxy(options: Credentials & {
  stimeMinutes?: number;
}): ProxyProvider<ProxyEndpoint>;

swiftProxy ​

Swiftproxy: USER_custom_zone_US on us.swiftproxy.net:7878. Sticky sessions are configured in their dashboard's endpoint generator (no documented username syntax), so every request rotates unless you use a sticky endpoint from there with proxyUrl().

ts
function swiftProxy(options: Credentials): ProxyProvider<ProxyEndpoint>;

zenrows ​

ZenRows Universal Scraper API, proxy mode (https://docs.zenrows.com/universal-scraper-api/features/proxy-mode). http://APIKEY:premium_proxy=true&[email protected]:8001.

ts
function zenrows(options: {
  apiKey: string;
  /** Residential IPs. Default true (required for `proxy_country`). */
  premiumProxy?: boolean;
  /** Extra ZenRows parameters, e.g. `{ js_render: 'true' }`. */
  params?: Record<string, string>;
}): ProxyProvider<ProxyEndpoint>;

scrapingBee ​

ScrapingBee, proxy mode (https://www.scrapingbee.com/documentation/proxy-mode/). http://APIKEY:render_js=False&[email protected]:8886.

ts
function scrapingBee(options: {
  apiKey: string;
  /** Residential IPs (required for `country_code`). Default true. */
  premiumProxy?: boolean;
  /** ScrapingBee's hardest-target pool. */
  stealthProxy?: boolean;
  params?: Record<string, string>;
}): ProxyProvider<ProxyEndpoint>;

scraperApi ​

ScraperAPI, proxy mode (https://docs.scraperapi.com/making-requests/proxy-port-method). http://scraperapi.country_code=us.session_number=123:[email protected]:8001.

ts
function scraperApi(options: {
  apiKey: string;
  premium?: boolean;
  params?: Record<string, string>;
}): ProxyProvider<ProxyEndpoint>;

crawlbase ​

Crawlbase Smart Proxy (https://crawlbase.com/docs/smart-proxy/): token as user; params as CrawlbaseAPI-* headers.

ts
function crawlbase(options: {
  token: string;
  javascript?: boolean;
}): ProxyProvider<ProxyEndpoint>;

scrapeDo ​

Scrape.do proxy mode (https://scrape.do/documentation/proxy-mode/): token as user, API params as the password.

ts
function scrapeDo(options: {
  token: string;
  render?: boolean;
  super?: boolean;
  params?: Record<string, string>;
}): ProxyProvider<ProxyEndpoint>;

scrapingAnt ​

ScrapingAnt proxy mode (https://docs.scrapingant.com/proxy-mode): user scrapingant&browser=false&…, API key as password. Verify country.

ts
function scrapingAnt(options: {
  apiKey: string;
  browser?: boolean;
  params?: Record<string, string>;
}): ProxyProvider<ProxyEndpoint>;

scrapeOps ​

ScrapeOps proxy aggregator (https://scrapeops.io/docs/proxy-aggregator/): user scrapeops.country=us, API key as password. Verify.

ts
function scrapeOps(options: {
  apiKey: string;
  renderJs?: boolean;
  params?: Record<string, string>;
}): ProxyProvider<ProxyEndpoint>;

scrapfly ​

Scrapfly proxy mode (https://scrapfly.io/docs/scrape-api/proxy-mode): dash-separated options as user, API key as password. Verify.

ts
function scrapfly(options: {
  apiKey: string;
  renderJs?: boolean;
  asp?: boolean;
}): ProxyProvider<ProxyEndpoint>;

oxylabsUnblocker ​

Oxylabs Web Unblocker (https://developers.oxylabs.io): unblock.oxylabs.io:60000, params as x-oxylabs-* headers.

ts
function oxylabsUnblocker(options: {
  username: string;
  password: string;
  render?: boolean;
}): ProxyProvider<ProxyEndpoint>;

zyteProxy ​

Zyte API itself, in proxy mode (https://docs.zyte.com/zyte-api/usage/proxy-mode.html), e.g. as the last provider in rotate(): API key as user, Zyte-Geolocation / Zyte-Session-ID headers.

ts
function zyteProxy(options: {
  apiKey: string;
}): ProxyProvider<ProxyEndpoint>;

zyteApi ​

Zyte API itself (https://docs.zyte.com/zyte-api/usage/reference.html): the request is forwarded unchanged, so every Zyte feature works, AI extraction included. Use it as the last resort: rotate([nodeMaven(…), zyteApi({ apiKey })]). A bad key or a suspended account (401, 403) moves a combinator on to its next provider; alone, it is the answer, with no retries.

ts
function zyteApi(options: ZyteApiOptions): ProxyProvider<ApiEndpoint>;

xyteApi ​

Another xyte server (xyte serve, e.g. a deployment with its own proxy): zyteApi() with a required endpoint that also reads its X-Xyte-Attempts / X-Xyte-Transfer-Bytes headers, so the local attempt carries the remote ones (upstream) and the bytes they moved. Default name xyte; give each deployment its own: xyteApi({ endpoint, apiKey, name: 'xyte-oxylabs' }). An attempt waits up to timeout seconds (default 150): keep the remote's retries × timeout under it. The client needs no browser: the remote renders.

ts
function xyteApi(options: ZyteApiOptions & {
  endpoint: string;
}): ProxyProvider<ApiEndpoint>;

ZyteApiOptions ​

Options for zyteApi() and xyteApi().

ts
interface ZyteApiOptions {
  apiKey: string;
  /** Default `https://api.zyte.com/v1/extract`; any Zyte-compatible `/v1/extract` works. */
  endpoint?: string;
  /**
   * The provider's name in reports, `rates`, circuits and budgets. Default `zyte-api`. Set one per
   * endpoint when a combinator has several, so each keeps its own numbers.
   */
  name?: string;
  /**
   * Seconds an attempt on this endpoint may take, instead of the client's `timeout`. `xyteApi()`
   * defaults to 150 (the remote server retries on its side, and 150 s is Modal's web limit).
   */
  timeout?: number;
  /** Resolved at call time: a stub put on `globalThis.fetch` later is used too. */
  fetch?: typeof fetch;
}

scrappey ​

Scrappey (https://docs.scrappey.com): POST publisher.scrappey.com/api/v1?key=…. Browser requests use requestType: "browser", HTTP ones "request". Verify the response fields on your account (they're mapped from solution.response / statusCode / currentUrl).

ts
function scrappey(options: {
  apiKey: string;
  fetch?: typeof fetch;
  params?: Record<string, unknown>;
}): ProxyProvider<ApiEndpoint>;

scrapingBot ​

ScrapingBot raw HTML API (https://www.scraping-bot.io/web-scraping-documentation/): POST api.scraping-bot.io/scrape/raw-html, basic auth username:apiKey, answers the page's HTML. Its options (useChrome, premiumProxy, proxyCountry…) are passed through options; verify the names in your dashboard. The target's status code is not reported (assumed 200).

ts
function scrapingBot(options: {
  username: string;
  apiKey: string;
  options?: Record<string, unknown>;
  fetch?: typeof fetch;
}): ProxyProvider<ApiEndpoint>;

brightDataBrowser ​

Bright Data Scraping Browser (https://docs.brightdata.com/products/scraping-browser): country in the username, port 9222.

ts
function brightDataBrowser(options: {
  customerId: string;
  zone: string;
  password: string;
}): Connect;

browserless ​

Browserless (https://docs.browserless.io): token in the query, residential proxy with proxyCountry / proxySticky when the request has a geolocation (or proxy: 'residential').

ts
function browserless(options: {
  token: string;
  /** e.g. `production-sfo`, `production-lon`, or a full `wss://` URL for self-hosted. */
  region?: string;
  proxy?: "residential";
  params?: Record<string, string>;
}): Connect;

zenrowsBrowser ​

ZenRows Scraping Browser (https://docs.zenrows.com/scraping-browser): wss://browser.zenrows.com?apikey=…&proxy_country=….

ts
function zenrowsBrowser(options: {
  apiKey: string;
  sessionTtl?: string;
}): Connect;

oxylabsBrowser ​

Oxylabs Unblocking Browser: wss://USER:[email protected]?p_cc=US.

ts
function oxylabsBrowser(options: {
  username: string;
  password: string;
}): Connect;

nodeMavenBrowser ​

NodeMaven Scraping Browser: wss://USER-country-us-sid-…:[email protected]:8080. Verify.

ts
function nodeMavenBrowser(options: {
  username: string;
  password: string;
}): Connect;

PROVIDER_NAMES ​

Names accepted by XYTE_PROVIDER.

ts
const PROVIDER_NAMES: readonly string[];

ProxyProvider ​

Picks the exit for each attempt. Presets resolve to a ProxyEndpoint; API adapters to an ApiEndpoint.

ts
interface ProxyProvider<E extends ProxyEndpoint | ApiEndpoint = ProxyEndpoint | ApiEndpoint> {
  readonly name: string;
  /**
   * The provider answers AI extraction fields itself (Zyte API): requests for them are not
   * rejected upfront for lack of a local extractor.
   */
  readonly extracts?: boolean;
  /**
   * The provider runs Zyte's own actions itself (`searchKeyword`, `setLocation`, `interaction`):
   * with `unsupportedActions: 'error'`, requests using them aren't rejected upfront, and attempts
   * can be passed over to reach it (docs/actions.md).
   */
  readonly zyteActions?: boolean;
  /** The proxy (or API) for this attempt, or null to connect directly. */
  resolve(context: ProxyContext): E | null | Promise<E | null>;
  /**
   * False while the provider shouldn't be used (an open circuit, a spent budget): combinators
   * skip it. Omitted means always available.
   */
  available?(context: ProxyContext): boolean;
  /** How an attempt this provider resolved went. Combinators forward it to the child they chose. */
  report?(context: ProxyContext, outcome: AttemptOutcome): void;
}

ProxyEndpoint ​

A proxy for one attempt: what a proxy provider's resolve() returns.

ts
interface ProxyEndpoint {
  /** `http://user:pass@host:port` (or `https://`). */
  url: string;
  /**
   * Skip TLS verification of the target. Scraping APIs in proxy mode (ZenRows, ScrapingBee,
   * ScraperAPI) re-sign traffic with their own certificate, so it is required for them.
   */
  insecure?: boolean;
  /**
   * Headers added to every request through this proxy, for scraping APIs that take their
   * parameters as headers in proxy mode (Crawlbase, Zyte, Oxylabs Web Unblocker).
   */
  headers?: Record<string, string>;
  /**
   * Reads what a scraping API in proxy mode adds to its answers (HTTP requests only): the final
   * URL after redirects it followed itself, and its own error answers, which are retried as a
   * provider failure instead of being returned as the target's status.
   */
  inspect?: (res: {
    status: number;
    headers: Headers;
  }) => {
    url?: string;
    failure?: string;
  } | undefined;
  /** The provider's bookkeeping headers, left out of `httpResponseHeaders`. */
  hiddenHeaders?: RegExp;
  /**
   * False when this exit ignores the request's `geolocation` (a fixed proxy URL). Such requests
   * still run, with a warning (`onWarning`, `X-Xyte-Warning`).
   */
  appliesCountry?: boolean;
}

ApiEndpoint ​

A scraping service with no proxy port (Scrappey, ScrapingBot, Zyte API itself): it takes the whole Zyte request and returns a Zyte-shaped response. Ban detection, retries, rotate() and extractors work on its answer as on a proxied download.

ts
interface ApiEndpoint {
  api: (request: ExtractRequest, info: {
    context: ProxyContext;
    signal: AbortSignal;
  }) => Promise<ApiResult>;
  /**
   * What `request` asks for that this service can't do (e.g. `['actions', 'screenshot']` for an
   * API that only returns a page's HTML). Inside a combinator the attempt is passed over for a
   * provider that can; alone, actions follow `unsupportedActions` and missing outputs are a 400.
   * Omitted: it can do everything.
   */
  unsupported?: (request: ExtractRequest) => readonly string[];
  /** Seconds an attempt on this API may take, instead of the client's `timeout`. */
  timeout?: number;
}

ApiResult ​

What an ApiEndpoint's api() resolves with: the service's answer in Zyte's shape.

ts
interface ApiResult {
  /** The Zyte-shaped response (url, statusCode, httpResponseBody / browserHtml, …). */
  response: ExtractResponse;
  /** The target's status, for ban detection. */
  statusCode: number;
  headers?: {
    name: string;
    value: string;
  }[];
  /** The page as text, for ban detection and extractors. */
  html?: string;
  /** The service already handled bans (Zyte): skip xyte's ban check on this answer. */
  trusted?: boolean;
  /** Bytes the service reported moving for it (another xyte server's `X-Xyte-Transfer-Bytes`). */
  transferBytes?: number;
  /** The service's own report of its attempts, when it is another xyte server (`xyteApi()`). */
  upstream?: UpstreamInfo;
}

ProxyContext ​

What a provider gets to pick an exit for one attempt of one request.

ts
interface ProxyContext {
  /** Target URL. */
  url: URL;
  /** ISO 3166-1 alpha-2, upper case, from the request's `geolocation`. */
  country?: string;
  /** From the request's `ipType`. */
  ipType?: "datacenter" | "residential";
  /**
   * Sticky-session key: the request's `session.id`, or a fresh id per attempt. Providers map it
   * to their session parameter so every hop of one attempt (redirects, browser sub-requests)
   * leaves from one IP, and a retry leaves from another.
   */
  sessionId: string;
  /** 0 for the first try, then 1, 2… */
  attempt: number;
  /** The attempt renders in a browser (every page sub-request goes through the proxy). */
  browser: boolean;
  /** Unique per `extract()` call: combinators use it to tell attempts of one request apart. */
  requestId?: string;
  /** Providers chosen so far by combinators, outermost first; the last is the one that served. */
  path?: string[];
}

EndpointOf ​

The endpoint type a provider resolves to.

ts
type EndpointOf<P> = P extends ProxyProvider<infer E> ? E : never;

AttemptOutcome ​

How an attempt went, reported to the provider that served it (and to onAttempt).

ts
interface AttemptOutcome {
  /**
   * `ok`: the page came back (a target's own 404 included). `ban`: a ban or challenge page.
   * `error`: the proxy or network failed. `aborted`: no verdict on the provider (an invalid
   * request, a cancelled call, another attempt's failure).
   */
  result: "ok" | "ban" | "error" | "aborted";
  /** Milliseconds the attempt took. */
  ms: number;
  statusCode?: number;
  /** Size of the page that came back (decoded body or browser HTML). */
  bytes?: number;
  /**
   * Bytes that crossed the wire, both ways: headers, compressed bodies, redirects and, in a
   * browser, every sub-request. What a per-GB proxy bills, minus TLS/tunnel overhead.
   */
  transferBytes?: number;
  reason?: string;
  /**
   * The provider can't serve any request until someone fixes it (a bad key, a suspended account):
   * `healthy()` opens its circuit at once.
   */
  fatal?: boolean;
  /** Through another xyte server (`xyteApi()`): its attempts, e.g. `oxylabs-residential:ban, oxylabs-residential:ok`. */
  upstream?: UpstreamInfo;
  /** Captchas met in a browser render with `browser.captcha`, and whether they were solved. */
  captchas?: {
    kind: string;
    solver: string;
    solved: boolean;
    ms: number;
    error?: string;
  }[];
}

Combining providers ​

Providers built from other providers. Routing explains each, with examples.

rotate ​

Tries providers in order across attempts: attempt 0 → first, attempt 1 → second… and wraps. A provider outage (e.g. a 407 on every request) then costs one attempt, not the request. Unavailable providers (an open healthy() circuit, a spent budget()) are skipped.

ts
function rotate<P extends ProxyProvider>(providers: readonly P[]): ProxyProvider<EndpointOf<P>>;

fallback ​

The primary while it works, then the fallbacks: fallback(zyteApi(…), [fleet], { primaryAttempts: 3 }). Each request gives the primary its first primaryAttempts attempts, then rotates through the fallbacks. The fallbacks get every attempt when the primary is unavailable (an open healthy() circuit, a spent budget()), and the rest of the request once it is passed over (an account problem, an action it can't run). Counted per request, from each attempt's report.

ts
function fallback<P extends ProxyProvider, F extends ProxyProvider>(primary: P, fallbacks: readonly F[], options?: FallbackOptions): ProxyProvider<EndpointOf<P | F>>;

FallbackOptions ​

Options for fallback().

ts
interface FallbackOptions {
  /**
   * Attempts the primary gets per request before the fallbacks take over. Default 3: with the
   * default `retries: 3` (4 attempts), the fallbacks get the last one.
   */
  primaryAttempts?: number;
}

route ​

Picks the provider by the target's host, so each site gets the cheapest exit that works on it: route({ 'doordash.com': zyteApi(…), 'wolt.com': direct() }, residential). A key matches the host and its subdomains (doordash.com matches www.doordash.com); the longest match wins. Every attempt of a request, retries included, stays on its route: nest rotate() inside a route for a fallback.

ts
function route<R extends Readonly<Record<string, ProxyProvider>>, F extends ProxyProvider>(routes: R, fallback: F): ProxyProvider<EndpointOf<R[keyof R] | F>>;

adaptive ​

Learns, per site, the cheapest provider that works. List providers cheapest first: adaptive([direct(), residential, zyteApi(…)]). Each request starts at the first provider whose success score on that site is at least minSuccess; a retry moves to the next one. Scores start optimistic and follow outcomes (bans and errors lower them), and a small share of requests re-tests the cheaper ones, so a site that stops blocking moves back down.

ts
function adaptive<P extends ProxyProvider>(providers: readonly P[], options?: AdaptiveOptions): AdaptiveProvider<EndpointOf<P>>;

byKind ​

One provider for plain HTTP requests, another for browser renders. Scraping APIs bill every sub-request of a render, residential proxies bill per GB: byKind({ http: scrapingBee(…), browser: oxylabs(…) }).

ts
function byKind<H extends ProxyProvider, B extends ProxyProvider>(providers: {
  http: H;
  browser: B;
}): ProxyProvider<EndpointOf<H | B>>;

byCountry ​

Picks the provider by the request's geolocation (ISO 3166-1 alpha-2): some networks are better, or cheaper, in some countries. byCountry({ US: oxylabs(…), FI: direct() }, decodo(…)).

ts
function byCountry<R extends Readonly<Record<string, ProxyProvider>>, F extends ProxyProvider>(countries: R, fallback: F): ProxyProvider<EndpointOf<R[keyof R] | F>>;

byIpType ​

Uses datacenter or residential per the request's ipType.

ts
function byIpType<D extends ProxyProvider = never, R extends ProxyProvider = never>(providers: {
  datacenter?: D;
  residential?: R;
  /** When the request sets no `ipType`. Default: `datacenter` if given, else `residential`. */
  default?: "datacenter" | "residential";
}): ProxyProvider<EndpointOf<D | R>>;

weighted ​

Splits traffic by weight, e.g. to try a new provider on 10% of requests: weighted([[current, 9], [candidate, 1]]). Each request picks once (by its id); its retries move on to the next provider, skipping unavailable ones.

ts
function weighted<P extends ProxyProvider>(entries: readonly (readonly [P, number])[]): ProxyProvider<EndpointOf<P>>;

healthy ​

A circuit breaker: after failures consecutive proxy/network errors (a 407 on every login, a gateway outage), the provider is reported unavailable for cooldown seconds, and rotate(), weighted() and adaptive() go straight to the others instead of losing an attempt on it. Bans don't count: they say the site blocked an IP, not that the provider is down. Used alone, an open circuit doesn't block requests (there is nothing to fall back to). The circuit belongs to the instance: use one instance everywhere it appears, and every use shares the circuit (in rotate([h, h, h, x]), h has one), or with a shared store, one key.

ts
function healthy<P extends ProxyProvider>(provider: P, options?: HealthyOptions): ProxyProvider<EndpointOf<P>>;

named ​

The same provider under another name, for reports, rates, circuits, budgets and adaptive(): named('oxylabs-us', oxylabs({ … })). Two instances of one preset in a combinator otherwise share a name, and with it their numbers.

ts
function named<P extends ProxyProvider>(name: string, provider: P): ProxyProvider<EndpointOf<P>>;

budget ​

Caps how many attempts go to a provider per time window, e.g. so a fallback to Zyte can't run up the bill when a site starts blocking the cheaper exits: rotate([residential, budget(zyteApi(…), { max: 500, per: 'hour' })]). Attempts that failed at the network level are given back. Once spent, rotate() and the others skip it; used alone (or when every option is spent), requests fail with a 429 /limits/over-user-limit, retryable later.

ts
function budget<P extends ProxyProvider>(provider: P, options: BudgetOptions): ProxyProvider<EndpointOf<P>>;

watch ​

Asks a provider how it is, ahead of requests: its usage API (credits left) or status page. An exhausted or down provider is reported unavailable, so rotate(), route(), weighted() and adaptive() skip it before an attempt is wasted on it. Checks run in the background every every seconds, never on the request path; until the first one answers, the provider counts as available. Used alone, an unavailable provider is still used (there is nothing to fall back to).

ts
function watch<P extends ProxyProvider>(provider: P, options: WatchOptions): WatchedProvider<EndpointOf<P>>;

AdaptiveOptions ​

Options of adaptive().

ts
interface AdaptiveOptions {
  /** A provider is used for a site while its success score there is at least this. Default 0.5. */
  minSuccess?: number;
  /** Share of requests that re-test a cheaper provider the site had stopped working on. Default 0.05. */
  explore?: number;
  /** How fast scores follow recent outcomes, 0–1. Default 0.2. */
  learningRate?: number;
  /** The key scores are kept by. Default: the host without `www.`. */
  keyOf?: (url: URL) => string;
  /**
   * Shared state: every instance learns from every other's outcomes. Updates go to the store as
   * they happen; each instance re-reads a site's scores in the background every `syncEvery`
   * seconds. Default: this process only.
   */
  store?: StateStore;
  /** The scores' key prefix in the store. Default: the adaptive provider's name. */
  key?: string;
  /** Seconds between background re-reads of a site's shared scores. Default 5. */
  syncEvery?: number;
}

AdaptiveProvider ​

What adaptive() returns: a provider that also reports its learned scores.

ts
type AdaptiveProvider<E extends ProxyEndpoint | ApiEndpoint> = ProxyProvider<E> & {
  /** Current scores, by site and provider (1: always works, 0: never). */
  scores(): Record<string, Record<string, number>>;
};

HealthyOptions ​

Options of healthy(), the circuit breaker.

ts
interface HealthyOptions {
  /**
   * Consecutive failed attempts (proxy or network errors, not bans) that open the circuit. Default
   * 5. An account problem (a bad key, a suspended account) opens it at once.
   */
  failures?: number;
  /** Seconds the circuit stays open. Then one attempt is let through: a failure reopens it. Default 60. */
  cooldown?: number;
  /** Called when the circuit opens (`true`) or closes again (`false`). */
  onChange?: (open: boolean, provider: string) => void;
  /** Shared state: a circuit one instance opens is open for all of them. Default: this process only. */
  store?: StateStore;
  /** The circuit's key in the store. Default: the provider's name. */
  key?: string;
}

BudgetOptions ​

Options of budget().

ts
interface BudgetOptions {
  /** Attempts allowed per window. */
  max: number;
  /** The window: seconds, or `minute`, `hour`, `day`. Windows are fixed (they start on the clock). */
  per: number | "minute" | "hour" | "day";
  /** Shared state: every instance spends from one budget. Default: this process only. */
  store?: StateStore;
  /** The budget's key in the store. Default: the provider's name. */
  key?: string;
}

WatchOptions ​

Options of watch().

ts
interface WatchOptions {
  /** One or more checks; the provider is available while all of them say so. */
  check: ProviderCheck | readonly ProviderCheck[];
  /** Seconds between checks. Default 60. */
  every?: number;
  /** Seconds a check may take. Default 10. */
  timeout?: number;
  /** Shared state: one instance checks, the others reuse its answer until it's `every` old. */
  store?: StateStore;
  /** The result's key in the store. Default: the provider's name. */
  key?: string;
  /** Called when availability changes (and on a first result that is unavailable). */
  onChange?: (result: CheckResult, provider: string) => void;
  /** Called when a check itself fails; the last result stays in place. */
  onError?: (error: unknown, provider: string) => void;
}

WatchedProvider ​

What watch() returns: a provider that also exposes its last check.

ts
type WatchedProvider<E extends ProxyEndpoint | ApiEndpoint> = ProxyProvider<E> & {
  /** The last check's result; undefined before the first one finished. */
  status(): CheckResult | undefined;
  /** Checks now (e.g. at startup, to know before the first request) and resolves with the result. */
  refresh(): Promise<CheckResult | undefined>;
};

Provider checks ​

Checks for watch(): routing.

scrapingBeeUsage ​

ScrapingBee credits: unavailable once fewer than minRemaining are left (default 0). Uses GET app.scrapingbee.com/api/v1/usage (checked against a live account).

ts
function scrapingBeeUsage(options: CheckOptions & {
  apiKey: string;
  minRemaining?: number;
}): ProviderCheck;

zenrowsUsage ​

ZenRows plan usage: unavailable when the subscription isn't active, its usage reaches maxPercent (default 100), or the key's own credit caps are spent. Uses GET api.zenrows.com/v1/subscriptions/self/details (from ZenRows's docs).

ts
function zenrowsUsage(options: CheckOptions & {
  apiKey: string;
  maxPercent?: number;
}): ProviderCheck;

scraperApiUsage ​

ScraperAPI requests: unavailable once requestCount reaches requestLimit. Uses GET api.scraperapi.com/account (from ScraperAPI's docs; not checked against an account).

ts
function scraperApiUsage(options: CheckOptions & {
  apiKey: string;
}): ProviderCheck;

oxylabsUsage ​

Oxylabs residential traffic this month, against your plan's limitGB (the API reports usage, not the limit): unavailable from maxPercent of it (default 100). Logs in to residential-api.oxylabs.io/v2 with the account's main proxy user (the first in the dashboard), which may differ from the user requests go out with. From Oxylabs's docs; not yet checked against an account.

ts
function oxylabsUsage(options: CheckOptions & {
  username: string;
  password: string;
  limitGB: number;
  maxPercent?: number;
}): ProviderCheck;

zyteSpend ​

Zyte API spend this billing cycle, against limitUsd (Zyte has no balance endpoint): unavailable from maxPercent of it (default 100), so a combinator moves to the next provider before Zyte's account runs dry. Reads cost_micro_total from Zyte's Stats API (zyte-api-stats.zyte.com/api/v1/stats), which takes the dashboard API key (not the Zyte API key) and your organizationId (the number in the dashboard URL, /o/000000). The cycle starts on cycleStartDay (1–28, UTC; default 1). The Stats API allows 20 requests a minute: keep watch()'s every at 60 or more. Checked against an account: its total matches /api/stats's cost_microusd_total.

ts
function zyteSpend(options: CheckOptions & {
  apiKey: string;
  organizationId: string | number;
  limitUsd: number;
  maxPercent?: number;
  cycleStartDay?: number;
  /** Default `USD`: the currency your organization is billed in. */
  currency?: string;
  /** Default `https://zyte-api-stats.zyte.com`. */
  baseUrl?: string;
}): ProviderCheck;

statusPage ​

A Statuspage page (https://status.zyte.com, IPRoyal, Webshare…): unavailable while its indicator is one of downOn (default major, critical; minor is a partial degradation).

ts
function statusPage(url: string, options?: CheckOptions & {
  downOn?: readonly string[];
}): ProviderCheck;

allChecks ​

Several checks as one, all of which must pass (what watch() does with an array).

ts
function allChecks(checks: readonly ProviderCheck[]): ProviderCheck;

STATUS_PAGES ​

Status pages known for providers xyte has presets for.

ts
const STATUS_PAGES: Readonly<Record<string, string>>;

ProviderCheck ​

Asks a provider's API or status page how it is. Built-in ones are in checks.

ts
type ProviderCheck = (signal: AbortSignal) => Promise<CheckResult>;

CheckResult ​

What a provider check found: whether to use the provider, and why.

ts
interface CheckResult {
  available: boolean;
  /** Human-readable state, e.g. `820/1000 credits used`. */
  detail?: string;
  /** What's left, in the check's own unit (credits, requests, GB), when it knows. */
  remaining?: number;
}

Shared state ​

Where budgets, circuits, checks, adaptive, sessions, the cache, jobs and stats keep their state: routing, lifecycle for closing a Redis store.

StateStore ​

Where budget(), healthy() and watch() keep their state. In memory (the default) each process counts on its own; with a shared store (Redis) every instance of a deployment sees the same budget, the same open circuits and the same provider checks.

ts
interface StateStore {
  /** Adds `by` (negative to give back) to a counter that expires `ttl` seconds after it's created. */
  incr(key: string, by: number, ttl: number): Promise<number>;
  get(key: string): Promise<string | null>;
  set(key: string, value: string, ttl: number): Promise<void>;
  /**
   * Releases what the store holds open, such as a Redis connection (which, with some clients, keeps
   * the process from exiting). Optional: memory stores and `redisStore(send)` have nothing to close.
   */
  close?(): Promise<void>;
}

memoryStore ​

State in this process. maxEntries caps it, dropping the least recently written first.

ts
function memoryStore(options?: {
  maxEntries?: number;
}): StateStore;

redisStore ​

State in Redis, through any client: pass how to send a raw command.

ts
redisStore((c) => bunRedis.send(c[0]!, c.slice(1)));   // Bun.redis / new Bun.RedisClient(url)
redisStore((c) => ioredis.call(c[0]!, ...c.slice(1))); // ioredis
redisStore((c) => nodeRedis.sendCommand(c));           // node-redis

Or redisStoreFromUrl(url), which picks one of those.

ts
function redisStore(send: RedisSend, options?: {
  prefix?: string;
}): StateStore;

redisStoreFromUrl ​

Redis at url (redis://…) with the client at hand: Bun's built-in one, else the ioredis or redis package if installed. The connection opens on first use; close() (or await using) closes it, which on Node with ioredis or redis is what lets the process exit.

ts
function redisStoreFromUrl(url: string, options?: {
  prefix?: string;
}): StateStore & {
  close(): Promise<void>;
} & AsyncDisposable;

resilient ​

A store that never fails a request: when the shared store errors (Redis down), the operation falls back to process memory and onError hears about it once per outage.

ts
function resilient(store: StateStore, onError?: (error: unknown) => void): StateStore;

RedisSend ​

Sends one Redis command, e.g. ['INCRBY', 'key', '1'], and resolves with its reply.

ts
type RedisSend = (command: string[]) => Promise<unknown>;

Actions ​

Zyte's own actions outside Zyte: actions.

ActionHandlers ​

The actions option: how searchKeyword, setLocation and interaction run outside Zyte. Each takes a handler, or a table by site (PerSite) or by id (ById). See docs/actions.md.

ts
interface ActionHandlers {
  /** `'*'` defaults to `searchBox`, a heuristic that finds the page's search box; `null` (or `false`) turns it off. */
  searchKeyword?: PerSite<SearchKeywordAction> | false;
  /** `'*'` defaults to none: forms to set a location are different on every site. */
  setLocation?: PerSite<SetLocationAction>;
  /** Your scripts by id; `'*'` defaults to none. */
  interaction?: ById<InteractionAction>;
}

ActionHandler ​

Runs one action on the page; throw to fail it (its onError then applies, as for any action).

ts
type ActionHandler<A extends ZyteOnlyAction> = (page: Page, action: A, context: ActionHandlerContext) => Promise<void>;

ActionHandlerContext ​

What an action handler gets besides the page and the action.

ts
interface ActionHandlerContext {
  /** The page's current URL. */
  url: URL;
  /** Milliseconds the action may take: 30 s, within the browser time left (these actions have no `timeout` of their own). */
  timeout: number;
  /** Adds a line to the action's `interactionLogs` in the response, as Zyte does with its scripts' `console.log`. */
  log: (message: string, level?: "debug" | "info" | "warning" | "error") => void;
}

ActionEntry ​

A handler, or null for explicitly none (no fallback, no built-in).

ts
type ActionEntry<A extends ZyteOnlyAction> = ActionHandler<A> | null;

PerSite ​

Handlers by site: keys are hosts (subdomains included, the longest key wins), '*' is every other site. A plain function is the same as { '*': fn }.

ts
type PerSite<A extends ZyteOnlyAction> = ActionHandler<A> | Readonly<Record<string, ActionEntry<A>>>;

ById ​

Handlers by the id the action names; '*' is every other id. A plain function is { '*': fn }.

ts
type ById<A extends ZyteOnlyAction> = ActionHandler<A> | Readonly<Record<string, ActionEntry<A>>>;

SearchKeywordAction ​

Zyte's searchKeyword action: type keyword into the page's search box and submit.

ts
type SearchKeywordAction = Extract<Action, {
  action: "searchKeyword";
}>;

SetLocationAction ​

Zyte's setLocation action: set the site's delivery or store location to an address.

ts
type SetLocationAction = Extract<Action, {
  action: "setLocation";
}>;

InteractionAction ​

Zyte's interaction action: run a script by id (on Zyte, one from its cloud IDE).

ts
type InteractionAction = Extract<Action, {
  action: "interaction";
}>;

ZyteOnlyAction ​

Zyte's own actions, which rely on Zyte's per-site knowledge (searchKeyword, setLocation) or on scripts stored in Zyte's cloud IDE (interaction). Zyte runs them on attempts that reach zyteApi(); elsewhere they run in xyte's browser through handlers: yours (which always run locally), or the built-in heuristic for searchKeyword. Without one, unsupportedActions decides (skip with a warning, pass over to Zyte, or a 400). See docs/actions.md.

ts
type ZyteOnlyAction = SearchKeywordAction | SetLocationAction | InteractionAction;

The default searchKeyword: finds the page's search box (search inputs, role=searchbox, a q/query/search field, a "search" placeholder or label; opening a search toggle first when the box is hidden behind one), types the keyword and presses Enter. Like Zyte's, it fails when the page has no search box; sites it gets wrong can have their own handler.

ts
const searchBox: ActionHandler<SearchKeywordAction>;

isZyteOnlyAction ​

Whether an action is one of Zyte's own three (searchKeyword, setLocation, interaction).

ts
const isZyteOnlyAction: (action: Action) => action is ZyteOnlyAction;

Browser ​

Renders: launching, stealth presets, remote browsers, captchas. Browser.

patchright ​

Patchright: Playwright patched against CDP-based detection (Chromium).

ts
function patchright(): BrowserOptions;

rebrowser ​

rebrowser-playwright: Playwright with the rebrowser patches (Chromium), a drop-in.

ts
function rebrowser(): BrowserOptions;

camoufox ​

Camoufox: a Firefox build that spoofs its fingerprint at the C++ level, through camoufox-js. Options go to Camoufox().

ts
function camoufox(options?: Record<string, unknown>): BrowserOptions;

cloakBrowser ​

CloakBrowser: a Chromium patched at the source level against fingerprinting. Options go to its launch().

ts
function cloakBrowser(options?: Record<string, unknown>): BrowserOptions;

STEALTH_BROWSERS ​

The presets by their XYTE_BROWSER names.

ts
const STEALTH_BROWSERS: Readonly<Record<string, () => BrowserOptions>>;

capsolver ​

CapSolver (capsolver.com): reCAPTCHA v2/v3 and Turnstile, without a proxy of its own. From its docs; not checked against an account.

ts
function capsolver(options: {
  apiKey: string;
  fetch?: typeof fetch;
  timeout?: number;
  pollEvery?: number;
}): CaptchaSolver;

twoCaptcha ​

2Captcha (2captcha.com), API v2: reCAPTCHA v2/v3, hCaptcha and Turnstile. From its docs; not checked against an account.

ts
function twoCaptcha(options: {
  apiKey: string;
  fetch?: typeof fetch;
  timeout?: number;
  pollEvery?: number;
}): CaptchaSolver;

createTaskSolver ​

A solver for services with the createTask / getTaskResult API (CapSolver, 2Captcha, and others that copied it): types maps each kind to the service's task type.

ts
function createTaskSolver(options: {
  name: string;
  endpoint: string;
  apiKey: string;
  types: Partial<Record<CaptchaKind, string>>;
  /** Seconds between result polls. Default 3. */
  pollEvery?: number;
  /** Seconds before giving up. Default 120. */
  timeout?: number;
  fetch?: typeof fetch;
}): CaptchaSolver;

CaptchaSolver ​

Solves captchas for browser renders (browser.captcha): capsolver(), twoCaptcha(), or your own.

ts
interface CaptchaSolver {
  readonly name: string;
  /** Resolves with the token the widget would have produced. */
  solve(task: CaptchaTask, signal?: AbortSignal): Promise<string>;
}

CaptchaTask ​

A captcha found on a page, as a solver gets it.

ts
interface CaptchaTask {
  kind: CaptchaKind;
  siteKey: string;
  /** The page the widget is on. */
  url: string;
  /** reCAPTCHA v3 / Turnstile `action`, when the widget declares one. */
  action?: string;
}

CaptchaKind ​

Captcha solving in browser renders: after the page loads (before your actions), a reCAPTCHA, hCaptcha or Cloudflare Turnstile widget on it is sent to a solving service, and its token is put where the page's form expects it, calling the widget's callback. Cloudflare's full-page "Just a moment…" challenge is a different mechanism (cookies and fingerprint), not handled here.

ts
type CaptchaKind = "recaptchaV2" | "recaptchaV3" | "hcaptcha" | "turnstile";

CaptchaOutcome ​

What happened to a captcha found on the page (in the attempt report).

ts
interface CaptchaOutcome {
  kind: CaptchaKind;
  solver: string;
  solved: boolean;
  ms: number;
  error?: string;
}

BrowserOptions ​

The browser option: how renders run. A local Chromium by default; launch for your own (Patchright, Camoufox…: see the stealth presets), connect for a hosted browser, plus page hooks, stealth and captcha solving. See docs/browser.md.

ts
interface BrowserOptions {
  /**
   * The Playwright-compatible browser type to launch. Default: `chromium` from `playwright`
   * (an optional peer dependency). Swap in a fork for stealth, e.g.
   * `(await import('patchright')).chromium`, `rebrowser-playwright`, or `playwright-extra`
   * with its stealth plugin.
   */
  browserType?: BrowserTypeLike | (() => BrowserTypeLike | Promise<BrowserTypeLike>);
  /** Options for `browserType.launch()`: `headless`, `channel: 'chrome'`, `args`, `executablePath`… */
  launchOptions?: LaunchOptions;
  /** Full control over the shared browser (overrides `browserType` / `launchOptions`). */
  launch?: () => Promise<unknown>;
  /**
   * Render each request in a remote browser instead: Bright Data Scraping Browser, Browserless,
   * Browserbase… Called per attempt with the country and session, so the endpoint can carry
   * them. The connection is closed after the request.
   */
  connect?: (context: ProxyContext) => string | RemoteBrowser | Promise<string | RemoteBrowser>;
  /**
   * Adjust the context options per request. Receives xyte's defaults (proxy, viewport, user
   * agent, locale…); return the options to use. Patchright, for instance, recommends dropping
   * the custom `userAgent` and `viewport`.
   */
  contextOptions?: (defaults: BrowserContextOptions, info: BrowserHookInfo) => BrowserContextOptions | Promise<BrowserContextOptions>;
  /** Runs on every new page before navigation: init scripts, routes, stealth patches. */
  onPage?: (page: Page, info: BrowserHookInfo) => void | Promise<void>;
  /**
   * Resource types to abort, to save proxy bandwidth (residential traffic is billed per GB),
   * e.g. `['image', 'media', 'font']`.
   */
  blockResources?: readonly string[];
  /** Total browser time per request, seconds. Zyte's limit is 60. */
  timeLimit?: number;
  /**
   * Solves reCAPTCHA, hCaptcha and Turnstile widgets found after the page loads, before the
   * actions: `capsolver({ apiKey })`, `twoCaptcha({ apiKey })`, or your own `CaptchaSolver`.
   */
  captcha?: CaptchaSolver;
  /**
   * After load and the actions, wait up to this many seconds for the network to go quiet, so
   * data the page fetches after `load` lands in `browserHtml` and `networkCapture`, as on Zyte.
   * Default 3; 0 returns right after load.
   */
  settleTime?: number;
}

BrowserHookInfo ​

What hooks get to see about the request being rendered.

ts
interface BrowserHookInfo {
  request: EngineAttempt["request"];
  proxy: ProxyEndpoint | null;
  context: ProxyContext;
}

BrowserTypeLike ​

The part of a Playwright BrowserType xyte uses. Typed loosely on purpose: forks (patchright, rebrowser-playwright…) declare their own, nominally different, Browser types.

ts
interface BrowserTypeLike {
  launch(options?: any): Promise<unknown>;
  connect(wsEndpoint: string, options?: any): Promise<unknown>;
  connectOverCDP(endpointURL: string, options?: any): Promise<unknown>;
}

RemoteBrowser ​

A remote browser to render one request in.

ts
interface RemoteBrowser {
  /** `wss://…` (or `http://…` for CDP discovery). */
  endpoint: string;
  /** `cdp` (default) for `connectOverCDP`, `playwright` for a Playwright server (`connect`). */
  protocol?: "cdp" | "playwright";
  headers?: Record<string, string>;
  /**
   * Also route the browser through xyte's `proxy`. Default false: remote browser services
   * pick their own exit (set the country in the endpoint).
   */
  useProxy?: boolean;
}

Extraction ​

AI extraction fields and Markdown: extraction.

Extractor ​

Computes one AI-extraction output (e.g. product) from the downloaded page.

ts
type Extractor = (input: {
  url: string;
  html: string;
  request: ExtractRequest;
  /** Options of the field, e.g. `productOptions`. */
  options: unknown;
}) => unknown | Promise<unknown>;

llmExtractors ​

Extractors for new Xyte({ extractors: llmExtractors({ complete }) }). Each runs one model call per requested field.

ts
function llmExtractors(options: LlmExtractorOptions): Partial<Record<ExtractionField, Extractor>>;

openAICompatible ​

complete for any OpenAI-compatible chat completions API:

openAICompatible({ baseURL: 'https://openrouter.ai/api/v1', apiKey, model: 'openai/gpt-5-mini' })
openAICompatible({ baseURL: 'http://localhost:11434/v1', model: 'qwen3' })   // Ollama
ts
function openAICompatible(options: {
  model: string;
  apiKey?: string;
  /** Default `https://api.openai.com/v1`. */
  baseURL?: string;
  headers?: Record<string, string>;
  temperature?: number;
  /** Ask for `response_format: json_schema` (default) or plain `json_object` for servers without it. */
  responseFormat?: "json_schema" | "json_object";
  fetch?: typeof fetch;
}): Complete;

Complete ​

Calls your model: gets a CompletionRequest, returns the answer (text, an object, or a Completion). openAICompatible() makes one for any OpenAI-compatible API.

ts
type Complete = (request: CompletionRequest) => Promise<string | object | Completion>;

CompletionRequest ​

One model call llmExtractors() makes: a prompt for one extraction field, and the schema to answer in.

ts
interface CompletionRequest {
  /** The extraction field, e.g. `product`. */
  field: ExtractionField;
  system: string;
  prompt: string;
  /** JSON Schema the answer must match. */
  schema: object;
}

Completion ​

A model's answer, with its token usage if known.

ts
interface Completion {
  /** The model's answer: JSON text (code fences are tolerated) or an already-parsed object. */
  output: string | object;
  usage?: {
    inputTokens?: number;
    outputTokens?: number;
  };
}

LlmExtractorOptions ​

Options of llmExtractors().

ts
interface LlmExtractorOptions {
  complete: Complete;
  /** Fields to provide. Default: all AI fields (product…pageContent, serp, customAttributes). */
  fields?: readonly ExtractionField[];
  /** Page text sent to the model, in characters (~4 per token). Default 60 000. */
  maxInputChars?: number;
}

parseJson ​

Parses a model answer, tolerating ```json fences and text around the object.

ts
function parseJson(output: string | object): Record<string, unknown>;

htmlToMarkdown ​

The built-in markdown extraction: HTML to Markdown, the main content only (Zyte's default) or the whole page. Relative links and images are resolved against url.

ts
function htmlToMarkdown(html: string, options?: MarkdownOptions): string;

MarkdownOptions ​

Options of htmlToMarkdown().

ts
interface MarkdownOptions {
  /** Base for relative links and images. */
  url?: string;
  /** Zyte's `markdownOptions.scope`. Default `mainContent`. */
  scope?: "mainContent" | "fullPage";
}

OUTPUT_SCHEMAS ​

Zyte's JSON Schema for each AI extraction field (product, article…), as llmExtractors() asks models to answer in.

ts
const OUTPUT_SCHEMAS: Record<OutputSchemaField, object>;

OutputSchemaField ​

The extraction fields with an output schema.

ts
type OutputSchemaField = "product" | "productList" | "productNavigation" | "article" | "articleList" | "articleNavigation" | "forumThread" | "jobPosting" | "jobPostingNavigation" | "pageContent" | "serp";

Reports, costs, cache and pacing ​

What each request took and cost: observability; the cache and rate limits: configuration.

RequestReport ​

What one extract() took and how it ended: onRequest / onReport.

ts
interface RequestReport {
  requestId: string;
  url: string;
  host: string;
  /** The request's `tags` (Zyte's way to label requests for its stats), for grouping reports. */
  tags?: ExtractRequest["tags"];
  /** The request's `jobId`, if it had one. */
  jobId?: string;
  /** `ok`: a response came back (the target's own 4xx/5xx included); `error`: an `XyteError`; `aborted`: cancelled. */
  outcome: "ok" | "error" | "aborted";
  /** The target's status, when a response came back. */
  statusCode?: number;
  /** The problem it failed with. */
  error?: XyteProblem;
  /** The provider that served the response (the last attempt's). */
  provider?: string;
  /** The last attempt rendered in a browser. */
  browser: boolean;
  /** `browserFallback` moved it from HTTP to a browser. */
  escalated: boolean;
  /** Milliseconds waiting for a `concurrency` slot, then working. */
  queuedMs: number;
  /**
   * With `cache`: `hit` (answered from the cache: no attempts), `shared` (an identical request
   * in flight answered it), or `miss` (fetched, and stored if cacheable).
   */
  cache?: "hit" | "shared" | "miss";
  ms: number;
  attempts: AttemptInfo[];
  /** Totals over the attempts. */
  transferBytes: number;
  /** With `rates`: what the request cost, in USD (0 for a cache hit). */
  costUsd?: number;
  byProvider: Record<string, ProviderUsage>;
  warnings: string[];
}

AttemptInfo ​

One attempt of an extract(): which provider, how it went, how long, how many bytes, what it cost. Passed to onAttempt, and listed in RequestReport.attempts.

ts
interface AttemptInfo extends AttemptOutcome {
  /** The `extract()` call it belongs to (same id as its `RequestReport`). */
  requestId: string;
  url: string;
  /** 0 for the first try. */
  attempt: number;
  /** The combinators' choices down to `provider`, outermost first, e.g. `['rotate(oxylabs-residential,zyte-api)', 'zyte-api']`. */
  path: string[];
  /** The provider that served the attempt: the leaf a combinator chose, e.g. `oxylabs-residential`. */
  provider: string;
  /** Rendered in a browser (requested, or `browserFallback`). */
  browser: boolean;
  /** The request's `tags` (as on Zyte, e.g. `{ spider: 'yelp' }`), to group attempts by in metrics. */
  tags?: ExtractRequest["tags"];
  /** With `rates`: what this attempt cost, in USD. */
  costUsd?: number;
}

ProviderUsage ​

Attempts and bytes of one provider within a request (or across reports, summed).

ts
interface ProviderUsage {
  attempts: number;
  ok: number;
  ban: number;
  error: number;
  aborted: number;
  /** Bytes on the wire (`transferBytes`), summed. */
  transferBytes: number;
  /** Time spent in its attempts, summed. */
  ms: number;
  /** With `rates`: what its attempts cost, in USD. */
  costUsd?: number;
}

usageByProvider ​

Sums attempts per provider.

ts
function usageByProvider(attempts: readonly AttemptInfo[]): Record<string, ProviderUsage>;

Rate ​

What a provider costs you. All in USD; any combination.

ts
interface Rate {
  /** Per GB on the wire (`transferBytes`): residential and datacenter proxies. */
  perGB?: number;
  /** Per attempt that got an answer (`ok` or `ban`): APIs that bill every request. */
  perRequest?: number;
  /** Per `ok` attempt only: APIs that bill successful requests. */
  perSuccess?: number;
}

Rates ​

Your prices by provider name (oxylabs, zyte-api…); a name also covers its variants (oxylabs → oxylabs-residential).

ts
type Rates = Readonly<Record<string, Rate>>;

rateFor ​

The rate for a provider: its exact name, else the longest key it starts with (oxylabs → oxylabs-residential).

ts
function rateFor(rates: Rates, provider: string): Rate | undefined;

attemptCost ​

What one attempt cost at rate. Errors and pass-overs got no answer: nothing to bill.

ts
function attemptCost(rate: Rate | undefined, attempt: Pick<AttemptOutcome, "result" | "transferBytes">): number;

CacheOptions ​

The cache option: keep responses for a while and answer repeated requests from them.

ts
interface CacheOptions {
  /**
   * Seconds a response stays fresh: one value for every site, or per site keyed by host
   * (subdomains included, the longest key wins) with `'*'` for the rest. 0 doesn't cache.
   */
  ttl: number | Readonly<Record<string, number>>;
  /** Where responses are kept. Default: this process's memory (1,000 responses). Redis shares them across instances. */
  store?: StateStore;
  /** Which responses to keep. Default: the target answered below 400, or 404 / 410 (gone stays gone). */
  cacheable?: (response: ExtractResponse) => boolean;
  /** The cache key. Default: a hash of the request without `tags`, `jobId` and `echoData`, which don't change the page. */
  key?: (request: ExtractRequest) => string;
}

requestKey ​

The cache's default key: a hash of the request without its labels (tags, jobId, echoData), so the same request with different tags is a hit.

ts
function requestKey(request: ExtractRequest): string;

RateLimits ​

Keyed by host (subdomains included, the longest key wins), '*' for every other site, each on its own ('*': '10/s' is 10 per second per site). Values: '2/s', '120/m', '1000/h', '5/10s', or a SiteLimit.

ts
type RateLimits = Readonly<Record<string, string | SiteLimit>>;

SiteLimit ​

rate attempts per per seconds, at most concurrency at once.

ts
interface SiteLimit {
  rate?: number;
  /** Seconds. Default 1. */
  per?: number;
  concurrency?: number;
}

parseLimit ​

Parses a rate limit like '2/s', '120/m' or '5/10s' into { rate, per } (per in seconds).

ts
function parseLimit(value: string | SiteLimit): SiteLimit;

Server ​

Serving Zyte's APIs: deploy, mounting, proxy mode.

createHandler ​

A Web-standard handler serving Zyte's HTTP API: POST /v1/extract. Mount it with Bun.serve, Deno, Hono, a Cloudflare Worker… then point existing Zyte code at it by changing only the base URL (https://api.zyte.com/v1/extract → http://localhost:8011/v1/extract).

ts
function createHandler(xyte: Xyte, options?: HandlerOptions): FetchHandler;

HandlerOptions ​

Options of createHandler().

ts
interface HandlerOptions {
  /**
   * Accepted API keys, sent like Zyte's: HTTP basic auth with the key as the username. When
   * empty or omitted, any (or no) key is accepted.
   */
  apiKeys?: readonly string[];
  /**
   * Where the routes live, e.g. `/zyte` for `/zyte/v1/extract`. Default: any path ending in
   * `/v1/extract` (or `/healthz`), so the handler works mounted under a prefix whether or not the
   * framework strips it.
   */
  basePath?: string;
  /** Largest accepted request body. Default 10 MB. */
  maxBodyBytes?: number;
  /**
   * Reject targets on loopback, private, link-local and cloud-metadata addresses (IP literals,
   * `localhost`, `*.local`, `*.internal`), so a public endpoint can't be aimed at your internal
   * network. Hostnames that *resolve* to private IPs are not caught; with a proxy provider, the
   * proxy's network is what's reachable anyway. Default false; `xyte serve` turns it on.
   */
  blockPrivateTargets?: boolean;
  /**
   * `POST /v1/jobs` (a Zyte request plus an optional `callbackUrl`) answers `202 { id, status }`
   * at once; `GET /v1/jobs/{id}` returns the job, with the Zyte response when `done`; the
   * result is POSTed to `callbackUrl` when it finishes. Options: where jobs are kept, for how
   * long, the webhook secret. `false` turns the routes off. On by default, in memory.
   */
  jobs?: JobsOptions | false;
  /**
   * `GET /api/stats` and `/api/v1/stats` in the shape of Zyte's Stats API, from the requests this
   * handler served (extract and jobs). Pass a recorder to share it (e.g. with a Redis store, or to
   * also `record()` library requests); `false` turns the routes off. On by default, in memory.
   */
  stats?: StatsRecorder | false;
}

FetchHandler ​

A Web-standard request handler, as Bun.serve, Deno, Hono and Workers take.

ts
type FetchHandler = (request: Request) => Promise<Response>;

toNodeHandler ​

Adapts a fetch handler to Node's (req, res) signature, for node:http, Express, Fastify (reply.hijack() first) or Next.js API routes:

app.use('/zyte', toNodeHandler(createHandler(xyte)));

A body already parsed by middleware (e.g. express.json()) is re-serialized.

ts
function toNodeHandler(handler: FetchHandler, options?: {
  maxBodyBytes?: number;
}): (req: IncomingMessage & {
  body?: unknown;
}, res: ServerResponse) => Promise<void>;

serve ​

Serves handler on Bun (Bun.serve) or Node.js (node:http). Returns a function that stops the server.

ts
function serve(handler: FetchHandler, options?: {
  port?: number;
  hostname?: string;
}): Promise<{
  url: string;
  stop: () => Promise<void>;
}>;

isPrivateTarget ​

Loopback, RFC 1918, link-local (incl. 169.254.169.254), CGNAT, ULA, and internal names.

ts
function isPrivateTarget(target: string): boolean;

Jobs ​

Runs requests in the background (POST /v1/jobs): submit() answers at once with a job, get() reads it later, and a callbackUrl gets the result POSTed, signed (see verifyWebhook).

ts
class Jobs {
  constructor(xyte: Xyte, options?: JobsOptions);
  /** Queues a request and starts it in the background; resolves with the queued job. */
  submit(request: ExtractRequest, callbackUrl?: string): Promise<Job>;
  /** The job as it is now; undefined if unknown, or older than `ttl`. */
  get(id: string): Promise<Job | undefined>;
}

Job ​

An asynchronous job, as POST /v1/jobs and GET /v1/jobs/{id} return it.

ts
interface Job {
  id: string;
  status: "queued" | "running" | "done" | "failed";
  createdAt: string;
  startedAt?: string;
  finishedAt?: string;
  /** The Zyte response, when `done`. */
  response?: ExtractResponse;
  /** The Zyte problem, when `failed`. */
  error?: XyteProblem;
  callbackUrl?: string;
  /** How the webhook delivery went. */
  callback?: {
    status: "pending" | "delivered" | "failed";
    attempts: number;
    lastError?: string;
  };
}

JobsOptions ​

Asynchronous extraction: submit a request, get an id back at once, and fetch the result later or have it POSTed to a webhook. For renders that outlast an HTTP timeout, and for queues.

ts
interface JobsOptions {
  /** Where jobs are kept. Default: this process's memory. Redis lets any instance answer a job's status. */
  store?: StateStore;
  /** Seconds a job (and its result) is kept. Default 86400 (a day). */
  ttl?: number;
  /** Signs webhook deliveries: `X-Xyte-Signature: sha256=<hex HMAC-SHA256 of the body>`. */
  webhookSecret?: string;
  /** Webhook delivery attempts. Default 3, backing off 1 s, 2 s… */
  webhookAttempts?: number;
  /** For webhook deliveries (tests, proxies). */
  fetch?: typeof fetch;
  /** Each job's request report (the server feeds its stats with it). */
  onReport?: (report: RequestReport) => void;
}

verifyWebhook ​

Verifies a webhook delivery's X-Xyte-Signature against the raw body.

ts
function verifyWebhook(body: string, signature: string | null | undefined, secret: string): boolean;

StatsRecorder ​

Records each request's report and answers Zyte's Stats API queries over them (GET /api/stats). Keeps hourly buckets in a store, so with Redis every instance reports the same totals.

ts
class StatsRecorder {
  constructor(options?: StatsOptions);
  /** Adds a request; flushed to the store within a second. */
  record(report: RequestReport, at?: Date): void;
  /** Writes pending counts to the store (read, add, write per hour). */
  flush(): Promise<void>;
  /**
   * Stats in Zyte's shape. `currencies: 'v1'` gives `/api/v1/stats`'s per-currency `cost_micro_*`
   * arrays; otherwise `/api/stats`'s `cost_microusd_*` strings. Throws a 422-style Error for an
   * invalid query.
   */
  query(query: StatsQuery, format?: "usd" | "v1"): Promise<StatsResponse>;
}

StatsOptions ​

Usage stats in the shape of Zyte's Stats API (https://docs.zyte.com/zyte-api/usage/stats/), built from request reports: so a dashboard or a billing check written for Zyte reads xyte's numbers by changing its base URL. record() takes RequestReports (onRequest); query() answers /api/stats and /api/v1/stats parameters.

ts
interface StatsOptions {
  /** Where hourly buckets are kept. Default: this process's memory. Redis adds up every instance. */
  store?: StateStore;
  /** Days of stats kept. Default 31. */
  retentionDays?: number;
  /** The `organization_id` reported. Default 0; a query's own is echoed back. */
  organizationId?: number;
}

StatsQuery ​

The query parameters of GET /api/stats, as on Zyte.

ts
interface StatsQuery {
  organization_id?: string | number;
  start_time?: string;
  end_time?: string;
  page?: string | number;
  page_size?: string | number;
  /** Comma-separated hosts. */
  domains?: string;
  /** Comma-separated codes (`200,404`). */
  response_codes?: string;
  /** Comma-separated `tag:value` (that value) or `tag` (present), all required. */
  tags?: string;
  groupby_time?: "hour" | "day" | "month" | "year" | null;
  groupby_domain?: string | boolean;
  include_domain_health?: string | boolean;
}

StatsResponse ​

The Stats API's answer, in Zyte's shape.

ts
interface StatsResponse {
  page: number;
  page_size: number;
  results: Record<string, unknown>[];
  total_result_count: number;
}

StatsQueryError ​

An invalid Stats API query: the server answers it with a 400.

ts
class StatsQueryError extends Error {}

serveProxyMode ​

A proxy-mode server on its own port.

ts
function serveProxyMode(xyte: Xyte, options?: ProxyModeOptions & {
  port?: number;
  hostname?: string;
}): Promise<{
  port: number;
  caCert: string;
  stop: () => Promise<void>;
}>;

attachProxyMode ​

Adds proxy mode to a node:http server: absolute-URL requests (http://) and CONNECT tunnels (https://, decrypted with per-site certificates from the CA). Returns the CA's certificate.

ts
function attachProxyMode(server: Server, xyte: Xyte, options?: ProxyModeOptions): {
  caCert: string;
};

ProxyModeOptions ​

Zyte API's proxy mode (https://docs.zyte.com/zyte-api/usage/proxy-mode.html), served by xyte: clients use it as an HTTP proxy (curl -x xyte:8012 -U key: --cacert ca.crt https://site), options go in Zyte-* request headers, and the answer is the site's own response. HTTPS is decrypted with a certificate signed by the proxy's CA, which clients must trust, as with Zyte's.

ts
interface ProxyModeOptions {
  /** Accepted keys, as the proxy username (`Proxy-Authorization: Basic base64(key:)`). Empty: anyone. */
  apiKeys?: readonly string[];
  /** The CA that signs per-site certificates. Default: a new one (clients must trust it, so keep one). */
  ca?: CertificateAuthority;
  /** Reject targets on loopback, private and metadata addresses. */
  blockPrivateTargets?: boolean;
  /** Each request's report (for stats or logs). */
  onReport?: (report: RequestReport) => void;
}

proxiedRequest ​

The Zyte request a proxied request stands for: its URL, method, body, headers and Zyte-* options.

ts
function proxiedRequest(url: string, method: string, headers: IncomingMessage["headers"], body: Buffer): ExtractRequest;

createCertificateAuthority ​

A new CA (10 years). Keep it: clients trust this one certificate.

ts
function createCertificateAuthority(commonName?: string): CertificateAuthority;

issueCertificate ​

A certificate for host (a name or an IP), signed by the CA, valid for a year.

ts
function issueCertificate(ca: CertificateAuthority, host: string): {
  cert: string;
  key: string;
};

CertificateAuthority ​

A certificate authority for proxy mode, as Zyte has one: to apply bans, retries and providers to HTTPS traffic, the proxy must read it, so it presents a certificate for each site signed by this CA, which clients are told to trust. Built with node:crypto (P-256 keys, ECDSA-SHA256) and a minimal DER encoder, no dependencies.

ts
interface CertificateAuthority {
  /** PEM certificate: what clients install (curl --cacert, NODE_EXTRA_CA_CERTS, REQUESTS_CA_BUNDLE…). */
  cert: string;
  /** PEM private key: keep it secret; anyone holding it can impersonate sites to those clients. */
  key: string;
}

Configuration from the environment ​

What the CLI builds from process.env; each takes an env to read instead. Configuration.

proxyFromEnv ​

Builds the proxy from environment variables:

  • XYTE_PROXY_URL: one or more proxy URLs, comma-separated (rotated across retries), or
  • XYTE_PROXY_TEMPLATE: a template() format, e.g. http://user-{env:MY_USER}[-country-{country:lower}]-session-{session}:{env:MY_PASS}@host:7000, or
  • XYTE_PROVIDER: one or more provider names, comma-separated (a retry moves to the next one), each with its credentials (OXYLABS_USERNAME, ZENROWS_API_KEY, …). See PROVIDER_NAMES and docs/providers.md, or
  • XYTE_ADAPTIVE: provider names, cheapest first, for adaptive().

Neither set: no proxy. On top of that:

  • XYTE_ROUTES=doordash.com=zyte-api,wolt.com=direct,restaurantguru.com=oxylabs+zyte-api: a provider per site (route()), + rotating; other sites use the above.
  • XYTE_BROWSER_PROVIDER: provider names for browser renders (byKind()).
  • XYTE_CIRCUIT_BREAKER=5,60: wraps every named provider in healthy() (failures, cooldown s).
  • XYTE_BUDGET=zyte-api=500/hour,scrapingbee=1000/day: wraps those providers in budget().
  • XYTE_WATCH=true: wraps providers that have a usage API or status page in watch() (XYTE_WATCH_EVERY seconds, default 60). Oxylabs needs OXYLABS_LIMIT_GB (and, if the proxy user isn't the account's main one, OXYLABS_API_USERNAME / OXYLABS_API_PASSWORD). Zyte's spend is checked too with ZYTE_STATS_API_KEY (the dashboard key), ZYTE_ORGANIZATION_ID and ZYTE_LIMIT_USD (optional ZYTE_MAX_PERCENT, ZYTE_CYCLE_START_DAY).
  • XYTE_REDIS_URL=redis://…: budgets, circuits and checks are shared by every instance.

A provider named in several places is one instance, so its budget and circuit are shared.

ts
function proxyFromEnv(env?: Env): ProxyProvider;

browserFromEnv ​

Browser settings from environment variables:

  • XYTE_BROWSER: playwright (default), a stealth preset (patchright, rebrowser, camoufox, cloakbrowser), or any installed Playwright-compatible package name exporting chromium.
  • XYTE_BROWSER_CHANNEL: e.g. chrome to use installed Google Chrome.
  • XYTE_BROWSER_HEADLESS=false: headed (needs a display, e.g. xvfb).
  • XYTE_BROWSER_ARGS: extra Chromium flags, comma-separated.
  • XYTE_REMOTE_BROWSER: a hosted browser preset: brightdata, browserless, zenrows, oxylabs, nodemaven (with its credentials, see docs/browser.md), or
  • XYTE_BROWSER_WS: render in a remote browser at this CDP endpoint instead; accepts the same placeholders as XYTE_PROXY_TEMPLATE (e.g. {country:lower}, {session}). XYTE_BROWSER_WS_PROTOCOL=playwright for a Playwright server.
  • XYTE_BLOCK_RESOURCES: resource types to abort, e.g. image,media,font.
  • XYTE_BROWSER_TIME_LIMIT: seconds per request (default 60).
  • XYTE_BROWSER_SETTLE: seconds to wait for the network to go quiet after load (default 3, 0 to skip).
ts
function browserFromEnv(env?: Env): BrowserOptions;

extractorsFromEnv ​

AI extraction from environment variables (any OpenAI-compatible API): XYTE_LLM_MODEL (required to enable), XYTE_LLM_BASE_URL (default OpenAI), XYTE_LLM_API_KEY, XYTE_LLM_RESPONSE_FORMAT=json_object for servers without JSON-schema output. Unset: no AI extraction (those fields get a 400, as without an extractor).

ts
function extractorsFromEnv(env?: Env): Partial<Record<ExtractionField, Extractor>> | undefined;

cacheFromEnv ​

The response cache from XYTE_CACHE_TTL: seconds for every site (3600), or per site (wolt.com=600,*=3600). Shared through XYTE_REDIS_URL when set. Unset: no cache.

ts
function cacheFromEnv(env?: Env): CacheOptions | undefined;

rateLimitsFromEnv ​

Per-site pacing from XYTE_RATE_LIMITS: yelp.co.uk=2/s,doordash.com=30/m,*=10/s.

ts
function rateLimitsFromEnv(env?: Env): RateLimits | undefined;

ratesFromEnv ​

Your prices from XYTE_RATES: oxylabs=6/GB,zyte-api=0.0013/request,scrapingbee=0.0005/success, + to combine units for one provider (x=2/GB+0.001/request).

ts
function ratesFromEnv(env?: Env): Rates | undefined;

sessionsFromEnv ​

The sessionContext pool from the environment: shared through XYTE_REDIS_URL when set.

ts
function sessionsFromEnv(env?: Env): SessionPoolOptions | undefined;

jobsFromEnv ​

The server's job options: XYTE_WEBHOOK_SECRET, XYTE_JOBS_TTL (seconds), kept in XYTE_REDIS_URL when set.

ts
function jobsFromEnv(env?: Env): JobsOptions;

statsFromEnv ​

The server's stats: kept in XYTE_REDIS_URL when set (every instance adds up), else in memory.

ts
function statsFromEnv(env?: Env): StatsRecorder;

caFromEnv ​

Proxy mode's CA: XYTE_CA_CERT / XYTE_CA_KEY (PEM, or paths to PEM files), else the one in XYTE_CA_DIR (default ~/.xyte), created there on first use. It must stay the same across restarts and instances: clients trust this certificate.

ts
function caFromEnv(env?: Env): Promise<{
  ca: CertificateAuthority;
  source: string;
}>;

closeSharedStores ​

Closes the Redis connections the *FromEnv helpers opened for XYTE_REDIS_URL (one per URL, shared by all of them). Call it on shutdown, after xyte.close(): on Node with ioredis or redis, an open connection keeps the process from exiting.

ts
function closeSharedStores(): Promise<void>;

Tools ​

xyte doctor's checks, as functions.

doctor ​

Checks each provider end to end: its credentials are there, a request through it works (and exits where it should), and its quota or status, where it has a usage API or status page.

ts
function doctor(options?: DoctorOptions): Promise<DoctorResult[]>;

formatDoctor ​

The results as a table for a terminal.

ts
function formatDoctor(results: readonly DoctorResult[]): string;

DoctorOptions ​

Options of doctor(): what to check, and how.

ts
interface DoctorOptions {
  env?: Record<string, string | undefined>;
  /** Provider names (`XYTE_PROVIDER` names, or `direct`). Default: `direct` plus every one the environment configures. */
  providers?: readonly string[];
  /** What to fetch through each. Default `https://ipinfo.io/json`, which reports the exit IP and country. */
  url?: string;
  /** `geolocation` for the probe, to check providers apply it. */
  country?: string;
  /** Seconds per probe. Default 30. */
  timeout?: number;
}

DoctorResult ​

What doctor() found for one provider.

ts
interface DoctorResult {
  provider: string;
  ok: boolean;
  /** The exit the probe saw (with the default probe URL, ipinfo.io's answer). */
  ip?: string;
  country?: string;
  city?: string;
  org?: string;
  ms?: number;
  /** Why it failed: missing credentials, rejected credentials, unreachable… */
  error?: string;
  /** A note that doesn't fail it, e.g. the exit country isn't the one asked for. */
  note?: string;
  /** Its usage API or status page, where it has one. */
  quota?: CheckResult | {
    error: string;
  };
}

MIT licensed. A personal project, not affiliated with Zyte Group Ltd (disclaimer).

Copyright © 2026 Santiago Montoya