Using Proxlane from an agent
How to give an AI agent a scraping tool that fails honestly.
Agents are bad at ambiguous failure. Give one a scraper that returns 200 with a challenge page and it will summarise the challenge page. Proxlane exists to make the failure legible, which is exactly what an agent needs.
The one rule
Branch on X-Outcome-Class, and give the agent a different instruction per class.
const res = await fetch(
`${GATEWAY}/v1?url=${encodeURIComponent(url)}`,
{ headers: { Authorization: `Bearer ${key}` } },
);
switch (res.headers.get('x-outcome-class')) {
case 'ok':
return { text: await res.text() };
case 'target':
// The site answered. Do not retry, do not try another tool.
return { error: 'page does not exist or refused access', retryable: false };
case 'blocked':
case 'provider':
case 'gateway':
// Proxlane already failed over. Back off, do not loop.
return { error: 'temporarily unavailable', retryable: true };
case 'client':
return { error: 'the URL was rejected as invalid', retryable: false };
}
Six classes, and the set never grows. New outcomes land inside existing classes, so this switch does not need revisiting.
Why retryable matters
An agent that cannot tell "blocked" from "not found" does one of two bad things. It retries a 404 until it runs out of budget, or it gives up on a page that was merely blocked once.
X-Outcome-Class answers that in one string. X-Outcome tells you why, if you want it in
the log.
Tell the agent what it got
Pass the outcome through to the model rather than swallowing it. A tool result of
{ "ok": false, "outcome": "HARD_BLOCK", "attempts": 3, "retryable": true }
produces better behaviour than an empty string, because the model can say "that site is blocking us" instead of inventing a summary of nothing.
X-Attempts and X-Cost-Estimate are worth logging for the same reason. An agent loop that
quietly costs three times what you expected is a failover loop nobody surfaced.
From the command line
The CLI is a usable tool surface for agents that shell out.
proxlane scrape https://example.com --provider=scraperapi --json
--json is the point. Every command supports it, and it is the form a tool wrapper should
parse. Exit codes distinguish usage errors from scrape failures.
proxlane outcomes --json # the full taxonomy, for building a retry policy
proxlane providers --json # which providers are configured
proxlane doctor --json # diagnose a deployment
proxlane scrape calls a provider directly using keys from the environment. It does not go
through the gateway, so there is no failover. It is a debugging and inspection tool, not a
production path.
Machine-readable docs
llms.txt at the site root lists every documentation page in a form intended for models:
https://proxlane.dev/llms.txt
What does not exist yet
There is no MCP server and no published SDK. @proxlane/sdk is a placeholder.
Use the HTTP endpoint. It is a single GET, it needs no client library, and the headers are the interface.