Scrape

The Scrape API provides one-click web scraping without manually managing Session lifecycles. The system automatically creates a disposable Session, navigates to the target URL, waits for the page to load, returns LLM-ready content, and cleans up resources. Usage counts against your plan minute quota like a normal Session.

Scrape a Web Page

POST /v1/scrape
X-API-Key: <api_key>

{
  "url": "https://example.com/blog/post",   // required
  "formats": ["markdown", "links"],         // preferred over legacy format
  "onlyMainContent": true,                  // optional; mutex with selector
  "waitFor": "networkidle"                  // load | domcontentloaded | networkidle
  // "selector": "#main",                   // optional CSS subtree
  // "proxy": { "type": "http", "host": "...", "port": 8080 }
}

# Returns
{
  "url": "https://example.com/blog/post",
  "statusCode": 200,
  "content": "# Post title\n\nBody...",     // primary content (first content format)
  "markdown": "# Post title\n\nBody...",
  "links": ["https://example.com/..."],
  "metadata": {
    "title": "Post title",
    "description": "...",
    "ogImage": null,
    "contentProfile": "article",
    "mainContentFallback": false
  },
  "tokensEstimate": {
    "contentChars": 4200,
    "approxTokens": 1050
  },
  "sessionId": "ses_xxxx",
  "durationMs": 3120
}

Example: Article markdown + links

const result = await client.scrape.scrape({
  url: 'https://example.com/blog/post',
  formats: ['markdown', 'links'],
  onlyMainContent: true,
  waitFor: 'networkidle',
});
console.log(result.markdown);
console.log(result.links);
console.log(result.metadata.contentProfile);
console.log(result.tokensEstimate.approxTokens);

Example: Product page (keep full page)

// Product pages: disable aggressive clipping (or rely on contentProfile: product)
const result = await client.scrape.scrape({
  url: 'https://www.amazon.com/dp/B0EXAMPLE',
  formats: ['markdown', 'links'],
  onlyMainContent: false,
  waitFor: 'networkidle',
});
// If clipping still falls back: result.metadata.mainContentFallback === true

Example: Scrape through a proxy

const result = await client.scrape.scrape({
  url: 'https://geo-restricted-site.com',
  formats: ['markdown'],
  proxy: {
    type: 'http',
    host: 'proxy.example.com',
    port: 8080,
    username: 'user',
    password: 'pass',
  },
});

Parameters

Notes