> ## Documentation Index
> Fetch the complete documentation index at: https://hyperbrowser.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Advanced Guide

> End-to-end guide to fetch pages, search the web, and extract structured data

This guide shows how to use the Web APIs to fetch pages, search the web, and extract structured data. It also documents the most important parameters for advanced use cases.

> [!NOTE]
>
> You can also see dedicated pages for [Fetch](https://hyperbrowser.ai/docs/web/fetch.md) and [Search](https://hyperbrowser.ai/docs/web/search.md).
> For full schemas, see the API Reference.

## Fetching a web page

With just a URL, you can extract page contents in your chosen formats using the `/api/web/fetch` endpoint.

```typescript Node theme={null}
import { Hyperbrowser } from "@hyperbrowser/sdk";
import { config } from "dotenv";

config();

const client = new Hyperbrowser({
  apiKey: process.env.HYPERBROWSER_API_KEY,
});

const main = async () => {
  const result = await client.web.fetch({
    url: "https://example.com",
    outputs: {
      formats: ["markdown", "links"],
    },
  });
  console.log("Fetch result:", result);
};

main();
```

```python Python 1.0+ theme={null}
import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))

# Fetch a page
result = client.web.fetch(
    {
        "url": "https://example.com",
        "outputs": {"formats": ["markdown", "links"]},
    }
)
print("Fetch result:\n", result.model_dump_json(indent=2))
```

```python Python (legacy) theme={null}
import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser
from hyperbrowser.models import FetchParams, FetchOutputOptions

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))

# Fetch a page
result = client.web.fetch(
    FetchParams(
        url="https://example.com",
        outputs=FetchOutputOptions(formats=["markdown", "links"]),
    )
)
print("Fetch result:\n", result.model_dump_json(indent=2))
```

```bash cURL theme={null}
curl -X POST https://api.hyperbrowser.ai/api/web/fetch \
    -H 'Content-Type: application/json' \
    -H 'x-api-key: <YOUR_API_KEY>' \
    -d '{
        "url": "https://example.com",
        "outputs": {
            "formats": ["markdown", "links"]
        }
    }'
```

### Fetch Response

The fetch response includes a `jobId`, overall `status`, and the outputs under `data`:

```json theme={null}
{
  "jobId": "962372c4-a140-400b-8c26-4ffe21d9fb9c",
  "status": "completed",
  "data": {
    "metadata": {
      "title": "Example Domain",
      "sourceURL": "https://example.com"
    },
    "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
    "links": [
      "https://www.iana.org/domains/example"
    ]
  }
}
```

### Output Formats

The `outputs.formats` array specifies what data to return. You can mix simple strings with output objects:

| Output | Description |
| - | - |
| `markdown` | Page content as Markdown |
| `html` | Raw HTML |
| `links` | All links on the page |
| `screenshot` | Screenshot (configurable) |
| `json` | AI-extracted structured data |
| `branding` | Visual brand profile—colors, fonts, logo, components, personality |

#### Screenshot with Options

```json theme={null}
{
  "url": "https://example.com",
  "outputs": {
    "formats": [
      "markdown",
      {
        "type": "screenshot",
        "fullPage": true,
        "format": "webp",
        "cropToContent": false
      }
    ]
  }
}
```

> [!WARNING]
>
> Screenshot cropping rules:
>
> * `fullPage` and `cropToContent` are mutually exclusive.
> * If both `cropToContentMaxHeight` and `cropToContentMinHeight` are set, max must be ≥ min.
> * Crop dimensions must be between **100** and **8000** pixels.

#### JSON Extraction with Schema

```json theme={null}
{
  "url": "https://example.com",
  "outputs": {
    "formats": [
      {
        "type": "json",
        "schema": {
          "type": "object",
          "properties": {
            "heading": { "type": "string" },
            "description": { "type": "string" }
          },
          "required": ["heading", "description"],
          "additionalProperties": false
        }
      }
    ]
  }
}
```

***

## Fetch Options

The Fetch API supports comprehensive configuration for controlling browser behavior, navigation, and outputs.

**url** `string` *required*

The URL of the page to fetch.

**stealth** `string`

`"none" | "auto" | "ultra"` — controls the default proxy/stealth posture.

**outputs.formats** `Array\<string | object\>`

Output formats to return (e.g. `["markdown", "links"]`), plus structured outputs like `{"type":"json","schema":...}`.

**outputs.sanitize** `string`

`"none" | "basic" | "advanced"` — Sanitizes output for easier downstream parsing.

**outputs.includeSelectors** `string[]` (default: \[])

CSS selectors to include. Only matching elements are returned.

**outputs.excludeSelectors** `string[]` (default: \[])

CSS selectors to exclude from output.

**outputs.storageState** `object`

Pre-seed `localStorage` / `sessionStorage` as key/value pairs.

**navigation.waitUntil** `string`

Load condition: `"load"`, `"domcontentloaded"`, or `"networkidle"`.

**navigation.waitFor** `number` (default: 0)

Milliseconds to wait after navigation completes before collecting outputs (0–30000).

**navigation.timeoutMs** `number`

Max time (ms) to wait for navigation (1–60000).

**cache.maxAgeSeconds** `number`

Cache control for fetch results. Set `0` to bypass cache reads.

**browser.location** `object`

Localize via proxy location (`country`, `state`, `city`). If set, proxy is enabled automatically.

**browser.profileId** `string`

Reuse an existing browser profile.

**browser.solveCaptchas** `string`

Enable CAPTCHA solving.

**browser.screen** `object`

Set viewport dimensions (`width`, `height`).

### Example: Fetch with Advanced Options

```typescript Node theme={null}
const result = await client.web.fetch({
  url: "https://example.com",
  stealth: "ultra",
  outputs: {
    formats: ["html"],
    sanitize: "basic",
    excludeSelectors: ["nav", "footer", "aside"],
    storageState: {
      localStorage: {
        "example:key": "example:value",
      },
    },
  },
  navigation: {
    waitUntil: "networkidle",
    waitFor: 2000,
    timeoutMs: 60000,
  },
  cache: {
    maxAgeSeconds: 3600,
  },
  browser: {
    location: {
      country: "US",
      state: "CA",
    },
  },
});
```

```python Python 1.0+ theme={null}
import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))

result = client.web.fetch(
    {
        "url": "https://example.com",
        "stealth": "ultra",
        "outputs": {
            "formats": ["html"],
            "sanitize": "basic",
            "exclude_selectors": ["nav", "footer", "aside"],
            "storage_state": {"local_storage": {"example:key": "example:value"}},
        },
        "navigation": {
            "wait_until": "networkidle",
            "wait_for": 2000,
            "timeout_ms": 60000,
        },
        "cache": {"max_age_seconds": 3600},
        "browser": {"location": {"country": "US", "state": "CA"}},
    }
)

print("Fetch result:\n", result.model_dump_json(indent=2))
```

```python Python (legacy) theme={null}
import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser
from hyperbrowser.models import (
    FetchParams,
    FetchOutputOptions,
    FetchNavigationOptions,
    FetchCacheOptions,
    FetchBrowserOptions,
    FetchBrowserLocationOptions,
    FetchStorageStateOptions,
)

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))

result = client.web.fetch(
    FetchParams(
        url="https://example.com",
        stealth="ultra",
        outputs=FetchOutputOptions(
            formats=["html"],
            sanitize="basic",
            exclude_selectors=["nav", "footer", "aside"],
            storage_state=FetchStorageStateOptions(
                local_storage={"example:key": "example:value"}
            ),
        ),
        navigation=FetchNavigationOptions(
            wait_until="networkidle", wait_for=2000, timeout_ms=60000
        ),
        cache=FetchCacheOptions(max_age_seconds=3600),
        browser=FetchBrowserOptions(
            location=FetchBrowserLocationOptions(country="US", state="CA")
        ),
    )
)

print("Fetch result:\n", result.model_dump_json(indent=2))
```

```bash cURL theme={null}
curl -X POST https://api.hyperbrowser.ai/api/web/fetch \
  -H 'Content-Type: application/json' \
  -H 'x-api-key: <YOUR_API_KEY>' \
  -d '{
    "url": "https://example.com",
    "stealth": "ultra",
    "outputs": {
      "formats": ["html"],
      "sanitize": "basic",
      "excludeSelectors": ["nav", "footer", "aside"],
      "storageState": {
        "localStorage": {
          "example:key": "example:value"
        }
      }
    },
    "navigation": {
      "waitUntil": "networkidle",
      "waitFor": 2000,
      "timeoutMs": 60000
    },
    "cache": {
      "maxAgeSeconds": 3600
    },
    "browser": {
      "location": {
        "country": "US",
        "state": "CA"
      }
    }
  }'
```

***

## Searching the web

Use the Search API to query the web and get structured results—titles, URLs, and descriptions—ready to use in your application.

```typescript Node theme={null}
import { Hyperbrowser } from "@hyperbrowser/sdk";
import { config } from "dotenv";

config();

const client = new Hyperbrowser({
  apiKey: process.env.HYPERBROWSER_API_KEY,
});

const main = async () => {
  const result = await client.web.search({
    query: "hyperbrowser browser automation",
  });
  console.log("Search result:", result);
};

main();
```

```python Python 1.0+ theme={null}
import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))

# Search the web
result = client.web.search({"query": "hyperbrowser browser automation"})
print("Search result:\n", result.model_dump_json(indent=2))
```

```python Python (legacy) theme={null}
import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser
from hyperbrowser.models import WebSearchParams

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))

# Search the web
result = client.web.search(WebSearchParams(query="hyperbrowser browser automation"))
print("Search result:\n", result.model_dump_json(indent=2))
```

```bash cURL theme={null}
curl -X POST https://api.hyperbrowser.ai/api/web/search \
    -H 'Content-Type: application/json' \
    -H 'x-api-key: <YOUR_API_KEY>' \
    -d '{
        "query": "hyperbrowser browser automation"
    }'
```

### Search Response

The response includes search results with titles, URLs, and descriptions:

```json theme={null}
{
  "status": "completed",
  "data": {
    "query": "hyperbrowser browser automation",
    "results": [
      {
        "title": "Hyperbrowser - Browser Automation Platform",
        "url": "https://hyperbrowser.ai",
        "description": "Hyperbrowser provides cloud browsers for AI agents and web scraping..."
      },
      {
        "title": "Getting Started with Hyperbrowser",
        "url": "https://docs.hyperbrowser.ai/quickstart",
        "description": "Learn how to use Hyperbrowser for browser automation..."
      }
    ]
  }
}
```

***

## Search Options

The Search API supports pagination, location-based results, and advanced filtering.

**query** `string` *required*

The search query (max 500 characters).

**page** `number`

Page number for pagination (1-100, default: 1).

**maxAgeSeconds** `number`

Cache control—cached results older than this are treated as stale. Set to `0` to bypass cache reads.

**location.country** `string`

ISO-2 country code (e.g., `"US"`, `"GB"`, `"DE"`).

**location.state** `string`

State code (for supported countries).

**location.city** `string`

City name (max 200 characters).

### Search Filters

Advanced search filters allow you to refine your search results:

**filters.exactPhrase** `boolean`

Wrap query in quotes for exact match.

**filters.semanticPhrase** `boolean`

Use semantic search.

**filters.excludeTerms** `string[]`

Terms to exclude from results.

**filters.boostTerms** `string[]`

Terms to prioritize in results.

**filters.filetype** `string`

Filter by file type: `pdf`, `doc`, `docx`, `xls`, `xlsx`, `ppt`, `pptx`, `html`.

**filters.site** `string`

Limit results to a specific site.

**filters.excludeSite** `string`

Exclude results from a specific site.

**filters.intitle** `string`

Search term must appear in page title.

**filters.inurl** `string`

Search term must appear in URL.

> [!WARNING]
>
> `exactPhrase` and `semanticPhrase` cannot both be `true`.

### Example: Search with Filters

```typescript Node theme={null}
const result = await client.web.search({
  query: "machine learning tutorials",
  page: 1,
  location: {
    country: "US",
    state: "CA",
  },
  filters: {
    excludeTerms: ["beginner"],
    filetype: "pdf",
    site: "arxiv.org",
  },
});
```

```python Python 1.0+ theme={null}
import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))

result = client.web.search(
    {
        "query": "machine learning tutorials",
        "page": 1,
        "location": {"country": "US", "state": "CA"},
        "filters": {
            "exclude_terms": ["beginner"],
            "filetype": "pdf",
            "site": "arxiv.org",
        },
    }
)

print("Search result:\n", result.model_dump_json(indent=2))
```

```python Python (legacy) theme={null}
import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser
from hyperbrowser.models import (
    WebSearchParams,
    WebSearchLocation,
    WebSearchFilters,
)

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))

result = client.web.search(
    WebSearchParams(
        query="machine learning tutorials",
        page=1,
        location=WebSearchLocation(country="US", state="CA"),
        filters=WebSearchFilters(
            exclude_terms=["beginner"],
            filetype="pdf",
            site="arxiv.org",
        ),
    )
)

print("Search result:\n", result.model_dump_json(indent=2))
```

```bash cURL theme={null}
curl -X POST https://api.hyperbrowser.ai/api/web/search \
  -H 'Content-Type: application/json' \
  -H 'x-api-key: <YOUR_API_KEY>' \
  -d '{
    "query": "machine learning tutorials",
    "page": 1,
    "location": {
      "country": "US",
      "state": "CA"
    },
    "filters": {
      "excludeTerms": ["beginner"],
      "filetype": "pdf",
      "site": "arxiv.org"
    }
  }'
```

***

## Combining Search and Fetch

A common pattern is to search for pages and then fetch their content. Here's how to do it:

```typescript Node theme={null}
import { Hyperbrowser } from "@hyperbrowser/sdk";

const client = new Hyperbrowser({
  apiKey: process.env.HYPERBROWSER_API_KEY,
});

const main = async () => {
  // First, search for pages
  const searchResult = await client.web.search({
    query: "hyperbrowser documentation",
    filters: {
      site: "docs.hyperbrowser.ai",
    },
  });

  // Then fetch content from the top results
  const fetchPromises = searchResult.data.results.slice(0, 3).map((result) =>
    client.web.fetch({
      url: result.url,
      outputs: {
        formats: ["markdown"],
      },
    })
  );

  const fetchedPages = await Promise.all(fetchPromises);

  fetchedPages.forEach((page, index) => {
    console.log(`\n--- Page ${index + 1}: ${searchResult.data.results[index].title} ---`);
    console.log(page.data.markdown);
  });
};

main();
```

```python Python 1.0+ theme={null}
import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))

# First, search for pages
search_result = client.web.search(
    {
        "query": "hyperbrowser documentation",
        "filters": {"site": "docs.hyperbrowser.ai"},
    }
)

# Then fetch content from the top results
for i, result in enumerate(search_result.data.results[:3]):
    print(f"\n--- Page {i + 1}: {result.title} ---")

    fetch_result = client.web.fetch(
        {
            "url": result.url,
            "outputs": {"formats": ["markdown"]},
        }
    )

    print(fetch_result.data.markdown)
```

```python Python (legacy) theme={null}
import os
from dotenv import load_dotenv
from hyperbrowser import Hyperbrowser
from hyperbrowser.models import (
    WebSearchParams,
    WebSearchFilters,
    FetchParams,
    FetchOutputOptions,
)

load_dotenv()

client = Hyperbrowser(api_key=os.getenv("HYPERBROWSER_API_KEY"))

# First, search for pages
search_result = client.web.search(
    WebSearchParams(
        query="hyperbrowser documentation",
        filters=WebSearchFilters(site="docs.hyperbrowser.ai"),
    )
)

# Then fetch content from the top results
for i, result in enumerate(search_result.data.results[:3]):
    print(f"\n--- Page {i + 1}: {result.title} ---")

    fetch_result = client.web.fetch(
        FetchParams(
            url=result.url,
            outputs=FetchOutputOptions(formats=["markdown"]),
        )
    )

    print(fetch_result.data.markdown)
```

```bash cURL theme={null}
# First, search for pages
SEARCH_RESPONSE=$(curl -s -X POST https://api.hyperbrowser.ai/api/web/search \
  -H 'Content-Type: application/json' \
  -H 'x-api-key: <YOUR_API_KEY>' \
  -d '{
    "query": "hyperbrowser documentation",
    "filters": {
      "site": "docs.hyperbrowser.ai"
    }
  }')

# Extract first URL from results (requires jq)
URL=$(echo $SEARCH_RESPONSE | jq -r '.data.results[0].url')

# Fetch the page content
curl -X POST https://api.hyperbrowser.ai/api/web/fetch \
  -H 'Content-Type: application/json' \
  -H 'x-api-key: <YOUR_API_KEY>' \
  -d "{
    \"url\": \"$URL\",
    \"outputs\": {
      \"formats\": [\"markdown\"]
    }
  }"
```

***

## Browser & Stealth

Both Fetch and Search APIs support stealth configuration for bypassing anti-bot measures:

* **Stealth** (`stealth`): `"none" | "auto" | "ultra"`
* **Location** (`browser.location`): `{ country, state, city }` (enables proxy automatically)
* **CAPTCHA** (`browser.solveCaptchas`): boolean
* **Profiles** (`browser.profileId`): reuse an existing profile
* **Screen** (`browser.screen`): `{ width, height }`

> [!WARNING]
>
> Using proxy and CAPTCHA solving increases latency. Enable only when needed.

> [!NOTE]
>
> Using a custom location costs **10 credits** per search. Without a location specified, searches default to US and cost **5 credits**.

***

## Best Practices

### Caching

Use `cache.maxAgeSeconds` (Fetch) or `maxAgeSeconds` (Search) to control caching behavior:

* Set to `3600` (1 hour) for frequently updated content
* Set to `86400` (24 hours) for static content
* Set to `0` to always fetch fresh data

### Error Handling

Always check the `status` field in responses:

```typescript theme={null}
const result = await client.web.fetch({ url: "https://example.com" });

if (result.status === "failed") {
  console.error("Fetch failed:", result.error);
} else if (result.status === "completed") {
  console.log("Success:", result.data);
}
```

### Rate Limiting

The Web APIs are rate-limited. Use the SDK's built-in retry logic or implement your own exponential backoff strategy.

### Output Sanitization

Use `outputs.sanitize` to clean up HTML/Markdown:

* `"none"`: No sanitization (fastest)
* `"basic"`: Remove scripts, styles, and common noise
* `"advanced"`: Aggressive cleaning for LLM processing

### Selector Filtering

Use `includeSelectors` and `excludeSelectors` to focus on specific content:

```json theme={null}
{
  "url": "https://example.com/blog",
  "outputs": {
    "formats": ["markdown"],
    "includeSelectors": ["article", "main"],
    "excludeSelectors": ["nav", "footer", "aside", ".advertisement"]
  }
}
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.